Browse Prior Art Database

A Fine Grained Highly-scalable Resource Scheduling Framework based on Map Reduce

IP.com Disclosure Number: IPCOM000247136D
Publication Date: 2016-Aug-10
Document File: 7 page(s) / 80K

Publishing Venue

The IP.com Prior Art Database

Abstract

Resource scheduler is a critical component in datacenter for the purpose of increasing resource utilization, reducing operation cost, and improving application performance for various type of workloads. However, current schedulers always meets bottlenecks on collecting data and resource filtering. To mitigate the two bottlenecks, we propose a highly scalable scheduling framework with the concept of map reduce. The centralized scheduler is decomposed to several distributed components for collecting resource metrics, process filter() and aggregate() operation. The framework utilize map-reduce concept to maximize the parallelization.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 33% of the total text.

Page 01 of 7

A Fine Grained Highly-scalable Resource Scheduling Framework based on Map Reduce

Resource scheduler is a critical component in datacenter for the purpose of increasing resource utilization, reducing operationcost, and improving application performance for various type of workloads. Commonly resource scheduler includes three parts:

1. A resource monitoring agent that collects resource metrics in the host and reports to the controller.

2. A centralized controller that manages resource metrics reported from agents, process request from clients and call scheduler algorithm to get decision.

3. A scheduler algorithm that makes decision for a resource request.

There are two problems that none of the current resource schedulers has resolved.

1. Bottleneck of collecting data:current schedulers requires the centralized controllers (master) to get resource metrics from all agents using pull/push model. However, current datacenter has more than 10k servers in common, and the number of virtual resources is even 10-100 times larger. Hence it is not a scalable way to let a single controller (or with some replicated one) get global resource metrics in all.

2. Bottleneck of resource filtering: The scheduling algorithm varies, but there are two basic operation in it. 1) filter() is to exclude resources with a set of constraints leaving valid resources. 2) aggregate() (or say weight()) is to rate resource based on a certain policy. Most of the scheduler algorithm can be decomposed to a series of these two operations no matter how complicated it is. In current schedulers, the filter() operation always need to maintain a long list of resources, and check constraint for each element in it. Even though multi-threading can be used here to parallelize the filter operation, it is a programmatic solution rather than a systematic solution.

we propose a highly scalable scheduling framework with the concept of map reduce. The centralized scheduler is decomposed to several distributed components for collecting resource metrics, process filter() and aggregate() operation. The framework utilize map-reduce concept to maximize the parallelization. It has the following features:


 Parallelization: map-reduce concept to parallelize the resource metrics collection and filter operation


 Scalability: dynamic resource-filter worker mapping to easily add/remove resources and scale filter worker.


 Consistency: coordination mechanism to ensure the consistency of reported resource metrics.

1


Page 02 of 7

   Transparent to scheduling algorithm: the scheduling algorithm implementation does not need to consider the system architecture. Components that the framework includes:

Filter worker: collects resource metrics from resources and process filter() operation when receiving a filter request. Each filter worker


1.

only collects a sub set of resource metrics, and all filter workers cover the whole resource set. With this design, the bottleneck of metrics collection is...