Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Method and Apparatus for job scheduling with intelligent data awareness

IP.com Disclosure Number: IPCOM000241479D
Publication Date: 2015-May-05
Document File: 4 page(s) / 76K

Publishing Venue

The IP.com Prior Art Database

Abstract

Since the jobs and tasks need to deal with more and more data in the distributed environment, it becomes one big bottleneck and challenge about how to reduce the cost of data transmition and further improve the jobs performance and efficiency. This article provides one intelligent data awareness method and apparatus for job scheduling: it will construct one forecasting model based on the actual network traffic among nodes on the distributed environment and the cached or shared data each node owns as well as historic task-execution time information of each node, and then decide each incoming jobs/tasks should be placed on which node's waiting list to gain the best completion time for each jobs/tasks. In addition, it provides the monitor and threshold mechanism to solve the exceptional cases when the time that jobs/tasks really spent exceed the estimated completion time of those jobs/tasks.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 42% of the total text.

Page 01 of 4

Method and Apparatus for job scheduling with intelligent data awareness

No matter what type of data sharing, within the current job management framework, it's the essential step to transmit the necessary data from data nodes to compute nodes before they execute the jobs.

There have different and complicated computing environment:

data nodes and compute nodes are mixed together, namely these hosts both act as data nodes and compute nodes;

data nodes and compute nodes are separated together, namely these hosts act as either data nodes or compute nodes;

maybe there have data cache mechanism;

Since the jobs need to deal with more and more data, it becomes one big bottleneck and challenge about how to reduce the cost of data transmition and further improve the jobs performance and efficiency.

Current Solutions


Job Manager will base on priority and incoming time to schedule incoming jobs to the free compute nodes without considering data location;

Some better solution such as Hadoop will try to schedule jobs on their data-localized compute node based on the data locality; if the node is busying, then these jobs will be waiting list of that node until timeout and then be scheduled to other random nodes.

Disadvantage:


No consider data location;

The data locality rules are based on static manual topology info. So there are some weakness:
The info is no incorrect sometimes because of typo or physical location was changed;
no consider the real network traffic situation;

It's inflexible about the strategy that jobs are waiting for the data localized compute node until timeout.

No consider to reuse these cached data which is already downloaded to compute nodes when running jobs if there have cache mechanism; The invention is from the following two improved aspects
how to schedule the jobs to these compute nodes owning almost data which these jobs need;
how to schedule the jobs to the "nearest" data node;

Primary idea


Based on data node distribution and the historic data-transit speed as well as reusing these cached data had been already saved on the local disk if have, schedule the subsequent jobs to that most-suitable compute node which has the best balance between its most owning data and fastest data-transit speed so that jobs can be the fastest completed time.

Job Manager keeps and maintains record about which data node and compute node owns what shared data as well as data-transit speed among compute nodes and data nodes and each job execution time;

1.If the data is static, then the information about which data node owns what data before any schedule;

1



Page 02 of 4


2.If the data is generated or downloaded by one job and then can be reused by other subsequent jobs, then there has one notification from Node Manager on compute nodes to Job Manager about what data is ready on which compute node;

3.When each job is completed, then Node Manager on compute nodes will send the message to Job Manager about its execution time;

4.Node Manager on compu...