InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Data Affinity for Flexible Hierarchical Scheduling

IP.com Disclosure Number: IPCOM000236505D
Publication Date: 2014-Apr-30
Document File: 2 page(s) / 39K

Publishing Venue

The IP.com Prior Art Database


Disclosed is a method to provide data affinity amongst sessions that require the same data sets on top of a flexible hierarchical scheduling framework.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 52% of the total text.

Page 01 of 2

Data Affinity for Flexible Hierarchical Scheduling

In a complex environment, users may want to ensure that computing resources are distributed to the appropriate sessions in a way that reflects business needs. For example, certain projects, users, or user sessions within an organization may have higher priority than others may. An organization wants to allocate more resources to the higher-priority projects, users, and sessions.

In addition, large volumes of data result in performance bottlenecks in a large number of distributed applications . Multiple sessions may need to access the same data set(s). When tasks from these sessions execute within a service instance, those tasks may load the required data into service instance memory. This data loading may comprise a large portion of the task execution time.

Application performance suffers if nothing is done to optimize the sharing of service instances amongst sessions /tasks that use the same data.

Further, sessions may have workload patterns that occur in bursts, in which tasks are periodically submitted during the lifetime of the session; not all tasks are submitted up front. While these sessions have no tasks to process, the service instances can be shared with other sessions. Application performance suffers if nothing is done to optimize the sharing of service instances amongst sessions/tasks that use the same data.

An existing product offer hierarchical fair share; however, it does not support different policies at different nodes in the hierarchy . It only supports fair share policy throughout the whole hierarchy. Neither does it support dynamic data affinity for jobs that use similar data. It only supports a static host affinity for a job queue or partition. A user can specify a resource requirement (e.g., particular host(s) or hosts with a particular attribute) for a job queue or partition.

The solution contains one function, to support dynamic data affinity within the hierarchy policy. In this function, it deploys an extra layer of data sets to provide the behaviors for resource assignment, which is dynamic affinity, and preemptive tree rebalancing for maximum data affinity...