Browse Prior Art Database

System and Method to Maintain a Pool of Resource Negotiators for Running Applications in Clusters

IP.com Disclosure Number: IPCOM000241204D
Publication Date: 2015-Apr-02
Document File: 5 page(s) / 136K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed are a system and method to maintain a pool of resource negotiators that can run on any nodes in the cluster based on the resource availability. A pool manager launches a predefined number of resource negotiators at system initialization time, each of which is able to run a predefined number of applications, sequentially.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 31% of the total text.

Page 01 of 5

System and Method to Maintain a Pool of Resource Negotiators for Running Applications in Clusters

Applications running on clusters such as Hadoop* have to negotiate resources through an Application Master or Resource Negotiator (here, these terms are interchangeable).

An Application Master or Resource Negotiator typically performs the following:

• Connects to the Cluster Resource Manager • Requests the required resources to run the application • Waits for the resources to be allocated
• Launches the application on the allocated resources

In general, one Resource Negotiator is launched for each of the applications to be run in the cluster.

A considerable amount of time is used to start the Application Master each time an

application must run in the cluster and connect to the Resource Manager. The cost of total job run is equal to the cost to launch resource negotiator, plus the cost to acquire the resource, plus the cost to run the actual job. For large jobs, the cost to launch the resource negotiator and the cost to acquire resource is marginal compared to the cost to run the actual job. For small jobs, the cost to launch the resource negotiator and the cost to acquire resources significantly outweigh the cost to run the actual application. This effectively reduces the attractiveness of the cluster and efficiency for small jobs. A

method is needed to quickly run the applications and avoid or reduce the start-up cost.

Current solutions include a method to continuously run Resource Negotiators, a method to connect to the Resource Manager, and a method to wait for an application to connect and share resource request plan, and then proceed per the usual process. When hundreds or thousands of applications concurrently need to run, a single, continuously running Application Master cannot serve the purpose because it cannot take the entire load. One approach is to run multiple instances of such continuously running Resource Negotiators; however, this cannot determine how many resource negotiators are needed, which are available at the time of application request, or how long those resource negotiators must be kept running if users need to reclaim a system resource based on application usage pattern.

A method is needed to enable resource negotiator pooling that can start a group of

Resource Negotiators based on initial pool size configuration parameter, dynamically manage the Resource Negotiators' state and the pool, allocate each to different applications, and maintain balance of the pool size (e.g., start new ones when an existing one exits or crashes).

The novel contribution is a system and method to maintain a pool of resource negotiators that can run on any nodes in the cluster based on the resource availability.

A pool manager manages the pool of these resource negotiators. It launches a predefined number of resource negotiators at system initialization time, each of which is able to run a predefined number of applications,...