Browse Prior Art Database

Method and Apparatus of Multi-dimensional Resource De-fragmentation with Resource Proxy

IP.com Disclosure Number: IPCOM000246490D
Publication Date: 2016-Jun-13

Publishing Venue

The IP.com Prior Art Database

Abstract

This acritcle describes a method to handle the resource fragment in order to improve the resource utilization in multi-dimention resource sheduling policy

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 21% of the total text.

Page 01 of 12

Method and Apparatus of Multi-dimensional Resource De-fragmentation with Resource Proxy

In Big Data distributed computing scenario, the resource requirements are quite different among multi-tenant applications and different jobs and tasks. For example, a company maintains many of compute hosts to serve different big data applications and jobs, each application or job needs various hosts for different usage, someone wants 8 core and 16G memory hosts, someone wants 2 cores and 8G memory hosts, and the requirement may changes from time to time.

The Application Workload Manager will receive multi-dimensional resource from Cluster Resource Manager and serve for running big data applications. Since the big data workload is dynamic, resource fragments will be generated by multi-dimension scheduling inside Application Workload Manager. And the unavailable fragments will cause resource wasting. Since the resource fragment cannot be avoided, there needs a component and strategies to reduce unavailable fragment to increase resource utilization in Application Workload Manager.

This invention presents a resource de-fragmentation method by Resource Proxy component inside Application Workload Manager while allocating multi-dimensional resources to jobs/tasks. It aims to reduce unavailable resource fragments with Resource Proxy.

This invention is to implement the Resource Proxy that manages multi-dimensional resources in Application Workload Manger side and aims for de-fragmentation and fragment management.

Every Application Workload Manager will start an individual Resource Proxy.

Resource Proxy groups all jobs requirements and request resource from Cluster Resource Manager.

After Resource Proxy gets resource from Cluster Resource Manager(may only part of the resource requirements satisfied), Application Workload Manager allocates multi-dimensional resource to each jobs by job priority based polices.


When fragment resource not avoidable, Resource Proxy tries best to make the fragments available to other pending tasks.

This invented Resource Proxy can reduce resource unavailable fragments and provides configurable strategies for fragment management.

Claims:

1. Resource Proxy manages all resources got from Cluster Resource Manage. It collects and summarizes all jobs/applications' requirements and requests to Cluster Resource Manager in configurable interval.

A separated layer of Resource Proxy can decouple the resource request/release function from Application Workload Manager and make it reusable and extensible. Also can avoid communicating with Cluster Resource Manager by each application or job itself, significantly relieve the resource manager's load especially in multi-tenant scenario.

2. Resource Proxy manages resource got from Resource Manager as a resource pool and share those resource among applications/jobs/tasks inside one Application Workload Manger. Resource Proxy merges fragments on each host to single fragment

1


Pa...