Browse Prior Art Database

a new self-adaptive methodology for big data processing based on limited compute resources

IP.com Disclosure Number: IPCOM000248705D
Publication Date: 2016-Dec-28
Document File: 5 page(s) / 142K

Publishing Venue

The IP.com Prior Art Database

Abstract

big data applications are not well managed in resource consuming, most of the time, the cpu/memory are not fully occupied, and sometimes over-occupy happened. So as to address this issue, we provide a new methodology in dynamically adjust the resource usage by switch the program unit between parallel and sequential, so as to optimize the resource occupation.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 69% of the total text.

1

a new self-adaptive methodology for big data processing based on limited compute resources

recently the big data application in runtime is not stable in resource consuming, sometimes when plenty of online users, the job burst out, which cost tremendous computing resources, even beyond the existing hardware capability, e.g. out of memory or down, but sometime when user is very few, there are lots of resource wasting there which always without any performance enhancement. This disclosure describes a new self-adaptive methodology on balancing the current computing resource(cpu, memory, network) to gain best performance which can also avoid to the job disaster via real time logic computing refinement this disclosure includes following main points: 1> divide the big data computing logic into different dimensions, each related to different computing resource strategy(cpu, memory, network) 2> self adjust the computing logic in real time according to the current situation of the computing resources

advantages: 1> the existing computing resources can be fully make use of at any time 2> it can avoid the resource overload situation

Detail:

1> we analysis the program logic and generate all the DPUs (data process units., it can be one or more jobs) - analysis the job's dependency, check whether is can be parallel or not - analysis teh while, for... logic to check whether each loop inside logic is independent

2

2> when resource idled a lot, the DPUs logic will transformed into parallel...