Browse Prior Art Database

Method and System of Leader based Database Loading for Storm based High throughput NRT systems

IP.com Disclosure Number: IPCOM000243623D
Publication Date: 2015-Oct-06
Document File: 2 page(s) / 85K

Publishing Venue

The IP.com Prior Art Database

Related People

Haiyan Luo: INVENTOR [+4]

Abstract

A method and system is disclosed of leader based database loading for storm based high throughput near real time NRT systems The method and system allows workers on each storm node to dynamically select a leader worker to communicate with a database loader to retrieve database changes in a real time fashion to a local storage

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 53% of the total text.

Method and System of Leader-based Database Loading for Storm-based High-throughput NRT systems

Abstract

A method and system is disclosed of leader-based database loading for storm-based high-throughput near real-time (NRT) systems.  The method and system allows workers on each storm node to dynamically select a leader worker to communicate with a database loader to retrieve database changes in a real-time fashion to a local storage.  

Description

Disclosed is a method and system of leader-based database loading for storm-based high-throughput near real-time (NRT) systems.  The method and system allows workers on each storm node to dynamically select a leader worker to communicate with a database loader to retrieve database changes in a real-time fashion to a local storage.  The remaining non-leader storm workers on each storm node loads the data changes from the local storage to update an in-memory cache.

In accordance with the method and system, each storm node contains a leader worker and one or more non-leader storm workers.  The selection of leader worker can be performed using an open source code based application.  The method and system leverages a key-value store for the local storage.  Further, different spouts/bolts within the same worker share the same cache instances using static cache instances, in order to avoid duplicate cache instances.

Initially, the method and system leverages the open source code based application to perform a leader selection among all storm workers of each storm node.  The selected leader worker on each storm node directly communicates with the database loader through Restful web service to download...