Browse Prior Art Database

Method and System for Dynamic Message Routing Within a Distributed Stateful Data Transportation System

IP.com Disclosure Number: IPCOM000197955D
Publication Date: 2010-Jul-23
Document File: 8 page(s) / 658K

Publishing Venue

The IP.com Prior Art Database

Related People

Rohit Chandra: INVENTOR [+3]

Abstract

A method and system is disclosed for dynamic message routing to evenly distribute messages in a distributed message system among worker processes for stateful transformation of messages. The method and system disclosed herein preserves message ordering, database management, and reliability constraints.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 24% of the total text.

Method and System for Dynamic Message Routing Within a Distributed Stateful Data Transportation System

Abstract

A method and system is disclosed for dynamic message routing to evenly distribute  messages in a distributed message system among worker processes for stateful transformation of messages.  The method and system disclosed herein preserves message ordering, database management, and reliability constraints.

Description

Disclosed is a method and system for dynamic message routing to evenly distribute messages in a distributed message system among worker processes for stateful transformation of the messages.  While distributing the messages, the method and system disclosed herein preserves message ordering, database (DB) management, and reliability constraints.  In order to route the messages a scheduler is used.  The scheduler reads the messages from a repl file.  Repl is a file based message passing mechanism used by the distribute message system.  Thereafter, the messages are sent one at a time to the workers for the stateful transformation of messages.

The scheduler knows the commit state of the workers.  Based on the commit state, the scheduler or the worker decides that a commit is necessary.  After the completion of the commit, the scheduler is notified about the completion of the commit regardless of component that initiated the commit.  Additionally, the scheduler tracks which messages have been sent to a given worker but not yet committed to the DB.  Accordingly repl pointer is updated.

The scheduler remembers the list of account ids sent to each worker.  The list of account ids is discarded when a worker completes the commit operation.  All messages for a given account are processed by a single worker between commit operations.  When the commit operation is completed, a new set of messages for that account id may be assigned to a different worker.

In order to commit a set of messages, the scheduler divides messages into a sequence of “commit batches” and groups messages by account within each commit batch.  The commit batches are created by resetting the list of messages processed by each worker after each successful commit operation till the next commit.  Within the commit batches, the scheduler performs a simple grouping operation to collect messages by an account id.

The worker invokes a time-estimator function for the messages.  The time-estimator function is added to the CT plugin API called within the worker.  The time-estimator function is used to indicate that a long time is required by a given CT to process a given message.  Further, if there were messages processed by this worker for other accounts X, Y & Z, the worker will then force a DB commit and communicate this to the scheduler.  This allows the scheduler to assign messages in subsequent commit blocks for the accounts X, Y & Z to be processed by other workers.

Fig. 1A illustrates a worker flow in accordance with the method and system dis...