Browse Prior Art Database

Dynamic Messaging in Data Loading Services Disclosure Number: IPCOM000246593D
Publication Date: 2016-Jun-20
Document File: 5 page(s) / 172K

Publishing Venue

The Prior Art Database


Disclosed is a Dynamic Runtime Messaging (DRM) framework that provides an efficient dynamic messaging mechanism for cloud services that consumes less critical computing resources in the backend server than existing processes do, and eliminates the need of repeatedly downloading large log files from the backend server for cloud client applications. It enables cloud users to receive notification of some pre-defined and custom events upon occurrence, with minimal costs.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 22% of the total text.

Page 01 of 5

Dynamic Messaging in Data Loading Services

When processing data integration via services in the cloud, current methods for tracking

where the job is run and how to access the job log include relying on the Application

Programming Interface (API) to retrieve the job log as part of the response body, or using the web user interface (UI) to review the messages polled from the job log.

A job log may be very large, which can cause excessive overhead when downloading it

from the cloud along with hundreds of thousands of other jobs. Additionally, viewing the entire job log to find relevant information for a particular purpose can be time consuming, and some information in the log may be out of date soon after it is downloaded.

To get periodic updates, a user must constantly download the updated log files from the server. Most of the time, only a small portion of the messages in the log files may be relevant to a specific web UI display. It is often inefficient to filter out unwanted messages from a giant log file retrieved from the server, especially when it needs to be done every time after an updated log file is downloaded. Other approaches may send specific message filtering requests to the server so that log-filtering operations are carried out in the server on top of the generated log files to get the relevant log messages. In this case, the entire logs are not sent to the client. However, multiple message filtering operations may be repeatedly performed on the same centralized log files in the server. For batch applications, data processing jobs are normally executed in the backend clusters. Heavy log filtering operations could potentially consume significant central processing unit (CPU), memory, and input/output (I/O) bandwidth in the cluster, which can impact the overall job execution performance.

To avoid having to repeatedly download large log files or to unnecessarily consume critical resources in the backend, Dynamic Runtime Messaging (DRM) provides a message management framework that can be used to dynamically register message handlers, each of which defines one or more message types of interest for some specific client applications. The DRM framework provides a generic messaging service layer on top of multiple messaging brokers. It can be extended to use other custom messaging services, such as one that utilizes native message queue mechanisms. Only messages defined by active message handlers can be published to DRM queues.

A DRM client application registers itself as a DRM message consumer (or DRM consumer). A DRM consumer indicates the message type(s) it is interested to consume. Multiple DRM consumers can potentially consume exactly the same types of messages without worrying about not receiving the full set of messages of interest.

For example, one DRM consumer may only be interested in error and warning messages generated in backend job execution, while another DRM consumer may be

interested in percent completion st...