Browse Prior Art Database

Method and System for Demand-Driven Dataflow for Error Log Analysis

IP.com Disclosure Number: IPCOM000237233D
Publication Date: 2014-Jun-09
Document File: 2 page(s) / 129K

Publishing Venue

The IP.com Prior Art Database

Abstract

A method and system is disclosed for using demand-driven dataflow programming for processing multiple error log files.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 51% of the total text.

Page 01 of 2

Method and System for Demand-Driven Dataflow for Error Log Analysis

Disclosed is a method and system for using demand-driven dataflow programming for processing multiple error log files.

In accordance with the method and system, the demand-driven dataflow programming is a process for parallel processing. The method and system constructs a tree (or graph) of processing nodes. The processing nodes operates on a stream of tuples to do a simple operation such as, but not limited to, read a record from a file, filter out items, or merge two tuple streams together. Each tuple on a stream of tuples represents a log record. Leaf nodes of the tree log files are read by an input/output (IO) sources and then the log files data are sent upward in the tree for further processing. Thereafter, the log files data are merged together into a single stream of log record tuples. Then the stream of log record tuples is transmitted to the Tivoli Storage Manager (TSM*) Operations Center. The method and system manages the stream of tuples by using a series of buffer managers in between the processing nodes.

The figure below illustrates a node arrangement for a typical dataflow. The dataflow performing log retrieval and filtering of the method and system as disclosed herein.

Figure

1


Page 02 of 2

In an embodiment, the method and system retrieves and analyzes error logs by using processing nodes. For retrieving and analyzing error logs a few types of processing nodes like Parser Nodes, Filter Nodes, and Merge Nodes are required. The Parser Nodes read a file and parse the contents for log records. Thereafter, parsed contents are converted into a generalized tuple and sent through the output stream. The Filter Nodes read a tuple from the input stream and evaluate it against a filter rule. A filter

rule is associated with a filter node. If the filter rule matches the tuple, the tuple gets placed on the output stream otherwise gets dropped. The Merge Nodes read tuples from both the input streams and serialize the older tuple into the output stream followed by the newer tuple.

In accordance with the embodiment, the method and system manages integrated buffer manager and flow control elements by using streams. The integrated buffer...