Browse Prior Art Database

Integration of a 'dataflow pipeline' with a 'dataflow graph' programming model

IP.com Disclosure Number: IPCOM000032448D
Original Publication Date: 2004-Nov-05
Included in the Prior Art Database: 2004-Nov-05
Document File: 3 page(s) / 75K

Publishing Venue

IBM

Abstract

This article shows how a pipeline dataflow programming model and generalised branching dataflow models can be unified, such that general dataflow components can be used within the more restrictive pipeline programming model

This text was extracted from a PDF file.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 55% of the total text.

Page 1 of 3

Integration of a 'dataflow pipeline' with a 'dataflow graph' programming model

Pipeline dataflow models are often used as a simple component composition mechanism. Examples are:

JAX-RPC (Java* API for XML-based RPC) handlers.

    Unix** command piping Generalised dataflow networks, in which branches in the dataflow are allowed, are also commonly used; examples are:

APE

    MQSeries*** Integrator (WebSphere**** MQ Integrator, WebSphere Business Integrator Message Broker) and many others. This disclosure solves the problem of allowing these two programming models to be unified.

    A pipeline dataflow model is a specialised case of a dataflow network. Components on a pipeline have a single input port, a single (successful) output port, and a mechanism for indicating failure. An output port can be connected to a single input port.

    A general dataflow model allows components to have multiple input ports, multiple output ports, and output ports can be connected to more than one input port.

    The semantics of connecting more than one input port to an output port is that all connected input ports receive the same data.

    This disclosure solves the two issues with integrating these two programming models; allowing a Pipeline element to be used within a generalised dataflow model, and conversely, allowing a generalised dataflow to be used as a pipeline component. In a pipeline dataflow model, the pipeline elements have a single input port, a single output port and only a single connection may be made from an output port to an input port.

    Input to the pipeline and output from the pipeline is handled by the pipeline environment, and is not part of the pipeline itself, i.e. the mechanism that causes input and output to take place is implicit.

Dataflow pipeline programming model

dataflow from one pipeline element to another

stops dataflow and outputs error data Alternative outcomes

In a dataflow graph programming model. Each graph element may have more than one input port, more than one output port, and more than one connection between

failure

failure

external input external output

input Pipeline element

input Pipeline element

port

output port

output port

port

[This page contains 4 pictures or other non-text objects]

Page 2 of 3

input and output ports. Failures in the processing of an element are indicated by propagating data to a designated 'failure' output port. In the datagraph programming model, all inputs and outputs are explicitly modelled as nodes in the graph.

Datagraph element

Dataflow graph programming model

output port

input port

  Terminal datagraph node (no output ports)

input port

Datagraph element

output port

input port

failure port

input port

  Terminal datagraph node (no output por...