Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Gather/Scatter Pipe Interface

IP.com Disclosure Number: IPCOM000118317D
Original Publication Date: 1996-Dec-01
Included in the Prior Art Database: 2005-Apr-01
Document File: 4 page(s) / 141K

Publishing Venue

IBM

Related People

Bobak, RA: AUTHOR [+3]

Abstract

A gather/scatter interface was defined for data pipes. The gather function would be performed by a writer placing data into a pipe. The physical units of data would be gathered based on storage locations and lengths specified by the writer. The reader would use the scatter function to place data of various lengths in specified storage locations.

This text was extracted from an ASCII text file.
This is the abbreviated version, containing approximately 38% of the total text.

Gather/Scatter Pipe Interface

      A gather/scatter interface was defined for data pipes.  The
gather function would be performed by a writer placing data into a
pipe.  The physical units of data would be gathered based on storage
locations and lengths specified by the writer.  The reader would use
the scatter function to place data of various lengths in specified
storage locations.

      The interface consisted of external parameters to flag the
gather and scatter request, and to indicate the maximum size of the
gathered data.  The request was indicated on the pipe by
reading/writing a parameter list which described a table pointing to
the data areas. The  piping subsystem transferred data and
communicated back to the application through the parameter list.
Several processes and/or process  steps may be run to create multiple
files for input to applications which  then internally create a data
structure greater than the maximum block  size supported for
Input/Output (I/O) to real media for processing, and  then breakdown
that structure for output.  Since the pipe interface is  an access
method interface and in order to use this application and its  data
structure for pipeline construction and instantiation of parallelism
for the application with minimal application rewrite, this limitation
must be overcome.
  1.  Multiple records are often grouped physically into a logical
       grouping for processing; e.g., a customer account may
       consist of several records from different input sources.  To
       implement pipeline parallelism with this type of processing,
       the records have to be passed as a physical group.  For
       example, a front end reads records from multiple input
       files, and based on a change in the key (e.g., customer
       account number) recognizes the end of the records for the
       particular key.  This is the group of records that must be
       piped to the next stage.  If the next stage consists of
       replicated "identical" processes in order to implement
       parallelism, each process must get a group of records that
       represents the customer account.  Since the access to the
       pipe is asynchronous and with a destructive read, the
       records must be 'grouped' or physically 'blocked'
       together.  The gather/scatter interface is a means of this
       grouping.  The pipe width has to be set to the maximum size
       of the record grouping (pipe width and block size are used
       interchangeably).
  2.  When piping data through memory, data movement costs are a
       significant part of the overall data transfer.  There are
       instances when the data to be piped is sparse, therefore
       not all data needs to be piped.  Thus, if only relevant
       data can be piped at a trade off of a minimal amount...