Browse Prior Art Database

Distributed Parallel Build System for Hierarchically Organized Large Scale Software Systems

IP.com Disclosure Number: IPCOM000117794D
Original Publication Date: 1996-Jun-01
Included in the Prior Art Database: 2005-Mar-31
Document File: 6 page(s) / 307K

Publishing Venue

IBM

Related People

Knutson, JI: AUTHOR

Abstract

The following is a description of a build system which dramatically decreases build times of large scale software systems through the use of a task generator and a constraint based queuing system. The task generator is responsible for generating compilation tasks in the correct order. The scheduling system is responsible for the distribution and management of compilation tasks across many hosts.

This text was extracted from an ASCII text file.
This is the abbreviated version, containing approximately 19% of the total text.

Distributed Parallel Build System for Hierarchically Organized Large
Scale Software Systems

      The following is a description of a build system which
dramatically decreases build times of large scale software systems
through the use of a task generator and a constraint based queuing
system.  The task generator is responsible for generating compilation
tasks in the correct order.  The scheduling system is responsible for
the distribution and management of compilation tasks across many
hosts.

      This article describes a manager for distributed parallel
compilation.  The manager makes it easy to use many computers,
working in parallel, to build a large software system.  The parallel
make system, thus, allows a build to be completed in a fraction of
the time that would be required if the build were done in the
conventional way on a single computer.

From the viewpoint of parallel computation, the two essential
characteristics of a build are:
  o  The build consists of many small steps.  This makes it possible
      for more than one computer to work on the build at the same
time.
  o  There are some constraints on the order in which the tasks of
the
      build can be done.  For instance, TaskA cannot begin before
TaskC
      and TaskW have both completed.  For normal software builds, the
      constraints require some amount of task ordering, but they also
      allow a lot of parallel processing.  For example, TaskA, TaskD,
      and TaskF can begin any time, but they must finish before TaskQ
      can start.  The existence of the task ordering constraints
      requires that some manager or method be used to allow parallel
      processing while insuring that the ordering constraints are not
      violated.

      Although this article describes a manager for a build, the
concept can be used to manage distributed parallel processing of any
ordered collection of many small tasks that, while being subject to
some task order constraints, can be processed in parallel.  The next
portion of this article describes a general purpose parallel
processing queuing system followed by a description of a task
generator used to define a task list with constraints.

      The Queue System - The main components of the queue system
are the master process, the slave processes, and the task list.

Together, the master process, slave processes and the task list are
similar to other general Distributed Queuing Systems such as DQS (DQS
is a research project to produce a distributed queuing system in a
heterogeneous networked computing cluster) and includes similar
features such as querying task and build state, checkpointing the
build state, and distributed host attribute or resource matching, but
includes the additional concepts of ordering constraints for tasks
within a queue and restarting failed tasks that other tasks are
constrained by.

      The task list describes the ta...