Browse Prior Art Database

Improving Data Migration Performance for a Network Environment

IP.com Disclosure Number: IPCOM000114267D
Original Publication Date: 1994-Dec-01
Included in the Prior Art Database: 2005-Mar-28
Document File: 4 page(s) / 149K

Publishing Venue

IBM

Related People

Dewey, DW: AUTHOR [+2]

Abstract

The heightened capacities and slower disk speeds of optical drives exasperate the problem of how to efficiently move large numbers of files to and from an optical library used as a file server on a LAN. This paper presents a method to greatly improve the data rate when moving numerous small files.

This text was extracted from an ASCII text file.
This is the abbreviated version, containing approximately 33% of the total text.

Improving Data Migration Performance for a Network Environment

      The heightened capacities and slower disk speeds of optical
drives exasperate the problem of how to efficiently move large
numbers of files to and from an optical library used as a file server
on a LAN.  This paper presents a method to greatly improve the data
rate when moving numerous small files.

      Standard LAN interfaces involve the execution of Open, Write,
and Close calls to the file server to write a file.  When the data
integrity protocol requires the successful committing of the file to
disk before successfully closing, a minimum of two disk writes must
be done for each file.  When large numbers of small files are being
written, data is moved to the disk in small writes with large
directory overhead and data throughput becomes very low.  Some
systems have addressed this by relaxing the data integrity protocol
and using write caching.  This increases the data throughput to disk
but at the cost of data integrity.  Also the software overhead of
separate calls for each file, moving data into cache, and the
nonoptimal transfer of data over the LAN still remains.  This
software overhead is more critical in an optical library where a
single controller is managing multiple drives.  Caching also does not
help the throughput of reading scattered files.

      In the MVS attached versions of the 3995 optical library, where
the interface can be defined, this problem was addressed for writing
with the addition of a packed write interface.  This required
alterations to the file system code.  This paper presents a technique
to use the existing interfaces in LANs to accomplish a similar packed
write over the LAN interface.  We also present variants of how to
accomplish this packed write as well as the addition of a packed read
concept.  One variation does not require the alteration of the file
system, hence the technique may be used on existing native file
systems unlike the pure packed write approach.

      The solution involves using an application program on the
requestors of a LAN to write and read lists of files to and from the
file server.  The application uses existing file system calls over
the LAN to read or write one large file.  The one large file contains
the data of many smaller files and information identifying the files.
The server and the application cooperate in the packing and unpacking
of this file.

      The following two scenarios give one embodiment of how to
accomplish this using existing OS/2* calls for Open, Read, Write,
Close and Extended Attributes (EA).  In both examples, the contents
of the large file are:
  o  Identifying information which contains an entry for each file
      packed in the large file specifying name, attributes, size,
      locations within the large file of the smaller files data.
  o  Followed by sector size aligned data for the first file.
  o  Followed by sector size aligne...