Use of single placeholder for multiple messages in large file transfer
Original Publication Date: 2003-Mar-19
Included in the Prior Art Database: 2003-Mar-19
Sending large volumes of data already held persistently over a reliable transport involves two problems. (1) For efficiency, the messages must not be restored persistently by the transport (already handled by MQ 'reference messages'). (2) The data must be segmented, to prevent huge resend costs on failure (already handled by MQ message segmentation). However message segmentation must often be defined in advance by the application code; and even where reference segments are used, control data about each segment must be saved persistently. This can be expensive, and cause slow startup times, where there are many segments. This can be a particular problem when the user quickly realised the operation was in error, and wishes to abort. A lazy segmentation technique is described, where the segment control data is not realized at once, but on demand. The system stores a single block of control information that implies a set of reference control blocks; these segment reference blocks and thus the complete message segments are lazy realised as transmission occurs.