Browse Prior Art Database

Secure, Parallel Large Files transfer

IP.com Disclosure Number: IPCOM000210977D
Publication Date: 2011-Sep-19
Document File: 5 page(s) / 67K

Publishing Venue

The IP.com Prior Art Database

Abstract

Over time file sizes have both grown in size and frequency of transmission. Currently large files may be 100’s of MBs or TBs in size (referred to as Big Data). Multiple solutions have been devised to solve this problem.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 22% of the total text.

Page 01 of 5

Secure, Parallel Large Files transfer

An overview of the "Secure, Parallel Large Files transfer" concept is illustrated in figure 1 below. As illustrated in figure 1 the "Large File" is "logically" divided into multiple parts. Each "Large File part" is copied into a memory block. The "large file part/memory block" is again logically divided into multiple "sections". The memory block is then transmitted (copied) from the server to the client through multiple network connections. The number of network connections used is equal to the number of memory block sections. On the receiving (client) side once a complete memory block is received it is saved (written to DASD or other media). This process continues until all of the "Large File Parts" have been transmitted and re-assembled.

The transfer process throughput is enhanced by:
1. Using two pairs of memory blocks such that multiple operations can be performed concurrently. These operations (described at a high level) are:
a. Data is being transferred from the "Part n" of the Large File into the first memory block.
b. Data already stored in the second memory block is being sent over the TCP connection to the Clients second memory block.

c. Data stored in the Clients first memory block is being saved to the clients copy of the large file.
2. At startup a "maximum number" of connections are established between the client and the server, such that
a. As the transfer of each memory block of data is completed an evaluation is made as to the optimum number of the existing connections to be used for transferring the next memory block of data such that the data throughput would be increased. This results in the data between the pair memory blocks being transferred over 1 through "the maximum number" of available connections.

b. The established "maximum number of connections" are never closed or re-opened during the transfer of the large file parts. Thus eliminating the overhead of continually closing and re-opening the socket connections.

This process is described in more detail in the following figures.

Sender == Server

Figure 1: Overview of the "Secure, Parallelized Large File Transfer" concept

The description of the overall operation of the large file transfer mechanism assumes that the "client" is already logged on and authenticated to the server. The request for a large file transfer may be one of many requests sent from the client to the server.

The overall operation of the large file transfer mechanism can be summarized in several steps
• Establish the connection

1

Receiver == Client

Memory

Block 1

Memory

Block 2

Large File

Part 1

Part 2

Part n

Memory

Block 1

Memory

Block 2

Large File

Section 1

Part 1

Part n

Section 2

Section 3

Section 1

Section 2

Section 3

Active connections

In-active connections


Page 02 of 5

• While not end of file … loop

o Transfer the data
o Save transferred data o ….

• End the connection

Establish the Connection:

Figure 2 below describes the "Establish the Conn...