Browse Prior Art Database

A System for Efficient Storage Through Inline Transcoding

IP.com Disclosure Number: IPCOM000029502D
Original Publication Date: 2004-Jul-01
Included in the Prior Art Database: 2004-Jul-01
Document File: 4 page(s) / 15K

Publishing Venue

IBM

Abstract

This invention otlines a system where images are transcoded inline for more efficient storage. The advantage of the system is that the storage format can be private and tuned, while the images served for viewing are in one or more widely supported formats. This enables the system to support a numebr of widely available standard viewers, but still store the images in a most efficient way practicable.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 29% of the total text.

Page 1 of 4

A System for Efficient Storage Through Inline Transcoding

    1. Background: What is the problem solved by your invention? Describe known solutions to this problem (if any). What are the drawbacks of such known solutions, or why is an additional solution required? Cite any relevant technical documents or references. There are today a number of applications where large collections of documents have to be stored for a long period of time and also made available for interactive display. One example is the Viewpointe system, where the scanned checks are archived. In general, such systems use some type of standard format that is widely supported by possible viewers. In some cases, this may be Adobe PDF, in others, like Viewpointe, Huffman-encoded baseline JPEG.

     The reason for this choice of formats is that it is usually deemed infeasible to convert or transcode the documents with sufficient throughput to include both storage and retrieval. The penalties associated with this are quite severe, since the storage formats may require much more storage than necessary. For example, if Viewpointe would switch to arithmetic-encoded JPEG, the resulting 10% data size reduction would save on the order of $100,000 monthly in the reduced storage and bandwidth costs. Potential savings are much larger if the immense checks archives would also be transcoded.

     This invention outlines a system that can store the internal image in a non-presentation format.

    2. Summary of Invention: Briefly describe the core idea of your invention (saving the details for questions #3 below). Describe the advantage(s) of using your invention instead of the known solutions described above. The invention is based on three fundamental observations. First, the incoming document stream processing is highly parallelizable, since the incoming data consists of a large number of independent documents. Second, the storage/retrieval requirements are usually asymmetric, in the sense that only a fraction of documents is ever queried. Third, the current advances in the computer technology allow constructing very cost-effective parallel computer system that can provide the necessary throughput for inline processing based on simple commodity components.

     The proposed system is comprised of a computer cluster and a transcoding system running on it. This cluster is interposed between the data repository and the external world. As each document is received, it is transcoded into an internal format. Conversely, as each query is received, the requested documents are dynamically transcoded into an external format. The final part of the system is selection and tuning of the internal format.

    3. Description: Describe how your invention works, and how it could be implemented, using text, diagrams and flow charts as appropriate.

     The invention will de discussed in the context of our preferred embodiment, a check storage system such as Viewpointe. This system generally receives tens of millions of scanned ch...