Manual Financial Data Correction Using Idle/Interval-Based Blending Engine
Publication Date: 2010-Jul-27
The IP.com Prior Art Database
A method and system are described to reduce the impact of manual data correction on the performance of a running order book market data feed handler by queuing corrections and processing them when the server is idle.
Manual Financial Data Correction Using Idle /Interval-Based Blending Engine
Today's financial market data is arriving at rates previously unthinkable, and timely delivery of this data to the clients that thrive on it is of utmost importance. While enterprise systems maintain a high quality of data, rarely can a system run without introducing some incorrect data to the end users or applications. Therefore, customers still require a facility for manually manipulating information within their market data processing and storage applications. This is especially important for collated data, which is not necessarily overwritten by subsequent messages. An example of such data is so-called 'order book' data (a.k.a, 'depth-of-book', 'level 2'). Without a means for manually correcting such data, erroneous information may get 'stuck' in a system for the day or even longer. Current vendor systems provide ad hoc mechanisms for manual manipulation of direct financial market data, implemented separately for a number of server applications, and at a performance premium. The implementations include halting data processing when manual update commands arrive, applying those commands, and then continuing. This is programmed for each market data feed handler. This traditional method can introduce dreaded latency to a running market data system.
Disclosed is a method for eliminating some or all of the latency attributed to manual data correction in a running order book feed handler. This method consists of submitting manual updates to a waiting queue of the running system and delaying the application of the update to an appropriate time. The waiting queue is processed at a throttled rate whenever idle periods are encountered, or after a pre-defined interval when no such relief is seen. The embodiment of this method includes: a well-defined data structure representing a manual manipulation command that can be used for any and all order book feed handlers; a standard queue for storing such commands arriving in a dedicated administrative thread; and an evaluation mechanism for identifying the best time to execute these commands from within the feed processing thread.
The proposed system begins with a standard financial order book feed handler. Such products exist from multiple vendors, including IBM's Websphere* Front Office for Financial Markets. These products read a variety of proprietary market data protocols, process the arriving market data and leverage some middleware technology to distribute normalized data to interested downstream clients. The disclosed method requires an administrative agent running within said feed handler, which commonly is a constituent of the middleware component or a separate administrative technology, such as SNMP. The administrative agent runs in one or more independent threads within the market data feed handler, while feed processing occurs in one or more separate threads. System administrators typically use a graphical...