Flow Control and Overload Protection for Data Fusion Networks
Original Publication Date: 2004-May-19
Included in the Prior Art Database: 2004-May-19
This invention describes a control layer for message-oriented middleware that allows the orchestration of resources across multiple messaging channels.
Flow Control and Overload Protection for Data Fusion Networks Introduction
A data fusion network is an approach to collect data ("readings") from a large number of sensors, process and correlate this data in a decentralized way and provide a uniform access to the processed information. Message-oriented middleware such as IBM's MQ Series, Corba Event Notification Service, Gryphon, Siena and others is used to loosely couple the components in a data fusion network. For very large and/or dynamic systems, it is difficult to assess the amount of resources that are needed for the middleware components. Sensors, with their limited storage and processing capabilities, forward information whenever a "reading" is ready, no buffering can be done at the sensors. If the middleware is not ready to accept the sensors' readings, data loss will occur. In the case of an overload situation, existing middleware either discards data (for instance Corba's event notification system) or applies a flow control mechanism to throttle incoming data from the sensors. Both approaches lead to data loss.
In order to overcome this problem, a approach is proposed that is based on a control- and resource management system that reports failure situations, reconfigures the middleware, and/or adds new resources to the system dynamically. This solution essentially replaces flow-control mechanisms that operate per connection with a system-wide resource monitoring, resource allocation and load-balancing scheme. The advantages of the system are: -Data loss at the sensors is avoided since no flow control forces sensors to buffer/store data. -The client software at the sensors is less complex without flow control mechanism. -Resources are optimally used due to load-balancing.
Model of Message-oriented Middleware
The proposed model of event-based middleware is shown in Figure 1. It consists of sensors, event brokers or queue managers that manage queues, and processing nodes. Data, i.e. messages, are stored in queues from where they are retrieved for further processing and potentially re-publishing. The final destination of the messages is a server, which is also modeled as a processing node. The server typically is an access point for end-user applications, for example an application server. Brokers typically implement publish/subscribe systems where a message arrives and is immediately distributed to all attached consumers. The semantics here is that the broker is an active element that pushes messages to all consumers. Also, messages are typically not retained in pub/sub systems, that means that a consumer that connects to a broker typically can not consume messages that have been published before it joined. Queue managers, on the other hand, typically implement point-to-point systems. A message that arrives at the queue manager is stored in the queue until it is consumed (and removed) by a receiver. A single message then travels point-to-point, although multiple senders an...