Browse Prior Art Database

Processing of streaming data in relational databases

IP.com Disclosure Number: IPCOM000241113D
Publication Date: 2015-Mar-27
Document File: 6 page(s) / 90K

Publishing Venue

The IP.com Prior Art Database

Abstract

The present article describes an architectural enhancement of relational database management systems (RDBMS) and extensions to the structured query language (SQL) that allows querying and processing of streaming data also called data in motion. These enhancements and extensions enables a user to extract, filter, aggregate and analyze data in motion using SQL.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 32% of the total text.

Page 01 of 6

Processing of streaming data in relational databases

Introduction

Disclosed is a system and method to process streaming data using classical concepts of relational database systems. The term streaming data refers to data that is too large to be stored persistently in a traditional RDBMS. In the context of big data streaming data is also called data in motion.

Examples for streaming data are weather information to forecast natural disasters or sensor data on rail cars to analyze the condition of critical parts in real-time. This type of data is often outdated right after or shortly after it is processed.

Streaming data is usually processed with specialized products instead of a classical RDBMS. Thus different type of business data is managed and processed in different types of data management systems due to different characteristics. This approach has several obvious drawbacks

The present article describes how to enhance traditional RDBMS architecture and SQL to provide a holistic view on streaming data and traditional data providing well known industry concepts like tables and queries.

RDBMS architecture enhancements

The traditional RDBMS architecture is enhanced by the following new components and concepts:


1.) Stream reader

This component is responsible for accessing various arbitrary stream sources and transfer the stream data to main memory managed by the RDBMS. This way the stream reader provides the content of so called streaming tables. An example of a stream source is a sequential stream of JSON objects delivered on a TCPIP connection.


2.) Streaming table

A streaming table is a virtual table that only exists in main memory. Like traditional

tables a streaming table can be processed by SQL queries to extract, filter, aggregate and analyze the stream data. Such a table is associated with one and only one stream and can be accessed via standard SQL SELECT statements. Data manipulation language (DML) commands like INSERT, UPDATE and DELETE are not supported on streaming tables because data in a streaming table is per se volatile . Streaming tables support concurrent access like traditional tables .

To support streaming tables the SQL DDL and DML language is enhanced to support

1


Page 02 of 6

the following new concepts that are described in more details below:

STREAM


STALE RULE
STREAMING TABLE


3.) Streaming table cursor

Like traditional SQL queries streaming tables queries are based on so-called cursors.

Two types of streaming table cursors are introduced:

Sampling cursor which is the default implicit cursor for a streaming table query.

Accessing tables using an implicit cursor performs a sampling on stream data . Infinite cursor . This type of cursor is created explicitly and used to process all data provided by a stream continuously.

Both types of cursors are described in more detail below.


4.) Stale Data watchdog

This component ensures that data stored in a streaming table comply to so called stale rules. In case streaming d...