Browse Prior Art Database

Semantic Filtering and Compression of IoT Data Streams

IP.com Disclosure Number: IPCOM000250157D
Publication Date: 2017-Jun-07
Document File: 2 page(s) / 108K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed is a method to create a cost effective storage solution for an Internet of Things (IoT) application that requires real-time and deep analytics. This includes a method for semantic filtering of data streams as well as a method for adapting filtering with system resources and data stream characteristics.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 52% of the total text.

1

Semantic Filtering and Compression of IoT Data Streams

Internet of Things (IoT) applications need optimize machine and human-machine systems by performing real-time and deep analytics. Real-time analytics require expensive high-speed and low latency storage. Deep analytics requires large amounts of data that is best saved on low-cost storage. The number of IoT devices for a given application can rapidly grow, and planning for this growth can be cost prohibitive.

A method or system is needed to create a cost effective storage solution for an IoT application that requires real-time and deep analytics.

The amount of real-time and deep analytics data required for an application can be determined by the application data access pattern. Thus, the novel solution is a method and system to analyze the IoT data to determine its growth, and then combine that with workload analysis to determine the IoT application data requirements. The system saves the IoT data in the cloud. As data exceeds the real-time storage requirement age, the oldest data goes to deep analytics storage. As data exceeds the deep- analytics storage requirement, the system deletes the oldest data.

Figure: System diagram

The core novel idea presented in this disclosure is a method for semantic filtering of data streams. This is a method to selectively filter and compress data record streams at runtime using policies describing the data schema and predicates. To implement the method for semantic filtering of data streams:

1. Specify the data types and format for the data records in the data stream using the data schema in the policy

2. Specify the items in the data record and criteria for filtering and compression using the predicates in the policy

3. Decode the data records in the stream based on data type and formats specified in the data schema

4. Generate and apply a runtime function on the decoded data that filters out the data records that need not satisfy the filter predicate for exporting data over the network

2

5. Generate and apply a compression codec to encode the data based on the compression predicate for exporting data over the network. Filter and compression predicates capture the semantics of the data stream and system properties (e.g., retention time window, geographical location of data sources, device identifiers, data arrival rate, number of devices, etc.)

Another essential development is a method for adapting filtering with system resources and data stream chara...