System and Method for NoSQL-RDB Data Movement on Monitoring Data
Publication Date: 2016-Mar-28
The IP.com Prior Art Database
This disclosure proposes an idea of how to transfer monitoring data from NoSQL JSON document store to Relational Database for analysis. In practice, monitoring systems use JSON to record its log events and these events are more likely to store in JSON document store for it future analysis. However, most of JSON document stores are lack of powerful data analytics capability and therefore it is important for JSON document stores to provide an utility for transfering monitoring events to relational data format for data analytics.
Page 01 of 6
System and Method for NoSQL
System and Method for NoSQL- --RDB Data Movement on Monitoring Data RDB Data Movement on Monitoring Data
NoSQL Data stores are very popular because: 1) lower-cost data management platform for managing 100's of TBs - PBs of data; 2) elastic (automatic) scale in/out; 3) data models are quite flexible. However, there is an increasing business to implement data analytics on data in NoSQL. The native NoSQL is hard to complete this task, so it needs to leverage the feature of data warehouses. Thus, we have to move data from NoSQL data stores to data warehouse. That is to say, we have to transform JSON documents to record in Relational database.
In this disclosure, we focus on moving monitory data in JSON document to RDB for analysis. With the development of NoSQL databases, more and more monitory data are persisted in document stores. The properties of monitory data is as follows:
The core idea of this invention is to summarize the sensor descriptions by association rule mining, then create a core table to persist sensor description and several measurement table to persist the monitory data, and finally insert data to core table and measurement tables respectively. In the core table, each row contains the information about a sensor, and there is a corresponding measurement table for this sensor.
Page 02 of 6
NoSQL data store is coming up these years. There are a lot of material to introduce how to move data from RDB to NoSQL data store. However, based on our knowledge, there is no prior art related to migrate NoSQL data stores to RDB.
This invention is concerned with the problem of data movement from NoSQL data stores to relational databases (RDBs). More particular, this invention focuses monitoring data, such as sensor data from IoT appliances, monitoring data from monitoring services.
Previous study investigated how to move data from NoSQL data stores to RDBs by flattening the hierarchical structure of records in NoSQL data stores. However, the previous solution does not work well on monitoring data. The limitations of the previous method include:
1) Each record in the monitoring data contains the corresponding sensor description. With the previous method, there are multiple repeated sensor descriptions in the RDB. It is a waste of spaces.
2) The measurements from different sensors have different value types. The previous method provided two solutions to solve this problem. The
Page 03 of 6
first one is to fix the measurement's type when creating the table in the RDB. With this solution, the measurements with different value types cannot be inserted in the table. The second one is to set multiple types in one record when creating the table. Since one measurement is just in one type of value, the spaces are wasted.
3) With the previous method, all sensor data are pushed into one table. This requires an extra filter step before analytic.
To sum up, a data movement methodology for monitor...