Browse Prior Art Database

A Hybrid Optimization Method of Bigdata Effective Storage and Efficient Query for Power System

IP.com Disclosure Number: IPCOM000236390D
Publication Date: 2014-Apr-24
Document File: 3 page(s) / 137K

Publishing Venue

The IP.com Prior Art Database

Abstract

The disclosure discloses a hybird optimization method of bigdata effective storage and efficient query for Power system. In the method, the optimization process is devided into two parts, retrieval calibration and cache optimization. Retrieval calibration first designs the row key for the database according to the real power system then combines the pre-fixer and search range setting of row key;Cache optimization treats cache as the data block of one-time reading and RPC based communication and I/O data stream are also proposed.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 52% of the total text.

Page 01 of 3

A Hybrid Optimization Method of Bigdata Effective Storage and Efficient Query for Power System

Background:
Volume, Variety and Velocity of big data exist in the domain of power system. Hence, effective storage and efficient query are needed.

Current Solution and Problem

1


Page 02 of 3

   Traditional RDBMS can not handle PB size of data, and has poor performance of unstructured data analysis , poor expansibility and poor performance of data mining.

   Open source of Hbase can not meet the requirement of concrete situation of the power system.Experienced experts and engineers are needed to improve and optimize it according to industry characteristics. There are still a lot of innovation space, such as the perspective of the application(different industry characteristics and requirement, different databases, file systems, and tables) and perspective of parameters tuning (more than 190 parameters in hadoop and about 20 parameters are very important and need to tune).

   Search optimization for RDBMS can somehow improve the performance of reading, but the following three problems can not be solved. First, in the environment of bigdatait is hard to full scan; second, to query most of the dataindex range scan is poor; Third, too many Indexes causes more maintenance costs

Our Idea

   After analyzing the real power system, retrieval calibration and cache optimization are presented for bigdata effective storage and efficient query:

   1Retrieval calibration: According to the real power system, design the row key for the database. Combination of pre-fixer and search range setting of row key is proposed.

2Efficient query: Treat cache as the data block of one-time reading, RPC based communication and I/O data stream are also proposed.

Process of our idea

   First, retrieval calibration is performed. Search performa...