Browse Prior Art Database

A method of mixture storage pattern of partition table

IP.com Disclosure Number: IPCOM000247575D
Publication Date: 2016-Sep-18

Publishing Venue

The IP.com Prior Art Database

Abstract

The disclosure propose a idea about create a partition table which can flexibly be configured with the row-based store or column-based stored. That is to say, one table can include both row-based table partition and column-based partition to support different data access behavior for different data. This processing can be either manual specified or automatically identified. It defined new DDL to support the change and enhanced the Database engine. This idea will give best performance for different table partition,save the data storage and reduce Database Administrator/developers effort

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 31% of the total text.

Page 01 of 10

A method of mixture storage pattern of partition table

Problem:


Data management in modern business applications is one of the most challenging topics in today's software industry. High volumeof large data stored in big tables usually shows up with different characteristics and is required to be processed with different business applications. Take partition table for example, a time based table partition is usually populated with the short term data and middle/long term data in different partitions. Short-term data is often processed with insert/update/delete and high selectivity query operations, while middle/long term data seldom has insert/update/delete operation, instead, it will be massively analyzed with low selectivity queries (such as On-Line Analytical Processing(OLAP) queries) for Business Intelligence(BI) purposes. Another case would be a location based partition table, for example, a company has several locations and each city owns a partition in the FACT table. Suppose in Jan, activities mainly happened in Beijing and Shanghai, so partitions of Beijing and Shanghai would be frequently modified and processed with high selectivity queries while other partitions remain relatively stable and mainly processed with low selectivity queries. When comes to Feb, activities moved to Xian and Guangzhou, then the table partition activeness changed accordingly. Another case would be a partition is undergoing transactionprocessing in the daytime, while working with analytical processing in the nighttime. All these cases reveal that, data are changing with different characteristics, some data is relatively hot-spot which has frequent insert/update/delete operations and requires high selectivity query operations, some is so-called "static" or "inactive" due that it has few modifications and is usually analyzed with low selectivity queries for analysis purposes.

How to effectively process with different data/application characteristics is key in current BIG DATA environment. Existed Database resource management (DBRMs) utilized classic row based stores to meet transaction processing requirements, while column based data stores is emerging in analytical data process domain. We are proposing a system and method to mix the row based and column based storage for partition table or big table which contains high volume data of different characteristics and requires different processing.

Background Knowledge of row store and column store:


A database table is a two-dimensional data structure where the cells are organized in Rows and Columns. However as Computer memory is organized as a linear structure. To store a table in linear memory, two options exist, a row oriented storage stores a table as a sequence of records, each of which contain the fields of one row. Conversely, in a column store the entries of a column are stored in contiguous memory locations.

Row based tables have following benefits:
a) easier to insert and update
b) the data is...