Browse Prior Art Database

An efficient method for columnar data exchange in MPP query engine

IP.com Disclosure Number: IPCOM000246924D
Publication Date: 2016-Jul-15
Document File: 6 page(s) / 134K

Publishing Venue

The IP.com Prior Art Database

Abstract

In shared-nothing database architecture, data movement among nodes is often unavoidable, e.g., to perform non-collated joins. Minimizing data movement is often the key to query performance in such architecture. In this disclosure, we propose efficient methods for shipping columnar data among nodes in shared-nothing database. Such methods dramatically reduce the network traffic and also enables the possibility of query processing over encoded network data that further improves query performance.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 39% of the total text.

Page 01 of 6

An efficient method for columnar data exchange in MPP query engine

In shared-nothing database(MPP) architecture, data movement among nodes is often unavoidable, e.g., to perform non-collated joins. Minimizing data movement is often the key to query performance in such architecture. In this disclosure, we propose efficient methods for shipping columnar data among nodes in shared-nothing database. Such methods dramatically reduce the network traffic and also enables the possibility of query processing over encoded network data that further improves query performance.

We will use the following sample query plan to explain how columnar data are being shipped over network. In the plan below, we have two tables, fact and dim, three operators where BTQ(broadcast table queue) will transfer data among nodes, HSJN(hash join) will join the two tables. The query needs to perform a non-collocated join. As such, we need to ship the fact table after applying local predicates over to the nodes where the dim table locates. To minimize network traffic and to leverage the query performance benefit over compressed columnar data, it's desirable to ship encoded columnar data (without decoding) over the network such that the join can be performed in encoded fashion.

The encoded data of a column is currently represented by two Vectors (bi-format): a Vector of codes for the rows that are compressed on disk and a Vector of values for the rows that are unencoded on disk. For example, the following table shows a sample of bi-format Vectors for a 50% encoded column. A Vector has two parts - a Null Indicator part and a Raw Data part. The Null Indicator part indicates whether a given row is NULL or not. If not NULL, the corresponding entry in the Raw Data part represents the data (codes or unencoded value). In the table below, 'Ci ' represents a code and 'Vj ' represents an unencoded value and 'X' stands for a NULL. Note that an entry of 0 in the Null Indicator part indicates that the row is not NULL and an entry of 1

1


Page 02 of 6

represents a NULL value in the corresponding row.

Code Vector Value Vector

Null Indicator Raw Data

Null Indicator Raw Data

0

C1

1

X

0

C2

1

X

1

X

0

V1

1

X

0

V2

1

X

1

X

0

C3

1

X

0

C4

1

X

0

C5

1

X

0

C6

1

X

1

X

0

X

0

C7

1

X

Currently, we serialize the bi-format Vectors in their entirety and ship over the network. This is space in-efficient and incurs more data volume and network traffic than shipping a single Vector of decoded values!

We disclose an efficient method that dramatically reduce the amount of network traffic for columnar mpp query engine. It includes:1) Optimizing the shipping of Null Indicators, 2) Compacting Raw Data Based on Null Indicator and 3) Compaction Raw Data in the Code Vector based on code width
In this disclosure, we propose a family of methods to reduce the data volume over the network for shipping encoded data and thus to improve distributed query performance.

  1) Optimizing the shipping of Null Indicators
Curr...