Browse Prior Art Database

A Method for ETL Data Transform Issue Tracking with Visualized UI

IP.com Disclosure Number: IPCOM000236719D
Publication Date: 2014-May-13
Document File: 6 page(s) / 161K

Publishing Venue

The IP.com Prior Art Database

Abstract

In a complex ETL transformed data issue tracking case, it’s very difficult to find where did the issue meta data come from and which problem ETL script code was been executed. The new method is to generate the issue data transform tracking flow with visualized UI to help issue data tracking. It's based on categorized ETL execution trace log analysis, master data transform flow map and history issue data statistics. It could save a lot of time and cost to identify the issue data root cause, and improve the cusotmer satisfaction.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 59% of the total text.

Page 01 of 6

A Method for ETL Data Transform Issue Tracking with Visualized UI

According to the complexity of ETL script, different meta data would generate plenty of various data transform rules and flows possibility.

It causes the issue data tracking to be very complex and difficult. In some case is a impossible mission.


•Some case the developer has to review the ETL script step by step or try to reverse the transformed data one by one, to find the potential problem manually.

•It takes too much time on this complex issue data tracking with the conventional approach.

We need figure out a more efficiency method to support the issue data tracking. Especially in some urgent cases.


•The new method is to generate the issue data transform tracking flow with visualized UI to help issue data tracking.

It's based on categorized ETL execution trace log analysis, master data transform flow map and history issue data statistics.

1


Page 02 of 6

•Comparing with manually tracking it extremely shortens the time cost on the issue data invitation.

•With the data transform flow and code block matrix, we can directly identify the problem in ETL script block location or meta data source.

3-a) Before your ETL script go to production, execute ETL script with tracking Baseline Data, and generate the Master Tracking Data into DB or file.

The master tracking data contains the operation category, meta data source, and related ETL script code block mark.

2


Page 03 of 6

3-b) Analyze the MTD(M...