Browse Prior Art Database

Use of a social media networking & microblogging service for remote monitoring and control of ETL processes.

IP.com Disclosure Number: IPCOM000224182D
Publication Date: 2012-Dec-12
Document File: 3 page(s) / 94K

Publishing Venue

The IP.com Prior Art Database

Abstract

Real-time monitoring and control of Extract, Transformation and Load (ETL) processes is essential for critical business operations. This paper describes the use of an established internet based social media networking & microblogging service to provide a pervasive mechanism for remotely monitoring and controlling execution of ETL processes in a production environment.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 38% of the total text.

Page 01 of 3

Use of a social media networking & microblogging service for remote monitoring and control of ETL processes.

Extract, Transformation and Load (ETL) processes are typically executed in a "production" environment, where real-time monitoring and control of those processes is essential to ensure successful completion of critical business operations.

    Typically, solutions for monitoring and control of such operations, are implemented within an intranet environment, using a mix of both off-the-shelf and/or proprietary components. With the increasing use of mobile computing, users also require access to such solutions from anywhere in the world, demanding breadth and depth of coverage of supported computing devices (e.g. tablets, smartphones, laptops) and software (e.g. operating systems, web browsers).

    Aspects of the remote monitoring requirement (notifications) can be achieved using established industry standard components/services. For example, an ETL user can be notified of current status of processes being executed, using a mail system based on SMTP (Simple Mail Transfer Protocol). This allows the ETL process to generate status mails that the user can then pickup via a standard mail client and meets the requirements of a lightweight remote monitoring capability, without a large investment in provision of access to that monitoring information.

    However, the control requirement can lead to complex and costly implementations, as the solution architecture often requires several components, namely: a front-end user interface; infrastructure to support communication with the back-end services; authentication to ensure secure access control and mechanisms to enable access through corporate firewalls etc.

This proposed technique of this disclosure employs the use of an established inter net based social media networking & microblogging service (Twitter) to provide a mechanism for controlling execution of ETL processes in a production environment. As a service, Twitter is pervasive across all forms of mobile computing devices and is established as a means for communicating small amounts of information, traditionally for the purpose of social networking.

    The published Twitter api would be used to provide a gateway to services implemented within an ETL application to provide a simple query and control mechanism for ETL process execution. Coupled with a simple monitoring capability employing the same technology, a lightweight solution could be implemented to provide a broad enough basic level of ETL production monitoring and control.

    In addition to the core communication services and associated api, Twitter provides an established web based front-end for accessing its "user account" based application. A user account is protected with an authentication mechanism and whilst Twitter services are primarily intended as a way to widely broadcast information to a large number of people (known as "followers"), it also provides privacy controls to limit...