Browse Prior Art Database

Heterogeneous Database Log Collection System Disclosure Number: IPCOM000238100D
Publication Date: 2014-Aug-01

Publishing Venue

The Prior Art Database


This document proposes a system to collect logs which are formed by information stored within a database and across different types of databases. This system is useful for products that need to collect logs from databases, for example SIEM(security information and event management), IPS(Intrusion prevention system) and IDS(Intrusion detection system) products.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 15% of the total text.

Page 01 of 15

Heterogeneous Database Log Collection System


Current SIEM technology provides real-time analysis on information generated by hardware and applications. The generated information, which is known as "log", is a representation of certain events happening on devices(user login, device error, service starting etc). Logs will be collected and processed by SIEM products. A general log process workflow can be event normalization, event correlation, vulnerability detection, threat reporting and etc.

Logs can be created by various devices(IDS/IPS, firewall, database, workstation, router etc). Due to the device nature, logs can be in various formats, for instance data packet(SNMP), binary file, text(syslog), database record.

TARGET PROBLEM 1: How To Efficiently Collect Logs Formed By Information In Multiple Tables Within One Database

Single Table Case(background)

One of the most common ways of storing logs is storing them in a database. In a simple case, the information which formed the log is stored within a table. After user has configured(through ui, config file, command terminal etc) connection information, the table name and the columns name into the log collector's setting, log collector can easily generate a simple query and retrieve the interested logs.

An example of automatically generated SQL query can be:


Page 02 of 15

Multiple Table Case(actual problem)

The complexity of configuration increases significantly if the log is formed by the information from multiple tables. In practice, it is common that the interested logs are not readily stored in a single column of a particular table. The interested logs may be formed by information spread across multiple columns or tables within a database.


One of the current solution is to provide users option to define free formed SQL query in log collector.


Page 03 of 15

However, this option highly increases the possibility of user error, which may lead to unmanageable consequences. For example, if users configure a DELETE or UPDATE query in log collector, the source database may be modified. It is risky for users to have a log collector with the capability of deleting or modifying records in their database.


Another current solution is to create a view in user/customer database.

This converts multiple table case to single table case. The log collector is then able to automatically generate a simple query to collect the relevant logs(exactly same mechanism as single table case).


Page 04 of 15

Due to security concern and maintenance reason, this solution is not always practical as customers/users are generally reluctant to make changes to their database schema for external party.


Another current solution is hardcoding complex SQL query in log collector, to accurately retrieve relevant logs. This is done by SIEM product developer with proper review process.

Using this solu...