Browse Prior Art Database

Collection System for Rapidly Rotated Webserver Logs

IP.com Disclosure Number: IPCOM000015007D
Original Publication Date: 2002-Oct-12
Included in the Prior Art Database: 2003-Jun-20
Document File: 1 page(s) / 42K

Publishing Venue

IBM

Abstract

A solution is disclosed that is capable of collecting logs from webservers when those logs are rotated on the order of minutes. This solution introduces a set of log collection clients and a set of log collection servers which can be configured into a "tree" for handling log file transfers to a central location for later processing. This system guarantees reliable end-to-end transfer of log files from the clients to the central processing location.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 51% of the total text.

Page 1 of 1

Collection System for Rapidly Rotated Webserver Logs

A solution is disclosed that is capable of collecting logs from webservers when those logs are rotated on the order of minutes. This solution introduces a set of log collection clients and a set of log collection servers which can be configured into a "tree" for handling log file transfers to a central location for later processing. This system guarantees reliable end-to-end transfer of log files from the clients to the central processing location.

For a cluster of webservers operating at high volume, it can be advantageous for log processing if the webserver rapidly rotates it access logs. When webserver log rotation is done in the order of minutes, the number of log files created per day quickly increases into the tens of thousands of logs when operating a high volume website. In order to process these logs (a CPU and I/O intense activity), it is desirable to transfer the logging data form the webserver node to a log processing node. Current systems addressing this have several problems: current systems have a difficult time determining if a log is ready to be transferred, and thus cannot rapidly process the logs. This can result in a slowdown in log transfer and subsequent log processing, or alternatively could result in a log transfer which is done before the webserver has completed writing the log, resulting in lost data. current systems do not guarantee end-to-end transfer of the log file. current systems, if they perform log file compression, generally perform that compression on the webserver node, which can negatively impact the performance of the webserver itself.

This solution addresses these shortcomings by introducing a log collection client and a log collection server. The log collection client is responsible fo...