Coffea Works

Building a Central Logging Service out of Common Parts

Using the Log4j Framework, Syslog, and Swatch to create a Responsive Central Logging Service

by Alan Berg

Log files are potential weeds in the system administrator's garden. This article shows one way of turning these weeds into a good source of event notification nourishment. Log4j in combination with Syslog and Swatch can act as an efficient and cheap means of log file maintenance and harvesting. This combination is particularly interesting for small to middle-sized companies that wish for a simple and quick solution for plumbing Java applications into a standard UNIX/Linux based infrastructure.

Introduction
Log4j is a common framework for Java that allows a low cost approach to creating a consistent structure for highly configurable logging across numerous Java applications. Many well-known products like the uPortal project use the Log4j framework. Among Log4j's various strengths is its ability to output event messages to different destinations. Log4j can e mail an event, write to file or socket, a windows event service or even the standard UNIX Syslog service. Swatch is a Syslog real time analysis script. The script is written in Perl, a language much liked by a large section of the system administrator population. In this article we will explore the potential of plumbing Log4j with Syslog and reacting to events centrally via implementing Swatch scripts.

Within the enterprise, log files have a tendency to accumulate locally on application systems. There may be instances when the various files are rotated, harvested, and analyzed into reports. However, they usually just lie around, grow old, consume space, and fragment hard drives. This article highlights a potential solution spanning across two languages - Java and Perl, and across potentially different problem domains - system administration, application writing, and configuration.

Figure 1 highlights this central logging solution concept, which, if correctly planned, costs only a little time and effort to build.

Figure 1: A simple design for a central logging service

The concept is straightforward - Java based applications use the Log4j framework. While a significant number of businesses deploy a Linux or UNIX infrastructure with a Syslog daemon enabled by default, many system administrators insist on getting down and dirty with Perl scripting. Swatch is an ancient and reliable Perl script that waits, as a daemon, for a defined pattern to occur within a specified Syslog file. This article will show you how easy it is to glue these parts of the jigsaw together.

Log4j
Log4j is a popular Java framework for logging application events. It has the reputation for efficiency and is elegantly built and logically divided. The framework separates the formatting and the destination of logging from the programming of the logging code. The choice of destinations (called adapters) and decoration is placed in the configuration file. There is also a facility to send log messages to the console, files, JDBC connector, socket, SMTP, telnet, NT event logger, JMS and more adapters. The framework is so successful that it has been ported to Perl, PHP, C and other programming languages. You only need to write six lines of configuration script to Log4j to use the UNIX based Syslog mechanism as a destination. Further, since the adapter is buffered by default, the application will not have to wait for the transmission of the logging event.

Syslog
Syslog is a UDP protocol for transmitting log events. Syslog daemons listen and act on the events, by writing to log files or by passing the event to other daemons, or both. Syslog is the standard logging methodology for UNIX and you would be hard pressed to find a UNIX/Linux box without the facility.

Swatch
Swatch is a Perl script that waits around as a daemon checking the input from a given Syslog file (or other log types). Swatch gets into action as soon as a specified pattern occurs. Swatch has a number of nice features: the configuration file has a simple format, the pattern recognition is based on regex expressions which are the staple food of many system administrators, and Swatch may run scripts depending on the pattern found thus making it easy to plumb into. Best of all, from the stability perspective, Swatch is a thoroughly debugged product.

Example Configuration
The following is an effective, but primitive, working configuration: a Tomcat server or application server is running a basic servlet that sends logging events to the Syslog server. Each time the servlet is browsed, Syslog writes to a file; then, Swatch acts on the incoming log update. The configuration of Syslog is based on a Debian Linux server. The Tomcat server can be on any operating system that supports Java 1.5.

Note: It is assumed that the mail subsystem on the central logging server is functioning.

Syslog and Swatch Configuration
The following recipe assumes the Sarge variant of Debian Linux. For other operating systems, you may need to make a few minor modifications:

Modify /etc/hosts.allow for tcpwrappers so that the war application can reach the Syslog server
Within /etc/Sysklogd modify line to SYSLOGD="-r" from "-m 360", where r stands for remote
Modify /etc/Syslog.conf to send events on to the backup Syslog server by adding a line at top *.* @host_ip, where host_ip is the IP address of the backup server
Enable the backup server for remote logging via repeating steps 2 and 3
Install Swatch via apt-get install swatch
Make a startup script to read the syslog.log file: swatch --tail-file=/var/log/syslog --awk-field-syntax --pid-file=/var/run/swatch.pid -daemon
Write configuration file .swatchrc in the home directory of the runner of the startup script

Swatch is amazingly simple to configure. The following three lines of code are about waiting for an ERROR to appear in the log file. If an error message pops up, an e-mail is send to the distribution list mentioned in the event in the body of the message. Throttling is also is enacted, and if no other errors have occurred in the last two minutes, then the e-mail is sent across; otherwise, the event is ignored, hence avoiding repetition.

watchfor /ERROR/

mail addresses=swatch \@dev.null,subject=Ping\ of\ death

throttle 0:2:0

Other possible commands include echo, bell and exec (execute). The pattern matching is based on Perl regex. The Swatch distribution file includes more examples.

Under the Debian flavor of Linux many applications are not enabled by default to be seen or used via the network. This is true for MySQL and Syslog. Therefore if logging does not occur, start by checking your tcpwrappers and Sysklogd files.

Note that the Swatch script keeps an eye on only one log file. So you would need to run a number of instances for more complex scenarios.

GUI enabled tools such as chainsaw and lumbermill can listen, via sockets, for Log4j events. Adding an extra socket adapter on top of the local file and syslog appenders allows you to fine tune and test the Swatch configuration visually. The configuration allows you to see, live, the ripples of events in a distributed staging or development application environment.

Web Application (Log4JPing servlet)
Deploy the .WAR file included in the source file (see
Resources section). The example code is quite brief and understandable. The code is for a ping servlet which returns your IP address to your browser when you browse the correct
url /Log4jPing/ping. The servlet code sends two log messages to Syslog - a warning message and an error message. The script is shown in Listing 1.

Listing 1

import java.io.IOException;

import java.io.PrintWriter;

import javax.servlet.ServletContext;

import javax.servlet.ServletException;

import javax.servlet.http.*;

import org.apache.log4j.Logger;

import org.apache.log4j.PropertyConfigurator;



public class LogPingServlet extends HttpServlet

{



public LogPingServlet()

{

}



public void init()

throws ServletException

{

PropertyConfigurator.configure(getServletContext().getRealPath("/") + "WEB-INF/log.properties");

}



protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException

{

doPost(request, response);

}



protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException

{

response.setContentType("text/html");

PrintWriter out = response.getWriter();

String message = "Ping from: " + request.getRemoteAddr();

out.println(message);

logger.warn(message);

logger.error(message);

}



static Logger logger;

static {logger = Logger.getLogger(LogPingServlet.class.getName());}

}

To avoid unnecessary system dependencies, it is recommended to place the logging configuration file in a path relative to the web application and not absolute to the physical file system. The line that loads the configuration file in such a manner is:

PropertyConfigurator.configure(getServletContext().getRealPath("/") + "WEB-INF/log.properties");

Log4j (Configured for the File System and Syslog)
The following code snippet configures the log4j application to send the events to a local file and a Syslog server to the specified IP address. Both debug and error messages are passed on and the conversion pattern for the file logging is different to that for Syslog:

log4j.debug=true



log4j.rootLogger=DEBUG, FILE, SYSLOG

log4j.appender.FILE=org.apache.log4j.ConsoleAppender

log4j.appender.FILE.layout=org.apache.log4j.PatternLayout

log4j.appender.FILE.layout.ConversionPattern=%d [%t] %-5p %c - %m%n



log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender

log4j.appender.SYSLOG.syslogHost=xxxx.xxxx.xxxx.xxxx

log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout

log4j.appender.SYSLOG.layout.ConversionPattern=%p: %c - %m

log4j.appender.SYSLOG.Facility=USER

The line log4j.appender.SYSLOG.Facility=USER defines which filter action Syslog itself reacts to and is mentioned in /etc/Syslog.conf. By default, you will see the logging in the /ect/syslog.log file.

Potential Refinements
Swatch can execute arbitrary scripts. The script described in this article can be modified into a script that generates static web pages and summary per event seen. The significant advantage of this approach would be that once an administrator receives an e-mail about a serious error, she can browse the static web site to view the context of the event with respect to other monitored Swatch patterns. For example, even if two applications fail at the same time and both are connected to the same back end database, the error can be easily pinpointed.

Alan Berg, Bsc. MSc. MSc. PGCE, has been a lead developer at the Central Computer Services at the University of Amsterdam for the last seven years. In his spare time he writes computer articles. He has a degree, two masters and a teaching qualification. In previous incarnations he was a technical writer, an Internet/Linux course writer, and a science teacher. He likes to get his hands dirty with the building and gluing of systems. He remains agile by playing computer games with his kids who (sadly) consistently beat him.

Resources