Log parser and report generator

July 4, 2009

Starting writing a new tool today that’s kinda neat.

One of the problems at work is that we have a ton of logs, and just as many log formats. Each subsystem uses something different. And everytime that I need to go through the logs, I have to take a moment to re-familiarize myself with the format, which fields are of interest, etc etc, and formulate a search.

Today I decided to write an object oriented Perl programme (who knew that Perl was so 2000?) where each “system” is expressed as a child class of the main Logger package, and a Factory class instantiates the appropriate worker class for parsing the input.

But, it doesn’t just parse the input, but it also groups some predefined fields, and stores them in named variables. You can then pass in a formatting string, that will output the named variables in whatever way you please.

logger So far I’ve just scratched the surface, but it should be fun writing more and more packages for this.

Here’s the code for the IMF system, with just the “user unknown” subsystem coded in:

use Logger;                        #Children must know about their parents
package Logger::IMF;
 
BEGIN{@ISA = qw ( Logger );}       #Declare this a child of the Logger class
 
# This allows us to extract the groups in the regex into particular named variable
# that are common to many of the reports
sub variablemap {
	my ($self) = $_[0];
 
	my $varmap;
	# We're looking for user unknown
	$varmap = {	1 =&gt; 'time',
			2 =&gt; 'ip',
			3 =&gt; 'recipient',
			4 =&gt; 'sender',
			5 =&gt; 'helo'
		  } if $self-&gt;{'root'}-&gt;{'subsys'} eq 'uu';
 
	$self-&gt;{'varmap'} = $varmap;
	return( $varmap );
}
 
sub regex {
	my ($self) = $_[0];
 
	return '\S+\s+\d+\s+?(\S+?)\s+?.+RCPT from \S+?\[(\d+\.\d+\.\d+\.\d+?)\]:.+Recipient address rejected:.+?from=\&lt;(.+?)\&gt;.*to=\&lt;(.*?)\&gt;.*helo=\&lt;(.*?)\&gt;' if $self-&gt;{'root'}-&gt;{'subsys'} eq 'uu';
	return undef;
}
 
1;
__END__

Tagged with: perl, programming, work

Categorised as: Perl