[JaxPM] log checking script once again

[Nate Campi]

On the jacksonville-pm-list; Jax.PM'er Nate Campi <nate at campin.net> wrote -


Hello again list. I'm now working at Lycos, well Wired News actually, but
it's all one company. Anyways, I'm using my log reporting script that I
built at the last job, but I need to fix a hack that I'm not sure how to
fix.

My script takes a log file, in syslog format, and compacts the messages
so that the same message isn't just repeated over and over.

An example follows: 

May 30 16:04:10, 16:09:16, 16:14:23, 16:19:29, 16:24:35, 16:29:41,
16:34:46, 16:39:52, 16:44:58, 16:50:08, 16:55:14 goose ipop3d: Logout
user=??? host=skitzo.campin.net [63.198.180.27] 

As you can see from this example, the BBNET host 63.198.180.27 (my home
machine) checked the POP3 service on host "goose" 11 times, but it only
showed up in one line of the report.

My method was the result of a suggestion from J Proctor (thanks!). It puts
the message into a hash, with the key being the actual message (goose
ipop3d: Logout user=??? host=skitzo.campin.net [63.198.180.27]) and the
value being the time(s) it was reported. Since each key has to be unique,
it works like a charm.

The problem with it is this: I couldn't think of a good way to capture the
month and day for output into the final report, so I hacked it badly,
and just capture the month and day into variables (and I even get them
inefficiently at that, over and over again) and print them into the final
report.

The whole script is short, so I'll just paste it in here:

#!/usr/bin/perl -w

my $LOGCHECK_DIR = "/usr/local/psionic/logcheck";

# open the output from the logtail program
open(LOG, "$LOGCHECK_DIR/tmp/check");

while (<LOG>) {
        next if /^$/;        # skip blank lines (shouldn't be any)
        @msg = split(/[ ]+/);# split it up for easy parsing
        $month = $msg[0];    # HACK, $month and $day are used at the bottom
        $day = $msg[1];      #  this isn't a problem for an hourly report,
                             #  but if you run it less often than 
			     #  once a day, you'd have
                             #  to change this hack to get the right date
                             #  -if you do it daily, do it at midnight ;)
        $hostname = $msg[3]; # get the hostname
        $message = "";       # null out the log message variable

        for( $i = 3 ; $i <= $#msg ; $i++ ){
                # put everything from the hostname till
                # the end of the log message into the KEY
                $message .= " " . $msg[$i];
        }
        # $message is the actual log message
        $message =~ s/\[\d+\]//; # get rid PID, or no messages will match
        # the value is the time with a leading comma
        ${ $hostname }{$message} .= ", $msg[2]";
        # put the hash for the host into the "hostname" hash, 
        #  for use when printing out all the messages later
        $hostname{$hostname} = 1; # we never use the value, just the key
}

foreach ( keys %hostname) {
        $host = $_;     # each key in this hash is a reference to a hash
        open (HOSTFILE, ">> $LOGCHECK_DIR/tmp/hosts/$host");

        # do the actual printing to the individual host files 
	# for newlogcheck.sh to analyze
        while(($key, $val)=(each %{ $host } )){
                $val =~ s/^, //;  # lose the leading comma and space 
	# hacked $month and $day variables 
        #  all your logs have to be from the same day
                print HOSTFILE "$month $day $val $key";
        }
}
exit 0;

So you see that the way I get the month and day is inefficient and
imprecise. I'd like to incorporate them into the hash as I go, but I don't
really know the best way to do it.

I want something like:
 
${ $hostname }{$message} = "$month $day " . ${ $hostname }{$message};

...but if I put it in the while loop, I'll stick it in there each time
through. Plus then when I print it to the final report, I'd have to split
the key into another array and print the times in there where they need to
be (to look like a normal syslog message).

I'm really at a loss here, I just think I need to get pointed in the right
direction by some of you guru-types.

Thanks in advance,

--
Nate


Jax.PM Moderator's Note:
This message was posted to the Jacksonville Perl Monger's Group listserv.
The group manager can be reached at -- owner-jacksonville-pm-list at pm.org
to whom send all praises, complaints, or comments...