[Chicago-talk] writing log files to a DB

Jay Strauss me at heyjay.com
Sat Mar 26 22:05:17 PST 2005

you shouldn't prepare your statement in every iteration of the loop,
that's a big database killer.

Why don't you sort your input file by timestamp decending (hi -> lo),
then insert while your timestamp from the file is >= the max(timestamp)
in your database.  Sorta like:

sort -r -n -t, --key=7,7  log_files | perl your_file.pl

#cat your_file.pl

my $q = "REPLACE INTO master_log VALUES (?,?,?,?,?,?,?,?)";
my $sth = $dbh->prepare($q);

$TS # Populate this with a select max(timestamp) from your db

while (<>) {

    my ($src, $dsc, $clid, $chn, $PU, $HU, $timestamp,$dur) =

    exit if $TS > $timestamp;

    $sth->execute($src, $dsc, $clid, $chn, $PU, $HU, $timestamp, $dur);



More information about the Chicago-talk mailing list