[Chicago-talk] writing log files to a DB

[Jay Strauss]

you shouldn't prepare your statement in every iteration of the loop,
that's a big database killer.

Why don't you sort your input file by timestamp decending (hi -> lo),
then insert while your timestamp from the file is >= the max(timestamp)
in your database.  Sorta like:

sort -r -n -t, --key=7,7  log_files | perl your_file.pl

#cat your_file.pl

my $q = "REPLACE INTO master_log VALUES (?,?,?,?,?,?,?,?)";
my $sth = $dbh->prepare($q);

$TS # Populate this with a select max(timestamp) from your db

while (<>) {

    my ($src, $dsc, $clid, $chn, $PU, $HU, $timestamp,$dur) =
split(",",$_);

    exit if $TS > $timestamp;

    $sth->execute($src, $dsc, $clid, $chn, $PU, $HU, $timestamp, $dur);

}

Jay