[Chicago-talk] writing log files to a DB

[Jay Strauss]

> If I understand this correctly, this begins to enter
> entries in the file from newest to older until it
> comes to an entry that was already in the table and
> stops.  Am I understanding it correctly?
> 
> Does it then put the entries/records in the table from
> oldest newest to oldest? Does this matter? Do I need
> to index the file?

Your database does not guarantee the order of rows returned, unless you 
specifically ask for them to be sequenced (via order by).  So it 
shouldn't matter what order you insert the rows.  Without knowing 
anything about your database I can't tell you what indexes you may or 
may not need.

But in general indexes help you find rows fast, when constraining (in 
the where clause) by the columns in the index.  Same thing goes with 
when querying with an "order by".

Steve makes a couple of points that are important.  You probably want to 
build a unique constraint on some set of columns of this table that 
uniquely identify the rows.  That is, there are some subset of the 
columns in this table, that when used in combination, make each row 
unique.  This set of columns it commonly called the "primary key", you 
typically have a primary key on every table.  You should go thru the 
exercise of identifying them.

Also the idea of a scratch/temp table moves the duplicate checking out 
of your perl.  Though the sql to move the data from the scratch table to 
the real table becomes more complicated