Log file parsing

[Tim Chambers]

Bill,

> I am scoured around looking for a
> fast way to pull lines out of very large log files.  These files are on
the
> order of 300+ mb.

> Unfortunately, there is not enouhg memory to slurp the whole thing into
ram
> and process that way.  Though, if I could how would I go about that?

Have you thought about slurping it into a database? If the logs don't change
often, that's the way to go. Then the database queries would be very
efficient. I'd use mySQL.

As for slurping the whole file into memory -- just give it a try. 300 megs
isn't much swap -- even for Windoze. Create a data structure (probably an
associative array) and do your queries on that. You might find that the data
structure is significantly smaller than the raw file.

<>< Tim