Log file parsing
timc+perl at divide.net
Tue Jul 1 11:29:43 CDT 2003
> I am scoured around looking for a
> fast way to pull lines out of very large log files. These files are on
> order of 300+ mb.
> Unfortunately, there is not enouhg memory to slurp the whole thing into
> and process that way. Though, if I could how would I go about that?
Have you thought about slurping it into a database? If the logs don't change
often, that's the way to go. Then the database queries would be very
efficient. I'd use mySQL.
As for slurping the whole file into memory -- just give it a try. 300 megs
isn't much swap -- even for Windoze. Create a data structure (probably an
associative array) and do your queries on that. You might find that the data
structure is significantly smaller than the raw file.
More information about the Pikes-peak-pm