Log file parsing

bill lewis william.l.lewis at usa.net
Tue Jul 1 11:15:58 CDT 2003


Hi all,
  Long time listener, first time poster.  I am scoured around looking for a
fast way to pull lines out of very large log files.  These files are on the
order of 300+ mb.  

I am finding a list of Proc/pid combinations that we log, and creating an
array of those then foreach'ing over that list looking through the log for
matching lines.

Each new proc/pid means a new search on the entire file which is obviously
time consuming.  Right now, I am using the OS grep to pipe into perl to get
the lines for each procpid and then processing those lines.

Unfortunately, there is not enouhg memory to slurp the whole thing into ram
and process that way.  Though, if I could how would I go about that? 

Anyone done anything like this before?

thanks,
Bill Lewis


-----
William L. Lewis
email: william.l.lewis at usa.net
"They that can give up essential liberty to obtain a little temporary safety
deserve neither liberty nor safety."
--Ben Franklin





More information about the Pikes-peak-pm mailing list