SPUG: Sorting a big file
DeRykus, Charles E
charles.e.derykus at boeing.com
Fri Aug 27 15:19:29 CDT 2004
> I have a large file (~1 million lines, ~142MB) which I need to sort
> (any order is fine, just so the lines are in a repeatable order).
> Just using perl's 'sort' on the file read into an array eats up all
> the RAM and swap on my box and crashes. I'm also trying tying the
> file as an array, but it looks like that is also going to use up all
> the memory. Does anyone know some other methods I could try?
I've used the DBM DB_File (with DB_Tree) to sort large files.
(lexical sorting by default but can be customized.). Suprisingly
fast too.
hth,
--
Charles DeRykus
More information about the spug-list
mailing list