SPUG: Sorting a big file

DeRykus, Charles E charles.e.derykus at boeing.com
Fri Aug 27 15:19:29 CDT 2004


> I have a large file (~1 million lines, ~142MB) which I need to sort 
> (any order is fine, just so the lines are in a repeatable order).

> Just using perl's 'sort' on the file read into an array eats up all 
> the RAM and swap on my box and crashes.  I'm also trying tying the 
> file as an array, but it looks like that is also going to use up all 
> the memory. Does anyone know some other methods I could try?

I've used the DBM DB_File (with DB_Tree) to sort large files.
(lexical sorting by default but can be customized.). Suprisingly
fast too.

hth,
--
Charles DeRykus




More information about the spug-list mailing list