SPUG: Sorting a big file
bri at ifokr.org
Fri Aug 27 15:51:40 CDT 2004
> I have a large file (~1 million lines, ~142MB) which I need to sort (any
> order is fine, just so the lines are in a repeatable order).
> Just using perl's 'sort' on the file read into an array eats up all the
> RAM and swap on my box and crashes. I'm also trying tying the file as an
> array, but it looks like that is also going to use up all the memory. Does
> anyone know some other methods I could try?
split $FILE section.
for file in section.*
sort $file > $file.sorted
sort -m section.*.sorted > sorted-version
Brian Hatch "Londo, do you know where you are?"
Systems and "Either in Medlab, or in Hell.
Security Engineer Either way, the decor needs work."
Every message PGP signed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: Digital signature
Url : http://mail.pm.org/pipermail/spug-list/attachments/20040827/9bcd3dbf/attachment.bin
More information about the spug-list