SPUG: Sorting a big file

Brian Hatch bri at ifokr.org
Fri Aug 27 15:51:40 CDT 2004



> 
> I have a large file (~1 million lines, ~142MB) which I need to sort (any
> order is fine, just so the lines are in a repeatable order).
> 
> Just using perl's 'sort' on the file read into an array eats up all the
> RAM and swap on my box and crashes.  I'm also trying tying the file as an
> array, but it looks like that is also going to use up all the memory. Does
> anyone know some other methods I could try?

#!/bin/sh
FILE=whatever

split $FILE section.
for file in section.*
do
	sort $file > $file.sorted
done
sort -m section.*.sorted > sorted-version

rm section.*


-- 
Brian Hatch                  "Londo, do you know where you are?"
   Systems and               "Either in Medlab, or in Hell.
   Security Engineer          Either way, the decor needs work."
http://www.ifokr.org/bri/

Every message PGP signed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.pm.org/pipermail/spug-list/attachments/20040827/9bcd3dbf/attachment.bin


More information about the spug-list mailing list