Phoenix.pm: update of HUGE text file

[Kevin Buettner]

On Oct 2, 11:39am, Mark A. Sharkey wrote:

> Is there a way in perl to do an update on a huge (100MB+ text file) without
> bringing the whole thing into memory?
> 
> I have client that is using a text file as a make-shift database table.  The
> client needs to be able to add/modify/delete lines in the text file.  He does a
> search against the file, edits the results, and then needs to make the edits
> "stick".
> 
> Can I make modifications to certain lines in the file, without reading the whole
> thing into memory?

I think you're asking if it's possible to modify the file in place.

You can certainly sysseek (lseek() in C) to the right place and then
modify the file.  The catch is that this will only work if the total
number of characters in the file stays the same.  I.e, if you add or
delete any characters, you will still need to rewrite the rest of the
file (i.e, the part after the point of modification.) This will
necessarily entail bringing the rest of the file into memory and
writing it back out.  Also, when growing the file, you'll need to be
careful to not overwrite data that will need to be "shifted" when
making the initial modification.  When shrinking the file, you'll need
to make sure that you truncate the file to the new size when you're
finished.

Seeking to the correct point in the file and modifying it in place in
this way will certainly be helpful if most of the modifications are
going to be near the end of the file, but of little help whatsoever if
they're likely to be near the front.

I think maybe you'd better investigate the use of a proper database. 

Kevin