problem with opening up huge sized file

[corliss at alaskapm.org]

On Fri, 19 Jan 2001, Rick J wrote:

> Hi, Perl Mongers!
> 
> I have a question regarding opening up a huge sized
> file.
> 
> I was trying to open up a 2.6G file on Unix Solaris
> 5.6, 32bit. Perl barfed out "Variable value too
> large."
> 
> I tried different ways to open like "open (F,
> "filename")" and sysopen. Eventually I tried using cat
> and redirect to the file handle like open(F, "cat
> filename | "), which worked. 
> 
> I thought it was memory problem, cuz I believed that
> when PERL opens up a file, it reads whole content into
> its memory. Since the momory is low, it can't suck in
> all the lines. Just like when you create an array too
> big @array = (1..100_000_000), it will complain "out
> of memory". But someone thought it's the OS problem,
> because my Solaris is 32bit, and it can only handle
> the file smaller than 2.6G something. It it's true,
> why I can open the file with 'textedit' or 'vi' or
> even with command like 'more' or 'grep' search?
> 
> I wonder why I can use cat and pipe in open function
> to open up a huge file, is it because when it cats and
> pipes a file, it only sends 64k(?) bytes?
> 
> Please advise the reason and if there are other ways
> to open a large file.

That's something I've never tried, I'll have to try it on some of these other
machines.  Can you send the output of 'perl -V' so we know exactly what beast
we're dealing with?

One quick tidbit:  Perl (only the first letter is capitalised) can read
portions of a file in whatever size you ask it to.  It doesn't read the entire
thing into memory unless you do something foolish like:

	@file = <FILE>;

If you had done:

	$line = <FILE>;

you can read one line at a time.  It is possible, however, that it could have 
read the entire thing if it didn't find it's default line separator (\n on
most platforms).  Which isn't likely if it's a text file, but it's possible.

	--Arthur Corliss
	  Perl Monger/Alaska Perl Mongers
	  http://www.alaskapm.org/

=================================================
Mailing list info:  If at any time you wish to (un|re)subscribe to
the list send the request to majordomo at hfb.pm.org.  All requests
should be in the body, and look like such
                  subscribe anchorage-pm-list
                  unsubscribe anchorage-pm-list