From pisium at yahoo.com Sat Jan 20 01:56:17 2001 From: pisium at yahoo.com (Rick J) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file Message-ID: <20010120075617.12337.qmail@web1404.mail.yahoo.com> Hi, Perl Mongers! I have a question regarding opening up a huge sized file. I was trying to open up a 2.6G file on Unix Solaris 5.6, 32bit. Perl barfed out "Variable value too large." I tried different ways to open like "open (F, "filename")" and sysopen. Eventually I tried using cat and redirect to the file handle like open(F, "cat filename | "), which worked. I thought it was memory problem, cuz I believed that when PERL opens up a file, it reads whole content into its memory. Since the momory is low, it can't suck in all the lines. Just like when you create an array too big @array = (1..100_000_000), it will complain "out of memory". But someone thought it's the OS problem, because my Solaris is 32bit, and it can only handle the file smaller than 2.6G something. It it's true, why I can open the file with 'textedit' or 'vi' or even with command like 'more' or 'grep' search? I wonder why I can use cat and pipe in open function to open up a huge file, is it because when it cats and pipes a file, it only sends 64k(?) bytes? Please advise the reason and if there are other ways to open a large file. Thanks, Ricky __________________________________________________ Do You Yahoo!? Yahoo! Auctions - Buy the things you want at great prices. http://auctions.yahoo.com/ ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From corliss at alaskapm.org Sat Jan 20 02:44:48 2001 From: corliss at alaskapm.org (corliss@alaskapm.org) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file In-Reply-To: <20010120075617.12337.qmail@web1404.mail.yahoo.com> Message-ID: On Fri, 19 Jan 2001, Rick J wrote: > Hi, Perl Mongers! > > I have a question regarding opening up a huge sized > file. > > I was trying to open up a 2.6G file on Unix Solaris > 5.6, 32bit. Perl barfed out "Variable value too > large." > > I tried different ways to open like "open (F, > "filename")" and sysopen. Eventually I tried using cat > and redirect to the file handle like open(F, "cat > filename | "), which worked. > > I thought it was memory problem, cuz I believed that > when PERL opens up a file, it reads whole content into > its memory. Since the momory is low, it can't suck in > all the lines. Just like when you create an array too > big @array = (1..100_000_000), it will complain "out > of memory". But someone thought it's the OS problem, > because my Solaris is 32bit, and it can only handle > the file smaller than 2.6G something. It it's true, > why I can open the file with 'textedit' or 'vi' or > even with command like 'more' or 'grep' search? > > I wonder why I can use cat and pipe in open function > to open up a huge file, is it because when it cats and > pipes a file, it only sends 64k(?) bytes? > > Please advise the reason and if there are other ways > to open a large file. That's something I've never tried, I'll have to try it on some of these other machines. Can you send the output of 'perl -V' so we know exactly what beast we're dealing with? One quick tidbit: Perl (only the first letter is capitalised) can read portions of a file in whatever size you ask it to. It doesn't read the entire thing into memory unless you do something foolish like: @file = ; If you had done: $line = ; you can read one line at a time. It is possible, however, that it could have read the entire thing if it didn't find it's default line separator (\n on most platforms). Which isn't likely if it's a text file, but it's possible. --Arthur Corliss Perl Monger/Alaska Perl Mongers http://www.alaskapm.org/ ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From michael at shoebox.net Sun Jan 21 05:22:34 2001 From: michael at shoebox.net (Michael Fowler) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file In-Reply-To: <20010120075617.12337.qmail@web1404.mail.yahoo.com>; from pisium@yahoo.com on Fri, Jan 19, 2001 at 11:56:17PM -0800 References: <20010120075617.12337.qmail@web1404.mail.yahoo.com> Message-ID: <20010121022234.A439@shoebox.net> On Fri, Jan 19, 2001 at 11:56:17PM -0800, Rick J wrote: > I was trying to open up a 2.6G file on Unix Solaris > 5.6, 32bit. Perl barfed out "Variable value too > large." Can you post the exact code you're using? Preferably the smallest amount that will duplicate the error. I'm thinking your issue is with large file support in perl, or the lack thereof. The perl -V output as Arthur mentioned would also be helpful. > I thought it was memory problem, cuz I believed that > when PERL opens up a file, it reads whole content into > its memory. When perl opens a file it doesn't do any reading. You the programmer have to do the reading if you want to get any data. It's up to you how much of the file you read in at a time. > But someone thought it's the OS problem, > because my Solaris is 32bit, and it can only handle > the file smaller than 2.6G something. It it's true, > why I can open the file with 'textedit' or 'vi' or > even with command like 'more' or 'grep' search? It's not necessarily an OS problem, more of a problem with perl interfacing with your OS. What I'm guessing the problem is is that your perl was not compiled with large file support. Originally, the C library functions that dealt with file sizes (stat, seek, tell, etc.) had an upper limit on the file size they could deal with. In order to preserve backwards compatability with older programs that used these functions, many C libraries also define equivalent functions that deal with increased file sizes. On the systems with such C libraries the program (perl, grep, vi, etc.) has to use these equivalent functions, or set certain macros during compilation, in order to properly deal with large files. However, I'm guessing here. I can't say for sure until I see your program and your perl -V output. Michael -- Administrator www.shoebox.net Programmer, System Administrator www.gallanttech.com -- ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From pisium at yahoo.com Mon Jan 22 11:49:24 2001 From: pisium at yahoo.com (Rick J) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file Message-ID: <20010122174924.14136.qmail@web1403.mail.yahoo.com> Hi, there! Sorry for being late. I had to do it in the office. I used a very simple open subroutine as it is:" open (F, "file_path") or die "$!; while (){ do something; #like print; } The error is: Can't open the file Value too large for defined data type at test.pl line 3. And now the output for perl -V: Summary of my perl5 (5.0 patchlevel 4 subversion 4) configuration: Platform: osname=solaris, osvers=2.6, archname=sun4-solaris uname='sunos rodin 5.6 generic sun4u sparc ' hint=recommended, useposix=true, d_sigaction=define bincompat3=y useperlio=undef d_sfio=undef Compiler: cc='gcc', optimize='-O', gccversion=2.8.1 cppflags='-I/usr/local/include' ccflags ='-I/usr/local/include' stdchar='unsigned char', d_stdstdio=define, usevfork=false voidflags=15, castflags=0, d_casti32=define, d_castneg=define intsize=4, alignbytes=8, usemymalloc=y, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib libs=-lsocket -lnsl -ldl -lm -lc -lcrypt libc=/lib/libc.so, so=so useshrplib=false, libperl=libperl.a Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' ' cccdlflags='-Kpic', lddlflags='-G -L/usr/local/lib' Characteristics of this binary (from libperl): Built under solaris Compiled at Dec 16 1997 17:04:23 @INC: /usr/local/lib/perl5/sun4-solaris/5.00404 /usr/local/lib/perl5 /usr/local/lib/perl5/site_perl/sun4-solaris /usr/local/lib/perl5/site_perl . Thank you, all! Ricky __________________________________________________ Do You Yahoo!? Yahoo! Auctions - Buy the things you want at great prices. http://auctions.yahoo.com/ ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From michael at shoebox.net Mon Jan 22 13:09:02 2001 From: michael at shoebox.net (Michael Fowler) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file In-Reply-To: <20010122174924.14136.qmail@web1403.mail.yahoo.com>; from pisium@yahoo.com on Mon, Jan 22, 2001 at 09:49:24AM -0800 References: <20010122174924.14136.qmail@web1403.mail.yahoo.com> Message-ID: <20010122100902.C439@shoebox.net> On Mon, Jan 22, 2001 at 09:49:24AM -0800, Rick J wrote: > open (F, "file_path") or die "$!; > while (){ > do something; #like print; > } > > The error is: > Can't open the file > Value too large for defined data type at test.pl line > 3. Your code and your error don't match. Does test.pl look more like this? (The numbers to the left are line numbers.) 1: #!/usr/bin/perl 2: 3: open (F, "file_path") or die("Can't open the file\n$!") 4: while (){ 5: do something; 6: } > Summary of my perl5 (5.0 patchlevel 4 subversion 4) > configuration: It would appear your problem lies in the lack of large file support. I'm not even sure perl 5.00404 has large file support. If you can, convince your administrator to upgrade Perl to 5.00503 or 5.6.0, and compile in large file support while he or she is at it. Michael -- Administrator www.shoebox.net Programmer, System Administrator www.gallanttech.com -- ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From pisium at yahoo.com Mon Jan 22 14:01:19 2001 From: pisium at yahoo.com (Rick J) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file Message-ID: <20010122200119.3086.qmail@web1403.mail.yahoo.com> Hi, Michael, Your code and your error don't match. Does test.pl look more like this? (The numbers to the left are line numbers.) 1: #!/usr/bin/perl 2: 3: open (F, "file_path") or die("Can't open the file\n$!") 4: while (){ 5: do something; 6: } Sorry that I just skipped some new lines. Yes, the error occurs at line3. As matter of fact, the same error appears even without while loop, which means the script only contains open sub. Because of that, I guess at least perl's trying to open up the file with open sub, and set the filehandle position at first line? That's why at first I thought perl tried to read in all the contents of that giant file. Thanks, Rick __________________________________________________ Do You Yahoo!? Yahoo! Auctions - Buy the things you want at great prices. http://auctions.yahoo.com/ ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list From michael at shoebox.net Mon Jan 22 14:28:42 2001 From: michael at shoebox.net (Michael Fowler) Date: Wed Aug 4 23:56:48 2004 Subject: problem with opening up huge sized file In-Reply-To: <20010122200119.3086.qmail@web1403.mail.yahoo.com>; from pisium@yahoo.com on Mon, Jan 22, 2001 at 12:01:19PM -0800 References: <20010122200119.3086.qmail@web1403.mail.yahoo.com> Message-ID: <20010122112842.D439@shoebox.net> On Mon, Jan 22, 2001 at 12:01:19PM -0800, Rick J wrote: > Because of that, I guess at least perl's trying to > open up the file with open sub, and set the filehandle > position at first line? That's why at first I thought > perl tried to read in all the contents of that giant > file. No, some system call somewhere is failing with EOVERFLOW. As I said, this is probably due to lack of large file support; the system call being used cannot handle such a large file. Once again, the issue is not a bug in perl, it's due to perl not being compiled with large file support. Perl itself does not do any seeking on a file when you open it; where the file position starts depends on your open mode, and that behaviour is implemented by your OS. Michael -- Administrator www.shoebox.net Programmer, System Administrator www.gallanttech.com -- ================================================= Mailing list info: If at any time you wish to (un|re)subscribe to the list send the request to majordomo@hfb.pm.org. All requests should be in the body, and look like such subscribe anchorage-pm-list unsubscribe anchorage-pm-list