Hey Shlomi,<br><br>Thanks for taking the time.<br><br>Yes, there is more than enough RAM to load the thing....and<br>I was only trying that in desperation.<br><br>Given that the day is over, I will try a smaller dump file to write <br>
my script to.<br><br>Onward and upward,<br><br>Chris<br><br><div class="gmail_quote">On Thu, Nov 11, 2010 at 1:30 AM, Shlomi Fish <span dir="ltr"><<a href="mailto:shlomif@iglu.org.il">shlomif@iglu.org.il</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">On Thursday 11 November 2010 04:56:12 Christopher Hahn wrote:<br>
> Hey team,<br>
><br>
> I am trying to parse a huge (7 Gb) file that is line oriented but has<br>
> large sections<br>
> that are any kind of binary character.<br>
><br>
> (this is a p42svn dump file of a large perforce repository)<br>
><br>
> I tried several smarter things, but found the after running for a while<br>
> my console would just close....dead, gone:<br>
> ============================<br>
> administrator@cmSVNDumper-09:/p42svn/testing$ ./<a href="http://p4dump-parse-new.pl" target="_blank">p4dump-parse-new.pl</a><br>
> Killed<br>
> ============================<br>
><br>
> I am sure that there are odd chars in the file that are doing this....<br>
><br>
> I tried setting binmode on the input file handle, and just loading the<br>
> entire file into a buffer, just as a test, as we have enough memory to do<br>
> this.<br>
><br>
> The result:<br>
> ===========================================<br>
> open(OUTF, ">SM_amanda_238037_fixed.dump")<br>
> or die "Opening output file failed: $!";<br>
><br>
> open(INF, "SM_amanda_238037_bad.dump")<br>
> or die "Opening input file failed: $!";<br>
> binmode INF;<br>
><br>
> my @buffer = <INF>;<br>
><br>
<br>
</div>Are you sure you want to load the many lines of a 7GB file into an array? Perl<br>
arrays have a lot of overhead, and doing this would be very memory wasteful.<br>
How much RAM do you have? You'll need much more than 7 GB for that.<br>
<br>
Regards,<br>
<br>
Shlomi Fish<br>
<br>
--<br>
-----------------------------------------------------------------<br>
Shlomi Fish <a href="http://www.shlomifish.org/" target="_blank">http://www.shlomifish.org/</a><br>
Stop Using MSIE - <a href="http://www.shlomifish.org/no-ie/" target="_blank">http://www.shlomifish.org/no-ie/</a><br>
<br>
<rindolf> She's a hot chick. But she smokes.<br>
<go|dfish> She can smoke as long as she's smokin'.<br>
<br>
Please reply to list if it's a mailing list post - <a href="http://shlom.in/reply" target="_blank">http://shlom.in/reply</a> .<br>
</blockquote></div><br><br clear="all"><br>-- <br>Realisant mon espoir, je me lance vers la gloire.<br>Christopher Hahn == <a href="mailto:xrz1138@gmail.com">xrz1138@gmail.com</a><br>