David,<br><br>The split function is not going to make things any faster. In fact, without resorting to the use of another language, I can't think of a faster way of doing it than you have suggested. Even if you were to split on something like a quote followed by a space (/" /) and then reattach the quote to the end of each resulting element (work that is vastly simpler than regex matching), the process would end up being slower than regex matching because the regex maching happens in machine language and the more efficient work happens in Perl. I'm convinced also that even if you were to use the index function, your Perl code would still be slower than the regex-based solution you described.<br>
<br>In the past, I have tried a number of tricks to try to beat simple regex matching for this type of work and I've seldom been able to beat the regex matching. (When I write "this type of work", I am of course excluding regular Apache-like log files and other files that are designed to be easy and fast to parse. I'm talking about more thoughtless file designs, such as the one you described.) <br>
<br>You could roll out your own C extension, but that's just ridiculous because the hardware to process the slower and more general Perl regex would be less expensive than your time. <br><br>I don't know how you timed the split function, but I suspect that it was much faster because its regex was probably much simpler. If you try the split function with a more complicated regex, I'm sure you'll find that split isn't so fast any more. <br>
<br>You do need the /g at the end, of course.<br><br>--Donnie<br><br><br><div class="gmail_quote">On Mon, Oct 27, 2008 at 5:27 PM, B. Estrade <span dir="ltr"><<a href="mailto:estrabd@mailcan.com">estrabd@mailcan.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">David,<br>
<br>
Maybe the following will help:<br>
<br>
<a href="http://oreilly.com/catalog/perlwsmng/chapter/ch08.html" target="_blank">http://oreilly.com/catalog/perlwsmng/chapter/ch08.html</a><br>
<br>
I don't know a whole bunch about parsing tons of text with regexes. I do have one immediate suggestion - that "/g" might not be necessary. Anyway, take a look at that link; it might help.<br>
<br>
Cheers,<br>
Brett<br>
<div><div></div><div class="Wj3C7c"><br>
On Mon, Oct 27, 2008 at 03:59:32PM -0500, David B. John wrote:<br>
> (perl v5.8.8 on Ubuntu Hardy Heron/2.6.24-21-generic.)<br>
><br>
> I have an http logfile I'm trying to parse which looks like:<br>
><br>
> 2008:10:24-00:00:06 x.x.x.x httpproxy[4997]: id="0001" severity="info"<br>
> sys="SecureWeb" sub="http" name="http access" action="pass" method="GET"<br>
> srcip="x.x.x.x" user="" statuscode="200" cached="0" profile="profile_0"<br>
> filteraction="action_REF_DefaultHTTPCFFAction" size="78632" time="105<br>
> ms" request="0x90075b60"<br>
> url="<a href="http://sb.google.com/safebrowsing/update?client=navclient-auto-ffox&appver=2.0.0.17&version=goog-white-domain:1:481,goog-white-url:1:371,goog-black-url:1:25401,goog-black-enchash:1:62374" target="_blank">http://sb.google.com/safebrowsing/update?client=navclient-auto-ffox&appver=2.0.0.17&version=goog-white-domain:1:481,goog-white-url:1:371,goog-black-url:1:25401,goog-black-enchash:1:62374</a>" error="" category="175,178" categoryname="Software/Hardware,Internet Services" content-type="text/html"<br>
><br>
><br>
> (Nothing Spectacular.)<br>
><br>
> If I loop through the log file and do:<br>
><br>
> my ($date,$fwip,$proxy,$id,$severity,$sys,$sub,$name,$action,$method,<br>
> $srcip,$user,$statuscode,$cached,$profile,$filteraction,$size,$time,<br>
> $request,$url,$error,$category,$category_name,$content_type) =<br>
> $_ =~ /\w+=".*?"|\S+/g;<br>
><br>
> life is good but could be better (~ 75 seconds on a Hp dc7700S for a<br>
> compressed 500 MB logfile).<br>
><br>
> However, I'd really like to use split b/c it's so much faster (~ 15<br>
> seconds). The problem is if I split, sometimes $name, $time or<br>
> $category_name will include a space within the quotes which I don't want<br>
> to split on (see above).<br>
><br>
> I used Text::ParseWords but gave up after waiting 5 minutes.<br>
><br>
> I know I can use a regex with split but I'm stumped as to how I would go<br>
> about writing it. E.g. split on a space except when enclosed in quotes.<br>
> Also, would it theoretically be any faster than the example above since<br>
> it's using regex or should I just live with it?<br>
><br>
> Thanks.<br>
><br>
> David<br>
><br>
><br>
> _______________________________________________<br>
> NewOrleans-pm mailing list<br>
> <a href="mailto:NewOrleans-pm@pm.org">NewOrleans-pm@pm.org</a><br>
> <a href="http://mail.pm.org/mailman/listinfo/neworleans-pm" target="_blank">http://mail.pm.org/mailman/listinfo/neworleans-pm</a><br>
<br>
</div></div><font color="#888888">--<br>
B. Estrade<br>
Louisiana Optical Network Initiative<br>
+1.225.578.1920 aim: bz743<br>
:wq<br>
</font><div><div></div><div class="Wj3C7c">_______________________________________________<br>
NewOrleans-pm mailing list<br>
<a href="mailto:NewOrleans-pm@pm.org">NewOrleans-pm@pm.org</a><br>
<a href="http://mail.pm.org/mailman/listinfo/neworleans-pm" target="_blank">http://mail.pm.org/mailman/listinfo/neworleans-pm</a><br>
</div></div></blockquote></div><br>