<html>
<body>
<font size=3>At 01:00 PM 3/30/2015, you wrote:<br><br>
<blockquote type=cite class=cite cite="">
<blockquote type=cite class=cite cite="">LWP::Simple & LWP::Useragent
returned the page, but the pages are fairly dense with a lot of embedded
javascript, embedded forms and ads the are being served up. All of which
I don't need. It's just taking a lot of time and memory. I was just
looking for something that would just give me a 404 or 200 or stop
reading at the some place like the end of the /head tag. I'm trying to
test out thousands of URLs which is the real problem. (This may not be
possible.)</blockquote><br>
You can use the LWP::Simple head() function like David said, but head()
vs. get() is all-or-nothing. There’s no way to say “Give me the
page up to the and of the <head> tag”.</font></blockquote><br>
That's what I thought.<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>I’m curious as to
how these pages are taking a lot of memory. You’re not storing
them, are you? What memory problems are you running into?<br>
</font></blockquote><br>
Ouch. Just fixed the memory problem. I was stupidly adding instead of
re-using the string. That fixed that.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>What’s the
problem that you’re actually trying to solve? Is it taking too
long to do those 100x 0 URL checks? How long is it taking, and how
long would you like it to take?</font></blockquote><br>
The fetches were taking anywhere from 20-60 seconds each. The remote side
is taking a long time to fetch the parts of the page out of several
databases before it starts sending results. That's a problem of poor
design on their side that I was hoping to work around.<br>
Thanks for your help.<br><br>
<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>--<br>
Andy Lester => <a href="http://www.petdance.com">www.petdance.com</a>
<br><br>
_______________________________________________<br>
Chicago-talk mailing list<br>
Chicago-talk@pm.org<br>
<a href="http://mail.pm.org/mailman/listinfo/chicago-talk" eudora="autourl">
http://mail.pm.org/mailman/listinfo/chicago-talk</a></font></blockquote>
</body>
</html>