If you wanted to stick to perl, you may be able to pull this off with Mozilla::Mechanize. I have never used it, but from the description it sounds like it will run any javascript.<br><br><a href="http://cpan.uwinnipeg.ca/htdocs/Mozilla-Mechanize/Mozilla/Mechanize.html">
http://cpan.uwinnipeg.ca/htdocs/Mozilla-Mechanize/Mozilla/Mechanize.html</a><br><br>-- <br>Lee<br><br><div class="gmail_quote">On Nov 12, 2007 11:55 AM, David Young <<a href="mailto:davidy@nationalcycle.com">davidy@nationalcycle.com
</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi,<br><br>I'm on another mailing list of colleagues, and this question came up. What
<br>would you all recommend?<br><br>Ydy<br><br><br>-----Original Message-----<br>Sent: Monday, November 12, 2007 10:32 AM<br>To: techies@xxxxxxxxxxxxxxxxxxxx<br>Subject: How would you do this?<br><br>More and more, I'm finding the need to do some page-scraping of web pages
<br>that are Web 2.0-ey.<br><br>My normal scraping tool is perl's WWW::Mechanize and such. But it is<br>completely javascript brain-dead.<br><br>For example...I'd like to write something that goes to YouTube, looks up
<br>a set of videos, analyzes the comments and ratings, and presents a<br>summary report. That's just an example of something that is conceptually<br>simple (easily done manually) but can't be done with WWW::Mechanize.
<br><br>So if you wanted to do something like that, what would you use?<br><br>I'm not adverse to using something other than perl...something Windows-based<br>would be my last resort. I'd consider a commercial product, though
<br>preferably not a multi-hundred-dollar one...<br><br>Obviously, something that submits as well as scrapes would be ideal.<br><br><br><br><br><br>_______________________________________________<br>Chicago-talk mailing list
<br><a href="mailto:Chicago-talk@pm.org">Chicago-talk@pm.org</a><br><a href="http://mail.pm.org/mailman/listinfo/chicago-talk" target="_blank">http://mail.pm.org/mailman/listinfo/chicago-talk</a><br></blockquote></div>