[Melbourne-pm] libwww and perl threads

Toby Corkindale toby.corkindale at strategicdata.com.au
Tue Dec 13 17:03:42 PST 2011


Hi,
Has anyone here tried using libwww (ie. LWP::UserAgent and co) and 
WWW::Mechanize via Perl threading?

I have encountered a situation where I see unusual 404 errors - in 
between 0.03% to 0.10% of requests. Errors are randomly spaced on random 
pages, but over time the average amounts are quite consistent.
Error rates initially increase with the number of simultaneous threads, 
but seem to top off at .1%. (ie. One in a thousand requests)

The 404 errors are reported on the distant webserver as well, for URLs 
that are definitely not 404. (as the identical URL is being requested 
successfully tens of thousands of times in the same period).

Scale: This is typically running 40 threads, all going flat-out on an 
8-core system; issues show up whenever you get over ~6 threads though.


The only reason I don't think this is a problem with the network or 
webserver is that the problems don't show up if I use fork() instead of 
threads. (On otherwise identical code; and the same overall throughput 
rates are reached)


This was running on Perl 5.14.1 and current versions of the above 
modules. (I don't think there are any bugfixes listed in 5.14.2 that 
would affect this issue? [1])


Any thoughts?

Cheers,
Toby

[1 http://search.cpan.org/~flora/perl-5.14.2/pod/perldelta.pod ]


More information about the Melbourne-pm mailing list