[Melbourne-pm] libwww and perl threads
Toby Corkindale
toby.corkindale at strategicdata.com.au
Tue Dec 13 17:03:42 PST 2011
Hi,
Has anyone here tried using libwww (ie. LWP::UserAgent and co) and
WWW::Mechanize via Perl threading?
I have encountered a situation where I see unusual 404 errors - in
between 0.03% to 0.10% of requests. Errors are randomly spaced on random
pages, but over time the average amounts are quite consistent.
Error rates initially increase with the number of simultaneous threads,
but seem to top off at .1%. (ie. One in a thousand requests)
The 404 errors are reported on the distant webserver as well, for URLs
that are definitely not 404. (as the identical URL is being requested
successfully tens of thousands of times in the same period).
Scale: This is typically running 40 threads, all going flat-out on an
8-core system; issues show up whenever you get over ~6 threads though.
The only reason I don't think this is a problem with the network or
webserver is that the problems don't show up if I use fork() instead of
threads. (On otherwise identical code; and the same overall throughput
rates are reached)
This was running on Perl 5.14.1 and current versions of the above
modules. (I don't think there are any bugfixes listed in 5.14.2 that
would affect this issue? [1])
Any thoughts?
Cheers,
Toby
[1 http://search.cpan.org/~flora/perl-5.14.2/pod/perldelta.pod ]
More information about the Melbourne-pm
mailing list