[Melbourne-pm] libwww and perl threads

[Toby Corkindale]

Following up on my own post..
I'm now pretty much convinced this is a Perl threads issue.

I can't definitively *prove* it, which is the annoying part -- trying to 
build a stand-alone test case just makes the bug move around or hide, 
but that is typical of threading race conditions..

I used to call them Heisenbugs when I worked in threaded C code.

I converted all the code to use fork() and RabbitMQ for IPC, and the 
code achieves faster throughput, uses less memory.. and critically, 
doesn't get any errors at all from the webserver.
And as an added bonus, now the code can be distributed over multiple 
machines and it just works.

(I intend to give a talk in RabbitMQ and STOMP at MPM soon too)

-Toby

On 14/12/11 12:03, Toby Corkindale wrote:
> Hi,
> Has anyone here tried using libwww (ie. LWP::UserAgent and co) and
> WWW::Mechanize via Perl threading?
>
> I have encountered a situation where I see unusual 404 errors - in
> between 0.03% to 0.10% of requests. Errors are randomly spaced on random
> pages, but over time the average amounts are quite consistent.
> Error rates initially increase with the number of simultaneous threads,
> but seem to top off at .1%. (ie. One in a thousand requests)
>
> The 404 errors are reported on the distant webserver as well, for URLs
> that are definitely not 404. (as the identical URL is being requested
> successfully tens of thousands of times in the same period).
>
> Scale: This is typically running 40 threads, all going flat-out on an
> 8-core system; issues show up whenever you get over ~6 threads though.
>
>
> The only reason I don't think this is a problem with the network or
> webserver is that the problems don't show up if I use fork() instead of
> threads. (On otherwise identical code; and the same overall throughput
> rates are reached)
>
>
> This was running on Perl 5.14.1 and current versions of the above
> modules. (I don't think there are any bugfixes listed in 5.14.2 that
> would affect this issue? [1])
>
>
> Any thoughts?
>
> Cheers,
> Toby
>
> [1 http://search.cpan.org/~flora/perl-5.14.2/pod/perldelta.pod ]
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm


-- 
.signature