[Buffalo-pm] Multi-Threaded Perl...
DANIEL MAGNUSZEWSKI
dmagnuszewski at mandtbank.com
Tue Jun 14 07:10:09 PDT 2005
Thanks guys!
>>> "Kevin Eye" <eye at buffalo.edu> 06/14/05 10:02 AM >>>
Dan,
I tried to write a fairly complicated perl app with threads a few years
ago and gave up (I used forking eventually). First know that you may
have to (re)compile perl with thread support before you can get started
-- I don't think most distributions of perl have threading enabled
(except maybe ActivePerl). Also, when I was using threads (around perl
5.6.1), it was not a brand new feature, but it was somewhat
experimental. I think it still is.
Basically, it worked. But as the app got more complicated, it crashed a
lot. The crashes were due to race conditions where multiple threads
would access the same data at the same time (e.g. one reading and one
writing) leading to null/invalid pointers and spontaneous seg
faults/bus errors. The way to prevent these race conditions (in perl
and other threaded systems) is to add locks to the code, but this is
very tedious. Also, the more locked sections, the more linearized the
execution of your program becomes, which partially defeats the purpose
of threading.
Also, many modules are not thread-safe. This can sometimes be worked
around by locking around calls to the entire module, but again, that
defeats the purpose of threading.
Compounding the problem is that the crashes truly are spontaneous.
Since they are caused by thread interactions, which are unpredictable,
the program can run fine one time and crash the next. Or you might get
it running fine on your system every time, but bugs will be uncovered
when you run it on a different system. In theory once all the proper
locking is in place, there will be no more crashes, but I found it
nearly impossible to get it right.
Forking is probably a better option in general for perl, but it is more
resource intensive and more difficult to get your processes to
communicate. There are fancy ways of making inter-process communication
more streamlined (e.g. shared memory -- see IPC::Shareable), but they
have some of the drawbacks of shared variables in a threaded app. On
the up-side, if you use some sort of stream for IPC (sockets, pipes,
etc.) your process will be naturally suited to run distributed on a
cluster of machines if that becomes necessary.
There are a couple of other options that may help. POE is a perl module
distribution that implements a cooperative multitasking environment
(rather than preemptive like typical threading). In this type of
multitasking, you yield to other tasks at specific points in your code
(rather than being interrupted at any time) greatly reducing the
locking problem. POE has a steep learning curve, though. I've never
used it, so I can't say first hand if it's worthwhile.
On the simpler side, there are modules like Event::Lib, Event, and
IO::Select that can help you implement non-blocking IO without
threading or forking. With a system like this, you would still have one
linear code path, but you would not block on each network IO call in
order. Instead, you would put them all out there, then handle each one
as it's ready. This is potentially the most efficient and simplest
approach, but you'll need to have an SNMP module that supports
non-blocking IO. I'd try this approach first if possible. It could be
later combined with forking (e.g. 10 processes monitoring 10 machines
each) if the performance isn't enough.
Good luck.
- Kevin
On Jun 13, 2005, at 4:09 PM, DANIEL MAGNUSZEWSKI wrote:
> Mongers,
>
> I am in the planning stages of writing a program that will make
queries
> to a device (via SNMP), wait for a response, then do something with
the
> information received (namely store it into a database), and move on
to
> the next device in the list. Obviously, when having to do this
> thousands
> of times per a 5 minute time frame, a single threaded program can be
> restricted by the blocking for the I/O Requests. I figured that in
> order
> to maximize the speed, it would need to be multithreaded.
>
> My real question is whether anyone has written any multithreaded
Perl
> programs, along with the pluses and minuses or using Perl for
> multithreading. Hopefully I won't have to use C or C++, in which
case,
> I'd probably rather bang my head against the wall. Although, having
a
> high quality UB Computer Science education, it wouldn't be
impossible
> ;-)
>
> Any help or insight would be great! Thanks.
>
> -Dan
>
> _______________________________________________
> Buffalo-pm mailing list
> Buffalo-pm at pm.org
> http://mail.pm.org/mailman/listinfo/buffalo-pm
>
--
Kevin Eye
Web Applications Developer
Creative Services and Marketing
University at Buffalo
330 Crofts Hall
Buffalo, NY 14260
eye at buffalo.edu
phone (716) 645-5000 x1435
fax (716) 645-3765
More information about the Buffalo-pm
mailing list