[Raleigh-talk] Parallel::ForkManager and progress
Trevor Little
trevormg19 at gmail.com
Fri Dec 19 11:37:04 PST 2008
I've used IPC::Sharelite and it worked well in development but eventually
ran into problems because we use a hardened version of GCC on our
production servers that did NOT play well with it (segfaults). One thing
you could do if have each process send status data to a named pipe that
your monitor program could read. You could even have them only start
writing the status data in response to a SIGUSR or some other signal. That
similar to what I ended up with when IPC::Sharelite was eliminated as an
option. I setup a Boss/Worker module as a parent class for deamons we run
to do stuff. Basically the Boss finds work to do and starts up worker
deamons to do it. The worker deamons hang around as long as there is work
to do and the boss passes them "Jobs" (basically a serialized data struct)
by printing to pipes. It works, but has been buggy and the IPC::Sharelite
version was a whole lot cleaner.
So basiacally, I'd say give IPC::Sharelite a try and then maybe look at
pipes. Of course, if it's something you thing will grow into something
larger in the furture, it may save you some time in the long run to just
have it dump the data into a database.
Trevor
On Fri, 19 Dec 2008 13:56:00 -0500, Michael Peters <mpeters at plusthree.com>
wrote:
> I've got a long running batch process that uses Parallel::ForkManager to
> split the work up and take advantage of the multiple CPU/Cores on the
> box. This part works fine (have I mentioned how much I love P::FM?). But
> I'd like to have some sort of a running progress on the screen when run
> manually.
>
> So, any ideas on how best to do this? Since each worker is a separate
> process I could use something like Net::Shared or IPC::ShareLite to
> collect stats that are then reported on. Anyone have any success with
> these modules or have recommendations on shared memory modules?
>
> I've also thought about using the database to store the progress. That
> has the benefit of being something I can query even if the process isn't
> being run manually. But it has the drawback of having to connect to the
> database (which won't be on the same machine) and sending it lots of
> little queries to update a couple of counters.
>
> Thoughts?
More information about the Raleigh-talk
mailing list