[Raleigh-talk] Parallel::ForkManager and progress

Trevor Little trevormg19 at gmail.com
Fri Dec 19 11:37:04 PST 2008


I've used IPC::Sharelite and it worked well in development but eventually  
ran into problems because we use a hardened version of GCC on our  
production servers that did NOT play well with it (segfaults). One thing  
you could do if have each process send status data to a named pipe that  
your monitor program could read. You could even have them only start  
writing the status data in response to a SIGUSR or some other signal. That  
similar to what I ended up with when IPC::Sharelite was eliminated as an  
option. I setup a Boss/Worker module as a parent class for deamons we run  
to do stuff. Basically the Boss finds work to do and starts up worker  
deamons to do it. The worker deamons hang around as long as there is work  
to do and the boss passes them "Jobs" (basically a serialized data struct)  
by printing to pipes. It works, but has been buggy and the IPC::Sharelite  
version was a whole lot cleaner.

So basiacally, I'd say give IPC::Sharelite a try and then maybe look at  
pipes. Of course, if it's something you thing will grow into something  
larger in the furture, it may save you some time in the long run to just  
have it dump the data into a database.

Trevor

On Fri, 19 Dec 2008 13:56:00 -0500, Michael Peters <mpeters at plusthree.com>  
wrote:

> I've got a long running batch process that uses Parallel::ForkManager to  
> split the work up and take advantage of the multiple CPU/Cores on the  
> box. This part works fine (have I mentioned how much I love P::FM?). But  
> I'd like to have some sort of a running progress on the screen when run  
> manually.
>
> So, any ideas on how best to do this? Since each worker is a separate  
> process I could use something like Net::Shared or IPC::ShareLite to  
> collect stats that are then reported on. Anyone have any success with  
> these modules or have recommendations on shared memory modules?
>
> I've also thought about using the database to store the progress. That  
> has the benefit of being something I can query even if the process isn't  
> being run manually. But it has the drawback of having to connect to the  
> database (which won't be on the same machine) and sending it lots of  
> little queries to update a couple of counters.
>
> Thoughts?




More information about the Raleigh-talk mailing list