SPUG: Daemonized work queue handling

Jonathan rise at knavery.net
Tue May 6 13:23:51 PDT 2008


We've got a process at work that handles generating and sending out
about 20M monthly newsletter emails to site members distributed over a
small cluster and I'm looking for a better way to run it.  Currently
each machine starts a cron job that forks semi-daemonized child
processes and redirects their output to a log file in /tmp.  They've
been converted to pull work blocks from a db-side queue and get all
their data from it as well, so there's no local state to care about.
Any recommendations or warnings for something that'll play well with
starting from an init script and running for months?

Things that look like they have potential:

Poe::Queue / POE::XS::Queue::Array
  http://search.cpan.org/~rcaputo/POE-1.0001/lib/POE.pm#POE::Queue_and_POE::Queue::Array
(possibly using POE::Wheel::Curses and dtach for a management/debugging REPL)

Daemon::Simple
  http://search.cpan.org/~khs/Daemon-Simple-0.02/lib/Daemon/Simple.pm
Pretty minimal solution

Daemon::Generic
  http://search.cpan.org/~muir/Daemon-Generic-0.51/lib/Daemon/Generic.pod
A little more featureful and feels more traditional
  
Net::Daemon
  http://search.cpan.org/~muir/Daemon-Generic-0.51/lib/Daemon/Generic.pod

TaskForest
  http://search.cpan.org/~enoor/TaskForest-1.09/lib/TaskForest.pm
Looks a little crufty, but that may just be the job control file syntax

-- 
Jonathan Conway                                                rise at knavery.net
    Checking whether build environment is sane ... build environment is
          grinning and holding a spatula.  Guess not. - xkcd #371


More information about the spug-list mailing list