SPUG: Daemonized work queue handling
Jonathan
rise at knavery.net
Tue May 6 13:23:51 PDT 2008
We've got a process at work that handles generating and sending out
about 20M monthly newsletter emails to site members distributed over a
small cluster and I'm looking for a better way to run it. Currently
each machine starts a cron job that forks semi-daemonized child
processes and redirects their output to a log file in /tmp. They've
been converted to pull work blocks from a db-side queue and get all
their data from it as well, so there's no local state to care about.
Any recommendations or warnings for something that'll play well with
starting from an init script and running for months?
Things that look like they have potential:
Poe::Queue / POE::XS::Queue::Array
http://search.cpan.org/~rcaputo/POE-1.0001/lib/POE.pm#POE::Queue_and_POE::Queue::Array
(possibly using POE::Wheel::Curses and dtach for a management/debugging REPL)
Daemon::Simple
http://search.cpan.org/~khs/Daemon-Simple-0.02/lib/Daemon/Simple.pm
Pretty minimal solution
Daemon::Generic
http://search.cpan.org/~muir/Daemon-Generic-0.51/lib/Daemon/Generic.pod
A little more featureful and feels more traditional
Net::Daemon
http://search.cpan.org/~muir/Daemon-Generic-0.51/lib/Daemon/Generic.pod
TaskForest
http://search.cpan.org/~enoor/TaskForest-1.09/lib/TaskForest.pm
Looks a little crufty, but that may just be the job control file syntax
--
Jonathan Conway rise at knavery.net
Checking whether build environment is sane ... build environment is
grinning and holding a spatula. Guess not. - xkcd #371
More information about the spug-list
mailing list