[DFW.pm] Concurrency not helping? Maybe you're doing it wrong

kevin kbrannen at pwhome.com
Fri Jan 10 00:15:01 PST 2014

On 01/05/2014 12:51 PM, Tommy Butler wrote:
> Just for fun, I thought I'd post this to the list.  It's a discussion on
> perlmonks about why the reference code (threaded version) doesn't seem
> to have large gains over the non-threaded version.  If you are also
> using concurrency in your hackathon code to try to speed things up, it
> could be a matter of either or both of: you don't need it (really you
> may not), or you're doing it wrong.
> http://perlmonks.org/?node_id=1069338


Sorry I'm slow, but I read the post and thought I'd give you this 
example. It's quite probable that you no longer need this as you have 
everything working, but perhaps it'll help someone. I've been meaning to 
post this to Perl Monks, but haven't gotten around to it.

I remember wanting to do some things with threads some time back and 
having a devil of a time trying to find a simple example that showed 
everything I needed to do. So I cobbled this together.

I think this has the "architecture" you were looking for, i.e. a parent 
in control, the children picking up tasks as fast as they could do them, 
so there's no real gate keeper. As with so many things, picking the 
right values for your task with number of children & number of items in 
the input queue is up to you. :) I think it's reasonably well 
documented, considering it's a test example, which I tend not to 
document at all, much to my detriment later. ;) It's sort of put 
together much in the manner I would have written back in the day when I 
used IPCs in C on a weekly basis.


# Example program for threads and queues.
# The parent will create an input queue for work to be done. It will be
# a number of tasks ($num_tasks). To do the work, it will create a number
# of children ($num_children) to do the work. Because this is a simple
# example, we'll just put a set of random numbers in the queue (from
# 1 to $task_time) for the children to work on. By convention, they know
# to sleep that number of seconds then consider the work done and return a
# "result" to the parent, before grabbing the next task to be done.
# Once all the tasks are done, they end ... as does the parent.
# In real life, the parent could add more work, the children could sleep
# until a specific command was given, etc.

use strict;
use warnings;
use threads qw(stringify);
use Thread::Queue;

my $num_children = 3;
my $num_tasks = 10;
my $task_time = 5;

# create communication queues
my $inputq = Thread::Queue->new();
my $outputq = Thread::Queue->new();

# queue up tasks
for (1 .. $num_tasks)
     $inputq->enqueue(int(rand($task_time) + 1));

# make worker threads
my @children;
my $thr;
for (1 .. $num_children)
     $children[$_] = $thr=threads->create(\&child, $_);

# watch for output from children
while (my $val = $outputq->dequeue())
     # we got something...
     print "parent received [$val]\tprocess check: ";

     # see if we're done and show state of children
     # could probably simplify this to "last if (($inputq->pending() == 
0) && ($outputq->pending() == 0));"
     #   because if both queues are empty, then we're really finished, 
but this way
     #   allows me to see the state of each child
     my $running = 0;
     for (my $c=1 ; $c <= $num_children ; $c++)
         my $state = 'x';
         if ($children[$c])
             if ($children[$c]->is_running())
                 $state = 'R';
                 $children[$c] = 0;
         print "$c=$state ";
     print "\n";
     last if (!$running);
print "parent done\n";


# the work our children will do
sub child
     my ($id) = @_;
     my $arg;

     # while there is more work to do and we can pull something off
     while ($inputq->pending() && ($arg = $inputq->dequeue()))
         # do work

         # tell our parent the results
         $outputq->enqueue("child $id had $arg");

     # all the work is done, exit
     print "child $id exiting\n";
     return 1;

More information about the Dfw-pm mailing list