SPUG: Forked Children talking back to the parent?
Jason Lamport
jason at strangelight.com
Wed Jan 17 13:13:09 CST 2001
First, you should consider carefully whether splitting the task into
multiple processes will actually speed things up. Remember, forking
a bunch of processes will not magically make your computer hardware
any faster. If the bottleneck is from a single local resource, such
as CPU time or hardware I/O to a single physical device, then using
multiple processes is not likely to speed things up, and may even
slow things down. On the other hand, there are certainly many cases
in which multiple processes can be much more efficient: e.g. if your
processes are spending most of their time waiting for replies from
internet servers, then 50 processes can wait for 50 replies in about
the same amount of time that one process could wait for one reply.
If you decide that using multiple processes is the way to go:
As usual, TMTOWTDI -- I'm not even going to try to summarize or
suggest which is best. Peruse
perldoc perlipc
for lots of info. The sections "Safe Pipe Opens" and "Bidirectional
Communication with Yourself" look like they may be most relevant, but
there's lots of good info throughout.
As for your design: I don't think you want the children talking
amongst themselves. I would guess the best design is a client/server
system, in which the parent process acts as a "task server", which
delegates the to-do items to its children, and when the children find
a new task, they simply report it back to the server.
#example pseudo-code -- this is just one possible design.
#note that you could use two completely different IPC methods
#for parent->child communication and for child->parent communication.
$number_of_children = 10;
for (1..$number_of_children) {
last unless $pid = fork();
push @child_pids, $pid;
}
if ( @child_pids ) { #I'm the parent/server
while ( @master_to_do_list ) {
$next_task = shift( @master_to_do_list );
delegate( $next_task );
#send this task to one of
#the children using whatever IPC
#method seems best
push @master_to_do_list, get_new_tasks_from_children();
}
} else { #I'm a child
@my_to_do_list = get_tasks_from_parent();
while ( @my_to_do_list ) {
do_this( shift @my_to_do_list );
if ( I find something else to do ) {
tell_parent_about( $this_new_thing );
#send this task to the parent
#using whatever IPC
#method seems best
}
push @my_to_do_list, get_new_tasks_from_parent();
}
}
__END__
Whatever your design, take care to consider whether two children
might attempt the same task at the same time, or discover the same
new task at the same time, and what happens if/when they do.
-jason
At 7:43 AM -0800 1/17/01, John Cokos wrote:
>Working on a long running perl script, that I think would be better
>suited to using fork() than running on it's own.
>
>Basically, the pseudocode for it is something like this:
>
>
>
>code:------------------------------------------------------------------------------my
>@to_do_list
>
>push @to_do_list, $something;
>
>foreach my $item ( @to_do_list ) {
>
> .. do some work.
> if ( I find something else to do) {
> push @to_do_list, $this_new_thing;
> }
>
>}
>------------------------------------------------------------------------------
>
>
>So, you can see that it's an array that can be added to in
>mid-stream, and continually grows. Depending on the amount of things
>"found to do" in the inside loop, it can take hours and hours to
>run, as that array grows larger.
>
>What I'm wanting to do, is fork off 10 child processes, and have
>each of them doing the same loop, each aware of the master array's
>contents, and each able to push things onto it, so that the other
>children, when they're in the loop see the new todo items...
>
>Can the children intercommunicate?
>
>If not, can they at least return a value back to the parent?
>In this case, we could have it fork 10, give them a task, get back
>the new todo items found, and die. Then, fork 10 new kids, etc
>....
>
>Thanks, John
>========================================
> John Cokos, President / CEO: iWeb Inc.
> http://www.iwebsys.com
> jcokos at ccs.net
>========================================
>
>
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> POST TO: spug-list at pm.org PROBLEMS: owner-spug-list at pm.org
> Subscriptions; Email to majordomo at pm.org: ACTION LIST EMAIL
> Replace ACTION by subscribe or unsubscribe, EMAIL by your Email-address
> For daily traffic, use spug-list for LIST ; for weekly, spug-list-digest
> Seattle Perl Users Group (SPUG) Home Page: http://www.halcyon.com/spug/
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
POST TO: spug-list at pm.org PROBLEMS: owner-spug-list at pm.org
Subscriptions; Email to majordomo at pm.org: ACTION LIST EMAIL
Replace ACTION by subscribe or unsubscribe, EMAIL by your Email-address
For daily traffic, use spug-list for LIST ; for weekly, spug-list-digest
Seattle Perl Users Group (SPUG) Home Page: http://www.halcyon.com/spug/
More information about the spug-list
mailing list