SPUG: Forked Children talking back to the parent?

[Jason Lamport]

First, you should consider carefully whether splitting the task into 
multiple processes will actually speed things up.  Remember, forking 
a bunch of processes will not magically make your computer hardware 
any faster.  If the bottleneck is from a single local resource, such 
as CPU time or hardware I/O to a single physical device, then using 
multiple processes is not likely to speed things up, and may even 
slow things down.  On the other hand, there are certainly many cases 
in which multiple processes can be much more efficient: e.g. if your 
processes are spending most of their time waiting for replies from 
internet servers, then 50 processes can wait for 50 replies in about 
the same amount of time that one process could wait for one reply.

If you decide that using multiple processes is the way to go:

As usual, TMTOWTDI -- I'm not even going to try to summarize or 
suggest which is best.  Peruse

perldoc perlipc

for lots of info.  The sections "Safe Pipe Opens" and "Bidirectional 
Communication with Yourself" look like they may be most relevant, but 
there's lots of good info throughout.

As for your design:  I don't think you want the children talking 
amongst themselves.  I would guess the best design is a client/server 
system, in which the parent process acts as a "task server", which 
delegates the to-do items to its children, and when the children find 
a new task, they simply report it back to the server.

#example pseudo-code -- this is just one possible design.
#note that you could use two completely different IPC methods
#for parent->child communication and for child->parent communication.

$number_of_children = 10;

for (1..$number_of_children) {
	last unless $pid = fork();
	push @child_pids, $pid;
}

if ( @child_pids ) {	#I'm the parent/server
	while ( @master_to_do_list ) {
		$next_task = shift( @master_to_do_list );
		delegate( $next_task );
			#send this task to one of
			#the children using whatever IPC
			#method seems best
		push @master_to_do_list, get_new_tasks_from_children();
	}
} else {	#I'm a child
	@my_to_do_list = get_tasks_from_parent();
	while ( @my_to_do_list ) {
		do_this( shift @my_to_do_list );
		if ( I find something else to do ) {
			tell_parent_about( $this_new_thing );
				#send this task to the parent
				#using whatever IPC
				#method seems best
		}
		push @my_to_do_list, get_new_tasks_from_parent();
	}
}

__END__

Whatever your design, take care to consider whether two children 
might attempt the same task at the same time, or discover the same 
new task at the same time, and what happens if/when they do.

-jason



At 7:43 AM -0800 1/17/01, John Cokos wrote:
>Working on a long running perl script, that I think would be better 
>suited to using fork() than running on it's own.
>
>Basically, the pseudocode for it is something like this:
>
>
>
>code:------------------------------------------------------------------------------my 
>@to_do_list
>
>push @to_do_list, $something;
>
>foreach my $item ( @to_do_list ) {
>
>     .. do some work.
>     if ( I find something else to do)  {
>        push @to_do_list, $this_new_thing;
>     }
>
>}
>------------------------------------------------------------------------------
>
>
>So, you can see that it's an array that can be added to in 
>mid-stream, and continually grows. Depending on the amount of things
>"found to do" in the inside loop, it can take hours and hours to 
>run, as that array grows larger.
>
>What I'm wanting to do, is fork off 10 child processes, and have 
>each of them doing the same loop, each aware of the master array's
>contents, and each able to push things onto it, so that the other 
>children, when they're in the loop see the new todo items...
>
>Can the children intercommunicate?
>
>If not, can they at least return a value back to the parent?
>In this case, we could have it fork 10, give them a task, get back 
>the new todo items found, and die. Then, fork 10 new kids, etc
>....
>
>Thanks, John
>========================================
>   John Cokos, President / CEO: iWeb Inc.
>   http://www.iwebsys.com
>   jcokos at ccs.net
>========================================
>
>
>
>  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>      POST TO: spug-list at pm.org       PROBLEMS: owner-spug-list at pm.org
>       Subscriptions; Email to majordomo at pm.org:  ACTION  LIST  EMAIL
>   Replace ACTION by subscribe or unsubscribe, EMAIL by your Email-address
>  For daily traffic, use spug-list for LIST ;  for weekly, spug-list-digest
>   Seattle Perl Users Group (SPUG) Home Page: http://www.halcyon.com/spug/


 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     POST TO: spug-list at pm.org       PROBLEMS: owner-spug-list at pm.org
      Subscriptions; Email to majordomo at pm.org:  ACTION  LIST  EMAIL
  Replace ACTION by subscribe or unsubscribe, EMAIL by your Email-address
 For daily traffic, use spug-list for LIST ;  for weekly, spug-list-digest
  Seattle Perl Users Group (SPUG) Home Page: http://www.halcyon.com/spug/