[VPM] Perl threads recursion

[abez]

Forkbomb alert.

Trust me, you don't want that running 200+ threads waiting on I/O operations
won't get you anywhere. Also the reason it is time consuming is because you're
waiting on I/O operations which are often times thousands to millions of times
slower than your cpu.

Yes you can make some gains doing certain operations in parallel because it
will read the harddisk as it travels about the platter searching for your
inodes.

What I think you really want is to have a limited size threadpool and have
these calls using waiting on threads in the pool. Your thread pool can either
just be a integer which counts how many threads or it can be an actual
collection of threads ready and waiting to attack the current problem (e.g.
waiting to run the recursive code on a parameter).

Another issue is you have to be clear what threads you are using. If you're
using user level threads your blocking operations will block all the threads
and you will gain nothing.

If it's kernel level threads then you'll be fine.

Basically if I was doing something you were doing I'd have this:

1. A queue of the current command to run. Threads will be blocked if there is
nothing on this queue.
2. Each thread goes like this
	do
		get new parameter from queue #blocking or polling
		run subroutine
			inside subroutine if we have to spawn again add that to the
			queue
		end subroutine
	loop

---- Totall Untested Example code ---------
use threads;
use threads::shared;
my $qmutex : shared = 1;
sub run {
	for(;;) {
		my $command="";
		{
			lock($qmutex);
			$command = shift(@queue);
		} #unlocks mutex
		if ($command) {
			subroutine($command);
		}
	}
}

sub addToQ { #call this inside subroutine to add your thing to the queue
	my ($val) = @_;
	lock($qmutex);
	push @queue,$val;
}

perldoc  perlthrtut

On Wed, 7 May 2003, Jeremy Aiyadurai wrote:

> hi all,
>
> I have the following problem, involving recursion and threads. i have a
> recursive subroutine that scans a directory structure from a given root, and
> performs operations on files it finds which meet a certain criteria. I find,
> that running this in a single thread is rather time consuming, especially
> when there are alot of files to process and directories to go through.
>
> I want a new thread created for each recursive call to the subroutine. The
> subroutine is called when a sub directory is found. I am not an expert on
> threads or forks in perl, and have had alot of errors, when trying to apply
> them. I am using perl 5.8 from Activestate. My OS is Windows 2000 Pro.
>
> Your help is much appreciated.
> thanks in advance,
>
> Jeremy A.
>
> P.S: below this line is the recursive subroutine.
> ------------------------------------------------------------------------------
> sub rgrab ($$) {
>     my $current = $_[0];
>     my $path    = "$_[1]/$current";
>     mkdir($current);
>     chdir($current);
>     opendir(DDD,".");
>     my @files = readdir(DDD);
>     foreach (@files) {
>         if (-f $_) # if a file
>         {
>            if($_ =~ /filenamehere/ig)
>            {
> # do stuff to file here
>            }
>         }
>         elsif(-d $_) #if a directory
>         {
>          #print "$_\n";
>          if (($_ ne ".") && ($_ ne ".."))
>          {
>          print "changing to directory:$_\n";
>          &rgrab( $_, $path );
>          chdir("..");
>          }
>         }
>     }
>     closedir(DDD);
> }
>
> _________________________________________________________________
> The new MSN 8: advanced junk mail protection and 2 months FREE*
> http://join.msn.com/?page=features/junkmail
>

-- 
abez ------------------------------------------
http://www.abez.ca/ Abram Hindle (abez at abez.ca)
------------------------------------------ abez