[Melbourne-pm] I <3 map & grep too :)

[Anneli Cuss]

> And indeed, I picked up this style of programming from Scala, which does do
> those operations in parallel - in both senses.
> It can start running the later operations before the early ones have
> finished - and it can use multiple CPUs to process each stage in parallel.

I misunderstood the point you were making, but I might as well share
what I've got, as it could serve as a springboard for others. It
shouldn't be too hard to feed results into subsequent operations early
(though you might need to ugly up the syntax or use something like
iterators to make it happen).

I'm sure there's already this on the CPAN (or ten of them), but maybe
it'll be as fun for others to read as it was for me to write. A simple
parallel map:

use threads;
use threads::shared;

our $PARALLEL = 4;

sub pmap (&@) {
    my $fun = shift;
    my @args :shared = @_;

    my $each = @args / $PARALLEL;
    my @threads;

    for ($i = 0; $i < $PARALLEL; ++$i) {
	my $from = $each * $i;
	my $to = ($i < $PARALLEL - 1) ? ($each * ($i + 1) - 1) : $#args;
	print STDERR "from $from to $to\n";

	my ($thr) = threads->create(sub {
	    map &$fun, @args[$from..$to]
	});
	push @threads, $thr;
    }

    print STDERR "joining\n";
    my @results;
    push @results, $_->join for (@threads);
    @results;
}

Adjust $PARALLEL to the number of cores your CPU has.

SERVING SUGGESTION:

sub dumb_fib {
    my $n = shift;
    $n <= 2 ? 1 : dumb_fib($n-1) + dumb_fib($n-2)
}

my @a = (39) x 8;
my @r = pmap { dumb_fib $_ } @a;
print join ', ', @r;

With $PARALLEL = 8 on a suitable server:

$ time perl pmap.pl
from 0 to 0
from 1 to 1
from 2 to 2
from 3 to 3
from 4 to 4
from 5 to 5
from 6 to 6
from 7 to 7
joining
63245986, 63245986, 63245986, 63245986, 63245986, 63245986, 63245986, 63245986
real	1m20.436s
user	10m36.575s
sys	0m0.188s
$

If I replace 'pmap' with 'map':

$ time perl map.pl
63245986, 63245986, 63245986, 63245986, 63245986, 63245986, 63245986, 63245986
real	5m56.631s
user	5m56.570s
sys	0m0.006s
$

So, half the time spent in the CPU (not managing context switches),
but 5 times longer on the wall!

- Anneli