[tpm] Single thread Perl 5.8.8 vs. Multi-threaded Perl 5.10.1

[Shane Boyce]

Just wondering if anyone has experience with multi-threaded Perl? And if the following benchmarks are typical of people's experiences?

I'm using a Solaris 10 system on a sun-sparc workstation and just went from the precompiled package from www.sunfreeware.com of Perl 5.8.8 to a compiled source version of Perl 5.10.1 /w multi-threading turned on.

Benchmarks of a program I used went like this:

SUMMARY:
########
I had to got to 12 threads to make it worth my while! That's a bit nuts. Is multi-threaded Perl 5.10.1 really so much slower than single threaded Perl 5.8.8? Or have I likely hindered it somehow? (I pretty sure debug mode was turned off) Thanks.

Final Results D5, Initial was A1:
        (A1) [1:13.57 min] "real" wall clock time
        (D5) [0:32.46 min] "real" wall clock time
But that is still using real seconds (summing thread totals of):
        (A1) [1:03.13 + 0.64] = [1:03.77 min] User + Sys time
        (D5) [2:09.34 + 2.97] = [2:12.31 min] User + Sys time

So D5 ends up being faster and takes up 44% of original time, but uses 209% more CPU time and +12 CPU threads to do it!

DETAILS:
########

Measurements using Benchmark.pm /w Hi::Res timing inside the file to benchmark individual subroutines are not shown. Shown here are the "timex" values of the main process' overall performance.

System:
        1 CPU /w 8 core /w 4 threads per core = 32 available threads.
        16 GB RAM (8 GB available for apps)
        Oracle backend running (using 8 GB of RAM) on same server as Perl application.

Data:
        - 468 tests
        - 1270 devices tested with those tests
        - 565003 individual measurements in total
          (Lower than 468x1270=594360 as not all devices were tested for all tests).

A1) Non-optimized, /wo use strict version:
       real     1:13.57
       user     1:03.13
       sys         0.64

A2) Original single threaded Perl 5.8.8:
       real     1:00.90
       user       54.53
       sys         0.63

B) Original code, but switched to multi-threaded Perl 5.10.1 as compiler:
       real     1:24.91
       user     1:17.95
       sys         0.61

I then went multi-threaded, but did it for a section of the code after many 100's of MB of data had been fetched (not a great idea once I understood that the threads copy EVERYTHING, not just what's passed to the thread subroutine):

    C1) #Timing: Perl 5.10.1 /w threading enabled (2 threads):
       real     1:17.28
       user     1:43.31
       sys         2.93

    C2) #Timing: Perl 5.10.1 /w threading enabled (8 threads):
       real     1:23.73
       user     1:43.18
       sys        10.35

    C3) #Timing: Perl 5.10.1 /w threading enabled (4 threads):
       real     1:10.04
       user     1:37.59
       sys         1.26

Then I got a bit smarter and went multi-threaded before fetching data, and only fetched the data from Oracle required by each thread (so results in more database accesses, but less memory copies):

    Used Thread::Queue to feed test list to threads. Each thread then processed that test and moved on to next one in the queue.

    D1) #Timing: Perl 5.10.1 /w threading enabled (2 thread) (thread whole fetch process per test):
       real     1:51.23
       user     3:04.13
       sys         4.71

    D2) #Timing: Perl 5.10.1 /w threading enabled (4 thread) (thread whole fetch process per test):
       real     1:01.82
       user     2:19.19
       sys         3.40

    D3) #Timing: Perl 5.10.1 /w threading enabled (6 thread) (thread whole fetch process per test):
       real       46.43
       user     2:08.68
       sys         3.08

    D4) #Timing: Perl 5.10.1 /w threading enabled (8 thread) (thread whole fetch process per test):
       real       42.35
       user     2:09.36
       sys         3.09

    D5) #Timing: Perl 5.10.1 /w threading enabled (12 thread) (thread whole fetch process per test):
       real       32.46
       user     2:09.34
       sys         2.97

    D6) #Timing: Perl 5.10.1 /w threading enabled (16 thread) (thread whole fetch process per test):
       real       30.63
       user     2:11.11
       sys         2.94

    D7) #Timing: Perl 5.10.1 /w threading enabled (32 thread) (thread whole fetch process per test):
       real       31.26
       user     2:13.50
       sys         3.13


Thanks,
Shane K. Boyce

This communication contains confidential information intended only for the addressee(s). If you have received this communication in error, please notify us immediately and delete this communication from your mail box.