From diona at studio12a.com Wed Jul 13 10:16:46 2005 From: diona at studio12a.com (Diona Kidd) Date: Wed, 13 Jul 2005 13:16:46 -0400 Subject: [Charlotte.PM] issues with Test::Simple Message-ID: <926594887809b065583e68659ca6dd4c@studio12a.com> Has anyone had issues with Test::Simple or other modules in perl with regards to threads? When I install this module bundle, I see the error "lock can only be used on shared values" from several of the module's tests and it errors out. The error seems to come from with perl itself and I'm wondering if it has something to do with threads. It's happening on both platforms I've tried, Redhat and Mac OS X. They both appear to have the default system perl. For some reason, I was thinking threads were disabled by default?? Meanwhile, I'm compiling a new instance of perl 5.8.7 with threads disabled. While talking to another developer, he mentioned that a lot of modules on CPAN aren't thread-safe and that this is a little dangerous with regards to Apache. Does anyone know more about this? Regards, Diona Kidd [ http://www.modernflow.com ] From william at knowmad.com Wed Jul 13 11:27:44 2005 From: william at knowmad.com (William McKee) Date: Wed, 13 Jul 2005 14:27:44 -0400 Subject: [Charlotte.PM] issues with Test::Simple In-Reply-To: <926594887809b065583e68659ca6dd4c@studio12a.com> References: <926594887809b065583e68659ca6dd4c@studio12a.com> Message-ID: <20050713182744.GF4024@knowmad.com> On Wed, Jul 13, 2005 at 01:16:46PM -0400, Diona Kidd wrote: > Has anyone had issues with Test::Simple or other modules in perl with > regards to threads? I try to avoid threads ATM except when running ActiveState Perl on Windows which is compiled with threads. > When I install this module bundle, I see the error > "lock can only be used on shared values" from several of the module's > tests and it errors out. The error seems to come from with perl itself > and I'm wondering if it has something to do with threads. I've never seen that one. Have you reported this error in CPAN's bug tracker? > For some reason, I was thinking threads were disabled by default?? What's the output from `perl -V`? > Meanwhile, I'm compiling a new instance of perl 5.8.7 with threads > disabled. While talking to another developer, he mentioned that a lot > of modules on CPAN aren't thread-safe and that this is a little > dangerous with regards to Apache. Does anyone know more about this? I've done a bit of research on this issue. My understanding is that this was true for the original implementation of threads but that the newer version should make all modules thread-safe by default[1]. I'd be interested to hear from someone who has more experience. Good luck, William [1] http://perldoc.perl.org/Thread.html -- Knowmad Services Inc. http://www.knowmad.com From diona at studio12a.com Wed Jul 13 12:16:35 2005 From: diona at studio12a.com (Diona Kidd) Date: Wed, 13 Jul 2005 15:16:35 -0400 Subject: [Charlotte.PM] issues with Test::Simple In-Reply-To: <20050713182744.GF4024@knowmad.com> References: <926594887809b065583e68659ca6dd4c@studio12a.com> <20050713182744.GF4024@knowmad.com> Message-ID: >> For some reason, I was thinking threads were disabled by default?? > > What's the output from `perl -V`? threading on both...Looks like compiling from source with threading turned off has solved the issue. I'll look into reporting the issue on cpan.. Thanks, Diona From william at knowmad.com Sat Jul 16 06:31:15 2005 From: william at knowmad.com (William McKee) Date: Sat, 16 Jul 2005 09:31:15 -0400 Subject: [Charlotte.PM] Meeting Reminder - No meeting in July Message-ID: <20050716133115.GK616@knowmad.com> Hi Mongers, This is a quick reminder email that we are not meeting next week. Meetings will resume in August. Stay tuned for a great Fall line-up with presentations by local companies who are using Perl for serious application development. Enjoy your summer, William -- Knowmad Services Inc. http://www.knowmad.com From diona at studio12a.com Sun Jul 17 12:52:04 2005 From: diona at studio12a.com (Diona Kidd) Date: Sun, 17 Jul 2005 15:52:04 -0400 Subject: [Charlotte.PM] issues with Test::Simple In-Reply-To: <36540a84050717121714b6ec6b@mail.gmail.com> References: <926594887809b065583e68659ca6dd4c@studio12a.com> <20050713182744.GF4024@knowmad.com> <36540a84050717121714b6ec6b@mail.gmail.com> Message-ID: <45302549b64b1da524661323aca8c3b0@studio12a.com> Turns out there is a bug in Apache::Test on any calls to Test::Builder due to a difference in method calls. Apache::Test is calling Test::Builder with class methods when object methods are needed. The update is in the Apache CVS but not yet on CPAN. I've attached the patch and test I received from Michael Schwern, the maintainer of Test::Builder. I've already installed this patch on three systems (2 linux, 1 mac) with no issues. -------------- next part -------------- A non-text attachment was scrubbed... Name: at.patch Type: application/octet-stream Size: 1204 bytes Desc: not available Url : http://mail.pm.org/pipermail/charlotte/attachments/20050717/ccff07ff/at.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: testmore.t Type: application/octet-stream Size: 294 bytes Desc: not available Url : http://mail.pm.org/pipermail/charlotte/attachments/20050717/ccff07ff/testmore.obj -------------- next part -------------- Regards, Diona Kidd On Jul 17, 2005, at 3:17 PM, Kurt W. Whitsel wrote: > FYI, > > I ran into this exact same problem when trying to install Test::Simple > from cpan > > On 7/13/05, Diona Kidd wrote: >>>> For some reason, I was thinking threads were disabled by default?? >>> >>> What's the output from `perl -V`? >> >> threading on both...Looks like compiling from source with threading >> turned off has solved the issue. I'll look into reporting the issue on >> cpan.. >> >> Thanks, >> >> Diona >> >> _______________________________________________ >> charlotte mailing list >> charlotte at pm.org >> http://mail.pm.org/mailman/listinfo/charlotte >> From drewhead at drewhead.org Tue Jul 26 20:20:08 2005 From: drewhead at drewhead.org (drewhead@drewhead.org) Date: Tue, 26 Jul 2005 23:20:08 -0400 (EDT) Subject: [Charlotte.PM] Help me tune this for speed Message-ID: <200507270320.j6R3K8eL017182@keep.drewhead.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Here's an interesting problem I've hit recently [and totally diffrent from the recent template discussions :) ] I have a set of X,Y coordinates. I'm trying to describe these coordinates. One of the measures I'm trying to make is a 'remoteness' factor. I'm going to define this factor as a set of statistical metrics (mean, median, stddev) on the set of edge between 'neighbooring' points. A edge is the distance between to points. A point has definable neighboors by having an edge that no shorter edge that also describes a neighboor intersects. So my goal is to find the complete list of non intersected edges between neighboors. After that the stats become trivial. I'll compute all the unique edges first, then loop though them sorted smallest to largest. Per edge I'll look through all previously found non intersecting edges to prove/disprove intersection. Since the smallest edge HAS to to be non intersecting bu a smaller one [non exist] this should work looking through the list in sorted order. The problem is that this algorithm is O(N^3). That's kind of nasty cause it scales badly. The example I'm providing is 301 points (I'll need to process several thousand) which baloones up to 27+ million caculations at the limit. I'm going to use Math::Geometry::Planar which has a SegmentIntersection function. I've tinkere dwith this for about a week, and think I'm loosing more time in my storage/control loops than the actual math. The example data file processes on my XP2500 (linux) in just shy of 25 minutes. I ran a 2001 point grid over the weekend that never came back. So, here are the example code and an datafile with 301 coordinates. Anyone have any ideas of how to speed this up? Here's my output: Found unique Edges = 90300 1 wallclock secs ( 1.28 usr + 0.06 sys = 1.34 CPU) Unintersected edges 1778 1487 wallclock secs (1458.49 usr + 1.35 sys = 1459.84 CPU) - -- Drew Dowling Drewhead http://www.drewhead.org drewhead at drewhead.org | | WWW / \ Alpha Phi Omega Concord, North Carolina | | Nimat / \ Gamma Lambda CLEMSON UNIVERSITY ALUMNI | | Apatschin /_____\ TIGER BAND! | VGAP4 Hosting at http://vgap.drewhead.org -----BEGIN PGP SIGNATURE----- Comment: Public key available at http://www.drewhead.org iD8DBQFC5vxV8J7U7yHE638RAoJUAKCiV2SPMTo4x0LcWnZ4kd2AFiYDOACdEKqL IdAbgBgseXlIRWLxNXANq6A= =lKdU -----END PGP SIGNATURE----- -------------- next part -------------- #!/usr/bin/perl ###################################################################### # CPAN modules ###################################################################### use YAML qw(Dump Load); use Carp; use Benchmark; use Math::Geometry::Planar; use strict; ###################################################################### # A few globals ###################################################################### my $points; # hash ref holding the raw data {'1' => [X,Y], '2' => [X,Y], ...} my @edges; # Array to keep refrences to all edges [edge, X1, Y1, X2, Y2] ... my $neighborEdges; # hash ref to hold sorted neighbor edges {'1' => edge1, '2' => edge2, ...} ###################################################################### # # MAIN # ###################################################################### # Read in the X,Y coordinates from supplied data { local $/; open(my $fh, "dump_301.dat") or die "Unable to open file: $!\n"; $points = Load( <$fh> ); croak($@) if $@; close($fh); } # Find all the unique edges eval { my $start_distance = Benchmark->new; my $found = {}; # p1 is starting or anchor point of the line segment foreach my $p1 (@{$points}) { # p1 is end point of the line segment foreach my $p2 (@{$points}) { # We don't need to caculate if anchor and end are the same # or we have already seen these two pairs [reversed] before unless ( ($p1 == $p2) || $found->{$p2}->{$p1} ) { # Compute the edge my $edge = sqrt( ($p1->[0] - $p2->[0])**2 + ($p1->[1] - $p2->[1])**2 ); # Push a ref of the whole thing on a stack push (@edges, [ $edge, $p1->[0], $p1->[1], $p2->[0], $p2->[1] ]); # Keep track of the edges we've already computed (no need to do so twice) $found->{$p2}->{$p1} = 1; } } } print "Found unique Edges = ", scalar(@edges), "\n"; print timestr(timediff(Benchmark->new, $start_distance)), "\n"; }; croak($@) if ($@); # Compute the neighboors my $start_intersect = Benchmark->new; # Need an integer to keep our sort order my $neighboorCNT = 1; # Loop trough all found unique edges from smallest to largest foreach my $aref ( sort {$a->[0] <=> $b->[0] } @edges) { my $neighbor = 1; # default true # Loop through all previously found edges in order for (my $n=1;$n<=$neighboorCNT;$n++) { if ($neighborEdges->{$n}) { my $points_ref = []; # Data structure to pass to SegmentIntersection push (@{$points_ref}, [$aref->[1],$aref->[2]]); # line 1 point A X,Y push (@{$points_ref}, [$aref->[3],$aref->[4]]); # line 1 point B X,Y push (@{$points_ref}, [$neighborEdges->{$n}->[1],$neighborEdges->{$n}->[2]]); # line 2 point A X,Y push (@{$points_ref}, [$neighborEdges->{$n}->[3],$neighborEdges->{$n}->[4]]); # line 2 point B X,Y my @p = SegmentIntersection($points_ref); # If a intersect is found, set false and quit checking if (SegmentIntersection($points_ref)) { $neighbor = 0; last; } } } if ($neighbor) { $neighborEdges->{$neighboorCNT} = $aref; $neighboorCNT++; } } print "Unintersected edges ", scalar(keys %{$neighborEdges}), "\n"; print timestr(timediff(Benchmark->new, $start_intersect)), "\n"; -------------- next part -------------- --- - - 3000 - 3000 - - 3173 - 3667 - - 3817 - 2706 - - 2309 - 3198 - - 3613 - 2917 - - 2953 - 3366 - - 3023 - 2702 - - 2558 - 2708 - - 2918 - 2353 - - 2948 - 3210 - - 3662 - 2469 - - 2457 - 3585 - - 3277 - 2893 - - 3000 - 2818 - - 2974 - 3494 - - 2988 - 3171 - - 2528 - 2910 - - 2154 - 3854 - - 3126 - 3718 - - 3087 - 3790 - - 2787 - 3257 - - 1872 - 3383 - - 3143 - 2742 - - 2741 - 2767 - - 3333 - 3356 - - 3115 - 3414 - - 2427 - 2329 - - 3313 - 2925 - - 2531 - 3534 - - 2743 - 3226 - - 2849 - 3265 - - 3945 - 2518 - - 2839 - 3287 - - 2449 - 2498 - - 3125 - 3792 - - 3503 - 3135 - - 3063 - 2492 - - 3424 - 3125 - - 3761 - 3385 - - 2439 - 2904 - - 2818 - 2839 - - 3148 - 3186 - - 2925 - 2216 - - 2965 - 3256 - - 2882 - 3324 - - 3185 - 2991 - - 2723 - 3811 - - 3309 - 3115 - - 3275 - 2773 - - 3059 - 3718 - - 3388 - 3022 - - 3337 - 3383 - - 3588 - 2823 - - 3463 - 3149 - - 3633 - 3660 - - 3086 - 3541 - - 2959 - 2441 - - 3222 - 2574 - - 2962 - 2793 - - 2238 - 2594 - - 3337 - 3032 - - 2938 - 2713 - - 2936 - 3359 - - 2644 - 2735 - - 2589 - 3791 - - 2683 - 2494 - - 3183 - 2552 - - 2974 - 3120 - - 2706 - 4180 - - 2730 - 2529 - - 2730 - 3205 - - 2911 - 3524 - - 3181 - 2675 - - 3048 - 2286 - - 2651 - 2772 - - 2799 - 2900 - - 2912 - 2464 - - 3587 - 3147 - - 3251 - 3181 - - 2799 - 2156 - - 3004 - 3928 - - 3780 - 2813 - - 2745 - 2736 - - 3157 - 3181 - - 2991 - 2591 - - 3557 - 2763 - - 2925 - 2696 - - 3010 - 3300 - - 2686 - 2532 - - 3331 - 2724 - - 3032 - 3834 - - 2686 - 2880 - - 3076 - 3314 - - 2774 - 2911 - - 2857 - 2416 - - 2955 - 2762 - - 2535 - 2981 - - 2598 - 2767 - - 3459 - 3010 - - 3078 - 3539 - - 2756 - 3453 - - 2721 - 3711 - - 2609 - 2133 - - 2565 - 3090 - - 2763 - 3231 - - 2701 - 2645 - - 2791 - 2963 - - 3154 - 2673 - - 2512 - 2749 - - 2348 - 2689 - - 3097 - 3297 - - 2304 - 2918 - - 3065 - 3281 - - 2907 - 2777 - - 3495 - 2896 - - 3451 - 2845 - - 3145 - 2405 - - 3167 - 3509 - - 3238 - 3624 - - 2824 - 3314 - - 3361 - 2782 - - 2717 - 3126 - - 2934 - 3239 - - 2826 - 3373 - - 3217 - 2854 - - 3975 - 2611 - - 2560 - 2850 - - 3138 - 3555 - - 2949 - 2863 - - 3118 - 3395 - - 2538 - 3119 - - 3056 - 2398 - - 3191 - 2578 - - 2590 - 3183 - - 3539 - 3345 - - 2802 - 2990 - - 2707 - 3257 - - 3580 - 3157 - - 3299 - 3205 - - 2712 - 2922 - - 2901 - 2539 - - 2992 - 3249 - - 2499 - 3111 - - 3103 - 3120 - - 3031 - 2933 - - 3298 - 2955 - - 3465 - 3452 - - 3617 - 3627 - - 2645 - 3446 - - 2876 - 2847 - - 2929 - 3014 - - 3481 - 3329 - - 3082 - 2934 - - 2953 - 2711 - - 3239 - 2108 - - 3293 - 3168 - - 2230 - 2904 - - 3657 - 2852 - - 2046 - 2934 - - 3291 - 3243 - - 3748 - 2576 - - 3475 - 3455 - - 2586 - 2869 - - 3683 - 2615 - - 3260 - 2988 - - 3672 - 2938 - - 2178 - 2317 - - 3455 - 3307 - - 3197 - 2996 - - 3215 - 3746 - - 2771 - 3208 - - 2454 - 2732 - - 3344 - 2609 - - 2706 - 2313 - - 2772 - 2693 - - 3114 - 3374 - - 3482 - 2927 - - 2887 - 2625 - - 2474 - 3002 - - 2712 - 1858 - - 2979 - 3304 - - 3251 - 2732 - - 2828 - 3071 - - 3342 - 3747 - - 2192 - 3265 - - 3160 - 2658 - - 2394 - 3358 - - 2667 - 3390 - - 2612 - 2497 - - 2753 - 3108 - - 3521 - 2839 - - 2840 - 2920 - - 3742 - 3082 - - 2710 - 3233 - - 3070 - 2840 - - 3309 - 2894 - - 3033 - 2684 - - 2464 - 2876 - - 2826 - 3492 - - 2069 - 2379 - - 3446 - 2943 - - 2500 - 3122 - - 2748 - 3024 - - 2813 - 2591 - - 2573 - 2871 - - 3325 - 3017 - - 3595 - 3255 - - 3569 - 2639 - - 2714 - 3475 - - 3133 - 3024 - - 2155 - 3133 - - 3320 - 3303 - - 3999 - 2247 - - 3601 - 3322 - - 2619 - 2782 - - 2709 - 2670 - - 3081 - 2985 - - 3623 - 3498 - - 3088 - 3076 - - 2795 - 3799 - - 2344 - 3187 - - 3029 - 3348 - - 2257 - 2628 - - 2910 - 3563 - - 3206 - 2674 - - 3447 - 3079 - - 2966 - 3759 - - 3446 - 3207 - - 3904 - 2947 - - 3320 - 3086 - - 3063 - 2716 - - 2638 - 2919 - - 2380 - 2913 - - 3210 - 3027 - - 3333 - 2668 - - 3200 - 1955 - - 3285 - 2637 - - 2537 - 2582 - - 3678 - 3411 - - 2871 - 3594 - - 2757 - 3157 - - 3384 - 2871 - - 2727 - 3279 - - 4219 - 2948 - - 3207 - 2274 - - 2273 - 3298 - - 2728 - 2875 - - 2995 - 3091 - - 2775 - 2624 - - 2901 - 3334 - - 2548 - 4252 - - 3063 - 2879 - - 2800 - 2398 - - 2681 - 2964 - - 3231 - 3185 - - 3578 - 3063 - - 3255 - 2697 - - 2433 - 2821 - - 2592 - 2147 - - 3300 - 2580 - - 2738 - 4050 - - 2651 - 3082 - - 2948 - 2703 - - 2963 - 3084 - - 3054 - 3372 - - 2778 - 3497 - - 2722 - 3325 - - 3370 - 3582 - - 3403 - 3704 - - 2685 - 3075 - - 2071 - 3389 - - 2909 - 2435 - - 3474 - 2698 - - 2628 - 2260 - - 2868 - 3273 - - 2177 - 2833 - - 2188 - 2026 - - 3125 - 2615 - - 2723 - 2649 - - 2350 - 3173 - - 2166 - 2800 - - 2533 - 2361 - - 2237 - 3359 - - 3533 - 2768 - - 2771 - 2453 - - 2758 - 3087 - - 2614 - 2869 - - 3141 - 2617 - - 2756 - 2719 - - 2947 - 3278 - - 3325 - 3045 - - 2588 - 2559 - - 3679 - 2870 - - 2948 - 3108 - - 2996 - 3065 - - 3195 - 2784 - - 2885 - 2780 - - 2729 - 3023 - - 3024 - 2611 - - 2819 - 3254 - - 2525 - 2495 From steven at knowmad.com Wed Jul 27 06:16:07 2005 From: steven at knowmad.com (Steven Bush) Date: Wed, 27 Jul 2005 09:16:07 -0400 Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <200507270320.j6R3K8eL017182@keep.drewhead.org> References: <200507270320.j6R3K8eL017182@keep.drewhead.org> Message-ID: <42E78917.7030301@knowmad.com> Hi Drew, I've found one point that I think will help speed your process up. When you loop through all found unique edges from smallest to largest and again through all previously found edges in order, you build an array ref each time ($points_ref). When you are populating this array with data, the first two things you put into it do not use any information that is generated by the second loop. So you are basically preforming a pair of calculations 90300 times that only needs to be calculated 1778 times. I've re-written the associated area of code (lines 84-87, 94 in my code) to fix this problem, and it is included below. I have not yet tried to run this code, so I'm not sure how much help it will be, but I think it should help some. ~Steven #!/usr/bin/perl ###################################################################### # CPAN modules ###################################################################### use YAML qw(Dump Load); use Carp; use Benchmark; use Math::Geometry::Planar; use strict; ###################################################################### # A few globals ###################################################################### my $points; # hash ref holding the raw data {'1' => [X,Y], '2' => [X,Y], ...} my @edges; # Array to keep refrences to all edges [edge, X1, Y1, X2, Y2] ... my $neighborEdges; # hash ref to hold sorted neighbor edges {'1' => edge1, '2' => edge2, ...} ###################################################################### # # MAIN # ###################################################################### # Read in the X,Y coordinates from supplied data { local $/; open(my $fh, "dump_301.dat") or die "Unable to open file: $!\n"; $points = Load( <$fh> ); croak($@) if $@; close($fh); } # Find all the unique edges eval { my $start_distance = Benchmark->new; my $found = {}; # p1 is starting or anchor point of the line segment foreach my $p1 (@{$points}) { # p1 is end point of the line segment foreach my $p2 (@{$points}) { # We don't need to caculate if anchor and end are the same # or we have already seen these two pairs [reversed] before unless ( ($p1 == $p2) || $found->{$p2}->{$p1} ) { # Compute the edge my $edge = sqrt( ($p1->[0] - $p2->[0])**2 + ($p1->[1] - $p2->[1])**2 ); # Push a ref of the whole thing on a stack push (@edges, [ $edge, $p1->[0], $p1->[1], $p2->[0], $p2->[1] ]); # Keep track of the edges we've already computed (no need to do so twice) $found->{$p2}->{$p1} = 1; } } } print "Found unique Edges = ", scalar(@edges), "\n"; print timestr(timediff(Benchmark->new, $start_distance)), "\n"; }; croak($@) if ($@); # Compute the neighboors my $start_intersect = Benchmark->new; # Need an integer to keep our sort order my $neighboorCNT = 1; # Loop trough all found unique edges from smallest to largest foreach my $aref ( sort {$a->[0] <=> $b->[0] } @edges) { my $neighbor = 1; # default true my $line_1_ref = []; push (@{$line_1_ref}, [$aref->[1],$aref->[2]]); # line 1 point A X,Y push (@{$line_1_ref}, [$aref->[3],$aref->[4]]); # line 1 point B X,Y # Loop through all previously found edges in order for (my $n=1;$n<=$neighboorCNT;$n++) { if ($neighborEdges->{$n}) { my $points_ref = []; # Data structure to pass to SegmentIntersection push (@{$points_ref}, @{$line_1_ref}); # line 1 points push (@{$points_ref}, [$neighborEdges->{$n}->[1],$neighborEdges->{$n}->[2]]); # line 2 point A X,Y push (@{$points_ref}, [$neighborEdges->{$n}->[3],$neighborEdges->{$n}->[4]]); # line 2 point B X,Y my @p = SegmentIntersection($points_ref); # If a intersect is found, set false and quit checking if (SegmentIntersection($points_ref)) { $neighbor = 0; last; } } } if ($neighbor) { $neighborEdges->{$neighboorCNT} = $aref; $neighboorCNT++; } } print "Unintersected edges ", scalar(keys %{$neighborEdges}), "\n"; print timestr(timediff(Benchmark->new, $start_intersect)), "\n"; From steven at knowmad.com Wed Jul 27 07:29:54 2005 From: steven at knowmad.com (Steven Bush) Date: Wed, 27 Jul 2005 10:29:54 -0400 Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <42E78917.7030301@knowmad.com> References: <200507270320.j6R3K8eL017182@keep.drewhead.org> <42E78917.7030301@knowmad.com> Message-ID: <42E79A62.7040706@knowmad.com> I've run my modified version of the script. My system is not as fast as an XP2500, but here is the output (it converts to roughly 708 wall clock seconds if you use the first loop ratio to convert it): Found unique Edges = 90300 3 wallclock secs ( 2.48 usr + 0.05 sys = 2.53 CPU) Unintersected edges 1778 2126 wallclock secs (1909.88 usr + 8.53 sys = 1918.41 CPU) If my math is correct, this should about half the execution time for you. I hope this helped. ~Steven From william at knowmad.com Wed Jul 27 10:02:27 2005 From: william at knowmad.com (William McKee) Date: Wed, 27 Jul 2005 13:02:27 -0400 Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <200507270320.j6R3K8eL017182@keep.drewhead.org> References: <200507270320.j6R3K8eL017182@keep.drewhead.org> Message-ID: <20050727170227.GC1258@knowmad.com> On Tue, Jul 26, 2005 at 11:20:08PM -0400, drewhead at drewhead.org wrote: > Here's an interesting problem I've hit recently [and totally diffrent from > the recent template discussions :) ] Thanks for the mind-bender. It is certainly a refreshing change from the template discussions. I tried profiling the code following the article at Perl.com[1]. Using the 301 edges, I had a 2.2Gb output file which was way too much for my poor workstation. I dumbed down the input file to just 30 edges which generated the results I've included below. Basically, it looks like most of the time in the script is spent in Math::Geometry::Planar. Even using Steven's version of the code, I still see much the same results. It looks like you need to look at ways to limit the calls to SegmentIntersection() or improve the algorithms in Math::Geometry::Planar::SegmentIntersection. HTH, William [1] http://www.perl.com/pub/a/2004/06/25/profiling.html william at zeus:~$ perl -d:DProf find_neighboor_edges.pl Found unique Edges = 870 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Unintersected edges 158 6 wallclock secs ( 5.83 usr + 0.14 sys = 5.97 CPU) william at zeus:~$ dprofpp Exporter::export_to_level has -1 unstacked calls in outer Exporter::export has -1 unstacked calls in outer POSIX::load_imports has 1 unstacked calls in outer POSIX::AUTOLOAD has -1 unstacked calls in outer Exporter::Heavy::heavy_export_to_level has 1 unstacked calls in outer Exporter::Heavy::heavy_export has 1 unstacked calls in outer Total Elapsed Time = 2.320667 Seconds User+System Time = 1.970667 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 77.4 1.527 2.227 95534 0.0000 0.0000 Math::Geometry::Planar::SegmentInt ersection 35.5 0.700 0.700 286602 0.0000 0.0000 Math::Geometry::Planar::Determinan t 4.57 0.090 0.169 5 0.0180 0.0339 main::BEGIN 1.01 0.020 0.030 19 0.0010 0.0016 YAML::BEGIN 1.01 0.020 0.019 91 0.0002 0.0002 YAML::_parse_next_line 0.51 0.010 0.010 1 0.0100 0.0100 Math::Geometry::Planar::GPC::boots trap 0.51 0.010 -0.000 1 0.0100 - POSIX::AUTOLOAD 0.51 0.010 0.010 1 0.0100 0.0100 overload::BEGIN 0.51 0.010 0.010 2 0.0050 0.0050 DynaLoader::BEGIN 0.51 0.010 0.010 17 0.0006 0.0006 Exporter::import 0.51 0.010 0.040 6 0.0017 0.0066 Math::Geometry::Planar::BEGIN 0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export_to_l evel 0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export 0.00 0.000 0.000 1 0.0000 0.0000 POSIX::load_imports 0.00 - -0.000 1 - - YAML::is_valid -- Knowmad Services Inc. http://www.knowmad.com From william at knowmad.com Wed Jul 27 10:23:57 2005 From: william at knowmad.com (William McKee) Date: Wed, 27 Jul 2005 13:23:57 -0400 Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <20050727170227.GC1258@knowmad.com> References: <200507270320.j6R3K8eL017182@keep.drewhead.org> <20050727170227.GC1258@knowmad.com> Message-ID: <20050727172357.GD1258@knowmad.com> On Wed, Jul 27, 2005 at 01:02:27PM -0400, William McKee wrote: > Basically, it looks like most of the time in the script is spent in > Math::Geometry::Planar. Even using Steven's version of the code, I still > see much the same results. It looks like you need to look at ways to > limit the calls to SegmentIntersection() or improve the algorithms in > Math::Geometry::Planar::SegmentIntersection. After a little more exploration, we realized that the first call to SegmentIntersection is unnecessary. Removing that call dropped the calculation time in half and similarly improved the profile output from dprofpp. Steven suggests looking at Ch 7 of _Perl Objects, References, and Modules_. Our invoice is in the mail :). Have fun, William -- Knowmad Services Inc. http://www.knowmad.com From drewhead at drewhead.org Wed Jul 27 11:08:58 2005 From: drewhead at drewhead.org (drewhead@drewhead.org) Date: Wed, 27 Jul 2005 14:08:58 -0400 (EDT) Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <42E79A62.7040706@knowmad.com> Message-ID: <200507271808.j6RI8wkR026121@keep.drewhead.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > I've run my modified version of the script. My system is not as fast as > an XP2500, but here is the output (it converts to roughly 708 wall clock > seconds if you use the first loop ratio to convert it): > Found unique Edges = 90300 > 3 wallclock secs ( 2.48 usr + 0.05 sys = 2.53 CPU) > Unintersected edges 1778 > 2126 wallclock secs (1909.88 usr + 8.53 sys = 1918.41 CPU) > If my math is correct, this should about half the execution time for > you. I hope this helped. Cool, that will certianlly be a help. Thanks for the response. This evening I'll set up some tests on a couple of diffrent boxes with both code blocks and see if I can get a decent multiplier showing what the improvement is. - -- Drew Dowling Drewhead http://www.drewhead.org drewhead at drewhead.org | | WWW / \ Alpha Phi Omega Concord, North Carolina | | Nimat / \ Gamma Lambda CLEMSON UNIVERSITY ALUMNI | | Apatschin /_____\ TIGER BAND! | VGAP4 Hosting at http://vgap.drewhead.org -----BEGIN PGP SIGNATURE----- Comment: Public key available at http://www.drewhead.org iD8DBQFC582z8J7U7yHE638RAozRAJwNDLMwuFaIqU5jV0yWByTxqgG3HACeJDqE 9L3A6hpZcT5QpOCtnGVAJrU= =CMPf -----END PGP SIGNATURE----- From drewhead at drewhead.org Wed Jul 27 11:29:53 2005 From: drewhead at drewhead.org (drewhead@drewhead.org) Date: Wed, 27 Jul 2005 14:29:53 -0400 (EDT) Subject: [Charlotte.PM] Help me tune this for speed In-Reply-To: <20050727172357.GD1258@knowmad.com> Message-ID: <200507271829.j6RITrde026432@keep.drewhead.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > On Wed, Jul 27, 2005 at 01:02:27PM -0400, William McKee wrote: > > Basically, it looks like most of the time in the script is spent in > > Math::Geometry::Planar. Even using Steven's version of the code, I still > > see much the same results. It looks like you need to look at ways to > > limit the calls to SegmentIntersection() or improve the algorithms in > > Math::Geometry::Planar::SegmentIntersection. > After a little more exploration, we realized that the first call to > SegmentIntersection is unnecessary. Removing that call dropped the > calculation time in half and similarly improved the profile output from > dprofpp. Steven suggests looking at Ch 7 of _Perl Objects, References, > and Modules_. Our invoice is in the mail :). Doh! Drew left in Debug code! that first call shouldn't have been there. I've choped this thing to bits so much over the last 10 days I've forgotton what's real and whats in there 'just to see'. Yes, that would make things run longer, removing it unfortuantly doesn't solve my scale problem. However... the link and hints about profiling are pure gold. Cool! I never knew that was out there. Learning of these is worth the price of the OP. I'm being vacated for 48 hours by my family starting tomorow. Guess what I'll be sitting in my basement doing. :) Given that the intersection math seems to be where time was spent, something I had discounted given my own [incorrect] tests, then I suppose that I need to investigate one of the Math modules to improve upon performance, or resign to farming this out to C code (as was orginally sugested by the guy that came up with the algorithm.) There is another algorithm that uses triangles that we think is O(N log N) but it's quite a bit more complex and we're still working on making it function. :) - -- Drew Dowling Drewhead http://www.drewhead.org drewhead at drewhead.org | | WWW / \ Alpha Phi Omega Concord, North Carolina | | Nimat / \ Gamma Lambda CLEMSON UNIVERSITY ALUMNI | | Apatschin /_____\ TIGER BAND! | VGAP4 Hosting at http://vgap.drewhead.org -----BEGIN PGP SIGNATURE----- Comment: Public key available at http://www.drewhead.org iD8DBQFC59KV8J7U7yHE638RAnjFAJ93ebSpbAaD/g4AYzdNrd2EHwZfuwCghi4P ijvvghiCfRQ61rRpoeu3CTY= =kpzt -----END PGP SIGNATURE-----