From trevormg19 at gmail.com  Wed Dec  3 19:44:41 2008
From: trevormg19 at gmail.com (Trevor Little)
Date: Wed, 3 Dec 2008 22:44:41 -0500
Subject: [Raleigh-talk] OT: Hurricanes Tickets for Thursday Dec 4
Message-ID: <944d53050812031944i1c714512ude190caeecd8265a@mail.gmail.com>

Hey Everybody,
     I've got 2 tickets to the Penguins vs Hurricanes game on Thursday
Dec 4 that I can't use. If anyone wants one or both their yours for
$25 each (obo). They're they're behind the goal. I can probably meet
somewhere around lunch time or after work to do the hand-off.

Trevor

From bradoaks at gmail.com  Tue Dec  9 13:52:38 2008
From: bradoaks at gmail.com (Brad Oaks)
Date: Tue, 9 Dec 2008 16:52:38 -0500
Subject: [Raleigh-talk] Fwd: [Charlotte.PM] Fwd: Higher-Order Perl full text
	now available online
In-Reply-To: <20081209205640.GM7908@knowmad.com>
References: <20081209205640.GM7908@knowmad.com>
Message-ID: <df7afb5e0812091352g65adb838oe384fdf4ce6011c9@mail.gmail.com>

Yay, download the full PDF or each chapter as a PDF.
--bradoaks


---------- Forwarded message ----------
From: William McKee <william at knowmad.com>
Date: Tue, Dec 9, 2008 at 3:56 PM
Subject: [Charlotte.PM] Fwd: Higher-Order Perl full text now available online
To: "Charlotte.PM" <charlotte at mail.pm.org>


----- Forwarded message from Mark Jason Dominus <mjd at plover.com> -----

> To: Higher-Order Perl announcement list <mjd-book at plover.com>
> From: Mark Jason Dominus <mjd at plover.com>
> Date: Tue, 09 Dec 2008 18:43:29 +0000
> Subject: Higher-Order Perl full text now available online
> Organization: Plover Systems
>
> If you forgot what this list is about, or you don't know why you're
> getting this message, please see
>
>         http://hop.perl.plover.com/
>
> To unsubscribe, send a blank message to mjd-book-unsubscribe at plover.com.
>
> ----------------------------------------------------------------
>
>
>
> Higher-Order Perl is now available online.
>
> But first, a word from our sponsor:
>
>         Higher-Order Perl is copyright 2005 by Elsevier
>         Inc.  Unauthorized reproduction or distribution is absolutely
>         forbidden.
>
> In particular, note that although the text is available for free,
> Higher-Order Perl is not in the public domain and is not available
> under a free license of any sort.  I distribute it from my web site by
> virtue of special permission from the publisher.  You, most likely, do
> not have any such permission.
>
>
> YOU CAN GET IT FROM:
>
>                    http://hop.perl.plover.com/book/
>
> There are two versions available.  You have your choice of structure
> or presentation, but not both.
>
>
> PDF VERSION
>
>                  http://hop.perl.plover.com/book/pdf/
>
> This is the publisher's own PDF proof of the second version, which was
> sent to the printers in August 2005.
>
> This is better than the bootleg copies available from download sites
> in at least three ways:
>
>         * It is the complete text of the *second* printing, which
>           incorporates many minor corrections; the bootleg copies are
>           all bootlegs of the first printing.
>
>         * It does not have  a nasty little grafitto advertising a vainglorious
>           bootlegger plastered on every page.
>
>         * It was come by honestly, not stolen from the printer.
>
>
>
> MOD VERSION
>
>                  http://hop.perl.plover.com/book/mod/
>
> These are the original sources that I used to generate the manuscript
> draft that I first sent to the publisher.  They are written in MOD, a
> variation on the "pod" markup in which Perl's documentation is
> written.  It should be self-explanatory.  If not, drop me a note.
>
> The big drawback of this version is that it not only omits the
> corrections that were added between the first and second printings,
> but also the much more significant corrections that were added before
> the first printing.
>
>
>
> FREQUENTLY-ASKED QUESTION
>
> Q. What took you so long?
>
> A.      I ran outta gas. I had a flat tire. I didn't have enough money for cab
>         fare. My tux didn't come back from the cleaners. An old friend came in
>         from outta town. Someone stole my car. There was an earthquake, a
>         terrible flood, locust's. It wasn't my fault!! I swear to  God!!
>
>    No, that's not right.  It's totally my fault.  I'd explain, but it
>    would feel too much like making excuses.  Also you probably don't
>    care about the mundane details of my life for the past four years.
>    And anyway, even if I rehearsed the whole thing, you would be quite
>    justified in asking, at the end of the long story, "So I don't get
>    it.  What too you so long?"
>
>    You may remember I wanted to turn the book into a wiki.  That would
>    have been awesome.  But the book's fourth anniversary is coming up
>    this spring and I have to admit to myself that I'm not gonna get
>    the wiki together.  So I'm posting the thing already.
>
>    If anyone else wants to try to turn it into a wiki, drop me a note.
>
>
> Enjoy.
>
>

----- End forwarded message -----

--
Knowmad Technologies - Open Source Web Solutions
W: http://www.knowmad.com | E: william at knowmad.com | P: 704.343.9330
_______________________________________________
charlotte mailing list
charlotte at pm.org
http://mail.pm.org/mailman/listinfo/charlotte

From andrew at eiknet.com  Fri Dec 19 07:04:15 2008
From: andrew at eiknet.com (Andrew Rankin)
Date: Fri, 19 Dec 2008 10:04:15 -0500
Subject: [Raleigh-talk] Sharing data between modules
Message-ID: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>

Hi All.

I have question about sharing some data between some Perl modules and
the best way to accomplish it.

I have written an application for tracking our IBM bladecenters and
Xen VMs, basically it gives you a list of "containers" (the Domain-O
and the Blade centers) and lets you list the servers and information
on those servers within that container:
http://eiknet.com/glovebox01.png.  The backend is perl (obviously :)).
 My issues is that I have a set of data that I'd like to share with
the applications perl modules, they are layed out like this:

                                                  ->
Interfaces::VIRT.pm -> Shared::Library & DB
AJAX -> index.pl -> Glovebox.pm -> Interfaces::SNMP.pm -> Shared::Library & DB
                                                  -> Shared::Library.pm
                                                  -> DB

Within Glovebox.pm's new() I open a database connection, collect the
cgi paramaters, a list of interfaces and open the shared library.
Which looks something like this:

$VAR1 = {
                 'interfaces' => {
                                   'bladecenters' => {
                                                       'deldev' => '0',
                                                       'name' =>
'BladeCenter SNMP',
                                                       'id' => '1',
                                                       'class' =>
'Glovebox::Interface::SNMP',
                                                       'type' => 'bladecenters',
                                                       'enabled' => '1'
                                                     },
                                   'virt' => {
                                               'deldev' => '1',
                                               'name' => 'Xen Dom-0',
                                               'id' => '2',
                                               'class' =>
'Glovebox::Interface::VIRT',
                                               'type' => 'virt',
                                               'enabled' => '1'
                                             }
                                 },
                 'db' => {
                           'database' => 'systems',
                           'lock_id' => 8797996,
                           'db_user' => 'theuser',
                           'dsn' => 'DBI:mysql:database=systems;host=localhost',
                           'db_hostname' => 'localhost',
                           'dbh' => bless( {}, 'DBI::db' ),
                           'db_password' => 'notreallythepw'
                         },
                 'DevLib' => bless( {
                                      'db' => {
                                                'database' => 'systems',
                                                'lock_id' => 7291450,
                                                'db_user' => 'theuser',
                                                'dsn' =>
'DBI:mysql:database=systems;host=localhost',
                                                'db_hostname' => 'localhost',
                                                'dbh' => bless( {}, 'DBI::db' ),
                                                'db_password' =>
'notreallythepw'
                                              },
                                      'container' => undef
                                    }, 'Glovebox::Shared::DevLib' ),
                 'param' => {
                              'type' => 'virt'
                            }
               };


I'd like to share that data between all the modules to avoid the
multiple database connections and multiple shared library's being
opened. Whats the best way of doing this?  So far I've had suggested
using Exporter or defining a variable using 'our' in Glovebox.pm and
just referencing back to it.  Thoughts?

- Andrew Rankin

From trevormg19 at gmail.com  Fri Dec 19 07:26:30 2008
From: trevormg19 at gmail.com (Trevor Little)
Date: Fri, 19 Dec 2008 10:26:30 -0500
Subject: [Raleigh-talk] Sharing data between modules
In-Reply-To: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
References: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
Message-ID: <op.ume06gux6wiec6@bruce.sgi.int>

Andrew,
     I'm not sure I've got all the details right. When you say that the  
modules are "layed out like this", what to the arrows signify? Is that  
inheritance, or one module calling methods in another, or what? Sorry if  
I'm being dense. If it's inheritance and your creating data in a  
constructor of a parent Class that you want to be available in child  
classes it seems you could store it in the object as instance data. But  
I'm guessing that that's not the situation or there's a reason why that  
won't work for you. One thing I often forget is that every module can  
access any data in any other module (well, for the most part) by using the  
full name-space (ie $Glovebox::some_variable). It's not always pretty, but  
sometimes it a better alternative to exporting stuff and polluting  
name-spaces. Anyway, I'm not sure if any of that is helpful, cause I feel  
I might have misunderstood what you're asking. I'll be in #raleigh.pm on  
irc.perl.org today (along with a bunch smarter people) if you want to pop  
in and explain in more detail.

Trevor

On Fri, 19 Dec 2008 10:04:15 -0500, Andrew Rankin <andrew at eiknet.com>  
wrote:

> Hi All.
>
> I have question about sharing some data between some Perl modules and
> the best way to accomplish it.
>
> I have written an application for tracking our IBM bladecenters and
> Xen VMs, basically it gives you a list of "containers" (the Domain-O
> and the Blade centers) and lets you list the servers and information
> on those servers within that container:
> http://eiknet.com/glovebox01.png.  The backend is perl (obviously :)).
>  My issues is that I have a set of data that I'd like to share with
> the applications perl modules, they are layed out like this:
>
>                                                   ->
> Interfaces::VIRT.pm -> Shared::Library & DB
> AJAX -> index.pl -> Glovebox.pm -> Interfaces::SNMP.pm ->  
> Shared::Library & DB
>                                                   -> Shared::Library.pm
>                                                   -> DB
>
> Within Glovebox.pm's new() I open a database connection, collect the
> cgi paramaters, a list of interfaces and open the shared library.
> Which looks something like this:
>
> $VAR1 = {
>                  'interfaces' => {
>                                    'bladecenters' => {
>                                                        'deldev' => '0',
>                                                        'name' =>
> 'BladeCenter SNMP',
>                                                        'id' => '1',
>                                                        'class' =>
> 'Glovebox::Interface::SNMP',
>                                                        'type' =>  
> 'bladecenters',
>                                                        'enabled' => '1'
>                                                      },
>                                    'virt' => {
>                                                'deldev' => '1',
>                                                'name' => 'Xen Dom-0',
>                                                'id' => '2',
>                                                'class' =>
> 'Glovebox::Interface::VIRT',
>                                                'type' => 'virt',
>                                                'enabled' => '1'
>                                              }
>                                  },
>                  'db' => {
>                            'database' => 'systems',
>                            'lock_id' => 8797996,
>                            'db_user' => 'theuser',
>                            'dsn' =>  
> 'DBI:mysql:database=systems;host=localhost',
>                            'db_hostname' => 'localhost',
>                            'dbh' => bless( {}, 'DBI::db' ),
>                            'db_password' => 'notreallythepw'
>                          },
>                  'DevLib' => bless( {
>                                       'db' => {
>                                                 'database' => 'systems',
>                                                 'lock_id' => 7291450,
>                                                 'db_user' => 'theuser',
>                                                 'dsn' =>
> 'DBI:mysql:database=systems;host=localhost',
>                                                 'db_hostname' =>  
> 'localhost',
>                                                 'dbh' => bless( {},  
> 'DBI::db' ),
>                                                 'db_password' =>
> 'notreallythepw'
>                                               },
>                                       'container' => undef
>                                     }, 'Glovebox::Shared::DevLib' ),
>                  'param' => {
>                               'type' => 'virt'
>                             }
>                };
>
>
> I'd like to share that data between all the modules to avoid the
> multiple database connections and multiple shared library's being
> opened. Whats the best way of doing this?  So far I've had suggested
> using Exporter or defining a variable using 'our' in Glovebox.pm and
> just referencing back to it.  Thoughts?
>
> - Andrew Rankin
> _______________________________________________
> Raleigh-talk mailing list
> Raleigh-talk at pm.org
> http://mail.pm.org/mailman/listinfo/raleigh-talk



From mpeters at plusthree.com  Fri Dec 19 07:25:45 2008
From: mpeters at plusthree.com (Michael Peters)
Date: Fri, 19 Dec 2008 10:25:45 -0500
Subject: [Raleigh-talk] Sharing data between modules
In-Reply-To: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
References: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
Message-ID: <494BBCF9.7050407@plusthree.com>

Andrew Rankin wrote:

> I'd like to share that data between all the modules to avoid the
> multiple database connections and multiple shared library's being
> opened. 

First off how are you running this? Is it under mod_perl? Some other persistant framework like 
FastCGI? Or is it normal vanilla CGI?

The other question is why. Most databases should be able to handle thousands of connections without 
a problem. And loading a library multiple times might not be a bad thing. It all depends on what 
you're trying to do and why. If you're trying to remove bottlenecks, have you actually done any 
measuring?

> Whats the best way of doing this?  So far I've had suggested
> using Exporter or defining a variable using 'our' in Glovebox.pm and
> just referencing back to it. 

Just to clarify things, are you worried that having "use MyModule" in lots of different places will 
load different copies of that module within the same process? If so, don't worry about it. Perl will 
only load 1 copy. But if your design has a single object that should be shared "globally" among the 
same code in the same process then you're probably looking at a Singleton pattern, so check out 
Class::Singleton.

But really, I need more information about what you're trying to do, not so much about what your code 
looks like.

-- 
Michael Peters
Plus Three, LP


From andrew at eiknet.com  Fri Dec 19 07:49:27 2008
From: andrew at eiknet.com (Andrew Rankin)
Date: Fri, 19 Dec 2008 10:49:27 -0500
Subject: [Raleigh-talk] Sharing data between modules
In-Reply-To: <494BBCF9.7050407@plusthree.com>
References: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
	<494BBCF9.7050407@plusthree.com>
Message-ID: <1bdec63c0812190749v64d45338qc7077477ba7cf7d@mail.gmail.com>

Thanks for the responses guys.

Michael -

Its just normal old vanilla CGI, nothing special.  My reasoning on the
sharing of a database connection was purely cleanliness and preference
- why have multiple if I can just have one...  The database and shared
library were actually low on my list of concerns, what I really wanted
to have was the containers data shared throughout the modules since
they all used it - and if it changes one place, all modules see the
change (if that makes sense).  I was figuring if I was sharing that
data, I'd just share the other items at the same time.

Andrew


On Fri, Dec 19, 2008 at 10:25 AM, Michael Peters <mpeters at plusthree.com> wrote:
> Andrew Rankin wrote:
>
>> I'd like to share that data between all the modules to avoid the
>> multiple database connections and multiple shared library's being
>> opened.
>
> First off how are you running this? Is it under mod_perl? Some other
> persistant framework like FastCGI? Or is it normal vanilla CGI?
>
> The other question is why. Most databases should be able to handle thousands
> of connections without a problem. And loading a library multiple times might
> not be a bad thing. It all depends on what you're trying to do and why. If
> you're trying to remove bottlenecks, have you actually done any measuring?
>
>> Whats the best way of doing this?  So far I've had suggested
>> using Exporter or defining a variable using 'our' in Glovebox.pm and
>> just referencing back to it.
>
> Just to clarify things, are you worried that having "use MyModule" in lots
> of different places will load different copies of that module within the
> same process? If so, don't worry about it. Perl will only load 1 copy. But
> if your design has a single object that should be shared "globally" among
> the same code in the same process then you're probably looking at a
> Singleton pattern, so check out Class::Singleton.
>
> But really, I need more information about what you're trying to do, not so
> much about what your code looks like.
>
> --
> Michael Peters
> Plus Three, LP
>
> _______________________________________________
> Raleigh-talk mailing list
> Raleigh-talk at pm.org
> http://mail.pm.org/mailman/listinfo/raleigh-talk
>

From james.oden at gmail.com  Fri Dec 19 07:53:20 2008
From: james.oden at gmail.com (James Olin Oden)
Date: Fri, 19 Dec 2008 10:53:20 -0500
Subject: [Raleigh-talk] Sharing data between modules
In-Reply-To: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
References: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
Message-ID: <ac921f990812190753j7e4002c8m184e91684d9deff0@mail.gmail.com>

On Fri, Dec 19, 2008 at 10:04 AM, Andrew Rankin <andrew at eiknet.com> wrote:
> Hi All.
>
> I have question about sharing some data between some Perl modules and
> the best way to accomplish it.
So rather than use globals, or horrors, a singleton, simply create a
class whose objectes
would contain all this information.   You can either:

   1) create it once, and pass it to the constructors of all the other objects.
   2) Create yet another class that acts as a cache for your objects,
and then have
       every object that needs the information pull from the cache
(only one object would
       seed the cache).

I use the latter model quite often as away to avoid singleton, because
almost every singleton
I've encountered, I've had a need to use two or more of at some
instance (i.e. seperate distinct objects
not two references to the same object).

Cache's are very easy to implement.  They can take the strategy of a
singleton interface to a particular class
(i.e. there is one and only one object tracked in the cache for a
particular process), or they can take
an approach of providing a cache of multiple objects each retrievable
by a unique key.

Does that make sense or do I need to be more concrete?

Cheers...james

From andrew at eiknet.com  Fri Dec 19 08:02:07 2008
From: andrew at eiknet.com (Andrew Rankin)
Date: Fri, 19 Dec 2008 11:02:07 -0500
Subject: [Raleigh-talk] Sharing data between modules
In-Reply-To: <ac921f990812190753j7e4002c8m184e91684d9deff0@mail.gmail.com>
References: <1bdec63c0812190704v692fa609u15cbad216cd0325c@mail.gmail.com>
	<ac921f990812190753j7e4002c8m184e91684d9deff0@mail.gmail.com>
Message-ID: <1bdec63c0812190802u6c9968a0t8098bcd7a98afa0e@mail.gmail.com>

That makes sense, I'll give a cache a try.  Thanks James.

Andrew

On Fri, Dec 19, 2008 at 10:53 AM, James Olin Oden <james.oden at gmail.com> wrote:
> On Fri, Dec 19, 2008 at 10:04 AM, Andrew Rankin <andrew at eiknet.com> wrote:
>> Hi All.
>>
>> I have question about sharing some data between some Perl modules and
>> the best way to accomplish it.
> So rather than use globals, or horrors, a singleton, simply create a
> class whose objectes
> would contain all this information.   You can either:
>
>   1) create it once, and pass it to the constructors of all the other objects.
>   2) Create yet another class that acts as a cache for your objects,
> and then have
>       every object that needs the information pull from the cache
> (only one object would
>       seed the cache).
>
> I use the latter model quite often as away to avoid singleton, because
> almost every singleton
> I've encountered, I've had a need to use two or more of at some
> instance (i.e. seperate distinct objects
> not two references to the same object).
>
> Cache's are very easy to implement.  They can take the strategy of a
> singleton interface to a particular class
> (i.e. there is one and only one object tracked in the cache for a
> particular process), or they can take
> an approach of providing a cache of multiple objects each retrievable
> by a unique key.
>
> Does that make sense or do I need to be more concrete?
>
> Cheers...james
> _______________________________________________
> Raleigh-talk mailing list
> Raleigh-talk at pm.org
> http://mail.pm.org/mailman/listinfo/raleigh-talk
>

From mpeters at plusthree.com  Fri Dec 19 10:56:00 2008
From: mpeters at plusthree.com (Michael Peters)
Date: Fri, 19 Dec 2008 13:56:00 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
Message-ID: <494BEE40.80104@plusthree.com>

I've got a long running batch process that uses Parallel::ForkManager to split the work up and take 
advantage of the multiple CPU/Cores on the box. This part works fine (have I mentioned how much I 
love P::FM?). But I'd like to have some sort of a running progress on the screen when run manually.

So, any ideas on how best to do this? Since each worker is a separate process I could use something 
like Net::Shared or IPC::ShareLite to collect stats that are then reported on. Anyone have any 
success with these modules or have recommendations on shared memory modules?

I've also thought about using the database to store the progress. That has the benefit of being 
something I can query even if the process isn't being run manually. But it has the drawback of 
having to connect to the database (which won't be on the same machine) and sending it lots of little 
queries to update a couple of counters.

Thoughts?
-- 
Michael Peters
Plus Three, LP


From james.oden at gmail.com  Fri Dec 19 11:06:37 2008
From: james.oden at gmail.com (James Olin Oden)
Date: Fri, 19 Dec 2008 14:06:37 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <494BEE40.80104@plusthree.com>
References: <494BEE40.80104@plusthree.com>
Message-ID: <ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>

On Fri, Dec 19, 2008 at 1:56 PM, Michael Peters <mpeters at plusthree.com> wrote:
> I've got a long running batch process that uses Parallel::ForkManager to
> split the work up and take advantage of the multiple CPU/Cores on the box.
> This part works fine (have I mentioned how much I love P::FM?). But I'd like
> to have some sort of a running progress on the screen when run manually.
>
> So, any ideas on how best to do this?
You can always just write your stats to a file.  Have to add flock()
to the equation
but it isn't that bad....james

From mpeters at plusthree.com  Fri Dec 19 11:15:07 2008
From: mpeters at plusthree.com (Michael Peters)
Date: Fri, 19 Dec 2008 14:15:07 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>
References: <494BEE40.80104@plusthree.com>
	<ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>
Message-ID: <494BF2BB.2000105@plusthree.com>

James Olin Oden wrote:

> You can always just write your stats to a file.  Have to add flock()
> to the equation
> but it isn't that bad

I guess I should have said that this process will be doing hundreds of billions (with a B) 
comparisons, so I'd like to keep everything in memory or at least in sockets. Having all of those 
file reads/writes in there gives me the willies.

But I guess that means that using the DB is out too since going over the network is even worse than 
going to the spinning platter.

-- 
Michael Peters
Plus Three, LP


From bradoaks at gmail.com  Fri Dec 19 11:21:51 2008
From: bradoaks at gmail.com (Brad Oaks)
Date: Fri, 19 Dec 2008 14:21:51 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <494BF2BB.2000105@plusthree.com>
References: <494BEE40.80104@plusthree.com>
	<ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>
	<494BF2BB.2000105@plusthree.com>
Message-ID: <df7afb5e0812191121t4de35978secc6c51ce972ad25@mail.gmail.com>

On Fri, Dec 19, 2008 at 2:15 PM, Michael Peters <mpeters at plusthree.com> wrote:
> James Olin Oden wrote:
>
>> You can always just write your stats to a file.  Have to add flock()
>> to the equation
>> but it isn't that bad
>
> I guess I should have said that this process will be doing hundreds of
> billions (with a B) comparisons, so I'd like to keep everything in memory or
> at least in sockets. Having all of those file reads/writes in there gives me
> the willies.
>
> But I guess that means that using the DB is out too since going over the
> network is even worse than going to the spinning platter.

You could update your counter only once every 10_000 records are
processed.  I've done that in the past when I didn't want to overrun
my screen with status messages.  Same would work for this if you don't
mind the total being off by less than a percent (times the number of
processes) when you check it.

--bradoaks

From mpeters at plusthree.com  Fri Dec 19 11:21:40 2008
From: mpeters at plusthree.com (Michael Peters)
Date: Fri, 19 Dec 2008 14:21:40 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <df7afb5e0812191121t4de35978secc6c51ce972ad25@mail.gmail.com>
References: <494BEE40.80104@plusthree.com>	<ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>	<494BF2BB.2000105@plusthree.com>
	<df7afb5e0812191121t4de35978secc6c51ce972ad25@mail.gmail.com>
Message-ID: <494BF444.4010605@plusthree.com>

Brad Oaks wrote:

> You could update your counter only once every 10_000 records are
> processed.  I've done that in the past when I didn't want to overrun
> my screen with status messages.  Same would work for this if you don't
> mind the total being off by less than a percent (times the number of
> processes) when you check it.

Yeah, I definitely think I'll be going with a delayed update. So I guess that using a file isn't 
that big of a deal. But I think I'll go the shared memory route since it's something new to play with :)

-- 
Michael Peters
Plus Three, LP


From trevormg19 at gmail.com  Fri Dec 19 11:37:04 2008
From: trevormg19 at gmail.com (Trevor Little)
Date: Fri, 19 Dec 2008 14:37:04 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <494BEE40.80104@plusthree.com>
References: <494BEE40.80104@plusthree.com>
Message-ID: <op.umfcr2wl6wiec6@bruce.sgi.int>

I've used IPC::Sharelite and it worked well in development but eventually  
ran into problems because we use a hardened version of GCC on our  
production servers that did NOT play well with it (segfaults). One thing  
you could do if have each process send status data to a named pipe that  
your monitor program could read. You could even have them only start  
writing the status data in response to a SIGUSR or some other signal. That  
similar to what I ended up with when IPC::Sharelite was eliminated as an  
option. I setup a Boss/Worker module as a parent class for deamons we run  
to do stuff. Basically the Boss finds work to do and starts up worker  
deamons to do it. The worker deamons hang around as long as there is work  
to do and the boss passes them "Jobs" (basically a serialized data struct)  
by printing to pipes. It works, but has been buggy and the IPC::Sharelite  
version was a whole lot cleaner.

So basiacally, I'd say give IPC::Sharelite a try and then maybe look at  
pipes. Of course, if it's something you thing will grow into something  
larger in the furture, it may save you some time in the long run to just  
have it dump the data into a database.

Trevor

On Fri, 19 Dec 2008 13:56:00 -0500, Michael Peters <mpeters at plusthree.com>  
wrote:

> I've got a long running batch process that uses Parallel::ForkManager to  
> split the work up and take advantage of the multiple CPU/Cores on the  
> box. This part works fine (have I mentioned how much I love P::FM?). But  
> I'd like to have some sort of a running progress on the screen when run  
> manually.
>
> So, any ideas on how best to do this? Since each worker is a separate  
> process I could use something like Net::Shared or IPC::ShareLite to  
> collect stats that are then reported on. Anyone have any success with  
> these modules or have recommendations on shared memory modules?
>
> I've also thought about using the database to store the progress. That  
> has the benefit of being something I can query even if the process isn't  
> being run manually. But it has the drawback of having to connect to the  
> database (which won't be on the same machine) and sending it lots of  
> little queries to update a couple of counters.
>
> Thoughts?



From bradoaks at gmail.com  Fri Dec 19 11:40:41 2008
From: bradoaks at gmail.com (Brad Oaks)
Date: Fri, 19 Dec 2008 14:40:41 -0500
Subject: [Raleigh-talk] Parallel::ForkManager and progress
In-Reply-To: <494BF444.4010605@plusthree.com>
References: <494BEE40.80104@plusthree.com>
	<ac921f990812191106v6cea9a92w1a5a59599b55ce57@mail.gmail.com>
	<494BF2BB.2000105@plusthree.com>
	<df7afb5e0812191121t4de35978secc6c51ce972ad25@mail.gmail.com>
	<494BF444.4010605@plusthree.com>
Message-ID: <df7afb5e0812191140u75f44246r14252b8294b565d5@mail.gmail.com>

On Fri, Dec 19, 2008 at 2:21 PM, Michael Peters <mpeters at plusthree.com> wrote:
> Brad Oaks wrote:
>
>> You could update your counter only once every 10_000 records are
>> processed.  I've done that in the past when I didn't want to overrun
>> my screen with status messages.  Same would work for this if you don't
>> mind the total being off by less than a percent (times the number of
>> processes) when you check it.
>
> Yeah, I definitely think I'll be going with a delayed update. So I guess
> that using a file isn't that big of a deal. But I think I'll go the shared
> memory route since it's something new to play with :)

One of my projects at MI was to replace some caching that was using
IPC::ShareLite to use Memcached instead.  The old way was working
fine, we only changed because we had a large number of machines and
needed to make the jump from a cache shared between processes to one
shared across machines too.

I appreciate the something-new-to-play-with aspect.

--bradoaks