[Raleigh-talk] Parallel::ForkManager and progress

[Brad Oaks]

On Fri, Dec 19, 2008 at 2:15 PM, Michael Peters <mpeters at plusthree.com> wrote:
> James Olin Oden wrote:
>
>> You can always just write your stats to a file.  Have to add flock()
>> to the equation
>> but it isn't that bad
>
> I guess I should have said that this process will be doing hundreds of
> billions (with a B) comparisons, so I'd like to keep everything in memory or
> at least in sockets. Having all of those file reads/writes in there gives me
> the willies.
>
> But I guess that means that using the DB is out too since going over the
> network is even worse than going to the spinning platter.

You could update your counter only once every 10_000 records are
processed.  I've done that in the past when I didn't want to overrun
my screen with status messages.  Same would work for this if you don't
mind the total being off by less than a percent (times the number of
processes) when you check it.

--bradoaks