[Melbourne-pm] Finding "Out of memory!"

Thu Oct 9 22:10:40 PDT 2014

In the past, I've found, Devel::Cycle, Devel::Leak and friends useful
for hunting such badness.  During development, I run my Catalyst apps
with the CatalystX::LeakCheaker plugin.  Perhaps those may give some
inspiration?

K

Kahlil (Kal) Hodgson                       GPG: C9A02289
Head of Technology                         (m) +61 (0) 4 2573 0382
DealMax Pty Ltd

Suite 1416
401 Docklands Drive
Docklands VIC 3008 Australia

"All parts should go together without forcing.  You must remember that
the parts you are reassembling were disassembled by you.  Therefore,
if you can't get them together again, there must be a reason.  By all
means, do not use a hammer."  -- IBM maintenance manual, 1925

On Fri, Oct 10, 2014 at 3:53 PM, Bradley Dean <bjdean at bjdean.id.au> wrote:
> Yep - doing that too. :)
>
> This idea started as "it would be handy to have a drop-in module that would
> help capture memory problems occurring while they are occurring and before
> they are fixed". Actually - the specific problem that started this line of
> thought is now history as the memory leak has been identified and fixed.
>
> I wasn't suggesting that an application that leaks memory should just be
> left to keep doing that - my idea was really of interest if said application
> only leaks memory sometimes and it's taking a while to find why (but the
> application needs to be run in the meantime).
>
> Cheerio,
>
>  Brad
>
>
> On Fri, Oct 10, 2014, at 13:06, Toby Corkindale wrote:
>
> When you encounter a problem that seems difficult to handle and has very
> little evidence of other people handling it, then it is a good idea to take
> a step back and look at the wider picture. Your overall architecture may
> simply be flawed, and a change elsewhere may simplify later issues.
>
> In this case, I suggest you fix your design so that out of memory errors
> stop occurring.
> Eg. Don't load entire massive files or data structures into memory, but
> instead use a specific data storage and retrieval system that is suited to
> the task and can manage the paging of data to disk with automatic caching to
> memory. (For instance, SQLite or LevelDB)
>
>
> ________________________________
>
> From: "Bradley Dean" <bjdean at bjdean.id.au>
> To: "Mathew Robertson" <mathew.blair.robertson at gmail.com>
> Cc: "Melbourne Perl Mongers" <melbourne-pm at pm.org>
> Sent: Friday, 10 October, 2014 12:51:11 PM
> Subject: Re: [Melbourne-pm] Finding "Out of memory!"
>
> Yes indeed - a non-zero exit code but not one that tells me anything about
> the reason (because I can't reliable catch the out of memory error to set a
> useful exit code, and perl isn't doing that either). :)
>
> I'm not claiming that searching for a magic string in STDERR is a good
> option, but at the moment I'm not sure there's a better one.
>
> Cheerio,
>
>  Brad
>
>
> On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote:
>
> The Perl script should exit with a non-zero exit code... it doesn't really
> matter why that exit code was non-zero, as any non-zero value will indicate
> a failure mode.
>
> eg: grep returns non-zero when it didn't actually grep for anything.... (you
> asked it to do something, but it couldn't).
>
> Tracing STDERR will be fraught with problems - how do you know some Perl
> library doesn't randomly spit out messages that you have yet to see?
>
>
> just my $0.02,
>
> Mathew
>
>
> On 10 October 2014 10:46, Bradley Dean <bjdean at bjdean.id.au> wrote:
>
> Yes - that's certainly a safer approach. In fact I've found that what
> worked for me in my example code was version sensitive.
>
> On the system I started with the last system call was an mmap2 with a
> ENOMEM error and this is what I found when checking $!. I then tested
> this on a different system and discovered that after the mmap2 failed
> (and before my perl code had a chance to look at $!) a bunch of
> rt_sigaction systems calls happened and one of those failed with EINVAL
> so there was no ENOMEM to find.
>
> I feel there's a hack with a wrapper script which greps for "Out of
> memory!" on STDERR coming my way as the exit code doesn't tell me
> anything useful either.
>
> Cheerio,
>
>  Brad
>
> On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote:
>> I really don't think this approach is going to pay dividends.
>> When you're hitting out of memory conditions in Perl, it's usually
>> game over. In some languages you could code up something that runs
>> without using any more memory, as a handler, but you don't have that
>> level of control in Perl.
>>
>> I'd look at wrapping your Perl script in another program - maybe just
>> a bash or even another Perl script - that waits for it to exit, and
>> then checks to see if it exited normally or with an error code.
>>
>> On 3 October 2014 11:54, Bradley Dean <bjdean at bjdean.id.au> wrote:
>> > Greetings folks,
>> >
>> > While looking at a problem where a variety of scripts might develop a
>> > memory leak and fatally terminate with "Out of memory!" I'm trying to
>> > work out if this seemingly simple code is in fact too simple:
>> >
>> > package NotifyOnOutOfMemory;
>> > use Errno;
>> > END {
>> >     # If the last errno indicates out of memory perl will now be
>> >     # terminating with "Out of memory!" so tell me about it.
>> >     if ( $! == Errno::ENOMEM ) {
>> >         # Do some low-memory thing that tells me a process has died due
>> >         to memory exhaustion
>> >         # eg. touch /path/to/some/file
>> >         #    or - exec("/path/to/script/which/emails/or/logs $$ $0
>> >         $someOtherDetail")
>> >      }
>> > }
>> > 1;
>> >
>> > Which would give me a drop in module for scripts with this problem.
>> >
>> > What am I missing? :)
>> >
>> > Cheerio,
>> >
>> >  Brad
>> >
>> > --
>> > Bradley Dean
>> > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
>> > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
>> > _______________________________________________
>> > Melbourne-pm mailing list
>> > Melbourne-pm at pm.org
>> > http://mail.pm.org/mailman/listinfo/melbourne-pm
>>
>>
>>
>> --
>> Turning and turning in the widening gyre
>> The falcon cannot hear the falconer
>> Things fall apart; the center cannot hold
>> Mere anarchy is loosed upon the world
>
>
> --
> Bradley Dean
> Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm
>
>
>
>
> --
> Bradley Dean
> Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
>
>
>
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm
>
>
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm
>
>
> --
> Bradley Dean
> Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
>
>
>
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm