[Melbourne-pm] Finding "Out of memory!"

[Bradley Dean]

Hi Sam,

Running on linux, and "yes" to all of your comments (we do monitor
memory usage, and we're restricting ulimits to avoid oom-killer joining
in). With ulimits set perl fails on malloc failing (the fatal error
case) but it does run END blocks. If you do something in the END block
which then causes perl to fail another malloc then you're out of luck.

In fact, oom-killer killing my process is more visible than the perl
killing itself because you can look for oom-killer in the the syslogs
(kern.log usually) , but nothing is logged when perl terminates when it
hits the ulimit because as far as the system is concerned all that has
happened is that a malloc call has failed.

A slight extension to the concept of that NotifyOnOutOfMemory which has
worked in my testing thus far (but which is also a bit hacky, and could
have unexpected consequences if included into a script which had other
BEGIN blocks):

package NotifyOnOutOfMemory;
use Errno;
use MyOrg::ErrorEmailingModule qw(senderroremail);
BEGIN {
    if ( grep { /email-out-of-memory/ } @ARGV ) {
        senderroremail(
            subject => "$0 out of memory",
            message => "Detected Errno::ENOMEM which terminated $0"
        );
    }
}
END {
    if ( $! == Errno::ENOMEM ) {
        exec("$0 email-out-of-memory");
    }
}
1;

Use of exec here is handy because the exec replaces the current process
such that you only need enough memory to run exec (as opposed to say
using MyOrg::ErrorEmailingModule::senderroremail in the END block in the
process which has hit the ulimit because senderroremail may try and
allocate memory).

Cheerio,

 Brad

On Fri, Oct 3, 2014, at 12:19, Sam Watkins wrote:
> What operating system are you running the scripts on?  Linux, at least,
> does not handle out-of-memory conditions very well.  It over-allocates
> memory, and if memory runs out the kernel kills likely processes.  You
> won't see an "out of memory" error from malloc (or whatever) under
> linux.  Also, when the system is low on memory things tend to grind to a
> halt with excessive swapping.  You might want to catch the problem
> before it gets to this stage.
> 
> Also, if a program "dies" with an error, the END {} block is not called.
> You would need a $SIG{__DIE__} handler or eval {} block to catch the
> error.  I'm not sure if perl needs memory to continue basic processing
> such as entering a function.
> 
> I suggest to find or write a program that can monitor your processes
> from the outside, and alert you if a process is using too much memory,
> or if a process dies unexpectedly.
> 
> 
> On Fri, Oct 03, 2014 at 11:54:12AM +1000, Bradley Dean wrote:
> > Greetings folks,
> > 
> > While looking at a problem where a variety of scripts might develop a
> > memory leak and fatally terminate with "Out of memory!" I'm trying to
> > work out if this seemingly simple code is in fact too simple:
> > 
> > package NotifyOnOutOfMemory;
> > use Errno;
> > END {
> >     # If the last errno indicates out of memory perl will now be
> >     # terminating with "Out of memory!" so tell me about it.
> >     if ( $! == Errno::ENOMEM ) {
> >         # Do some low-memory thing that tells me a process has died due
> >         to memory exhaustion
> >         # eg. touch /path/to/some/file
> >         #    or - exec("/path/to/script/which/emails/or/logs $$ $0
> >         $someOtherDetail")
> >      }
> > }
> > 1;
> > 
> > Which would give me a drop in module for scripts with this problem.
> > 
> > What am I missing? :)
> > 
> > Cheerio,
> > 
> >  Brad
> > 
> > -- 
> > Bradley Dean
> > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
> > _______________________________________________
> > Melbourne-pm mailing list
> > Melbourne-pm at pm.org
> > http://mail.pm.org/mailman/listinfo/melbourne-pm
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm


-- 
Bradley Dean
Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/