[Melbourne-pm] Finding "Out of memory!"

[Bradley Dean]

I should also have replied to:

> > Also, if a program "dies" with an error, the END {} block is not called.
> > You would need a $SIG{__DIE__} handler or eval {} block to catch the
> > error.  I'm not sure if perl needs memory to continue basic processing
> > such as entering a function.

END blocks do run after a regular die exception (but that's not what
happens when perl fails to malloc):

    An "END" code block is executed as late as possible, that is, after
    perl
    has finished running the program and just before the interpreter is
    being exited, even if it is exiting as a result of a die() function.
    (But not if it's morphing into another program via "exec", or being
    blown out of the water by a signal--you have to trap that yourself
    (if
    you can).) You may have multiple "END" blocks within a file--they
    will
    execute in reverse order of definition; that is: last in, first out
    (LIFO). "END" blocks are not executed when you run perl with the
    "-c"
    switch, or if compilation fails.
    
    See: perldoc perlmod

One failing of my NotifyOnOutOfMemory idea is that END blocks run LIFO
order. So if somewhere else other END blocks are defined and these try
and do something which gets in the way of the NotifyOnOutOfMemory END
block running then once again I'm out of luck. Perhaps I could mostly
avoid that problem by using "require NotifyOnOutOfMemory;" in an INIT
block.

Cheerio,

 Brad

On Fri, Oct 3, 2014, at 13:52, Bradley Dean wrote:
> Hi Sam,
> 
> Running on linux, and "yes" to all of your comments (we do monitor
> memory usage, and we're restricting ulimits to avoid oom-killer joining
> in). With ulimits set perl fails on malloc failing (the fatal error
> case) but it does run END blocks. If you do something in the END block
> which then causes perl to fail another malloc then you're out of luck.
> 
> In fact, oom-killer killing my process is more visible than the perl
> killing itself because you can look for oom-killer in the the syslogs
> (kern.log usually) , but nothing is logged when perl terminates when it
> hits the ulimit because as far as the system is concerned all that has
> happened is that a malloc call has failed.
> 
> A slight extension to the concept of that NotifyOnOutOfMemory which has
> worked in my testing thus far (but which is also a bit hacky, and could
> have unexpected consequences if included into a script which had other
> BEGIN blocks):
> 
> package NotifyOnOutOfMemory;
> use Errno;
> use MyOrg::ErrorEmailingModule qw(senderroremail);
> BEGIN {
>     if ( grep { /email-out-of-memory/ } @ARGV ) {
>         senderroremail(
>             subject => "$0 out of memory",
>             message => "Detected Errno::ENOMEM which terminated $0"
>         );
>     }
> }
> END {
>     if ( $! == Errno::ENOMEM ) {
>         exec("$0 email-out-of-memory");
>     }
> }
> 1;
> 
> Use of exec here is handy because the exec replaces the current process
> such that you only need enough memory to run exec (as opposed to say
> using MyOrg::ErrorEmailingModule::senderroremail in the END block in the
> process which has hit the ulimit because senderroremail may try and
> allocate memory).
> 
> Cheerio,
> 
>  Brad
> 
> On Fri, Oct 3, 2014, at 12:19, Sam Watkins wrote:
> > What operating system are you running the scripts on?  Linux, at least,
> > does not handle out-of-memory conditions very well.  It over-allocates
> > memory, and if memory runs out the kernel kills likely processes.  You
> > won't see an "out of memory" error from malloc (or whatever) under
> > linux.  Also, when the system is low on memory things tend to grind to a
> > halt with excessive swapping.  You might want to catch the problem
> > before it gets to this stage.
> > 
> > Also, if a program "dies" with an error, the END {} block is not called.
> > You would need a $SIG{__DIE__} handler or eval {} block to catch the
> > error.  I'm not sure if perl needs memory to continue basic processing
> > such as entering a function.
> > 
> > I suggest to find or write a program that can monitor your processes
> > from the outside, and alert you if a process is using too much memory,
> > or if a process dies unexpectedly.
> > 
> > 
> > On Fri, Oct 03, 2014 at 11:54:12AM +1000, Bradley Dean wrote:
> > > Greetings folks,
> > > 
> > > While looking at a problem where a variety of scripts might develop a
> > > memory leak and fatally terminate with "Out of memory!" I'm trying to
> > > work out if this seemingly simple code is in fact too simple:
> > > 
> > > package NotifyOnOutOfMemory;
> > > use Errno;
> > > END {
> > >     # If the last errno indicates out of memory perl will now be
> > >     # terminating with "Out of memory!" so tell me about it.
> > >     if ( $! == Errno::ENOMEM ) {
> > >         # Do some low-memory thing that tells me a process has died due
> > >         to memory exhaustion
> > >         # eg. touch /path/to/some/file
> > >         #    or - exec("/path/to/script/which/emails/or/logs $$ $0
> > >         $someOtherDetail")
> > >      }
> > > }
> > > 1;
> > > 
> > > Which would give me a drop in module for scripts with this problem.
> > > 
> > > What am I missing? :)
> > > 
> > > Cheerio,
> > > 
> > >  Brad
> > > 
> > > -- 
> > > Bradley Dean
> > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
> > > _______________________________________________
> > > Melbourne-pm mailing list
> > > Melbourne-pm at pm.org
> > > http://mail.pm.org/mailman/listinfo/melbourne-pm
> > _______________________________________________
> > Melbourne-pm mailing list
> > Melbourne-pm at pm.org
> > http://mail.pm.org/mailman/listinfo/melbourne-pm
> 
> 
> -- 
> Bradley Dean
> Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
> Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/
> _______________________________________________
> Melbourne-pm mailing list
> Melbourne-pm at pm.org
> http://mail.pm.org/mailman/listinfo/melbourne-pm


-- 
Bradley Dean
Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au
Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/