<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Yep - doing that too. :)<br></div>
<div> </div>
<div>This idea started as "it would be handy to have a drop-in module that would help capture memory problems occurring while they are occurring and before they are fixed". Actually - the specific problem that started this line of thought is now history as the memory leak has been identified and fixed.<br></div>
<div> </div>
<div>I wasn't suggesting that an application that leaks memory should just be left to keep doing that - my idea was really of interest if said application only leaks memory sometimes and it's taking a while to find why (but the application needs to be run in the meantime).<br></div>
<div> </div>
<div>Cheerio,<br></div>
<div> </div>
<div> Brad<br></div>
<div> </div>
<div> </div>
<div>On Fri, Oct 10, 2014, at 13:06, Toby Corkindale wrote:<br></div>
<blockquote type="cite"><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000"><div>When you encounter a problem that seems difficult to handle and has very little evidence of other people handling it, then it is a good idea to take a step back and look at the wider picture. Your overall architecture may simply be flawed, and a change elsewhere may simplify later issues.<br></div>
<div> </div>
<div>In this case, I suggest you fix your design so that out of memory errors stop occurring.<br></div>
<div>Eg. Don't load entire massive files or data structures into memory, but instead use a specific data storage and retrieval system that is suited to the task and can manage the paging of data to disk with automatic caching to memory. (For instance, SQLite or LevelDB)<br></div>
<div> </div>
<div> </div>
<div><hr><br></div>
<blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div><b>From: </b>"Bradley Dean" <bjdean@bjdean.id.au><br></div>
<div><b>To: </b>"Mathew Robertson" <mathew.blair.robertson@gmail.com><br></div>
<div><b>Cc: </b>"Melbourne Perl Mongers" <melbourne-pm@pm.org><br></div>
<div><b>Sent: </b>Friday, 10 October, 2014 12:51:11 PM<br></div>
<div><b>Subject: </b>Re: [Melbourne-pm] Finding "Out of memory!"<br></div>
<div> </div>
<div>Yes indeed - a non-zero exit code but not one that tells me anything about the reason (because I can't reliable catch the out of memory error to set a useful exit code, and perl isn't doing that either). :)<br></div>
<div> </div>
<div>I'm not claiming that searching for a magic string in STDERR is a good option, but at the moment I'm not sure there's a better one.<br></div>
<div> </div>
<div>Cheerio,<br></div>
<div> </div>
<div> Brad<br></div>
<div> </div>
<div> </div>
<div>On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote:<br></div>
<blockquote><div dir="ltr"><div>The Perl script should exit with a non-zero exit code... it doesn't really matter why that exit code was non-zero, as any non-zero value will indicate a failure mode.<br></div>
<div> </div>
<div>eg: grep returns non-zero when it didn't actually grep for anything.... (you asked it to do something, but it couldn't).<br></div>
<div> </div>
<div>Tracing STDERR will be fraught with problems - how do you know some Perl library doesn't randomly spit out messages that you have yet to see?<br></div>
<div> </div>
<div> </div>
<div>just my $0.02,<br></div>
<div> </div>
<div>Mathew<br></div>
<div> </div>
</div>
<div><div> </div>
<div><div>On 10 October 2014 10:46, Bradley Dean <span dir="ltr"><<a href="mailto:bjdean@bjdean.id.au" target="_blank">bjdean@bjdean.id.au</a>></span> wrote:<br></div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Yes - that's certainly a safer approach. In fact I've found that what<br></div>
<div>worked for me in my example code was version sensitive.<br></div>
<div> </div>
<div>On the system I started with the last system call was an mmap2 with a<br></div>
<div>ENOMEM error and this is what I found when checking $!. I then tested<br></div>
<div>this on a different system and discovered that after the mmap2 failed<br></div>
<div>(and before my perl code had a chance to look at $!) a bunch of<br></div>
<div>rt_sigaction systems calls happened and one of those failed with EINVAL<br></div>
<div>so there was no ENOMEM to find.<br></div>
<div> </div>
<div>I feel there's a hack with a wrapper script which greps for "Out of<br></div>
<div>memory!" on STDERR coming my way as the exit code doesn't tell me<br></div>
<div>anything useful either.<br></div>
<div> </div>
<div>Cheerio,<br></div>
<div> </div>
<div> Brad<br></div>
<div><div> </div>
<div>On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote:<br></div>
<div>> I really don't think this approach is going to pay dividends.<br></div>
<div>> When you're hitting out of memory conditions in Perl, it's usually<br></div>
<div>> game over. In some languages you could code up something that runs<br></div>
<div>> without using any more memory, as a handler, but you don't have that<br></div>
<div>> level of control in Perl.<br></div>
<div>><br></div>
<div>> I'd look at wrapping your Perl script in another program - maybe just<br></div>
<div>> a bash or even another Perl script - that waits for it to exit, and<br></div>
<div>> then checks to see if it exited normally or with an error code.<br></div>
<div>><br></div>
<div>> On 3 October 2014 11:54, Bradley Dean <<a href="mailto:bjdean@bjdean.id.au" target="_blank">bjdean@bjdean.id.au</a>> wrote:<br></div>
<div>> > Greetings folks,<br></div>
<div>> ><br></div>
<div>> > While looking at a problem where a variety of scripts might develop a<br></div>
<div>> > memory leak and fatally terminate with "Out of memory!" I'm trying to<br></div>
<div>> > work out if this seemingly simple code is in fact too simple:<br></div>
<div>> ><br></div>
<div>> > package NotifyOnOutOfMemory;<br></div>
<div>> > use Errno;<br></div>
<div>> > END {<br></div>
<div>> > # If the last errno indicates out of memory perl will now be<br></div>
<div>> > # terminating with "Out of memory!" so tell me about it.<br></div>
<div>> > if ( $! == Errno::ENOMEM ) {<br></div>
<div>> > # Do some low-memory thing that tells me a process has died due<br></div>
<div>> > to memory exhaustion<br></div>
<div>> > # eg. touch /path/to/some/file<br></div>
<div>> > # or - exec("/path/to/script/which/emails/or/logs $$ $0<br></div>
<div>> > $someOtherDetail")<br></div>
<div>> > }<br></div>
<div>> > }<br></div>
<div>> > 1;<br></div>
<div>> ><br></div>
<div>> > Which would give me a drop in module for scripts with this problem.<br></div>
<div>> ><br></div>
<div>> > What am I missing? :)<br></div>
<div>> ><br></div>
<div>> > Cheerio,<br></div>
<div>> ><br></div>
<div>> > Brad<br></div>
<div>> ><br></div>
<div>> > --<br></div>
<div>> > Bradley Dean<br></div>
<div>> > Email: <a href="mailto:bjdean@bjdean.id.au" target="_blank">bjdean@bjdean.id.au</a> Skype: <a href="mailto:skype@bjdean.id.au" target="_blank">skype@bjdean.id.au</a><br></div>
<div>> > Mobile(Aus): <a href="tel:%2B61-413014395" target="_blank">+61-413014395</a> WWW: <a href="http://bjdean.id.au/" target="_blank">http://bjdean.id.au/</a><br></div>
<div>> > _______________________________________________<br></div>
<div>> > Melbourne-pm mailing list<br></div>
<div>> > <a href="mailto:Melbourne-pm@pm.org" target="_blank">Melbourne-pm@pm.org</a><br></div>
<div>> > <a href="http://mail.pm.org/mailman/listinfo/melbourne-pm" target="_blank">http://mail.pm.org/mailman/listinfo/melbourne-pm</a><br></div>
<div>><br></div>
<div>><br></div>
<div>><br></div>
<div>> --<br></div>
<div>> Turning and turning in the widening gyre<br></div>
<div>> The falcon cannot hear the falconer<br></div>
<div>> Things fall apart; the center cannot hold<br></div>
<div>> Mere anarchy is loosed upon the world<br></div>
<div> </div>
<div> </div>
<div>--<br></div>
<div>Bradley Dean<br></div>
<div>Email: <a href="mailto:bjdean@bjdean.id.au" target="_blank">bjdean@bjdean.id.au</a> Skype: <a href="mailto:skype@bjdean.id.au" target="_blank">skype@bjdean.id.au</a><br></div>
<div>Mobile(Aus): <a href="tel:%2B61-413014395" target="_blank">+61-413014395</a> WWW: <a href="http://bjdean.id.au/" target="_blank">http://bjdean.id.au/</a><br></div>
<div>_______________________________________________<br></div>
<div>Melbourne-pm mailing list<br></div>
<div><a href="mailto:Melbourne-pm@pm.org" target="_blank">Melbourne-pm@pm.org</a><br></div>
<div><a href="http://mail.pm.org/mailman/listinfo/melbourne-pm" target="_blank">http://mail.pm.org/mailman/listinfo/melbourne-pm</a><br></div>
</div>
</blockquote></div>
<div> </div>
</div>
</blockquote><div> </div>
<div><div>--<br></div>
<div>Bradley Dean<br></div>
<div>Email: bjdean@bjdean.id.au Skype: skype@bjdean.id.au<br></div>
<div>Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/<br></div>
<div> </div>
</div>
<div> </div>
<div> </div>
<div>_______________________________________________<br></div>
<div>Melbourne-pm mailing list<br></div>
<div>Melbourne-pm@pm.org<br></div>
<div>http://mail.pm.org/mailman/listinfo/melbourne-pm<br></div>
</blockquote><div> </div>
</div>
<div><u>_______________________________________________</u><br></div>
<div>Melbourne-pm mailing list<br></div>
<div><a href="mailto:Melbourne-pm@pm.org">Melbourne-pm@pm.org</a><br></div>
<div><a href="http://mail.pm.org/mailman/listinfo/melbourne-pm">http://mail.pm.org/mailman/listinfo/melbourne-pm</a><br></div>
</blockquote><div> </div>
<div id="sig22637289"><div class="signature">--<br></div>
<div class="signature">Bradley Dean<br></div>
<div class="signature">Email: bjdean@bjdean.id.au Skype: skype@bjdean.id.au<br></div>
<div class="signature">Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/<br></div>
<div class="signature"> </div>
</div>
<div> </div>
</body>
</html>