From bjdean at bjdean.id.au Thu Oct 2 18:54:12 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 03 Oct 2014 11:54:12 +1000 Subject: [Melbourne-pm] Finding "Out of memory!" Message-ID: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> Greetings folks, While looking at a problem where a variety of scripts might develop a memory leak and fatally terminate with "Out of memory!" I'm trying to work out if this seemingly simple code is in fact too simple: package NotifyOnOutOfMemory; use Errno; END { # If the last errno indicates out of memory perl will now be # terminating with "Out of memory!" so tell me about it. if ( $! == Errno::ENOMEM ) { # Do some low-memory thing that tells me a process has died due to memory exhaustion # eg. touch /path/to/some/file # or - exec("/path/to/script/which/emails/or/logs $$ $0 $someOtherDetail") } } 1; Which would give me a drop in module for scripts with this problem. What am I missing? :) Cheerio, Brad -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ From sam at nipl.net Thu Oct 2 19:19:06 2014 From: sam at nipl.net (Sam Watkins) Date: Fri, 3 Oct 2014 12:19:06 +1000 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <20141003021003.GD2129@opal.nipl.net> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <20141003021003.GD2129@opal.nipl.net> Message-ID: <20141003021906.GE2129@opal.nipl.net> What operating system are you running the scripts on? Linux, at least, does not handle out-of-memory conditions very well. It over-allocates memory, and if memory runs out the kernel kills likely processes. You won't see an "out of memory" error from malloc (or whatever) under linux. Also, when the system is low on memory things tend to grind to a halt with excessive swapping. You might want to catch the problem before it gets to this stage. Also, if a program "dies" with an error, the END {} block is not called. You would need a $SIG{__DIE__} handler or eval {} block to catch the error. I'm not sure if perl needs memory to continue basic processing such as entering a function. I suggest to find or write a program that can monitor your processes from the outside, and alert you if a process is using too much memory, or if a process dies unexpectedly. On Fri, Oct 03, 2014 at 11:54:12AM +1000, Bradley Dean wrote: > Greetings folks, > > While looking at a problem where a variety of scripts might develop a > memory leak and fatally terminate with "Out of memory!" I'm trying to > work out if this seemingly simple code is in fact too simple: > > package NotifyOnOutOfMemory; > use Errno; > END { > # If the last errno indicates out of memory perl will now be > # terminating with "Out of memory!" so tell me about it. > if ( $! == Errno::ENOMEM ) { > # Do some low-memory thing that tells me a process has died due > to memory exhaustion > # eg. touch /path/to/some/file > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > $someOtherDetail") > } > } > 1; > > Which would give me a drop in module for scripts with this problem. > > What am I missing? :) > > Cheerio, > > Brad > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm From bjdean at bjdean.id.au Thu Oct 2 20:52:47 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 03 Oct 2014 13:52:47 +1000 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <20141003021906.GE2129@opal.nipl.net> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <20141003021003.GD2129@opal.nipl.net> <20141003021906.GE2129@opal.nipl.net> Message-ID: <1412308367.3024942.174632641.48DEDA13@webmail.messagingengine.com> Hi Sam, Running on linux, and "yes" to all of your comments (we do monitor memory usage, and we're restricting ulimits to avoid oom-killer joining in). With ulimits set perl fails on malloc failing (the fatal error case) but it does run END blocks. If you do something in the END block which then causes perl to fail another malloc then you're out of luck. In fact, oom-killer killing my process is more visible than the perl killing itself because you can look for oom-killer in the the syslogs (kern.log usually) , but nothing is logged when perl terminates when it hits the ulimit because as far as the system is concerned all that has happened is that a malloc call has failed. A slight extension to the concept of that NotifyOnOutOfMemory which has worked in my testing thus far (but which is also a bit hacky, and could have unexpected consequences if included into a script which had other BEGIN blocks): package NotifyOnOutOfMemory; use Errno; use MyOrg::ErrorEmailingModule qw(senderroremail); BEGIN { if ( grep { /email-out-of-memory/ } @ARGV ) { senderroremail( subject => "$0 out of memory", message => "Detected Errno::ENOMEM which terminated $0" ); } } END { if ( $! == Errno::ENOMEM ) { exec("$0 email-out-of-memory"); } } 1; Use of exec here is handy because the exec replaces the current process such that you only need enough memory to run exec (as opposed to say using MyOrg::ErrorEmailingModule::senderroremail in the END block in the process which has hit the ulimit because senderroremail may try and allocate memory). Cheerio, Brad On Fri, Oct 3, 2014, at 12:19, Sam Watkins wrote: > What operating system are you running the scripts on? Linux, at least, > does not handle out-of-memory conditions very well. It over-allocates > memory, and if memory runs out the kernel kills likely processes. You > won't see an "out of memory" error from malloc (or whatever) under > linux. Also, when the system is low on memory things tend to grind to a > halt with excessive swapping. You might want to catch the problem > before it gets to this stage. > > Also, if a program "dies" with an error, the END {} block is not called. > You would need a $SIG{__DIE__} handler or eval {} block to catch the > error. I'm not sure if perl needs memory to continue basic processing > such as entering a function. > > I suggest to find or write a program that can monitor your processes > from the outside, and alert you if a process is using too much memory, > or if a process dies unexpectedly. > > > On Fri, Oct 03, 2014 at 11:54:12AM +1000, Bradley Dean wrote: > > Greetings folks, > > > > While looking at a problem where a variety of scripts might develop a > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > work out if this seemingly simple code is in fact too simple: > > > > package NotifyOnOutOfMemory; > > use Errno; > > END { > > # If the last errno indicates out of memory perl will now be > > # terminating with "Out of memory!" so tell me about it. > > if ( $! == Errno::ENOMEM ) { > > # Do some low-memory thing that tells me a process has died due > > to memory exhaustion > > # eg. touch /path/to/some/file > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > $someOtherDetail") > > } > > } > > 1; > > > > Which would give me a drop in module for scripts with this problem. > > > > What am I missing? :) > > > > Cheerio, > > > > Brad > > > > -- > > Bradley Dean > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > _______________________________________________ > > Melbourne-pm mailing list > > Melbourne-pm at pm.org > > http://mail.pm.org/mailman/listinfo/melbourne-pm > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ From bjdean at bjdean.id.au Thu Oct 2 23:36:32 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 03 Oct 2014 16:36:32 +1000 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1412308367.3024942.174632641.48DEDA13@webmail.messagingengine.com> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <20141003021003.GD2129@opal.nipl.net> <20141003021906.GE2129@opal.nipl.net> <1412308367.3024942.174632641.48DEDA13@webmail.messagingengine.com> Message-ID: <1412318192.3075338.174663269.55EFBCCF@webmail.messagingengine.com> I should also have replied to: > > Also, if a program "dies" with an error, the END {} block is not called. > > You would need a $SIG{__DIE__} handler or eval {} block to catch the > > error. I'm not sure if perl needs memory to continue basic processing > > such as entering a function. END blocks do run after a regular die exception (but that's not what happens when perl fails to malloc): An "END" code block is executed as late as possible, that is, after perl has finished running the program and just before the interpreter is being exited, even if it is exiting as a result of a die() function. (But not if it's morphing into another program via "exec", or being blown out of the water by a signal--you have to trap that yourself (if you can).) You may have multiple "END" blocks within a file--they will execute in reverse order of definition; that is: last in, first out (LIFO). "END" blocks are not executed when you run perl with the "-c" switch, or if compilation fails. See: perldoc perlmod One failing of my NotifyOnOutOfMemory idea is that END blocks run LIFO order. So if somewhere else other END blocks are defined and these try and do something which gets in the way of the NotifyOnOutOfMemory END block running then once again I'm out of luck. Perhaps I could mostly avoid that problem by using "require NotifyOnOutOfMemory;" in an INIT block. Cheerio, Brad On Fri, Oct 3, 2014, at 13:52, Bradley Dean wrote: > Hi Sam, > > Running on linux, and "yes" to all of your comments (we do monitor > memory usage, and we're restricting ulimits to avoid oom-killer joining > in). With ulimits set perl fails on malloc failing (the fatal error > case) but it does run END blocks. If you do something in the END block > which then causes perl to fail another malloc then you're out of luck. > > In fact, oom-killer killing my process is more visible than the perl > killing itself because you can look for oom-killer in the the syslogs > (kern.log usually) , but nothing is logged when perl terminates when it > hits the ulimit because as far as the system is concerned all that has > happened is that a malloc call has failed. > > A slight extension to the concept of that NotifyOnOutOfMemory which has > worked in my testing thus far (but which is also a bit hacky, and could > have unexpected consequences if included into a script which had other > BEGIN blocks): > > package NotifyOnOutOfMemory; > use Errno; > use MyOrg::ErrorEmailingModule qw(senderroremail); > BEGIN { > if ( grep { /email-out-of-memory/ } @ARGV ) { > senderroremail( > subject => "$0 out of memory", > message => "Detected Errno::ENOMEM which terminated $0" > ); > } > } > END { > if ( $! == Errno::ENOMEM ) { > exec("$0 email-out-of-memory"); > } > } > 1; > > Use of exec here is handy because the exec replaces the current process > such that you only need enough memory to run exec (as opposed to say > using MyOrg::ErrorEmailingModule::senderroremail in the END block in the > process which has hit the ulimit because senderroremail may try and > allocate memory). > > Cheerio, > > Brad > > On Fri, Oct 3, 2014, at 12:19, Sam Watkins wrote: > > What operating system are you running the scripts on? Linux, at least, > > does not handle out-of-memory conditions very well. It over-allocates > > memory, and if memory runs out the kernel kills likely processes. You > > won't see an "out of memory" error from malloc (or whatever) under > > linux. Also, when the system is low on memory things tend to grind to a > > halt with excessive swapping. You might want to catch the problem > > before it gets to this stage. > > > > Also, if a program "dies" with an error, the END {} block is not called. > > You would need a $SIG{__DIE__} handler or eval {} block to catch the > > error. I'm not sure if perl needs memory to continue basic processing > > such as entering a function. > > > > I suggest to find or write a program that can monitor your processes > > from the outside, and alert you if a process is using too much memory, > > or if a process dies unexpectedly. > > > > > > On Fri, Oct 03, 2014 at 11:54:12AM +1000, Bradley Dean wrote: > > > Greetings folks, > > > > > > While looking at a problem where a variety of scripts might develop a > > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > > work out if this seemingly simple code is in fact too simple: > > > > > > package NotifyOnOutOfMemory; > > > use Errno; > > > END { > > > # If the last errno indicates out of memory perl will now be > > > # terminating with "Out of memory!" so tell me about it. > > > if ( $! == Errno::ENOMEM ) { > > > # Do some low-memory thing that tells me a process has died due > > > to memory exhaustion > > > # eg. touch /path/to/some/file > > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > > $someOtherDetail") > > > } > > > } > > > 1; > > > > > > Which would give me a drop in module for scripts with this problem. > > > > > > What am I missing? :) > > > > > > Cheerio, > > > > > > Brad > > > > > > -- > > > Bradley Dean > > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > _______________________________________________ > > > Melbourne-pm mailing list > > > Melbourne-pm at pm.org > > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > _______________________________________________ > > Melbourne-pm mailing list > > Melbourne-pm at pm.org > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ From tjc at wintrmute.net Thu Oct 9 16:37:49 2014 From: tjc at wintrmute.net (Toby Wintermute) Date: Fri, 10 Oct 2014 10:37:49 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> Message-ID: I really don't think this approach is going to pay dividends. When you're hitting out of memory conditions in Perl, it's usually game over. In some languages you could code up something that runs without using any more memory, as a handler, but you don't have that level of control in Perl. I'd look at wrapping your Perl script in another program - maybe just a bash or even another Perl script - that waits for it to exit, and then checks to see if it exited normally or with an error code. On 3 October 2014 11:54, Bradley Dean wrote: > Greetings folks, > > While looking at a problem where a variety of scripts might develop a > memory leak and fatally terminate with "Out of memory!" I'm trying to > work out if this seemingly simple code is in fact too simple: > > package NotifyOnOutOfMemory; > use Errno; > END { > # If the last errno indicates out of memory perl will now be > # terminating with "Out of memory!" so tell me about it. > if ( $! == Errno::ENOMEM ) { > # Do some low-memory thing that tells me a process has died due > to memory exhaustion > # eg. touch /path/to/some/file > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > $someOtherDetail") > } > } > 1; > > Which would give me a drop in module for scripts with this problem. > > What am I missing? :) > > Cheerio, > > Brad > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world From bjdean at bjdean.id.au Thu Oct 9 16:46:50 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 10 Oct 2014 10:46:50 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> Message-ID: <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> Yes - that's certainly a safer approach. In fact I've found that what worked for me in my example code was version sensitive. On the system I started with the last system call was an mmap2 with a ENOMEM error and this is what I found when checking $!. I then tested this on a different system and discovered that after the mmap2 failed (and before my perl code had a chance to look at $!) a bunch of rt_sigaction systems calls happened and one of those failed with EINVAL so there was no ENOMEM to find. I feel there's a hack with a wrapper script which greps for "Out of memory!" on STDERR coming my way as the exit code doesn't tell me anything useful either. Cheerio, Brad On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: > I really don't think this approach is going to pay dividends. > When you're hitting out of memory conditions in Perl, it's usually > game over. In some languages you could code up something that runs > without using any more memory, as a handler, but you don't have that > level of control in Perl. > > I'd look at wrapping your Perl script in another program - maybe just > a bash or even another Perl script - that waits for it to exit, and > then checks to see if it exited normally or with an error code. > > On 3 October 2014 11:54, Bradley Dean wrote: > > Greetings folks, > > > > While looking at a problem where a variety of scripts might develop a > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > work out if this seemingly simple code is in fact too simple: > > > > package NotifyOnOutOfMemory; > > use Errno; > > END { > > # If the last errno indicates out of memory perl will now be > > # terminating with "Out of memory!" so tell me about it. > > if ( $! == Errno::ENOMEM ) { > > # Do some low-memory thing that tells me a process has died due > > to memory exhaustion > > # eg. touch /path/to/some/file > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > $someOtherDetail") > > } > > } > > 1; > > > > Which would give me a drop in module for scripts with this problem. > > > > What am I missing? :) > > > > Cheerio, > > > > Brad > > > > -- > > Bradley Dean > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > _______________________________________________ > > Melbourne-pm mailing list > > Melbourne-pm at pm.org > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > -- > Turning and turning in the widening gyre > The falcon cannot hear the falconer > Things fall apart; the center cannot hold > Mere anarchy is loosed upon the world -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ From mathew.blair.robertson at gmail.com Thu Oct 9 17:19:31 2014 From: mathew.blair.robertson at gmail.com (Mathew Robertson) Date: Fri, 10 Oct 2014 11:19:31 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> Message-ID: The Perl script should exit with a non-zero exit code... it doesn't really matter why that exit code was non-zero, as any non-zero value will indicate a failure mode. eg: grep returns non-zero when it didn't actually grep for anything.... (you asked it to do something, but it couldn't). Tracing STDERR will be fraught with problems - how do you know some Perl library doesn't randomly spit out messages that you have yet to see? just my $0.02, Mathew On 10 October 2014 10:46, Bradley Dean wrote: > Yes - that's certainly a safer approach. In fact I've found that what > worked for me in my example code was version sensitive. > > On the system I started with the last system call was an mmap2 with a > ENOMEM error and this is what I found when checking $!. I then tested > this on a different system and discovered that after the mmap2 failed > (and before my perl code had a chance to look at $!) a bunch of > rt_sigaction systems calls happened and one of those failed with EINVAL > so there was no ENOMEM to find. > > I feel there's a hack with a wrapper script which greps for "Out of > memory!" on STDERR coming my way as the exit code doesn't tell me > anything useful either. > > Cheerio, > > Brad > > On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: > > I really don't think this approach is going to pay dividends. > > When you're hitting out of memory conditions in Perl, it's usually > > game over. In some languages you could code up something that runs > > without using any more memory, as a handler, but you don't have that > > level of control in Perl. > > > > I'd look at wrapping your Perl script in another program - maybe just > > a bash or even another Perl script - that waits for it to exit, and > > then checks to see if it exited normally or with an error code. > > > > On 3 October 2014 11:54, Bradley Dean wrote: > > > Greetings folks, > > > > > > While looking at a problem where a variety of scripts might develop a > > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > > work out if this seemingly simple code is in fact too simple: > > > > > > package NotifyOnOutOfMemory; > > > use Errno; > > > END { > > > # If the last errno indicates out of memory perl will now be > > > # terminating with "Out of memory!" so tell me about it. > > > if ( $! == Errno::ENOMEM ) { > > > # Do some low-memory thing that tells me a process has died due > > > to memory exhaustion > > > # eg. touch /path/to/some/file > > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > > $someOtherDetail") > > > } > > > } > > > 1; > > > > > > Which would give me a drop in module for scripts with this problem. > > > > > > What am I missing? :) > > > > > > Cheerio, > > > > > > Brad > > > > > > -- > > > Bradley Dean > > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > _______________________________________________ > > > Melbourne-pm mailing list > > > Melbourne-pm at pm.org > > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > > > > > -- > > Turning and turning in the widening gyre > > The falcon cannot hear the falconer > > Things fall apart; the center cannot hold > > Mere anarchy is loosed upon the world > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjdean at bjdean.id.au Thu Oct 9 18:51:11 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 10 Oct 2014 12:51:11 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> Message-ID: <1412905871.3409191.177288093.2EF8793B@webmail.messagingengine.com> Yes indeed - a non-zero exit code but not one that tells me anything about the reason (because I can't reliable catch the out of memory error to set a useful exit code, and perl isn't doing that either). :) I'm not claiming that searching for a magic string in STDERR is a good option, but at the moment I'm not sure there's a better one. Cheerio, Brad On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote: The Perl script should exit with a non-zero exit code... it doesn't really matter why that exit code was non-zero, as any non-zero value will indicate a failure mode. eg: grep returns non-zero when it didn't actually grep for anything.... (you asked it to do something, but it couldn't). Tracing STDERR will be fraught with problems - how do you know some Perl library doesn't randomly spit out messages that you have yet to see? just my $0.02, Mathew On 10 October 2014 10:46, Bradley Dean <[1]bjdean at bjdean.id.au> wrote: Yes - that's certainly a safer approach. In fact I've found that what worked for me in my example code was version sensitive. On the system I started with the last system call was an mmap2 with a ENOMEM error and this is what I found when checking $!. I then tested this on a different system and discovered that after the mmap2 failed (and before my perl code had a chance to look at $!) a bunch of rt_sigaction systems calls happened and one of those failed with EINVAL so there was no ENOMEM to find. I feel there's a hack with a wrapper script which greps for "Out of memory!" on STDERR coming my way as the exit code doesn't tell me anything useful either. Cheerio, Brad On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: > I really don't think this approach is going to pay dividends. > When you're hitting out of memory conditions in Perl, it's usually > game over. In some languages you could code up something that runs > without using any more memory, as a handler, but you don't have that > level of control in Perl. > > I'd look at wrapping your Perl script in another program - maybe just > a bash or even another Perl script - that waits for it to exit, and > then checks to see if it exited normally or with an error code. > > On 3 October 2014 11:54, Bradley Dean <[2]bjdean at bjdean.id.au> wrote: > > Greetings folks, > > > > While looking at a problem where a variety of scripts might develop a > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > work out if this seemingly simple code is in fact too simple: > > > > package NotifyOnOutOfMemory; > > use Errno; > > END { > > # If the last errno indicates out of memory perl will now be > > # terminating with "Out of memory!" so tell me about it. > > if ( $! == Errno::ENOMEM ) { > > # Do some low-memory thing that tells me a process has died due > > to memory exhaustion > > # eg. touch /path/to/some/file > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > $someOtherDetail") > > } > > } > > 1; > > > > Which would give me a drop in module for scripts with this problem. > > > > What am I missing? :) > > > > Cheerio, > > > > Brad > > > > -- > > Bradley Dean > > Email: [3]bjdean at bjdean.id.au Skype: [4]skype at bjdean.id.au > > Mobile(Aus): [5]+61-413014395 WWW: [6]http://bjdean.id.au/ > > _______________________________________________ > > Melbourne-pm mailing list > > [7]Melbourne-pm at pm.org > > [8]http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > -- > Turning and turning in the widening gyre > The falcon cannot hear the falconer > Things fall apart; the center cannot hold > Mere anarchy is loosed upon the world -- Bradley Dean Email: [9]bjdean at bjdean.id.au Skype: [10]skype at bjdean.id.au Mobile(Aus): [11]+61-413014395 WWW: [12]http://bjdean.id.au/ _______________________________________________ Melbourne-pm mailing list [13]Melbourne-pm at pm.org [14]http://mail.pm.org/mailman/listinfo/melbourne-pm -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ References 1. mailto:bjdean at bjdean.id.au 2. mailto:bjdean at bjdean.id.au 3. mailto:bjdean at bjdean.id.au 4. mailto:skype at bjdean.id.au 5. tel:%2B61-413014395 6. http://bjdean.id.au/ 7. mailto:Melbourne-pm at pm.org 8. http://mail.pm.org/mailman/listinfo/melbourne-pm 9. mailto:bjdean at bjdean.id.au 10. mailto:skype at bjdean.id.au 11. tel:%2B61-413014395 12. http://bjdean.id.au/ 13. mailto:Melbourne-pm at pm.org 14. http://mail.pm.org/mailman/listinfo/melbourne-pm -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby.corkindale at strategicdata.com.au Thu Oct 9 19:06:41 2014 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 10 Oct 2014 13:06:41 +1100 (EST) Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1412905871.3409191.177288093.2EF8793B@webmail.messagingengine.com> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> <1412905871.3409191.177288093.2EF8793B@webmail.messagingengine.com> Message-ID: <1988991306.120737.1412906801505.JavaMail.zimbra@strategicdata.com.au> When you encounter a problem that seems difficult to handle and has very little evidence of other people handling it, then it is a good idea to take a step back and look at the wider picture. Your overall architecture may simply be flawed, and a change elsewhere may simplify later issues. In this case, I suggest you fix your design so that out of memory errors stop occurring. Eg. Don't load entire massive files or data structures into memory, but instead use a specific data storage and retrieval system that is suited to the task and can manage the paging of data to disk with automatic caching to memory. (For instance, SQLite or LevelDB) ----- Original Message ----- > From: "Bradley Dean" > To: "Mathew Robertson" > Cc: "Melbourne Perl Mongers" > Sent: Friday, 10 October, 2014 12:51:11 PM > Subject: Re: [Melbourne-pm] Finding "Out of memory!" > Yes indeed - a non-zero exit code but not one that tells me anything about > the reason (because I can't reliable catch the out of memory error to set a > useful exit code, and perl isn't doing that either). :) > I'm not claiming that searching for a magic string in STDERR is a good > option, but at the moment I'm not sure there's a better one. > Cheerio, > Brad > On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote: > > The Perl script should exit with a non-zero exit code... it doesn't really > > matter why that exit code was non-zero, as any non-zero value will indicate > > a failure mode. > > > eg: grep returns non-zero when it didn't actually grep for anything.... > > (you > > asked it to do something, but it couldn't). > > > Tracing STDERR will be fraught with problems - how do you know some Perl > > library doesn't randomly spit out messages that you have yet to see? > > > just my $0.02, > > > Mathew > > > On 10 October 2014 10:46, Bradley Dean < bjdean at bjdean.id.au > wrote: > > > > Yes - that's certainly a safer approach. In fact I've found that what > > > > > > worked for me in my example code was version sensitive. > > > > > > On the system I started with the last system call was an mmap2 with a > > > > > > ENOMEM error and this is what I found when checking $!. I then tested > > > > > > this on a different system and discovered that after the mmap2 failed > > > > > > (and before my perl code had a chance to look at $!) a bunch of > > > > > > rt_sigaction systems calls happened and one of those failed with EINVAL > > > > > > so there was no ENOMEM to find. > > > > > > I feel there's a hack with a wrapper script which greps for "Out of > > > > > > memory!" on STDERR coming my way as the exit code doesn't tell me > > > > > > anything useful either. > > > > > > Cheerio, > > > > > > Brad > > > > > > On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: > > > > > > > I really don't think this approach is going to pay dividends. > > > > > > > When you're hitting out of memory conditions in Perl, it's usually > > > > > > > game over. In some languages you could code up something that runs > > > > > > > without using any more memory, as a handler, but you don't have that > > > > > > > level of control in Perl. > > > > > > > > > > > > > > I'd look at wrapping your Perl script in another program - maybe just > > > > > > > a bash or even another Perl script - that waits for it to exit, and > > > > > > > then checks to see if it exited normally or with an error code. > > > > > > > > > > > > > > On 3 October 2014 11:54, Bradley Dean < bjdean at bjdean.id.au > wrote: > > > > > > > > Greetings folks, > > > > > > > > > > > > > > > > While looking at a problem where a variety of scripts might develop a > > > > > > > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > > > > > > > work out if this seemingly simple code is in fact too simple: > > > > > > > > > > > > > > > > package NotifyOnOutOfMemory; > > > > > > > > use Errno; > > > > > > > > END { > > > > > > > > # If the last errno indicates out of memory perl will now be > > > > > > > > # terminating with "Out of memory!" so tell me about it. > > > > > > > > if ( $! == Errno::ENOMEM ) { > > > > > > > > # Do some low-memory thing that tells me a process has died due > > > > > > > > to memory exhaustion > > > > > > > > # eg. touch /path/to/some/file > > > > > > > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > > > > > > > $someOtherDetail") > > > > > > > > } > > > > > > > > } > > > > > > > > 1; > > > > > > > > > > > > > > > > Which would give me a drop in module for scripts with this problem. > > > > > > > > > > > > > > > > What am I missing? :) > > > > > > > > > > > > > > > > Cheerio, > > > > > > > > > > > > > > > > Brad > > > > > > > > > > > > > > > > -- > > > > > > > > Bradley Dean > > > > > > > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > > > > > > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > > > > > > _______________________________________________ > > > > > > > > Melbourne-pm mailing list > > > > > > > > Melbourne-pm at pm.org > > > > > > > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Turning and turning in the widening gyre > > > > > > > The falcon cannot hear the falconer > > > > > > > Things fall apart; the center cannot hold > > > > > > > Mere anarchy is loosed upon the world > > > > > > -- > > > > > > Bradley Dean > > > > > > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > > > > > > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > > > > _______________________________________________ > > > > > > Melbourne-pm mailing list > > > > > > Melbourne-pm at pm.org > > > > > > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjdean at bjdean.id.au Thu Oct 9 21:53:30 2014 From: bjdean at bjdean.id.au (Bradley Dean) Date: Fri, 10 Oct 2014 15:53:30 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1988991306.120737.1412906801505.JavaMail.zimbra@strategicdata.com.au> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> <1412905871.3409191.177288093.2EF8793B@webmail.messagingengine.com> <1988991306.120737.1412906801505.JavaMail.zimbra@strategicdata.com.au> Message-ID: <1412916810.3469310.177325637.7ED8B72E@webmail.messagingengine.com> Yep - doing that too. :) This idea started as "it would be handy to have a drop-in module that would help capture memory problems occurring while they are occurring and before they are fixed". Actually - the specific problem that started this line of thought is now history as the memory leak has been identified and fixed. I wasn't suggesting that an application that leaks memory should just be left to keep doing that - my idea was really of interest if said application only leaks memory sometimes and it's taking a while to find why (but the application needs to be run in the meantime). Cheerio, Brad On Fri, Oct 10, 2014, at 13:06, Toby Corkindale wrote: When you encounter a problem that seems difficult to handle and has very little evidence of other people handling it, then it is a good idea to take a step back and look at the wider picture. Your overall architecture may simply be flawed, and a change elsewhere may simplify later issues. In this case, I suggest you fix your design so that out of memory errors stop occurring. Eg. Don't load entire massive files or data structures into memory, but instead use a specific data storage and retrieval system that is suited to the task and can manage the paging of data to disk with automatic caching to memory. (For instance, SQLite or LevelDB) __________________________________________________________ From: "Bradley Dean" To: "Mathew Robertson" Cc: "Melbourne Perl Mongers" Sent: Friday, 10 October, 2014 12:51:11 PM Subject: Re: [Melbourne-pm] Finding "Out of memory!" Yes indeed - a non-zero exit code but not one that tells me anything about the reason (because I can't reliable catch the out of memory error to set a useful exit code, and perl isn't doing that either). :) I'm not claiming that searching for a magic string in STDERR is a good option, but at the moment I'm not sure there's a better one. Cheerio, Brad On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote: The Perl script should exit with a non-zero exit code... it doesn't really matter why that exit code was non-zero, as any non-zero value will indicate a failure mode. eg: grep returns non-zero when it didn't actually grep for anything.... (you asked it to do something, but it couldn't). Tracing STDERR will be fraught with problems - how do you know some Perl library doesn't randomly spit out messages that you have yet to see? just my $0.02, Mathew On 10 October 2014 10:46, Bradley Dean <[1]bjdean at bjdean.id.au> wrote: Yes - that's certainly a safer approach. In fact I've found that what worked for me in my example code was version sensitive. On the system I started with the last system call was an mmap2 with a ENOMEM error and this is what I found when checking $!. I then tested this on a different system and discovered that after the mmap2 failed (and before my perl code had a chance to look at $!) a bunch of rt_sigaction systems calls happened and one of those failed with EINVAL so there was no ENOMEM to find. I feel there's a hack with a wrapper script which greps for "Out of memory!" on STDERR coming my way as the exit code doesn't tell me anything useful either. Cheerio, Brad On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: > I really don't think this approach is going to pay dividends. > When you're hitting out of memory conditions in Perl, it's usually > game over. In some languages you could code up something that runs > without using any more memory, as a handler, but you don't have that > level of control in Perl. > > I'd look at wrapping your Perl script in another program - maybe just > a bash or even another Perl script - that waits for it to exit, and > then checks to see if it exited normally or with an error code. > > On 3 October 2014 11:54, Bradley Dean <[2]bjdean at bjdean.id.au> wrote: > > Greetings folks, > > > > While looking at a problem where a variety of scripts might develop a > > memory leak and fatally terminate with "Out of memory!" I'm trying to > > work out if this seemingly simple code is in fact too simple: > > > > package NotifyOnOutOfMemory; > > use Errno; > > END { > > # If the last errno indicates out of memory perl will now be > > # terminating with "Out of memory!" so tell me about it. > > if ( $! == Errno::ENOMEM ) { > > # Do some low-memory thing that tells me a process has died due > > to memory exhaustion > > # eg. touch /path/to/some/file > > # or - exec("/path/to/script/which/emails/or/logs $$ $0 > > $someOtherDetail") > > } > > } > > 1; > > > > Which would give me a drop in module for scripts with this problem. > > > > What am I missing? :) > > > > Cheerio, > > > > Brad > > > > -- > > Bradley Dean > > Email: [3]bjdean at bjdean.id.au Skype: [4]skype at bjdean.id.au > > Mobile(Aus): [5]+61-413014395 WWW: [6]http://bjdean.id.au/ > > _______________________________________________ > > Melbourne-pm mailing list > > [7]Melbourne-pm at pm.org > > [8]http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > -- > Turning and turning in the widening gyre > The falcon cannot hear the falconer > Things fall apart; the center cannot hold > Mere anarchy is loosed upon the world -- Bradley Dean Email: [9]bjdean at bjdean.id.au Skype: [10]skype at bjdean.id.au Mobile(Aus): [11]+61-413014395 WWW: [12]http://bjdean.id.au/ _______________________________________________ Melbourne-pm mailing list [13]Melbourne-pm at pm.org [14]http://mail.pm.org/mailman/listinfo/melbourne-pm -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ _______________________________________________ Melbourne-pm mailing list Melbourne-pm at pm.org http://mail.pm.org/mailman/listinfo/melbourne-pm _______________________________________________ Melbourne-pm mailing list [15]Melbourne-pm at pm.org [16]http://mail.pm.org/mailman/listinfo/melbourne-pm -- Bradley Dean Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ References 1. mailto:bjdean at bjdean.id.au 2. mailto:bjdean at bjdean.id.au 3. mailto:bjdean at bjdean.id.au 4. mailto:skype at bjdean.id.au 5. tel:%2B61-413014395 6. http://bjdean.id.au/ 7. mailto:Melbourne-pm at pm.org 8. http://mail.pm.org/mailman/listinfo/melbourne-pm 9. mailto:bjdean at bjdean.id.au 10. mailto:skype at bjdean.id.au 11. tel:%2B61-413014395 12. http://bjdean.id.au/ 13. mailto:Melbourne-pm at pm.org 14. http://mail.pm.org/mailman/listinfo/melbourne-pm 15. mailto:Melbourne-pm at pm.org 16. http://mail.pm.org/mailman/listinfo/melbourne-pm -------------- next part -------------- An HTML attachment was scrubbed... URL: From kahlil.hodgson at dealmax.com.au Thu Oct 9 22:10:40 2014 From: kahlil.hodgson at dealmax.com.au (Kahlil Hodgson) Date: Fri, 10 Oct 2014 16:10:40 +1100 Subject: [Melbourne-pm] Finding "Out of memory!" In-Reply-To: <1412916810.3469310.177325637.7ED8B72E@webmail.messagingengine.com> References: <1412301252.2990451.174607277.2CCA28DC@webmail.messagingengine.com> <1412898410.3379103.177257149.59E28E76@webmail.messagingengine.com> <1412905871.3409191.177288093.2EF8793B@webmail.messagingengine.com> <1988991306.120737.1412906801505.JavaMail.zimbra@strategicdata.com.au> <1412916810.3469310.177325637.7ED8B72E@webmail.messagingengine.com> Message-ID: In the past, I've found, Devel::Cycle, Devel::Leak and friends useful for hunting such badness. During development, I run my Catalyst apps with the CatalystX::LeakCheaker plugin. Perhaps those may give some inspiration? K Kahlil (Kal) Hodgson GPG: C9A02289 Head of Technology (m) +61 (0) 4 2573 0382 DealMax Pty Ltd Suite 1416 401 Docklands Drive Docklands VIC 3008 Australia "All parts should go together without forcing. You must remember that the parts you are reassembling were disassembled by you. Therefore, if you can't get them together again, there must be a reason. By all means, do not use a hammer." -- IBM maintenance manual, 1925 On Fri, Oct 10, 2014 at 3:53 PM, Bradley Dean wrote: > Yep - doing that too. :) > > This idea started as "it would be handy to have a drop-in module that would > help capture memory problems occurring while they are occurring and before > they are fixed". Actually - the specific problem that started this line of > thought is now history as the memory leak has been identified and fixed. > > I wasn't suggesting that an application that leaks memory should just be > left to keep doing that - my idea was really of interest if said application > only leaks memory sometimes and it's taking a while to find why (but the > application needs to be run in the meantime). > > Cheerio, > > Brad > > > On Fri, Oct 10, 2014, at 13:06, Toby Corkindale wrote: > > When you encounter a problem that seems difficult to handle and has very > little evidence of other people handling it, then it is a good idea to take > a step back and look at the wider picture. Your overall architecture may > simply be flawed, and a change elsewhere may simplify later issues. > > In this case, I suggest you fix your design so that out of memory errors > stop occurring. > Eg. Don't load entire massive files or data structures into memory, but > instead use a specific data storage and retrieval system that is suited to > the task and can manage the paging of data to disk with automatic caching to > memory. (For instance, SQLite or LevelDB) > > > ________________________________ > > From: "Bradley Dean" > To: "Mathew Robertson" > Cc: "Melbourne Perl Mongers" > Sent: Friday, 10 October, 2014 12:51:11 PM > Subject: Re: [Melbourne-pm] Finding "Out of memory!" > > Yes indeed - a non-zero exit code but not one that tells me anything about > the reason (because I can't reliable catch the out of memory error to set a > useful exit code, and perl isn't doing that either). :) > > I'm not claiming that searching for a magic string in STDERR is a good > option, but at the moment I'm not sure there's a better one. > > Cheerio, > > Brad > > > On Fri, Oct 10, 2014, at 11:19, Mathew Robertson wrote: > > The Perl script should exit with a non-zero exit code... it doesn't really > matter why that exit code was non-zero, as any non-zero value will indicate > a failure mode. > > eg: grep returns non-zero when it didn't actually grep for anything.... (you > asked it to do something, but it couldn't). > > Tracing STDERR will be fraught with problems - how do you know some Perl > library doesn't randomly spit out messages that you have yet to see? > > > just my $0.02, > > Mathew > > > On 10 October 2014 10:46, Bradley Dean wrote: > > Yes - that's certainly a safer approach. In fact I've found that what > worked for me in my example code was version sensitive. > > On the system I started with the last system call was an mmap2 with a > ENOMEM error and this is what I found when checking $!. I then tested > this on a different system and discovered that after the mmap2 failed > (and before my perl code had a chance to look at $!) a bunch of > rt_sigaction systems calls happened and one of those failed with EINVAL > so there was no ENOMEM to find. > > I feel there's a hack with a wrapper script which greps for "Out of > memory!" on STDERR coming my way as the exit code doesn't tell me > anything useful either. > > Cheerio, > > Brad > > On Fri, Oct 10, 2014, at 10:37, Toby Wintermute wrote: >> I really don't think this approach is going to pay dividends. >> When you're hitting out of memory conditions in Perl, it's usually >> game over. In some languages you could code up something that runs >> without using any more memory, as a handler, but you don't have that >> level of control in Perl. >> >> I'd look at wrapping your Perl script in another program - maybe just >> a bash or even another Perl script - that waits for it to exit, and >> then checks to see if it exited normally or with an error code. >> >> On 3 October 2014 11:54, Bradley Dean wrote: >> > Greetings folks, >> > >> > While looking at a problem where a variety of scripts might develop a >> > memory leak and fatally terminate with "Out of memory!" I'm trying to >> > work out if this seemingly simple code is in fact too simple: >> > >> > package NotifyOnOutOfMemory; >> > use Errno; >> > END { >> > # If the last errno indicates out of memory perl will now be >> > # terminating with "Out of memory!" so tell me about it. >> > if ( $! == Errno::ENOMEM ) { >> > # Do some low-memory thing that tells me a process has died due >> > to memory exhaustion >> > # eg. touch /path/to/some/file >> > # or - exec("/path/to/script/which/emails/or/logs $$ $0 >> > $someOtherDetail") >> > } >> > } >> > 1; >> > >> > Which would give me a drop in module for scripts with this problem. >> > >> > What am I missing? :) >> > >> > Cheerio, >> > >> > Brad >> > >> > -- >> > Bradley Dean >> > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au >> > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ >> > _______________________________________________ >> > Melbourne-pm mailing list >> > Melbourne-pm at pm.org >> > http://mail.pm.org/mailman/listinfo/melbourne-pm >> >> >> >> -- >> Turning and turning in the widening gyre >> The falcon cannot hear the falconer >> Things fall apart; the center cannot hold >> Mere anarchy is loosed upon the world > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > > > -- > Bradley Dean > Email: bjdean at bjdean.id.au Skype: skype at bjdean.id.au > Mobile(Aus): +61-413014395 WWW: http://bjdean.id.au/ > > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm From alexandre.andrade at ecetera.com.au Sun Oct 19 21:27:02 2014 From: alexandre.andrade at ecetera.com.au (Alexandre Andrade) Date: Mon, 20 Oct 2014 15:27:02 +1100 Subject: [Melbourne-pm] Use of each() on hash after insertion without resetting hash interator results in undefined behavior Message-ID: Hi Folks, I'm using Perl v5.20.0 for sun4-solaris-thread-multi-64 on Solaris 11. I have a main thread that reads from disk and adds some values to a Shared Hash and a secondary thread that goes through that Hash, uses some values and delete the used values from the hash. Something like this: # This Thread will run in paralel and check the Memory Hash for completed transactions and do some tricks my $thr = threads->new( \&check_time ); while (defined(my $line=$file->read)) { my @transaction_details : shared; . . . $transactions{ $transaction->{"RequestId"} } = \@ transaction_details; } sub check_time() { while (1) { { foreach my $key ( keys(%transactions) ) { doSomeMagic( $transactions{$key} ); delete($transactions{$key}); # Abandoning the Foreach Loop as I've changed the %transactions memory map last; } sleep(1); } } -- The code works but the warning: Use of each() on hash after insertion without resetting hash interator results in undefined behavior Is flooding the logs - I've tried to set "no warnings" or create a copy of %transactions and interact with it instead of %transaction itself but no luck so far. Any ideas? Regards, Alexandre Andrade | APM Consultant | Ecetera alexandre.andrade at ecetera.com.au | M: +61 41307 1370 Helping rid the World of badly behaving Apps by investigating my Customers problems and finding solutions to make things work faster and better. Connect to Ecetera on LinkedIn , Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From dean at fragfest.com.au Sun Oct 19 21:46:36 2014 From: dean at fragfest.com.au (Dean Hamstead) Date: Mon, 20 Oct 2014 15:46:36 +1100 Subject: [Melbourne-pm] Use of each() on hash after insertion without resetting hash interator results in undefined behavior In-Reply-To: References: Message-ID: <7d75065867a10fae84f88de5ce9c53fa@fragfest.com.au> Hash keys are in "random" order, so when a new one is inserted that order changes. When iterating through a hash (via each() etc), perl marks its spot when it moves from key to key. But when a new key is added, the spot that it marked may not have the same keys before and after it as prior to the addition. Adding threads makes this more strange, as you absolutely have a race condition without locking. You are probably better off pushing on to a stack (array) and popping each one off for processing. my @stack= ({ options => 'foo' }, {}, {}, {}); while (1) { my $foo = pop @stack or last; workon($foo) } Gavin Carr recently gave a talk in parallelising in perl at Sydney PM, see http://www.openfusion.net/talks/pwp/ The MCE module may be helpful Dean On 2014-10-20 15:27, Alexandre Andrade wrote: > Hi Folks, > I'm using Perl v5.20.0 for sun4-solaris-thread-multi-64 on Solaris 11. > I have a main thread that reads from disk and adds some values to a Shared Hash and a secondary thread that goes through that Hash, uses some values and delete the used values from the hash. > Something like this: > > # This Thread will run in paralel and check the Memory Hash for completed transactions and do some tricks > my $thr = threads->new( &check_time ); > while (defined(my $line=$file->read)) { > > my @transaction_details : shared; > . > . > . > $transactions{ $transaction->{"RequestId"} } = @transaction_details; > > } > > sub check_time() { > > while (1) { > { > foreach my $key ( keys(%transactions) ) { > > doSomeMagic( $transactions{$key} ); > delete($transactions{$key}); > # Abandoning the Foreach Loop as I've changed the %transactions memory map > last; > } > sleep(1); > } > > } > > -- > The code works but the warning: > > Use of each() on hash after insertion without resetting hash interator results in undefined behavior > > Is flooding the logs - I've tried to set "no warnings" or create a copy of %transactions and interact with it instead of %transaction itself but no luck so far. > > Any ideas? > > Regards, > > Alexandre Andrade | APM Consultant | Ecetera alexandre.andrade at ecetera.com.au | M: +61 41307 1370 > > Helping rid the World of badly behaving Apps by investigating my Customers problems and finding solutions to make things work faster and better. > Connect to Ecetera on LinkedIn [2], Twitter [3] > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm [1] Links: ------ [1] http://mail.pm.org/mailman/listinfo/melbourne-pm [2] http://www.linkedin.com/company/ecetera [3] http://www.twitter.com/#!/EceteraAU -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby.corkindale at strategicdata.com.au Sun Oct 19 21:52:32 2014 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 20 Oct 2014 15:52:32 +1100 (EST) Subject: [Melbourne-pm] Use of each() on hash after insertion without resetting hash interator results in undefined behavior In-Reply-To: References: Message-ID: <816960876.229550.1413780752888.JavaMail.zimbra@strategicdata.com.au> Hi Andre, In my experience, Perl threads don't work properly. Not just my opinion either --the official stance is that their use is "strongly discouraged". To avoid the error you're seeing, I think you'd need a mutex lock around the hash access, ie. so that the foreach loop can't run at the same time as the hash insertion -- except that if you're going to do that, then your program may as well be single threaded. Consider rewriting your code to use forking with a queue if you're after performance and reliability. If you do stick with threads, then you also need to take more care around race conditions - for instance, consider what happens if a requestid is added to the hash for a second time, while the first time is being processed in doSomeMagic(). Cheers, Toby ----- Original Message ----- > From: "Alexandre Andrade" > To: melbourne-pm at pm.org > Sent: Monday, 20 October, 2014 3:27:02 PM > Subject: [Melbourne-pm] Use of each() on hash after insertion without resetting hash interator results in undefined > behavior > > Hi Folks, > I'm using Perl v5.20.0 for sun4-solaris-thread-multi-64 on Solaris 11. > I have a main thread that reads from disk and adds some values to a Shared > Hash and a secondary thread that goes through that Hash, uses some values > and delete the used values from the hash. > Something like this: > > # This Thread will run in paralel and check the Memory Hash for completed > transactions and do some tricks > my $ thr = threads -> new ( \ &check_time ); > while ( defined ( my $ line = $ file -> read )) { > my @ transaction_details : shared ; > . > . > . > $ transactions { $ transaction ->{ "RequestId" } } = \ @ transaction_details > ; > } > sub check_time () { > while ( 1 ) { > { > foreach my $ key ( keys ( % transactions ) ) { > doSomeMagic( $ transactions { $ key } ); > delete ( $ transactions { $ key }); > # Abandoning the Foreach Loop as I've changed the %transactions memory map > last ; > } > sleep(1); > } > } > > -- > The code works but the warning: > > Use of each() on hash after insertion without resetting hash interator > results in undefined behavior > > Is flooding the logs - I've tried to set "no warnings" or create a copy of > %transactions and interact with it instead of %transaction itself but no > luck so far. > > Any ideas? > > Regards, > > Alexandre Andrade | APM Consultant | Ecetera > alexandre.andrade at ecetera.com.au | M: +61 41307 1370 > > Helping rid the World of badly behaving Apps by investigating my Customers > problems and finding solutions to make things work faster and better. > Connect to Ecetera on LinkedIn , Twitter > > > > > > > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm From alexandre.andrade at ecetera.com.au Mon Oct 20 00:37:45 2014 From: alexandre.andrade at ecetera.com.au (Alexandre Andrade) Date: Mon, 20 Oct 2014 18:37:45 +1100 Subject: [Melbourne-pm] Use of each() on hash after insertion without resetting hash interator results in undefined behavior In-Reply-To: <816960876.229550.1413780752888.JavaMail.zimbra@strategicdata.com.au> References: <816960876.229550.1413780752888.JavaMail.zimbra@strategicdata.com.au> Message-ID: Dean/Toby, Thank you a lot for you replies ! Dean, very clear explanation! But In this case I need to check if the values of the Hash item is complete (the main thread writes several times to each item before I can use it) so if I pop I have to push it back in case it is incomplete - And thanks for sharing Gavin's material - a most read ! Toby, Very well observed but I already have to lock command around the Hash interactions - I've cleaned them up to keep the code clear, that was a poor choice. I have a test case where I generate sequences, consume then using this scrip and compare the output with another processes and even after 20 hours of execution I got not fails - the only issue was that warning popping for most of the time - I was using a older version of Perl and I don't remember that message showing up. As I couldn't get rid of that and I am compiling Perl from source, I've decide to just comment that line from the "hv.c" and I will let my test code run for a few days to guarantee I'm not hiding a bug. Cheers, Alex On Mon, Oct 20, 2014 at 3:52 PM, Toby Corkindale < toby.corkindale at strategicdata.com.au> wrote: > Hi Andre, > In my experience, Perl threads don't work properly. Not just my opinion > either --the official stance is that their use is "strongly discouraged". > > To avoid the error you're seeing, I think you'd need a mutex lock around > the hash access, ie. so that the foreach loop can't run at the same time as > the hash insertion -- except that if you're going to do that, then your > program may as well be single threaded. > Consider rewriting your code to use forking with a queue if you're after > performance and reliability. > If you do stick with threads, then you also need to take more care around > race conditions - for instance, consider what happens if a requestid is > added to the hash for a second time, while the first time is being > processed in doSomeMagic(). > > Cheers, > Toby > > ----- Original Message ----- > > From: "Alexandre Andrade" > > To: melbourne-pm at pm.org > > Sent: Monday, 20 October, 2014 3:27:02 PM > > Subject: [Melbourne-pm] Use of each() on hash after insertion without > resetting hash interator results in undefined > > behavior > > > > Hi Folks, > > I'm using Perl v5.20.0 for sun4-solaris-thread-multi-64 on Solaris 11. > > I have a main thread that reads from disk and adds some values to a > Shared > > Hash and a secondary thread that goes through that Hash, uses some values > > and delete the used values from the hash. > > Something like this: > > > > # This Thread will run in paralel and check the Memory Hash for completed > > transactions and do some tricks > > my $ thr = threads -> new ( \ &check_time ); > > while ( defined ( my $ line = $ file -> read )) { > > my @ transaction_details : shared ; > > . > > . > > . > > $ transactions { $ transaction ->{ "RequestId" } } = \ @ > transaction_details > > ; > > } > > sub check_time () { > > while ( 1 ) { > > { > > foreach my $ key ( keys ( % transactions ) ) { > > doSomeMagic( $ transactions { $ key } ); > > delete ( $ transactions { $ key }); > > # Abandoning the Foreach Loop as I've changed the %transactions memory > map > > last ; > > } > > sleep(1); > > } > > } > > > > -- > > The code works but the warning: > > > > Use of each() on hash after insertion without resetting hash interator > > results in undefined behavior > > > > Is flooding the logs - I've tried to set "no warnings" or create a copy > of > > %transactions and interact with it instead of %transaction itself but no > > luck so far. > > > > Any ideas? > > > > Regards, > > > > Alexandre Andrade | APM Consultant | Ecetera > > alexandre.andrade at ecetera.com.au | M: +61 41307 1370 > > > > Helping rid the World of badly behaving Apps by investigating my > Customers > > problems and finding solutions to make things work faster and better. > > Connect to Ecetera on LinkedIn , Twitter > > > > > > > > > > > > > > > > > > _______________________________________________ > > Melbourne-pm mailing list > > Melbourne-pm at pm.org > > http://mail.pm.org/mailman/listinfo/melbourne-pm > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > -- Regards, Alexandre Andrade | APM Consultant | Ecetera alexandre.andrade at ecetera.com.au | M: +61 41307 1370 Helping rid the World of badly behaving Apps by investigating my Customers problems and finding solutions to make things work faster and better. Connect to Ecetera on LinkedIn , Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: