From janine at emazing.com Mon Jul 23 09:51:36 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... Message-ID: I need help from people with Big Brains. Can anyone offer suggestions for tracking down memory leaks in perl code? I've never needed to do this before, so I don't even know where to start. TIA, Janine From sml at zfx.com Mon Jul 23 09:55:13 2001 From: sml at zfx.com (Steve Lane) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... References: Message-ID: <3B5C3AD1.CC2E536E@zfx.com> Janine wrote: > I need help from people with Big Brains. Can anyone offer suggestions > for tracking down memory leaks in perl code? I've never needed to do > this before, so I don't even know where to start. i don't have a Big Brain, but i do have Big Experience :). i'll be happy to help, this is what i'll need: - a sample script - the evidence for the memory leak - the `perl -V` output -- Steve Lane From eek at eekeek.org Mon Jul 23 10:16:18 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > I need help from people with Big Brains. Can anyone offer suggestions > for tracking down memory leaks in perl code? I've never needed to do > this before, so I don't even know where to start. you want to look at Devel::Leak. also check for self-referential data structures. $data{badmonkey} = \%data; i doubt it will be that obvious :) also, is this perchance modperl with Apache::DBI? -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Mon Jul 23 10:36:42 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: > you want to look at Devel::Leak. Thanks. :-) > also check for self-referential data structures. > > $data{badmonkey} = \%data; > > i doubt it will be that obvious :) I grepped for badmonkey and found none. So it's okay, right? ;-) I have not yet checked the code for this. Perhaps I'll do that now. > also, is this perchance modperl with Apache::DBI? Nope. It's DBI::Oracle. Janine From eek at eekeek.org Mon Jul 23 11:00:54 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > > you want to look at Devel::Leak. > > Thanks. :-) Devel::Leak is kinda mysterious and i can lend some assistance if it fails to make sense. :) > I grepped for badmonkey and found none. So it's okay, right? ;-) > I have not yet checked the code for this. Perhaps I'll do that now. basically what you're looking for is anything which keeps a reference to data outside of scope. self-referential data wont get destroyed because the reference count is too high. other variables holding refs to your data can do the same thing. > > also, is this perchance modperl with Apache::DBI? > > Nope. It's DBI::Oracle. oh good. apache::dbi has strange memory issues. i hate it so much :) -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Mon Jul 23 12:11:00 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: > Devel::Leak is kinda mysterious and i can lend some assistance if it > fails to make sense. :) The documentation is very thin. Is it really as simple as just NoteSV, CheckSV? Janine From eek at eekeek.org Mon Jul 23 12:17:44 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > The documentation is very thin. Is it really as simple as just NoteSV, > CheckSV? yes it is actually. NoteSV logs how many SVs are out there. CheckSV compares the current count of SVs to what you noted earlier and yells about differences. -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Mon Jul 23 12:40:39 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: Cool. I've asked my sysadmin to install it. :-) Have you done anything with PERL_DEBUG_MSTATS? I'm trying to find out what the memory stats look like. Janine > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Matt Cashner > Sent: Monday, July 23, 2001 1:18 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: RE: LPM: Not so quiet on the western front... > > > On Mon, 23 Jul 2001, Janine wrote: > > > The documentation is very thin. Is it really as simple as > just NoteSV, > > CheckSV? > > yes it is actually. NoteSV logs how many SVs are out there. CheckSV > compares the current count of SVs to what you noted earlier and yells > about differences. > > -------- > Matt Cashner > Web Applications Developer > The Creative Group (http://www.cre8tivegroup.com) > eek@eekeek.org | Codito, ergo sum > > > > From eek at eekeek.org Mon Jul 23 12:50:12 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > Cool. I've asked my sysadmin to install it. :-) > > Have you done anything with PERL_DEBUG_MSTATS? I'm trying to find out > what the memory stats look like. cant say i have tho from the sound of things you might need a -DDEBUGGING build of perl which your box is not likely to have. lets hope you dont need that, eh? :) -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Mon Jul 23 13:04:01 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: Yeah, debugging and some malloc thinger too. John (our sysadmin) is going to bump my name to the top of his poop list by the end of the day, I'm sure. ;-) > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Matt Cashner > Sent: Monday, July 23, 2001 1:50 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: RE: LPM: Not so quiet on the western front... > > > On Mon, 23 Jul 2001, Janine wrote: > > > Cool. I've asked my sysadmin to install it. :-) > > > > Have you done anything with PERL_DEBUG_MSTATS? I'm trying > to find out > > what the memory stats look like. > > cant say i have tho from the sound of things you might need a > -DDEBUGGING build of perl which your box is not likely to have. lets > hope you dont need that, eh? :) > > -------- > Matt Cashner > Web Applications Developer > The Creative Group (http://www.cre8tivegroup.com) > eek@eekeek.org | Codito, ergo sum > > > > From eek at eekeek.org Mon Jul 23 13:20:28 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:07 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > Yeah, debugging and some malloc thinger too. John (our sysadmin) is > going to bump my name to the top of his poop list by the end of the day, > I'm sure. ;-) hunting down big memory leaks in perl is not much fun. mainly because perl leaks in a lot of places and it may not be yoru code that's at fault. its one of the most frustrating things about working with perl. OH. are you using eval anywhere? certain scenarios of eval leak like my head. -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Mon Jul 23 15:16:26 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: Nope, no evals. Here's the scoop: There's a huge select sorted by three fields. The values in these three fields are affectionately known as a Big3. We fetch rows, joining fields with a tab and pushing the resulting string onto an array, until the Big3 changes. At that point, we do "stuff." Then we undef the array and start fetching again. There are about 900 different Big3s. undef @bigarray is not freeing up memory as I expect. Watching the program's memory usage with top, it slowly creeps up and up and up - even after @bigarray is undef'd. Right now I'm looking at the "stuff". Something in there must be holding on to elements in @bigarray. *sigh* Janine > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Matt Cashner > Sent: Monday, July 23, 2001 2:20 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: RE: LPM: Not so quiet on the western front... > > > On Mon, 23 Jul 2001, Janine wrote: > > > Yeah, debugging and some malloc thinger too. John (our sysadmin) is > > going to bump my name to the top of his poop list by the > end of the day, > > I'm sure. ;-) > > hunting down big memory leaks in perl is not much fun. mainly because > perl leaks in a lot of places and it may not be yoru code that's at > fault. its one of the most frustrating things about working with perl. > > OH. are you using eval anywhere? certain scenarios of eval > leak like my > head. > > -------- > Matt Cashner > Web Applications Developer > The Creative Group (http://www.cre8tivegroup.com) > eek@eekeek.org | Codito, ergo sum > > > From eek at eekeek.org Mon Jul 23 15:28:02 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Mon, 23 Jul 2001, Janine wrote: > undef @bigarray is not freeing up memory as I expect. Watching the > program's memory usage with top, it slowly creeps up and up and up - > even after @bigarray is undef'd. > > Right now I'm looking at the "stuff". Something in there must be > holding on to elements in @bigarray. ouch. i've been there. you might try putting the routine in a block of its own. that should add extra scoping and yet another place for that array to get cleared. -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum From janine at emazing.com Tue Jul 24 15:14:02 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: Our standard Perl distribution (present on all servers) is 5.005_03. Yesterday the sysadmin installed 5.6 on a test box for us. The program is running now, and its memory usage is under control. Under 5.005 it creeps up to 600mb and over, but it's holding steady at 150mb under 5.6. That's not the kind of resolution that makes you smarter, but it gets the work done. *sigh* > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Matt Cashner > Sent: Monday, July 23, 2001 4:28 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: RE: LPM: Not so quiet on the western front... > > > On Mon, 23 Jul 2001, Janine wrote: > > > > undef @bigarray is not freeing up memory as I expect. Watching the > > program's memory usage with top, it slowly creeps up and up and up - > > even after @bigarray is undef'd. > > > > Right now I'm looking at the "stuff". Something in there must be > > holding on to elements in @bigarray. > > ouch. i've been there. you might try putting the routine in a block of > its own. that should add extra scoping and yet another place for that > array to get cleared. > > -------- > Matt Cashner > Web Applications Developer > The Creative Group (http://www.cre8tivegroup.com) > eek@eekeek.org | Codito, ergo sum > > > From eek at eekeek.org Tue Jul 24 15:21:58 2001 From: eek at eekeek.org (Matt Cashner) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: On Tue, 24 Jul 2001, Janine wrote: > Our standard Perl distribution (present on all servers) is 5.005_03. > Yesterday the sysadmin installed 5.6 on a test box for us. The program > is running now, and its memory usage is under control. Under 5.005 it > creeps up to 600mb and over, but it's holding steady at 150mb under 5.6. > > That's not the kind of resolution that makes you smarter, but it gets > the work done. yeah, a few of the p5p'ers have been beating on perl with Purify and other leak finders for a while now. 5.8 should be even better. (and perl6 will be perfect, right? :) and hey, rest assured that the problem took the might and thunder of people way smarter than the lot of us to solve. also, rest assured knowing that it wasnt your code that was the problem :) -------- Matt Cashner Web Applications Developer The Creative Group (http://www.cre8tivegroup.com) eek@eekeek.org | Codito, ergo sum