From oneiros at dcr.net Wed Aug 15 14:02:53 2001 From: oneiros at dcr.net (Joe Hourcle) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: Message-ID: <20010815145830.R95647-100000@andrew.dcr.net> It's a bit old, but well, I haven't been keeping up with this list, and it seems to be an unresolved issue. On Mon, 23 Jul 2001, Janine wrote: > Nope, no evals. > > Here's the scoop: > > There's a huge select sorted by three fields. The values in these three > fields are affectionately known as a Big3. We fetch rows, joining > fields with a tab and pushing the resulting string onto an array, until > the Big3 changes. At that point, we do "stuff." Then we undef the > array and start fetching again. There are about 900 different Big3s. > > undef @bigarray is not freeing up memory as I expect. Watching the > program's memory usage with top, it slowly creeps up and up and up - > even after @bigarray is undef'd. > > Right now I'm looking at the "stuff". Something in there must be > holding on to elements in @bigarray. Something similar went by the dc-pm list in June: >===== Original Message From dc@lists.pm.org ===== > This was unexpected. In the map method, I use the following line to > declare my hashref to use to hold db fetch values: > > my $href = undef; > > > even though I exit this method after each line and therefore $href should > be automatically destroyed by GC, it's not. However, changing that line > to > > my $href = (); > > fixed the problem. No idea what is going on here, though. > > --Curt So, I don't know if it's absolutely similar, but you might try setting your array to an empty list, instead of using undef, and see if that still sucks down the same amount of memory. -Joe From janine at emazing.com Wed Aug 15 14:20:34 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: Not so quiet on the western front... In-Reply-To: <20010815145830.R95647-100000@andrew.dcr.net> Message-ID: I tried undef and assigning to an empty list but the script behaved the same in both cases (that is, kept consuming more and more memory until the sysadmin's pager started to buzz :-). The resolution was to upgrade to version 5.6 from 5.003. The memory leak disappeared under the more recent version of Perl. Not a very satisfying answer, but an answer nonetheless. :-\ Janine > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Joe Hourcle > Sent: Wednesday, August 15, 2001 3:03 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: RE: LPM: Not so quiet on the western front... > > > > It's a bit old, but well, I haven't been keeping up with this > list, and it > seems to be an unresolved issue. > > > On Mon, 23 Jul 2001, Janine wrote: > > > Nope, no evals. > > > > Here's the scoop: > > > > There's a huge select sorted by three fields. The values > in these three > > fields are affectionately known as a Big3. We fetch rows, joining > > fields with a tab and pushing the resulting string onto an > array, until > > the Big3 changes. At that point, we do "stuff." Then we undef the > > array and start fetching again. There are about 900 > different Big3s. > > > > undef @bigarray is not freeing up memory as I expect. Watching the > > program's memory usage with top, it slowly creeps up and up and up - > > even after @bigarray is undef'd. > > > > Right now I'm looking at the "stuff". Something in there must be > > holding on to elements in @bigarray. > > Something similar went by the dc-pm list in June: > > > >===== Original Message From dc@lists.pm.org ===== > > This was unexpected. In the map method, I use the following line to > > declare my hashref to use to hold db fetch values: > > > > my $href = undef; > > > > > > even though I exit this method after each line and > therefore $href should > > be automatically destroyed by GC, it's not. However, > changing that line > > to > > > > my $href = (); > > > > fixed the problem. No idea what is going on here, though. > > > > --Curt > > > So, I don't know if it's absolutely similar, but you might try setting > your array to an empty list, instead of using undef, and see > if that still > sucks down the same amount of memory. > > -Joe > > From hempy at ket.org Tue Aug 21 15:39:25 2001 From: hempy at ket.org (David Hempy) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) Message-ID: <5.0.0.25.0.20010821154814.01fa8780@mail.ket.org> Well, since Y2K was such a fizzle, we get another chance at notoriety on September 8, 2001, when we exceed 999,999,999 seconds since the epoch. Shouldn't be a big deal, as that's only one less than 1,000,000,000 seconds. But yet, I'm facing it in one of my programs... I got a call from a user complaining that September 8th's program schedule was out of order. Everything after 9:30 PM (EDT) showed up first, before our first show at 8:00 AM. Sure enough, she was right: http://edit-www.ket.org/cgi-plex/schedule/ribbon_funnytime.pl?date=2001-09-08 It didn't take too long to figure notice the coincidence between the sorting problem and the 9+1 issue. After convincing myself that SQL Server didn't know how to sort these values correctly (duh) I looked at my own code. Turns out the problem was actually me not understanding how hash keys work. I was using epoch times as hash keys. It appears that hash keys are stored as strings, not the integers I was expecting. For example: $foo{1} = "one"; $foo{2} = "two"; $foo{3} = "three"; $foo{10} = "ten"; $foo{20} = "twenty"; $foo{30} = "thirty"; foreach my $key (sort keys %foo) { print "$key - $foo{$key}\n"; } This produces this output: 1 - one 10 - ten 2 - two 20 - twenty 3 - three 30 - thirty Any suggestions on a good way to sort hash keys by their integer values? -dave -- David Hempy Internet Database Administrator Kentucky Educational Television - Distance Learning Division -- (859)258-7164 -- (800)333-9764 From gcasillo at ket.org Tue Aug 21 16:07:54 2001 From: gcasillo at ket.org (Gregg Casillo) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) References: <5.0.0.25.0.20010821154814.01fa8780@mail.ket.org> Message-ID: <3B82CDAA.6030205@ket.org> My first (and presently only) idea would be to zero-pad those numbers so that they are a uniform length and thus will sort correctly. So if you have numbers like 3, 9, 10, 22, and 105, zero-pad them: 003, 009, 010, 022, and 105. Use sprintf(): $padded = sprintf("%03d", 23); # $padded = 023; I don't know how feasible it would be to go back and zero-pad all of your numbers in question though. Interesting, Gregg David Hempy wrote: > > Well, since Y2K was such a fizzle, we get another chance at notoriety > on September 8, 2001, when we exceed 999,999,999 seconds since the > epoch. Shouldn't be a big deal, as that's only one less than > 1,000,000,000 seconds. > > But yet, I'm facing it in one of my programs... > > I got a call from a user complaining that September 8th's program > schedule was out of order. Everything after 9:30 PM (EDT) showed up > first, before our first show at 8:00 AM. Sure enough, she was right: > > http://edit-www.ket.org/cgi-plex/schedule/ribbon_funnytime.pl?date=2001-09-08 > > > > It didn't take too long to figure notice the coincidence between the > sorting problem and the 9+1 issue. After convincing myself that SQL > Server didn't know how to sort these values correctly (duh) I looked > at my own code. > > Turns out the problem was actually me not understanding how hash keys > work. I was using epoch times as hash keys. It appears that hash > keys are stored as strings, not the integers I was expecting. For > example: > > $foo{1} = "one"; > $foo{2} = "two"; > $foo{3} = "three"; > $foo{10} = "ten"; > $foo{20} = "twenty"; > $foo{30} = "thirty"; > > foreach my $key (sort keys %foo) { > print "$key - $foo{$key}\n"; > } > > > This produces this output: > 1 - one > 10 - ten > 2 - two > 20 - twenty > 3 - three > 30 - thirty > > > Any suggestions on a good way to sort hash keys by their integer values? > > > -dave > From pstackhouse at ket.org Tue Aug 21 16:14:36 2001 From: pstackhouse at ket.org (Paul Stackhouse) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) References: <5.0.0.25.0.20010821154814.01fa8780@mail.ket.org> Message-ID: <3B82CF3C.C3A6C1B8@ket.org> It appears to only be a problem on the exact day of the cross-over. Once you get to the ninth, when all of the entries are ten digits, everything sorts properly again, right? I'm still curious, though, about the answer to your question. -paul David Hempy wrote: > > > Well, since Y2K was such a fizzle, we get another chance at notoriety on > September 8, 2001, when we exceed 999,999,999 seconds since the > epoch. Shouldn't be a big deal, as that's only one less than 1,000,000,000 > seconds. > > But yet, I'm facing it in one of my programs... > > I got a call from a user complaining that September 8th's program schedule > was out of order. Everything after 9:30 PM (EDT) showed up first, before > our first show at 8:00 AM. Sure enough, she was right: > > http://edit-www.ket.org/cgi-plex/schedule/ribbon_funnytime.pl?date=2001-09-08 > > It didn't take too long to figure notice the coincidence between the > sorting problem and the 9+1 issue. After convincing myself that SQL Server > didn't know how to sort these values correctly (duh) I looked at my own code. > > Turns out the problem was actually me not understanding how hash keys > work. I was using epoch times as hash keys. It appears that hash keys are > stored as strings, not the integers I was expecting. For example: > > $foo{1} = "one"; > $foo{2} = "two"; > $foo{3} = "three"; > $foo{10} = "ten"; > $foo{20} = "twenty"; > $foo{30} = "thirty"; > > foreach my $key (sort keys %foo) { > print "$key - $foo{$key}\n"; > } > > > This produces this output: > 1 - one > 10 - ten > 2 - two > 20 - twenty > 3 - three > 30 - thirty > > Any suggestions on a good way to sort hash keys by their integer values? > > -dave > > -- > David Hempy > Internet Database Administrator > Kentucky Educational Television - Distance Learning Division > -- (859)258-7164 -- (800)333-9764 -- Paul Stackhouse Webmaster Kentucky Educational TV Voice: 859-258-7135 600 Cooper Drive Fax: 859-258-7399 Lexington, KY 40502 http://www.ket.org From janine at emazing.com Tue Aug 21 16:26:00 2001 From: janine at emazing.com (Janine) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) In-Reply-To: <3B82CF3C.C3A6C1B8@ket.org> Message-ID: What about this: #!/usr/bin/perl -w use strict; my %nums = ( 1 => 'one', 2 => 'two', 10 => 'ten', 20 => 'twenty' ); print "The old way...\n"; foreach my $num (sort keys %nums) { print "$num ($nums{$num})\n"; } print "\n\n"; print "The new way...\n"; #my @sorted = sort ( map { sprintf("%012u",$_) } (keys %nums) ); #my @sorted = sort @mapped; foreach my $num (sort ( map { sprintf("%012u",$_) } (keys %nums) )) { my $orig = int $num; print "$num ($nums{$orig})\n"; } Janine > -----Original Message----- > From: owner-lexington-pm-list@pm.org > [mailto:owner-lexington-pm-list@pm.org]On Behalf Of Paul Stackhouse > Sent: Tuesday, August 21, 2001 5:15 PM > To: lexington-pm-list@happyfunball.pm.org > Subject: Re: LPM: The times they are a-changin' (or: integer > hash keys) > > > It appears to only be a problem on the exact day of the > cross-over. Once you > get to the ninth, when all of the entries are ten digits, > everything sorts > properly again, right? I'm still curious, though, about the > answer to your > question. > > -paul > > David Hempy wrote: > > > > > > Well, since Y2K was such a fizzle, we get another chance at > notoriety on > > September 8, 2001, when we exceed 999,999,999 seconds since the > > epoch. Shouldn't be a big deal, as that's only one less > than 1,000,000,000 > > seconds. > > > > But yet, I'm facing it in one of my programs... > > > > I got a call from a user complaining that September 8th's > program schedule > > was out of order. Everything after 9:30 PM (EDT) showed up > first, before > > our first show at 8:00 AM. Sure enough, she was right: > > > > http://edit-www.ket.org/cgi-plex/schedule/ribbon_funnytime.pl?date=2001- 09-08 > > It didn't take too long to figure notice the coincidence between the > sorting problem and the 9+1 issue. After convincing myself that SQL Server > didn't know how to sort these values correctly (duh) I looked at my own code. > > Turns out the problem was actually me not understanding how hash keys > work. I was using epoch times as hash keys. It appears that hash keys are > stored as strings, not the integers I was expecting. For example: > > $foo{1} = "one"; > $foo{2} = "two"; > $foo{3} = "three"; > $foo{10} = "ten"; > $foo{20} = "twenty"; > $foo{30} = "thirty"; > > foreach my $key (sort keys %foo) { > print "$key - $foo{$key}\n"; > } > > > This produces this output: > 1 - one > 10 - ten > 2 - two > 20 - twenty > 3 - three > 30 - thirty > > Any suggestions on a good way to sort hash keys by their integer values? > > -dave > > -- > David Hempy > Internet Database Administrator > Kentucky Educational Television - Distance Learning Division > -- (859)258-7164 -- (800)333-9764 -- Paul Stackhouse Webmaster Kentucky Educational TV Voice: 859-258-7135 600 Cooper Drive Fax: 859-258-7399 Lexington, KY 40502 http://www.ket.org From grdodson at lexmark.com Tue Aug 21 17:48:49 2001 From: grdodson at lexmark.com (Graydon Dodson) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) Message-ID: <200108212248.SAA07631@interlock2.lexmark.com> I am confused as to why this is "hard". This is a simple numeric sort from the old "pink camel book" days. The following works just fine for me: ---------------------------------------------------- #!/usr/bin/perl -w use strict; my %nums = ( 1 => 'one', 2 => 'two', 10 => 'ten', 20 => 'twenty' ); sub numericaly { $a <=> $b }; print "The numeric way...\n"; foreach my $num (sort numericaly keys %nums) { print "$num ($nums{$num})\n"; } ----------------------------------------------------- Graydon Dodson grdodson@lexmark.com Lexmark International Inc. From oneiros at dcr.net Thu Aug 23 10:35:21 2001 From: oneiros at dcr.net (Joe Hourcle) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) In-Reply-To: <3B82CDAA.6030205@ket.org> Message-ID: <20010823111415.K27259-100000@andrew.dcr.net> On Tue, 21 Aug 2001, Gregg Casillo wrote: > My first (and presently only) idea would be to zero-pad those numbers so > that they are a uniform length and thus will sort correctly. So if you > have numbers like 3, 9, 10, 22, and 105, zero-pad them: 003, 009, 010, > 022, and 105. Use sprintf(): > > $padded = sprintf("%03d", 23); # $padded = 023; > > I don't know how feasible it would be to go back and zero-pad all of > your numbers in question though. Assuming you're sure they're not going to be read as numbers, and only as strings, otherwise, you have issues with programs that think they're octal, but find 8/9 in there. > > Any suggestions on a good way to sort hash keys by their integer values? ##### my @array = qw(1 2 4 5 6 10 11 12 20 24 2000); # default print join(' ', sort @array), "\n"; # alpha print join(' ', sort { $a cmp $b } @array), "\n"; # number print join(' ', sort { $a <=> $b } @array), "\n"; # force to number, number sort print join(' ', sort { ($a+0) <=> ($b+0) } @array), "\n"; ##### I get the following output: 1 10 11 12 2 20 2000 24 4 5 6 1 10 11 12 2 20 2000 24 4 5 6 1 2 4 5 6 10 11 12 20 24 2000 1 2 4 5 6 10 11 12 20 24 2000 -Joe From oneiros at dcr.net Thu Aug 23 10:51:03 2001 From: oneiros at dcr.net (Joe Hourcle) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) In-Reply-To: <20010823111415.K27259-100000@andrew.dcr.net> Message-ID: <20010823113640.T27259-100000@andrew.dcr.net> On Thu, 23 Aug 2001, Joe Hourcle wrote: > print join(' ', sort @array), "\n"; > print join(' ', sort { $a cmp $b } @array), "\n"; > print join(' ', sort { $a <=> $b } @array), "\n"; > print join(' ', sort { ($a+0) <=> ($b+0) } @array), "\n"; A few other notes about this syntax... it can also come in handy for other abnormal sorting eg: ##### my %HASH = ( one => { sort => 1 }, two => { sort => 2 }, fourteen => { sort => 14 }, six => { sort => 6 }, neg_two => { sort => -2 } ); print join (' ', sort { $HASH{$a}{sort} <=> $HASH{$b}{sort} } keys %HASH), "\n"; ##### or, for reporting purposes: ##### our %PEOPLE = ( Bob => { gender => 'M', age => 42 }, Dave => { gender => 'M', age => 30 }, Debbie => { gender => 'F', age => 25 }, Sue => { gender => 'F', age => 36 }, Al => { gender => 'M', age => 35 } ); sub decend_by_age { $PEOPLE{$b}{age} <=> $PEOPLE{$a}{age}; } print "Age Sorted Decending\n"; foreach my $person (sort decend_by_age keys %PEOPLE) { print "$person\t$PEOPLE{$person}{age}\t$PEOPLE{$person}{gender}\n"; } ##### -Joe From hempy at ket.org Thu Aug 23 16:33:55 2001 From: hempy at ket.org (David Hempy) Date: Thu Aug 5 00:06:08 2004 Subject: LPM: The times they are a-changin' (or: integer hash keys) In-Reply-To: <20010823111415.K27259-100000@andrew.dcr.net> References: <3B82CDAA.6030205@ket.org> Message-ID: <5.0.0.25.0.20010823173302.0394eec0@mail.ket.org> At 11:35 AM 8/23/2001 -0400, you wrote: >print join(' ', sort { $a <=> $b } @array), "\n"; > >-Joe Perfect! Thanks for the courteous reply, Joe! -dave :-) -- David Hempy Internet Database Administrator Kentucky Educational Television - Distance Learning Division -- (859)258-7164 -- (800)333-9764