From mathew.blair.robertson at gmail.com Wed Jan 18 17:24:46 2017 From: mathew.blair.robertson at gmail.com (Mathew Robertson) Date: Thu, 19 Jan 2017 12:24:46 +1100 Subject: [Melbourne-pm] Perl forking & Copy-on-write Message-ID: Here is an interesting read on how Python's garbage-collector, causes Linux's copy-on-write, to become less effective than would otherwise: https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172#.25rzyh6im Essentially it amounts to the GC walking over Python's read-only variables, but still adjusting the reference-counters... thus causing Linux to see that the memory that is backing the Python instance, to be written to. Python's cycle-detection could also cause memory-writes due to the mark+sweep job. And it also has a generational collection, for short-term vs long-term objects. Adjusting lifetime lengths, would also cause a memory-write. Perl also uses reference counting. So I can also see that Perl's reference-counting would cause the references to be written, when a given Perl variable is copied. The question is... does anyone have any insight into the same CoW failings within the context of Perl 5 ? Perl 6 uses a generational collector (does it *also* use reference counting?), so the generational migration would impact CoW. But given the rather smart people working on that project, I can envisage that some solution may eventually get implemented. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dean at fragfest.com.au Wed Jan 18 22:03:37 2017 From: dean at fragfest.com.au (Dean Hamstead) Date: Thu, 19 Jan 2017 17:03:37 +1100 Subject: [Melbourne-pm] Perl forking & Copy-on-write In-Reply-To: References: Message-ID: perhaps TonyC on #australia (irc.perl.org) can provide some feedback he is seemingly perpetually working on tpf grants fixing core bugs D On 19/01/17 12:24, Mathew Robertson wrote: > Here is an interesting read on how Python's garbage-collector, causes > Linux's copy-on-write, to become less effective than would otherwise: > > https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172#.25rzyh6im > > Essentially it amounts to the GC walking over Python's read-only > variables, but still adjusting the reference-counters... thus causing > Linux to see that the memory that is backing the Python instance, to > be written to. > > Python's cycle-detection could also cause memory-writes due to the > mark+sweep job. > > And it also has a generational collection, for short-term vs long-term > objects. Adjusting lifetime lengths, would also cause a memory-write. > > > Perl also uses reference counting. So I can also see that Perl's > reference-counting would cause the references to be written, when a > given Perl variable is copied. > > > The question is... does anyone have any insight into the same CoW > failings within the context of Perl 5 ? > > > Perl 6 uses a generational collector (does it *also* use reference > counting?), so the generational migration would impact CoW. But given > the rather smart people working on that project, I can envisage that > some solution may eventually get implemented. > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaoru at slackwise.net Thu Jan 19 00:23:49 2017 From: kaoru at slackwise.net (Alex Balhatchet) Date: Thu, 19 Jan 2017 08:23:49 +0000 Subject: [Melbourne-pm] Perl forking & Copy-on-write In-Reply-To: References: Message-ID: I've had very good experiences using Linux Copy on Write semantics in Perl programs, both in the context of Apache 2.4 + mod_perl2 and Perl scripts using Parallel::ForkManager. The key is to set things up in the parent process - load modules and config files, pre-warm in-process caches or whatever. Then do not touch those variables again after you fork. Watch out for writes that look like reads, eg. hash key autovivification! Make good use of exists() In Apache you can use the PerlPostConfigHandler config option to run things pre-fork in the parent.[0] As an aside you can use PerlChildInitHandler for running things post-fork in the child - I recommend running 'srand' at the very least.[1] One of the most useful things you can do is to learn how different ops tools treat CoW. Unfortunately the answer is basically that they all[2] ignore CoW and will show all your child processes as using lots of memory. The most reliable source of truth is /proc//smaps which has {Shared/Private}/{Clean/Dirty} values.[4] One final point - I specified at the top of the email that I was talking about "Linux Copy on Write". That's because Perl has its own concept of Copy on Write[5] which is used to optimise assignments which are semantically copies but might not need to be. sub foo { my $arg = $_[0]; return $arg + 4; } @_ contains aliases, and $arg = $_[0] makes a new scalar containing a "copy" of the contents. But actually Perl doesn't copy immediately, it sets up a CoW variable which avoids actually duplicating the memory until it absolutely has to. Hope that helps :) I also hope it's all accurate... if anybody notices any errors please correct me :) - Alex [1] https://perl.apache.org/docs/2.0/user/handlers/server.html [2] http://blogs.perl.org/users/brian_phillips/2010/06/when-rand-isnt-random.html [3] top, free, etc. [4] https://www.brightbox.com/blog/2012/11/28/measuring-shared-ram-usage/ [5] http://perldoc.perl.org/perlguts.html#Copy-on-Write On 19 January 2017 at 06:03, Dean Hamstead wrote: > perhaps TonyC on #australia (irc.perl.org) can provide some feedback > he is seemingly perpetually working on tpf grants fixing core bugs > > > D > > > On 19/01/17 12:24, Mathew Robertson wrote: > > Here is an interesting read on how Python's garbage-collector, causes > Linux's copy-on-write, to become less effective than would otherwise: > > https://engineering.instagram.com/dismissing-python-garbage- > collection-at-instagram-4dca40b29172#.25rzyh6im > > Essentially it amounts to the GC walking over Python's read-only > variables, but still adjusting the reference-counters... thus causing Linux > to see that the memory that is backing the Python instance, to be written > to. > > Python's cycle-detection could also cause memory-writes due to the > mark+sweep job. > > And it also has a generational collection, for short-term vs long-term > objects. Adjusting lifetime lengths, would also cause a memory-write. > > > Perl also uses reference counting. So I can also see that Perl's > reference-counting would cause the references to be written, when a given > Perl variable is copied. > > > The question is... does anyone have any insight into the same CoW failings > within the context of Perl 5 ? > > > Perl 6 uses a generational collector (does it *also* use reference > counting?), so the generational migration would impact CoW. But given the > rather smart people working on that project, I can envisage that some > solution may eventually get implemented. > > > _______________________________________________ > Melbourne-pm mailing listMelbourne-pm at pm.orghttp://mail.pm.org/mailman/listinfo/melbourne-pm > > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: