From jarich at perltraining.com.au Thu May 6 23:49:18 2010 From: jarich at perltraining.com.au (Jacinta Richardson) Date: Fri, 07 May 2010 16:49:18 +1000 Subject: [Melbourne-pm] Next meeting: Wednesday 12th May 2010 Message-ID: <4BE3B7EE.3000900@perltraining.com.au> G'day folk, The next Melbourne Perl Mongers meeting will be hosted by Multiservice (courtesy of Hamish Carpenter): Multiservice Level 12 10-16 Queen Street Melbourne 3000 next week on Wednesday 12th May at 6:30pm. Doors lock at 6, so look out for a note on the door as to what number to ring to be let in. If you are one of the lucky few who haven't given in to having a mobile phone yet, try to get there earlier enough to get let in with other people who do have phones. :) Talks.... we don't have any yet. What would you like to talk about? Please send me or the list any talks you might be interested in giving and we can schedule them in. 20-40 minutes is a good limit. Failing all else, perhaps we could have a round table discussion about cool Perl/programming things people have done in the last few months. I won't be at the meeting (I'll be in Sydney again), but I will endeavour to send along some more camel biscuits. J PS: Suggestions for food/drinking afterwards are welcome. The Bond Lounge Bar on Ryrie Lane could be promising, but otherwise I have no suggestions myself. -- ("`-''-/").___..--''"`-._ | Jacinta Richardson | `6_ 6 ) `-. ( ).`-.__.`) | Perl Training Australia | (_Y_.)' ._ ) `._ `. ``-..-' | +61 3 9354 6001 | _..`--'_..-_/ /--'_.' ,' | contact at perltraining.com.au | (il),-'' (li),' ((!.-' | www.perltraining.com.au | From tjc at wintrmute.net Sun May 9 19:03:17 2010 From: tjc at wintrmute.net (Toby Wintermute) Date: Mon, 10 May 2010 12:03:17 +1000 Subject: [Melbourne-pm] Arcane operators Message-ID: Just something I felt like passing along.. (Background: I've been playing with Scala every now and then, as I've mentioned before. It has some nice advantages, like a great concurrency system, a good OO implementation, and it's quite fast once the jvm gets warmed up.) I've noticed a worrying trend in Scala libraries though. In Scala you can invent your own operators - they're just functions on the objects with those names. And it seems that Unicode is valid too. In a couple of actually-popular libraries I have seen the following operators. I had to copy-and-paste some of them because I don't know the right X Compose sequence to input them! 1) /: (was used to create a URL object from a string) 2) #:: (I couldn't work out what this did from the context) 3) >:> (Described as "Process Header as map in block") 4) ? (Couldn't work out what this did from context either) 5) ? (Also unsure.) 6) >| (Described as "Ignore response body") There are some cases where symbols seem to work well, such as: my_list_of_things.each( thing => { doSomething(thing) or somethingElse(thing) } ) or to build a map: var userIDs = Map(1 -> "root", 2 -> "daemon", 5 -> "games", 8 -> "mail", 9 -> "news") but for the most part, it feels like it's going a bit crazy. In Perl it seems like the received wisdom is to back away from writing code that looks like random line noise. If you could make up any operators you like, would you? Do you think it improves code readability or makes it worse? Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world From daniel at rimspace.net Sun May 9 19:52:26 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Mon, 10 May 2010 12:52:26 +1000 Subject: [Melbourne-pm] Arcane operators In-Reply-To: (Toby Wintermute's message of "Mon, 10 May 2010 12:03:17 +1000") References: Message-ID: <87hbmgjwnp.fsf@rimspace.net> Toby Wintermute writes: > (Background: > I've been playing with Scala every now and then, as I've mentioned before. > It has some nice advantages, like a great concurrency system, a good > OO implementation, and it's quite fast once the jvm gets warmed up.) > > I've noticed a worrying trend in Scala libraries though. > In Scala you can invent your own operators - they're just functions on > the objects with those names. And it seems that Unicode is valid too. Yay! APL and A+! How can you go past a language that supports things like: x[?x?6?40] ...and the kids today complain that Perl and Emacs commands looks like line noise. They don't know how good they have it. [...] > but for the most part, it feels like it's going a bit crazy. It probably is: the issue you mention is actually both less and more bad than the equivalent problem back in C++ when I worked there: back then you had a much more limited set of operators to overload, but the same enthusiasm for overloading them. The broader character set means that '*' is less likely to do something completely unexpected like "map", but you are more likely to see someone write APL style code. > In Perl it seems like the received wisdom is to back away from writing > code that looks like random line noise. APL is pretty much the extreme end of the Domain Specific Language spectrum, which is what operator overloading is a small step along the path to. Done right, this still is gold: you can't beat it. It is still the killer advantage of Lisp, that it is so trivial to define a useful DSL, so you tend to do it all along the way. OTOH, done poorly DSL code is terrible. Probably worse than, say, some of the "I don't understand OO and find Perl5 scary" Perl that I get to support. > If you could make up any operators you like, would you? Yes, absolutely. I also 'use overload' in Perl. > Do you think it improves code readability or makes it worse? There isn't a single answer. Naturally, when *I* do it I think it makes code readability better, but opinions vary about how that actually looks. :) Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From toby.corkindale at strategicdata.com.au Mon May 10 01:30:34 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 10 May 2010 18:30:34 +1000 Subject: [Melbourne-pm] Arcane operators In-Reply-To: <87hbmgjwnp.fsf@rimspace.net> References: <87hbmgjwnp.fsf@rimspace.net> Message-ID: <4BE7C42A.1090500@strategicdata.com.au> On 10/05/10 12:52, Daniel Pittman wrote: > Toby Wintermute writes: > >> (Background: >> I've been playing with Scala every now and then, as I've mentioned before. >> It has some nice advantages, like a great concurrency system, a good >> OO implementation, and it's quite fast once the jvm gets warmed up.) >> >> I've noticed a worrying trend in Scala libraries though. >> In Scala you can invent your own operators - they're just functions on >> the objects with those names. And it seems that Unicode is valid too. > > Yay! APL and A+! How can you go past a language that supports things like: > > x[?x?6?40] > > ...and the kids today complain that Perl and Emacs commands looks like line > noise. They don't know how good they have it. > > [...] > >> but for the most part, it feels like it's going a bit crazy. > > It probably is: the issue you mention is actually both less and more bad than > the equivalent problem back in C++ when I worked there: back then you had a > much more limited set of operators to overload, but the same enthusiasm for > overloading them. > > The broader character set means that '*' is less likely to do something > completely unexpected like "map", but you are more likely to see someone write > APL style code. > > >> In Perl it seems like the received wisdom is to back away from writing >> code that looks like random line noise. > > APL is pretty much the extreme end of the Domain Specific Language spectrum, > which is what operator overloading is a small step along the path to. > > Done right, this still is gold: you can't beat it. It is still the killer > advantage of Lisp, that it is so trivial to define a useful DSL, so you tend > to do it all along the way. > > OTOH, done poorly DSL code is terrible. Probably worse than, say, some of the > "I don't understand OO and find Perl5 scary" Perl that I get to support. > >> If you could make up any operators you like, would you? > > Yes, absolutely. I also 'use overload' in Perl. So do I - but I'm overloading them within the original meaning. ie. + is adding or appending, - is reducing, " is making a printable response, etc.. I am now tempted to create a library for Scala that entirely consists of smiley expressions, such as the sadface operator: a :( b the snowman operator: c ? d the wince operator: e >.< f the mind boggles operator: g o.O h and possibly the nethack operator: a ?..{.d.. at .]K### b hell, with enough custom operators, I could make a DSL for Nethack games. ? ( @ o.O @[Shopkeeper] ) { d => # } -Toby From daniel at rimspace.net Mon May 10 01:43:03 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Mon, 10 May 2010 18:43:03 +1000 Subject: [Melbourne-pm] Arcane operators In-Reply-To: <4BE7C42A.1090500@strategicdata.com.au> (Toby Corkindale's message of "Mon, 10 May 2010 18:30:34 +1000") References: <87hbmgjwnp.fsf@rimspace.net> <4BE7C42A.1090500@strategicdata.com.au> Message-ID: <87sk60du5k.fsf@rimspace.net> Toby Corkindale writes: > On 10/05/10 12:52, Daniel Pittman wrote: >> Toby Wintermute writes: [...] >>> In Perl it seems like the received wisdom is to back away from writing >>> code that looks like random line noise. >> >> APL is pretty much the extreme end of the Domain Specific Language spectrum, >> which is what operator overloading is a small step along the path to. >> >> Done right, this still is gold: you can't beat it. It is still the killer >> advantage of Lisp, that it is so trivial to define a useful DSL, so you tend >> to do it all along the way. >> >> OTOH, done poorly DSL code is terrible. Probably worse than, say, some of the >> "I don't understand OO and find Perl5 scary" Perl that I get to support. >> >>> If you could make up any operators you like, would you? >> >> Yes, absolutely. I also 'use overload' in Perl. > > So do I - but I'm overloading them within the original meaning. > ie. + is adding or appending, - is reducing, " is making a printable response, > etc.. ...but the original meaning of '+' is never "appending" inside Perl, although it is in some other languages; Perl has '.' for that purpose. :) Arguably, some of these extensions, like '*', are quite reasonable: matrix multiplication is well defined, and matrix calculus arguably extends some functional bits into the array space, so... In other words, I think "original meaning" is a weak definition, much like the classic "I can't define indecency, but I know it when I see it" argument. Potentially the best you can do, but not very solid ? and easy for us to disagree on. :) > I am now tempted to create a library for Scala that entirely consists of > smiley expressions, such as the sadface operator: a :( b > the snowman operator: c ? d > the wince operator: e >.< f > the mind boggles operator: g o.O h > and possibly the nethack operator: > a ?..{.d.. at .]K### b > > hell, with enough custom operators, I could make a DSL for Nethack games. > > ? ( @ o.O @[Shopkeeper] ) { d => # } I see a great need for a teco implementation in Scala. ;) Daniel ...I wonder if someone has ported A+ to it yet... -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From daniel at rimspace.net Mon May 10 19:57:21 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Tue, 11 May 2010 12:57:21 +1000 Subject: [Melbourne-pm] CPAN Testers and build_requires support. Message-ID: <877hnb4032.fsf@rimspace.net> G'day. I have recently uploaded my first module[1] to CPAN, and started to get CPAN?Testers reports back about it ? including a couple of fails, because one of the tests requires Test::More 0.88 and doesn't have it available. I used Dist::Zilla to package it up, and it emitted the testing requirements into the "build_requires" section of META.{yml,json}, which I understand is the correct approach. Dist::Zilla ships an EU::MM based Makefile.PL for this distribution. Anyway, the two reports I have so far are both 'fails', because they don't have the requisite version of Test::More available. I can't find any guidance on how wide-spread build_requires support is; is this about the level I should expect: 6 pass to 2 fail on day one of testing? Is there anything else I can do to improve this? I don't mind, for example, shipping a Module::Build script rather than EU:MM if that makes things more likely to succeed. Also, the approach Dist::Zilla takes with the 'configure_requires' and 'build_requires' keys is to remove them when running on an older version of EU:MM. Is a better approach to add them to 'requires', but only on older versions of EU::MM, so that the generated code has those extra details on platforms that don't support the newer fields? Oh, and finally, if anyone has any comments on the module ? for all that it is really about as trivial as a module can be and all ? I would welcome them publicly or privately. Daniel Footnotes: [1] http://search.cpan.org/~danielp/Log-Any-Adapter-Syslog-1.0/ -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From alecclews at gmail.com Tue May 11 15:31:08 2010 From: alecclews at gmail.com (alecclews at gmail.com) Date: Tue, 11 May 2010 22:31:08 +0000 (UTC) Subject: [Melbourne-pm] perl6 meeting Message-ID: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> an idea for tonights meeting. A Perl6 workshop. Bring along laptops and smartphones to install Perl6 and experiment. Do we have net access? Alec ---- Sent from my 3 mobile From toby.corkindale at strategicdata.com.au Tue May 11 17:30:31 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 12 May 2010 10:30:31 +1000 Subject: [Melbourne-pm] perl6 meeting In-Reply-To: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> References: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> Message-ID: <4BE9F6A7.20104@strategicdata.com.au> On 12/05/10 08:31, alecclews at gmail.com wrote: > an idea for tonights meeting. A Perl6 workshop. > > Bring along laptops and smartphones to install Perl6 and experiment. I'll bring the lappie along, sure. Do we have power? -- Strategic Data Pty Ltd Ph: 03 9340 9000 From hamish at hamishcarpenter.com Tue May 11 17:52:40 2010 From: hamish at hamishcarpenter.com (Hamish Carpenter) Date: Wed, 12 May 2010 10:52:40 +1000 Subject: [Melbourne-pm] perl6 meeting In-Reply-To: <4BE9F6A7.20104@strategicdata.com.au> References: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> <4BE9F6A7.20104@strategicdata.com.au> Message-ID: Sorry but unfortunately there is no Internet access available. There is power though. On Wed, May 12, 2010 at 10:30 AM, Toby Corkindale wrote: > On 12/05/10 08:31, alecclews at gmail.com wrote: >> >> an idea for tonights meeting. A Perl6 workshop. >> >> Bring along laptops and smartphones to install Perl6 and experiment. > > I'll bring the lappie along, sure. > > Do we have power? From pjf at perltraining.com.au Tue May 11 18:59:29 2010 From: pjf at perltraining.com.au (Paul Fenwick) Date: Wed, 12 May 2010 11:59:29 +1000 Subject: [Melbourne-pm] Melbourne Perl Mongers, TONIGHT, 6:30pm, Queen Street Message-ID: <4BEA0B81.5050602@perltraining.com.au> G'day everyone! A reminder that the next Melbourne Perl Mongers meeting is TONIGHT, hosted by Multiservice (courtesy of Hamish Carpenter): Multiservice Level 12 10-16 Queen Street Melbourne 3000 The meeting starts at 6:30pm, but doors lock at 6pm, so look out for a note on the door as to what number to ring to be let in, or get there early so you don't need to worry about calls. Tonight we have Sam Watkins talking about mark-down HTML generators, and Alec Clews tentatively leading (or discussing) a Perl 6 workshop. BYO laptop and Internet access. :) We'll also be looking for volunteers to co-ordinate the next three Melbourne.pm meetings, as both Jacinta and myself will be on world tour. Looking forward to seeing you there! Paul -- Paul Fenwick | http://perltraining.com.au/ Director of Training | Ph: +61 3 9354 6001 Perl Training Australia | Fax: +61 3 9354 2681 From toby.corkindale at strategicdata.com.au Tue May 11 19:31:09 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 12 May 2010 12:31:09 +1000 Subject: [Melbourne-pm] perl6 meeting In-Reply-To: References: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> <4BE9F6A7.20104@strategicdata.com.au> Message-ID: <4BEA12ED.4020208@strategicdata.com.au> On 12/05/10 10:52, Hamish Carpenter wrote: > Sorry but unfortunately there is no Internet access available. There > is power though. I guess we should pre-install Perl6, and perhaps bring the tarball on a USB stick. > > > On Wed, May 12, 2010 at 10:30 AM, Toby Corkindale > wrote: >> On 12/05/10 08:31, alecclews at gmail.com wrote: >>> >>> an idea for tonights meeting. A Perl6 workshop. >>> >>> Bring along laptops and smartphones to install Perl6 and experiment. >> >> I'll bring the lappie along, sure. >> >> Do we have power? -- Strategic Data Pty Ltd Ph: 03 9340 9000 From toby.corkindale at strategicdata.com.au Tue May 11 19:43:29 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 12 May 2010 12:43:29 +1000 Subject: [Melbourne-pm] perl6.org Message-ID: <4BEA15D1.3090704@strategicdata.com.au> This isn't going well. I went to http://dev.perl.org/perl6/ Then I clicked on the link to "Perl 6 site", which went to http://perl6.org/, which redirected to http://feather.perl6.nl/syn/ Which failed to load. I tried going to Pugs, which was linked as "Try Perl 6 today!", which goes to www.pugscode.org, which fails to load too. I am now at http://github.com/rakudo/rakudo/downloads which at least loads; we'll see how well it compiles. Toby From sam at nipl.net Tue May 11 20:15:43 2010 From: sam at nipl.net (Sam Watkins) Date: Wed, 12 May 2010 03:15:43 +0000 Subject: [Melbourne-pm] Melbourne Perl Mongers, TONIGHT, 6:30pm, Queen Street In-Reply-To: <4BEA0B81.5050602@perltraining.com.au> References: <4BEA0B81.5050602@perltraining.com.au> Message-ID: <20100512031543.GD11396@nipl.net> On Wed, May 12, 2010 at 11:59:29AM +1000, Paul Fenwick wrote: > G'day everyone! > > A reminder that the next Melbourne Perl Mongers meeting is TONIGHT, hosted > by Multiservice (courtesy of Hamish Carpenter): > > Multiservice > Level 12 > 10-16 Queen Street > Melbourne 3000 > > The meeting starts at 6:30pm, but doors lock at 6pm, so look out for a > note on the door as to what number to ring to be let in, or get there early > so you don't need to worry about calls. > > Tonight we have Sam Watkins talking about mark-down HTML generators, and > Alec Clews tentatively leading (or discussing) a Perl 6 workshop. BYO > laptop and Internet access. :) Hi guys, First, can anyone bring a 3.5mm to 3xRC AV cable to the talk? I left mine at home, and I'd rather not have to buy another or go home first. Some great news today for free software, the humble indie games bundle has made over $1,000,000 in a week, and to say thanks they are open sourcing Lugaru, Gish, Penumbra: Overture and Aquaria. These four just became my top four favourite open source games. This is the best news for free software I've heard this year. You can still buy the bundle if you didn't already, google "humble indie bundle" as for some stupid reason the spam filter is rejecting my email if I include the URL to the wolfire page. I think I'll demo my markdown thing using my phone (Nokia n900) it should plug into the projector ok if I have a suitable cable. The phone itself is more interesting than my talk! Anyone else has a phone clocked to 1.2Ghz with Debian Linux, X11, perl, python, ruby, gcc, quake 3, etc? :) It runs brace my programming language nicely, and my graphics demos. I'm seriously thinking to start a business selling n900+ phones, I'd buy n900s and set them up with heaps of games, programs, emulators, overclockable kernel, fix the partitions, provide rescue images and support, maybe an android dual boot, etc. I reckon there'd be a market for that even among hackers as it's a bit complicated setting all that up! but I have scripted most of it, in case I brick my phone again! Sam From toby.corkindale at strategicdata.com.au Tue May 11 21:12:13 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 12 May 2010 14:12:13 +1000 Subject: [Melbourne-pm] perl6.org In-Reply-To: <4BEA15D1.3090704@strategicdata.com.au> References: <4BEA15D1.3090704@strategicdata.com.au> Message-ID: <4BEA2A9D.9010702@strategicdata.com.au> On 12/05/10 12:43, Toby Corkindale wrote: > This isn't going well. > I went to http://dev.perl.org/perl6/ > Then I clicked on the link to "Perl 6 site", which went to > http://perl6.org/, which redirected to http://feather.perl6.nl/syn/ > Which failed to load. > > I tried going to Pugs, which was linked as "Try Perl 6 today!", which > goes to www.pugscode.org, which fails to load too. > > I am now at http://github.com/rakudo/rakudo/downloads which at least > loads; we'll see how well it compiles. It compiled OK, but note that it required network access during the build process, and also required that subversion was installed. They're not kidding about the performance of Perl6 being.. poor.. at the moment: $ time ./hello.pl Hello earth Hello sky real 0m0.021s user 0m0.016s sys 0m0.008s $ time ./hello.p6 Hello earth. Hello sky. real 0m2.075s user 0m1.928s sys 0m0.140s On the other hand, it seems to be doing slightly better than Scala :) $ time scala HelloScala Hello earth Hello sky real 0m2.178s user 0m2.192s sys 0m0.108s And for good measure, here's Google's Go language: $ time ./6.out Hello earth. Hello sky. real 0m0.002s user 0m0.000s sys 0m0.000s None of these are terribly good performance tests though.. they're more a measure of the start-up time! Go has none, since it's compiled to native code.. perl5 is interpreted, but otherwise written in native code. Both Scala and Perl6 are running on bytecode VMs. From steve.sabljak at gmail.com Tue May 11 20:49:24 2010 From: steve.sabljak at gmail.com (Steve Sabljak) Date: Wed, 12 May 2010 13:49:24 +1000 Subject: [Melbourne-pm] perl6.org In-Reply-To: <4BEA15D1.3090704@strategicdata.com.au> References: <4BEA15D1.3090704@strategicdata.com.au> Message-ID: On 12 May 2010 12:43, Toby Corkindale wrote: > This isn't going well. > I went to http://dev.perl.org/perl6/ > Then I clicked on the link to "Perl 6 site", which went to > http://perl6.org/, which redirected to http://feather.perl6.nl/syn/ > Which failed to load. > > I tried going to Pugs, which was linked as "Try Perl 6 today!", which goes > to www.pugscode.org, which fails to load too. > > I am now at http://github.com/rakudo/rakudo/downloads which at least loads; > we'll see how well it compiles. > > Toby Try going to rakudo.org and click on the 'Get Rakudo' link, -Steve From alec.clews at gmail.com Tue May 11 21:26:07 2010 From: alec.clews at gmail.com (Alec Clews) Date: Wed, 12 May 2010 14:26:07 +1000 Subject: [Melbourne-pm] perl6.org In-Reply-To: References: <4BEA15D1.3090704@strategicdata.com.au> Message-ID: <4BEA2DDF.7070605@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/10 13:49, Steve Sabljak wrote: > On 12 May 2010 12:43, Toby Corkindale > wrote: >> >> I am now at http://github.com/rakudo/rakudo/downloads which at least loads; >> we'll see how well it compiles. >> > > Try going to rakudo.org and click on the 'Get Rakudo' link, > No debs for Ubuntu and I have not looked for a PPA so I've cloned the GitHub repo and I'm building now Alec -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvqLd4ACgkQP+e5j5u7/HxPtQCgpNU0PzK/t7eTD9vpLHdp5/H9 BGsAoJ9sFDL7h2BynY8e7JtWfzhPK1wF =JVjs -----END PGP SIGNATURE----- From wayland at wayland.id.au Tue May 11 21:09:12 2010 From: wayland at wayland.id.au (Timothy S. Nelson) Date: Wed, 12 May 2010 14:09:12 +1000 (EST) Subject: [Melbourne-pm] perl6.org In-Reply-To: <4BEA2A9D.9010702@strategicdata.com.au> References: <4BEA15D1.3090704@strategicdata.com.au> <4BEA2A9D.9010702@strategicdata.com.au> Message-ID: On Wed, 12 May 2010, Toby Corkindale wrote: > They're not kidding about the performance of Perl6 being.. poor.. at the > moment: The good news is, they put almost no time into optimisation, so there's plenty of room for improvement :). I'm personally waiting for Rakudo Start (June, IIRC) before I give it another go. :) --------------------------------------------------------------------- | Name: Tim Nelson | Because the Creator is, | | E-mail: wayland at wayland.id.au | I am | --------------------------------------------------------------------- ----BEGIN GEEK CODE BLOCK---- Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+>++ PGP->+++ R(+) !tv b++ DI++++ D G+ e++>++++ h! y- -----END GEEK CODE BLOCK----- From toby.corkindale at strategicdata.com.au Tue May 11 22:02:01 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 12 May 2010 15:02:01 +1000 Subject: [Melbourne-pm] perl6.org In-Reply-To: References: <4BEA15D1.3090704@strategicdata.com.au> <4BEA2A9D.9010702@strategicdata.com.au> Message-ID: <4BEA3649.5020807@strategicdata.com.au> On 12/05/10 14:09, Timothy S. Nelson wrote: > On Wed, 12 May 2010, Toby Corkindale wrote: > >> They're not kidding about the performance of Perl6 being.. poor.. at >> the moment: > > The good news is, they put almost no time into optimisation, so there's > plenty of room for improvement :). I'm personally waiting for Rakudo > Start (June, IIRC) before I give it another go. How are the supporting libraries going? I mean, if I want to try out Perl6, has anyone ported DBI, libwww, Text::CSV and XML::LibXML over yet? I tried perldoc-ing on my new Rakudo build, but couldn't find anything that seemed to be perl6-ish. From dsk_gr at hotmail.com Tue May 11 22:11:54 2010 From: dsk_gr at hotmail.com (Kostas Avlonitis) Date: Wed, 12 May 2010 15:11:54 +1000 Subject: [Melbourne-pm] perl6 meeting In-Reply-To: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> References: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> Message-ID: <4BEA389A.8080707@hotmail.com> > Bring along laptops and smartphones to install Perl6 and experiment. > It does sound good. Just my 2 cents: There will likely be some people either coming straight from work or that for other reasons will not be able to bring laptops. So if there is a talk perhaps it could be planned with that in mind. Kostas Avlonitis From scottp at dd.com.au Tue May 11 23:49:38 2010 From: scottp at dd.com.au (Scott Penrose) Date: Wed, 12 May 2010 16:49:38 +1000 Subject: [Melbourne-pm] perl6 meeting In-Reply-To: <4BEA389A.8080707@hotmail.com> References: <17170944.20065.1273617069950.JavaMail.seven@ap6.p8.sca.7sys.net> <4BEA389A.8080707@hotmail.com> Message-ID: <6464224F-2F31-4806-A183-75244381D68D@dd.com.au> Hey Guys Just wanted to say good luck tonight. Unfortunately I can't make it, I would love to be there for a Perl 6 meeting, I really dig perl 6. Some useful things: * Yes you need subversion for automatic Parrot build, or you can manually download/build parrot then you do not need network access * Download / Git checkout perl6-examples - http://github.com/perl6/perl6-examples - you can get commit access on this, or pass any examples to me and I can upload * For modules (ala a baby type of CPAN) Download / Git checkout proto - http://github.com/masak/proto It has linked lots of modules: see - http://github.com/masak/proto/blob/master/projects.list Which also is a useful link to see what modules are really available and working for Perl 6 No DBI sorry, but SQLite apparently works (not one I have tried) * Checkout http://november-wiki.org/ - a wiki written in perl6 As for performance. Yes, bad... but I have started using Perl 6 for scripts - as 99% of the ones I write are either through away, or if they take 4 seconds in stead of .4 seconds, doesn't really matter. Some links are here - http://scott.dd.com.au/wiki/Perl6 And a basic file read, word wrap example - http://scott.dd.com.au/wiki/Perl_6_Examples_File Good luck. Scot From sam at nipl.net Wed May 12 01:10:31 2010 From: sam at nipl.net (Sam Watkins) Date: Wed, 12 May 2010 08:10:31 +0000 Subject: [Melbourne-pm] Melbourne Perl Mongers, TONIGHT, 6:30pm, Queen Street In-Reply-To: <20100512031543.GD11396@nipl.net> References: <4BEA0B81.5050602@perltraining.com.au> <20100512031543.GD11396@nipl.net> Message-ID: <20100512081031.GA25553@nipl.net> I'm going to be a bit late unfortunately! Sam From shlomif at iglu.org.il Wed May 12 05:22:46 2010 From: shlomif at iglu.org.il (Shlomi Fish) Date: Wed, 12 May 2010 15:22:46 +0300 Subject: [Melbourne-pm] perl6.org In-Reply-To: <4BEA2A9D.9010702@strategicdata.com.au> References: <4BEA15D1.3090704@strategicdata.com.au> <4BEA2A9D.9010702@strategicdata.com.au> Message-ID: <201005121522.46796.shlomif@iglu.org.il> On Wednesday 12 May 2010 07:12:13 Toby Corkindale wrote: > On 12/05/10 12:43, Toby Corkindale wrote: > > This isn't going well. > > I went to http://dev.perl.org/perl6/ > > Then I clicked on the link to "Perl 6 site", which went to > > http://perl6.org/, which redirected to http://feather.perl6.nl/syn/ > > Which failed to load. > > > > I tried going to Pugs, which was linked as "Try Perl 6 today!", which > > goes to www.pugscode.org, which fails to load too. > > > > I am now at http://github.com/rakudo/rakudo/downloads which at least > > loads; we'll see how well it compiles. > > It compiled OK, but note that it required network access during the > build process, and also required that subversion was installed. > > They're not kidding about the performance of Perl6 being.. poor.. at the > moment: > > $ time ./hello.pl > Hello earth > Hello sky > > real 0m0.021s > user 0m0.016s > sys 0m0.008s > > > $ time ./hello.p6 > Hello earth. > Hello sky. > > real 0m2.075s > user 0m1.928s > sys 0m0.140s > > On the other hand, it seems to be doing slightly better than Scala :) > $ time scala HelloScala > Hello earth > Hello sky > > real 0m2.178s > user 0m2.192s > sys 0m0.108s > > > And for good measure, here's Google's Go language: > $ time ./6.out > Hello earth. > Hello sky. > > real 0m0.002s > user 0m0.000s > sys 0m0.000s > > > > None of these are terribly good performance tests though.. they're more > a measure of the start-up time! Go has none, since it's compiled to > native code.. perl5 is interpreted, but otherwise written in native > code. Both Scala and Perl6 are running on bytecode VMs. Actually, perl 5 compiles Perl 5 into its own internal bytecode, and it has a virtual machine for executing it (though very different from either Parrot (Rakudo's virtual machine) or the JVM). I don't know why Scala is so slow, but it's possible you're affected by the notorious long Java startup time. I was told it was fixed in JDK 1.5.x, but I haven't tried it yet. That or it is possible that the Scala compiler has some overhead. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ The Case for File Swapping - http://shlom.in/file-swap God considered inflicting XSLT as the tenth plague of Egypt, but then decided against it because he thought it would be too evil. Please reply to list if it's a mailing list post - http://shlom.in/reply . From tjc at wintrmute.net Wed May 12 06:35:56 2010 From: tjc at wintrmute.net (Toby Wintermute) Date: Wed, 12 May 2010 23:35:56 +1000 Subject: [Melbourne-pm] perl6.org In-Reply-To: <201005121522.46796.shlomif@iglu.org.il> References: <4BEA15D1.3090704@strategicdata.com.au> <4BEA2A9D.9010702@strategicdata.com.au> <201005121522.46796.shlomif@iglu.org.il> Message-ID: >From what I've read, Scala is quite fast, but does indeed suffer from the JRE taking a long time to warm up. Some of the little apps I wrote that did more than say "hello world" seemed to run snappily, anyway :) On 12 May 2010 22:22, Shlomi Fish wrote: > On Wednesday 12 May 2010 07:12:13 Toby Corkindale wrote: >> On 12/05/10 12:43, Toby Corkindale wrote: >> > This isn't going well. >> > I went to http://dev.perl.org/perl6/ >> > Then I clicked on the link to "Perl 6 site", which went to >> > http://perl6.org/, which redirected to http://feather.perl6.nl/syn/ >> > Which failed to load. >> > >> > I tried going to Pugs, which was linked as "Try Perl 6 today!", which >> > goes to www.pugscode.org, which fails to load too. >> > >> > I am now at http://github.com/rakudo/rakudo/downloads which at least >> > loads; we'll see how well it compiles. >> >> It compiled OK, but note that it required network access during the >> build process, and also required that subversion was installed. >> >> They're not kidding about the performance of Perl6 being.. poor.. at the >> moment: >> >> $ time ./hello.pl >> Hello earth >> Hello sky >> >> real ?0m0.021s >> user ?0m0.016s >> sys ? 0m0.008s >> >> >> $ time ./hello.p6 >> Hello earth. >> Hello sky. >> >> real ?0m2.075s >> user ?0m1.928s >> sys ? 0m0.140s >> >> On the other hand, it seems to be doing slightly better than Scala :) >> $ time scala HelloScala >> Hello earth >> Hello sky >> >> real ?0m2.178s >> user ?0m2.192s >> sys ? 0m0.108s >> >> >> And for good measure, here's Google's Go language: >> $ time ./6.out >> Hello earth. >> Hello sky. >> >> real ?0m0.002s >> user ?0m0.000s >> sys ? 0m0.000s >> >> >> >> None of these are terribly good performance tests though.. they're more >> a measure of the start-up time! Go has none, since it's compiled to >> native code.. perl5 is interpreted, but otherwise written in native >> code. Both Scala and Perl6 are running on bytecode VMs. > > Actually, perl 5 compiles Perl 5 into its own internal bytecode, and it has a > virtual machine for executing it (though very different from either Parrot > (Rakudo's virtual machine) or the JVM). I don't know why Scala is so slow, but > it's possible you're affected by the notorious long Java startup time. I was > told it was fixed in JDK 1.5.x, but I haven't tried it yet. That or it is > possible that the Scala compiler has some overhead. > > Regards, > > ? ? ? ?Shlomi Fish > > -- > ----------------------------------------------------------------- > Shlomi Fish ? ? ? http://www.shlomifish.org/ > The Case for File Swapping - http://shlom.in/file-swap > > God considered inflicting XSLT as the tenth plague of Egypt, but then > decided against it because he thought it would be too evil. > > Please reply to list if it's a mailing list post - http://shlom.in/reply . > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world From T.Hunt at bom.gov.au Wed May 12 16:02:33 2010 From: T.Hunt at bom.gov.au (Timothy Hunt) Date: Thu, 13 May 2010 09:02:33 +1000 Subject: [Melbourne-pm] FW: ODTUG Welcomes the MySQL Community to Kaleidoscope 2010 [SEC=UNCLASSIFIED] Message-ID: <7548A47B0E2E8643BE38DBBD8FD9CA1AB51EDF2F09@BOM-VMBX-HO.bom.gov.au> So, the Oracle community is wanting to embrace the MySQL community. They say that Oracle will be answering questions about MySQL's future too... Cheers, Tim. ________________________________________ From: ODTUG [mailto:info at odtug.com] Sent: Thursday, 13 May 2010 06:40 To: Timothy Hunt Subject: ODTUG Welcomes the MySQL Community to Kaleidoscope 2010 ODTUG Welcomes the MySQL Community to Kaleidoscope 2010 ODTUG is pleased to welcome the MySQL community to participate in our annual Kaleidoscope Conference in Washington, D.C.? Following the successful formula of asking the experts within a community to create an agenda for new and relevant topic areas, newly named Oracle MySQL ACE Director Ronald Bradford and his colleagues have come up with a very compelling reason for MySQL developers to attend?this year's?conference.? This agenda should be of interest to anyone involved in development of applications that utilize the MySQL database, as well as those attendees that would just like to learn a little more about MySQL.? To see more details about this exciting addition to the ODTUG Kaleidoscope Conference, check out www.odtugkaleidoscope.com/MySQL.html. On Monday, June 28 from 5:45 p.m. to 6:45 p.m., all MySQL users are invited to attend the MySQL Sundown Session. Oracle MySQL ACE Directors, Ronald Bradford and Sheeri K. Cabral, will lead a lively session and answer your MySQL technical questions, as well as questions regarding the future of MySQL. Immediately following the Sundown Sessions, participants can continue their discussions and meet other Oracle ACE Directors and ACEs during a short reception. Light refreshments will be served. Sundown Sessions will also be offered on Middle Tier and Client-Side Development, Database Development, Hyperion, Essbase and BI, SOA and BPM, and Application Express. ODTUG Kaleidoscope 2010, June 27-July 1 at the Marriott Wardman Park Hotel in Washington, D.C. features more than 170 sessions, Hands-on Training, and four full-day Symposia. Major topics include: Application Express, Database Development, Essbase, Hyperion Applications, Hardcore Hyperion, Middle Tier and Client-Side Development, MySQL, Oracle Business Intelligence, Hyperion Reporting, Data Warehousing, and SOA and BPM. For more information go to www.odtugkaleidoscope.com.? Hotel Room Reservation Deadline - June 9 Don't forget to make your hotel reservations at the Marriott Wardman Park Hotel. Rates are $199 for all attendees (this is even lower than the approved government rate). But don't delay; the room reservation cut-off deadline is June 9, 2010.? You may call the Washington Marriott Wardman Park directly at 202.328.2000, and be sure to mention that you are with the ODTUG room block. You may also click here to make your hotel reservations.? ODTUG - 3208 Oleander Drive Suite C - Wilmington, NC 28403 From hamish at hamishcarpenter.com Wed May 12 20:44:33 2010 From: hamish at hamishcarpenter.com (Hamish Carpenter) Date: Thu, 13 May 2010 13:44:33 +1000 Subject: [Melbourne-pm] Bash prompt red on error Message-ID: As briefly discussed last night, I mentioned I'd changed my prompt to be red on error (return code != 0). This helps me detect failed commands and also helps in another case where you type a command then cancel it by pressing ctrl+c. This was what prompted me to change my prompt as someone watching your terminal cannot easily tell that command was canceled, especially if it is a command that would have no output on success (eg rf -f project_directory). add the following to your .bash_profile: # needs PS1 to be previously defined so add nearer the end of your .bashrc/.bash_profile # will turn entire prompt red (31) or green (32) # depending on previous output return code export PS1="\[\e[\`if [ \$? = 0 ]; then echo '0;32m'; else echo '0;31m'; fi\`\]$PS1\[\e[0m\]" hth, Hamish From sam at nipl.net Wed May 12 21:37:44 2010 From: sam at nipl.net (Sam Watkins) Date: Thu, 13 May 2010 04:37:44 +0000 Subject: [Melbourne-pm] Bash prompt red on error In-Reply-To: References: Message-ID: <20100513043744.GB25044@nipl.net> On Thu, May 13, 2010 at 01:44:33PM +1000, Hamish Carpenter wrote: > As briefly discussed last night, I mentioned I'd changed my prompt to > be red on error (return code != 0). This helps me detect failed > commands and also helps in another case where you type a command then > cancel it by pressing ctrl+c. This was what prompted me to change my > prompt as someone watching your terminal cannot easily tell that > command was canceled, especially if it is a command that would have no > output on success (eg rf -f project_directory). > > add the following to your .bash_profile: > > # needs PS1 to be previously defined so add nearer the end of your > .bashrc/.bash_profile > # will turn entire prompt red (31) or green (32) > # depending on previous output return code > export PS1="\[\e[\`if [ \$? = 0 ]; then echo '0;32m'; else echo > '0;31m'; fi\`\]$PS1\[\e[0m\]" cool, a friend showed me something similar, giving ^_^ for a success prompt, and O_o for failure :) # SmiliePrompt - from http://x-demon.org/ smiley() { if [ $? == 0 ]; then echo '^_^'; else echo '0_o'; fi; } export PS1='$(smiley) ' It's important to know exit status, so I use this all the time. The shell should really do something like this by default, maybe not the cute emoticon version though. enjoy! :) From alfiejohn at gmail.com Wed May 12 22:03:57 2010 From: alfiejohn at gmail.com (Alfie John) Date: Thu, 13 May 2010 15:03:57 +1000 Subject: [Melbourne-pm] Bash prompt red on error In-Reply-To: <20100513043744.GB25044@nipl.net> References: <20100513043744.GB25044@nipl.net> Message-ID: What I tend to do is have all text as red to indicate that I'm on a production box, but Hamish's prompt is nice and not as offensive as a screen full of red. Alfie On Thu, May 13, 2010 at 2:37 PM, Sam Watkins wrote: > On Thu, May 13, 2010 at 01:44:33PM +1000, Hamish Carpenter wrote: > > As briefly discussed last night, I mentioned I'd changed my prompt to > > be red on error (return code != 0). This helps me detect failed > > commands and also helps in another case where you type a command then > > cancel it by pressing ctrl+c. This was what prompted me to change my > > prompt as someone watching your terminal cannot easily tell that > > command was canceled, especially if it is a command that would have no > > output on success (eg rf -f project_directory). > > > > add the following to your .bash_profile: > > > > # needs PS1 to be previously defined so add nearer the end of your > > .bashrc/.bash_profile > > # will turn entire prompt red (31) or green (32) > > # depending on previous output return code > > export PS1="\[\e[\`if [ \$? = 0 ]; then echo '0;32m'; else echo > > '0;31m'; fi\`\]$PS1\[\e[0m\]" > > cool, a friend showed me something similar, giving > ^_^ for a success prompt, and O_o for failure :) > > # SmiliePrompt - from http://x-demon.org/ > smiley() { if [ $? == 0 ]; then echo '^_^'; else echo '0_o'; fi; } > export PS1='$(smiley) ' > > It's important to know exit status, so I use this all the time. The shell > should really do something like this by default, maybe not the cute > emoticon > version though. > > enjoy! :) > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toby.corkindale at strategicdata.com.au Wed May 12 23:18:50 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Thu, 13 May 2010 16:18:50 +1000 Subject: [Melbourne-pm] Next meeting: Wednesday 9th June 2010 Message-ID: <4BEB99CA.3050304@strategicdata.com.au> Hi everyone, As we discussed at the meeting last night, Paul and Jacinta will be away for a few months, and thus organisation of the meetings falls to the rest of us mere mortals for the next three meetings. Jacinta and Paul, thank you for always managing these meetings for as long as I've subscribed to the Melbourne.pm list. The next meeting will be on the 9th of June, and will be hosted by Strategic Data: Strategic Data Level 2 51-55 Johnston Street Fitzroy 3065 They have kindly offered to provide some snacks and pizza! Talks: I think we had a volunteer last night for a talk demonstrating Padre, the development environment for Perl, in Perl? I'm sure we'll get a couple of lightning talks again.. Are there any other suggestions or volunteers for talks? Mail me any time in the next few weeks if you come up with anything, or I'll find something else to ramble on about again :) Cheers, Toby From alec.clews at gmail.com Thu May 13 03:40:10 2010 From: alec.clews at gmail.com (Alec Clews) Date: Thu, 13 May 2010 20:40:10 +1000 Subject: [Melbourne-pm] Notes from Wed meetings In-Reply-To: <20100513043744.GB25044@nipl.net> References: <20100513043744.GB25044@nipl.net> Message-ID: <4BEBD70A.8010104@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Are at http://perl.net.au/wiki/Melbourne_Perl_Mongers/Meeting_History_2010#12th_May_2010.2C_Perl_6.2C_Sam.27s_custom_markdown Have at them and add more information please Alec -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvr1wkACgkQP+e5j5u7/HzHXACfaoxHyHUZYSYpgizjX0NpAEBj SmIAnRe61Xg0BxY/vFBan0eMopZcTUWH =vHVj -----END PGP SIGNATURE----- From sam at nipl.net Thu May 13 21:17:29 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 14 May 2010 04:17:29 +0000 Subject: [Melbourne-pm] Notes from Wed meetings In-Reply-To: <4BEBD70A.8010104@gmail.com> References: <20100513043744.GB25044@nipl.net> <4BEBD70A.8010104@gmail.com> Message-ID: <20100514041729.GC3217@nipl.net> On Thu, May 13, 2010 at 08:40:10PM +1000, Alec Clews wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Are at > > http://perl.net.au/wiki/Melbourne_Perl_Mongers/Meeting_History_2010#12th_May_2010.2C_Perl_6.2C_Sam.27s_custom_markdown > > Have at them and add more information please Ok I made a small demo for my form thing and put some links to it there. thanks, Sam From wayland at wayland.id.au Fri May 14 05:04:14 2010 From: wayland at wayland.id.au (Timothy S. Nelson) Date: Fri, 14 May 2010 22:04:14 +1000 (EST) Subject: [Melbourne-pm] perl6.org In-Reply-To: <4BEA3649.5020807@strategicdata.com.au> References: <4BEA15D1.3090704@strategicdata.com.au> <4BEA2A9D.9010702@strategicdata.com.au> <4BEA3649.5020807@strategicdata.com.au> Message-ID: On Wed, 12 May 2010, Toby Corkindale wrote: > On 12/05/10 14:09, Timothy S. Nelson wrote: >> On Wed, 12 May 2010, Toby Corkindale wrote: >> >>> They're not kidding about the performance of Perl6 being.. poor.. at >>> the moment: >> >> The good news is, they put almost no time into optimisation, so there's >> plenty of room for improvement :). I'm personally waiting for Rakudo >> Start (June, IIRC) before I give it another go. > > How are the supporting libraries going? I haven't been following much since February (excpet the mailing list). The people on #perl6 on Freenode are usually helpful (at the right time of day :) ). Having said that, I'll try to answer. The answer is: not very well. I think the big holdup is that the thing that makes Parrot work with C-based libraries is not automatic (ie. you have to hand code in a parrot-specific language for each library). When that becomes automatic, then things should take off in that area. > I mean, if I want to try out Perl6, has anyone ported DBI, libwww, Text::CSV > and XML::LibXML over yet? Having said that, I know that Dave Rolsky is working with the P6 team to redesign DateTime from the ground up, and that will probably be coming along well. In theory Tim Bunce is also designing DBI 2.0 to work with Perl 6, but I haven't heard about that in a while. > I tried perldoc-ing on my new Rakudo build, but couldn't find anything that > seemed to be perl6-ish. I don't understand what you're saying. HTH, --------------------------------------------------------------------- | Name: Tim Nelson | Because the Creator is, | | E-mail: wayland at wayland.id.au | I am | --------------------------------------------------------------------- ----BEGIN GEEK CODE BLOCK---- Version 3.12 GCS d+++ s+: a- C++$ U+++$ P+++$ L+++ E- W+ N+ w--- V- PE(+) Y+>++ PGP->+++ R(+) !tv b++ DI++++ D G+ e++>++++ h! y- -----END GEEK CODE BLOCK----- From toby.corkindale at strategicdata.com.au Sun May 16 21:40:15 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 17 May 2010 14:40:15 +1000 Subject: [Melbourne-pm] Perl 5.12.1 out Message-ID: <4BF0C8AF.8010800@strategicdata.com.au> I note that Perl 5.12.1 was released yesterday. According to perl5121delta.pod, it seems to be a bunch of bugfixes to 5.12.0, with no incompatible changes, and no user-visible changes to the core language. Worth using just so you can type: use 5.12.1; rather than use 5.10.0; use strict; use feature ':5.10; Warning are, unfortunately, still optional, except for deprecation warnings. From toby.corkindale at strategicdata.com.au Sun May 16 22:49:28 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 17 May 2010 15:49:28 +1000 Subject: [Melbourne-pm] Retry/Attempt Message-ID: <4BF0D8E8.30904@strategicdata.com.au> By the way, we discussed several modules for retrying/attempting a repeated task at the last meeting. My method is available at: http://search.cpan.org/~tjc/Retry-0.10/ or http://github.com/TJC/Retry Mark Fowler's Attempt was another one mentioned. Notably, his module is designed more to retry over and over quickly, whereas mine increases the delay between retries, and has a callback to report on failures as they happen, in case you need to provide feedback to the user, or record it for error logging. From pat at patspam.com Mon May 17 07:58:02 2010 From: pat at patspam.com (Patrick Donelan) Date: Mon, 17 May 2010 10:58:02 -0400 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: <4BF0D8E8.30904@strategicdata.com.au> References: <4BF0D8E8.30904@strategicdata.com.au> Message-ID: > > Notably, his module is designed more to retry over and over quickly, > whereas mine increases the delay between retries > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! Patrick http://patspam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From wigs at stirfried.org Mon May 17 18:06:54 2010 From: wigs at stirfried.org (wigs at stirfried.org) Date: Tue, 18 May 2010 11:06:54 +1000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: References: <4BF0D8E8.30904@strategicdata.com.au> Message-ID: <20100518010654.GA18726@stirfried.org> On Mon, May 17, 2010 at 10:58:02AM -0400, Patrick Donelan wrote: > Toby wrote: > > Notably, his module is designed more to retry over and over quickly, > > whereas mine increases the delay between retries > > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! Nah, simple doubling, not Fibonacci: sleep($delay); $delay *= 2; For a good discussion on the approach I recommend this article, http://dthain.blogspot.com/2009/02/exponential-backoff-in-distributed.html which goes into the theory of how to choose a good delay value based on the behaviours of your system. -- Wigs From daniel at rimspace.net Mon May 17 18:16:12 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Tue, 18 May 2010 11:16:12 +1000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: (Patrick Donelan's message of "Mon, 17 May 2010 10:58:02 -0400") References: <4BF0D8E8.30904@strategicdata.com.au> Message-ID: <87wrv2huw3.fsf@rimspace.net> Patrick Donelan writes: >> Notably, his module is designed more to retry over and over quickly, whereas >> mine increases the delay between retries > > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! Heh. Despite my bikeshedding ^W encouragement, this just doubles the delay every time, rather than using a Fibonacci back-off strategy. Alas. :) Daniel Retry::Fibonnaci, I see a great need. -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From sam at nipl.net Mon May 17 18:55:34 2010 From: sam at nipl.net (Sam Watkins) Date: Tue, 18 May 2010 01:55:34 +0000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: <87wrv2huw3.fsf@rimspace.net> References: <4BF0D8E8.30904@strategicdata.com.au> <87wrv2huw3.fsf@rimspace.net> Message-ID: <20100518015534.GB14302@nipl.net> On Tue, May 18, 2010 at 11:16:12AM +1000, Daniel Pittman wrote: > > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! > > Heh. Despite my bikeshedding ^W encouragement, this just doubles the delay > every time, rather than using a Fibonacci back-off strategy. Alas. :) What advantage would there be to using Fibonacci numbers instead of exponential backoff? IIRC the Fibonacci series is approximately exponential anyway. I use exponential growth and backoff for several things, e.g. my sshd, and to grow storage for buffers and vectors (dynamically sized arrays). Sam From alec.clews at gmail.com Mon May 17 20:12:17 2010 From: alec.clews at gmail.com (Alec Clews) Date: Tue, 18 May 2010 13:12:17 +1000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: <4BF0D8E8.30904@strategicdata.com.au> References: <4BF0D8E8.30904@strategicdata.com.au> Message-ID: <4BF20591.1060408@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17/05/10 15:49, Toby Corkindale wrote: > we discussed several modules for retrying/attempting a repeated task at > the last meeting... Added to http://perl.net.au/wiki/Melbourne_Perl_Mongers/Meeting_History_2010#12th_May_2010.2C_Perl_6.2C_Sam.27s_custom_markdown Alec -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvyBZEACgkQP+e5j5u7/Hz81wCfcFhdK7MdHsyQlxJP3JZ0POqp zt8AoJClq8NHmUauV1dCw3G886OOd5NP =t4c3 -----END PGP SIGNATURE----- From toby.corkindale at strategicdata.com.au Mon May 17 20:49:23 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Tue, 18 May 2010 13:49:23 +1000 Subject: [Melbourne-pm] Still performing well.. Message-ID: <4BF20E43.3010503@strategicdata.com.au> After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs Scala in more than just a trivial case. I setup a test to read in a large CSV file, performing some minor numeric and text manipulation upon each row, and outputing the results. I timed the results within the apps, so as to avoid the start-up time, and will report both below. Language : Process time : Total time Perl 5.10 : 1089 ms : 1153 ms Go : 1682 ms : 1687 ms Scala 2.8 : 2729 ms : 3114 ms Note that Perl (using Text::CSV) came out ahead of the natively-compiled Go (using a third-party CSV library)! That surprised me! I was also surprised at how slow Scala was, especially since I was lazy and just used String.split(",") rather than a proper CSV library, which should be faster. On the other hand - my test was clearly biassed towards Perl's text-processing strengths, and avoided known weaknesses like object creation. It would be interesting to create another test that performed real-time multi-threaded stream analysis of XML data instead. I should also note that Scala and Perl produced identical output, whereas Go's output had slight variances in a few of the floating point results. (I think it was their printf performing different rounding.) -Toby From daniel at rimspace.net Mon May 17 22:26:56 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Tue, 18 May 2010 15:26:56 +1000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: <20100518015534.GB14302@nipl.net> (Sam Watkins's message of "Tue, 18 May 2010 01:55:34 +0000") References: <4BF0D8E8.30904@strategicdata.com.au> <87wrv2huw3.fsf@rimspace.net> <20100518015534.GB14302@nipl.net> Message-ID: <87vdalhja7.fsf@rimspace.net> Sam Watkins writes: > On Tue, May 18, 2010 at 11:16:12AM +1000, Daniel Pittman wrote: >> > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! >> >> Heh. Despite my bikeshedding ^W encouragement, this just doubles the delay >> every time, rather than using a Fibonacci back-off strategy. Alas. :) > > What advantage would there be to using Fibonacci numbers instead of > exponential backoff? It retries faster in the early stages, and grows a little less quickly, so tends to give better recovery faster than you would otherwise see. > IIRC the Fibonacci series is approximately exponential anyway. Er, no, not really. Exponential grows enormously more quickly than Fibonacci does; starting at 1, at 10 steps you have 34 vs 256, at 15 steps 377 vs 8192. > I use exponential growth and backoff for several things, e.g. my sshd, and > to grow storage for buffers and vectors (dynamically sized arrays). Depending on your needs either might be appropriate; Fibonacci is much better when you want something to retry fairly frequently, or to grow only by smaller steps. Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From daniel at rimspace.net Mon May 17 22:28:04 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Tue, 18 May 2010 15:28:04 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <4BF20E43.3010503@strategicdata.com.au> (Toby Corkindale's message of "Tue, 18 May 2010 13:49:23 +1000") References: <4BF20E43.3010503@strategicdata.com.au> Message-ID: <87r5l9hj8b.fsf@rimspace.net> Toby Corkindale writes: > After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs > Scala in more than just a trivial case. > > I setup a test to read in a large CSV file, performing some minor numeric and > text manipulation upon each row, and outputing the results. When you say "large", do you mean 1MB, 10MB, 1GB, 1TB? I suspect that you will find the figures change relationship a little as you go up the first few orders of magnitude. Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From sam at nipl.net Mon May 17 22:56:04 2010 From: sam at nipl.net (Sam Watkins) Date: Tue, 18 May 2010 05:56:04 +0000 Subject: [Melbourne-pm] Retry/Attempt In-Reply-To: <87vdalhja7.fsf@rimspace.net> References: <4BF0D8E8.30904@strategicdata.com.au> <87wrv2huw3.fsf@rimspace.net> <20100518015534.GB14302@nipl.net> <87vdalhja7.fsf@rimspace.net> Message-ID: <20100518055603.GA22182@nipl.net> On Tue, May 18, 2010 at 03:26:56PM +1000, Daniel Pittman wrote: > Sam Watkins writes: > > On Tue, May 18, 2010 at 11:16:12AM +1000, Daniel Pittman wrote: > >> > I'm jealous, a genuine opportunity to use Fibonacci in a real-world program! > >> > >> Heh. Despite my bikeshedding ^W encouragement, this just doubles the delay > >> every time, rather than using a Fibonacci back-off strategy. Alas. :) > > > > What advantage would there be to using Fibonacci numbers instead of > > exponential backoff? > > It retries faster in the early stages, and grows a little less quickly, so > tends to give better recovery faster than you would otherwise see. > > > IIRC the Fibonacci series is approximately exponential anyway. > > Er, no, not really. Exponential grows enormously more quickly than Fibonacci > does; starting at 1, at 10 steps you have 34 vs 256, at 15 steps 377 vs 8192. The Fibonacci series does grow exponentially, there is a closed-form formula for F(n) and it is approximated by an exponential of base 1+sqrt(5)/2 or about 1.618 The base is lower than 2, so the series grows more slowly than the powers of 2. http://en.wikipedia.org/wiki/Fibonacci_number#Closed-form_expression F(n) ~= (1+sqrt(5)/2) ^ n / sqrt(5) i.e. F(n) ~= 1.618 ^ n / sqrt(5) I guess it's potentially simpler to use Fibonacci numbers (addition) compared to multiplying by 1.618, so Fibonacci might be good for hardware if you want a slower growth rate than the powers of 2. There's some "natural balance" or beauty or something about the Fibonacci numbers, too, something to do with the golden ratio. Or you might like to use e^x for faster growth and some different benefits / mathematical qualities! Sam From toby.corkindale at strategicdata.com.au Mon May 17 23:04:47 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Tue, 18 May 2010 16:04:47 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <87r5l9hj8b.fsf@rimspace.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> Message-ID: <4BF22DFF.6090809@strategicdata.com.au> On 18/05/10 15:28, Daniel Pittman wrote: > Toby Corkindale writes: > >> After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs >> Scala in more than just a trivial case. >> >> I setup a test to read in a large CSV file, performing some minor numeric and >> text manipulation upon each row, and outputing the results. > > When you say "large", do you mean 1MB, 10MB, 1GB, 1TB? Uh, 10,000 lines + header. > I suspect that you will find the figures change relationship a little as you > go up the first few orders of magnitude. Repeating test with 100,000 lines + header, we get: Perl - 10.962 sec Scala - 13.31 sec Go - 16.77 sec How curious.. Perl and Go scaled linearly with the size of the dataset, but Scala didn't. Now repeating the test with 1,000,000 rows. Also including resident memory size this time.. Perl - 111.3 sec (~6 MB) Scala - 115.5 sec (~73 MB) Go - 154.3 sec (~2 MB) How curious. It looks like Scala would overtake Perl if we upped the magnitude one more time! From sam at nipl.net Mon May 17 23:17:41 2010 From: sam at nipl.net (Sam Watkins) Date: Tue, 18 May 2010 06:17:41 +0000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <4BF20E43.3010503@strategicdata.com.au> References: <4BF20E43.3010503@strategicdata.com.au> Message-ID: <20100518061741.GB22182@nipl.net> On Tue, May 18, 2010 at 01:49:23PM +1000, Toby Corkindale wrote: > After the last Perlmongers meeting I was curious to benchmark Perl vs Go > vs Scala in more than just a trivial case. > > I setup a test to read in a large CSV file, performing some minor > numeric and text manipulation upon each row, and outputing the results. > > I timed the results within the apps, so as to avoid the start-up time, > and will report both below. > > > Language : Process time : Total time > Perl 5.10 : 1089 ms : 1153 ms > Go : 1682 ms : 1687 ms > Scala 2.8 : 2729 ms : 3114 ms Text::CSV use C code (XS) by default, but I think Go should still be faster, as Text::CSV has to create all those perl data structures. If you did it in pure Perl (Text::CSV_PP) it would be a lot slower. Perhaps Go is so much slower than C, that the cost of setting up perl data structures doesn't matter. Can you provide your benchmark code? What did you use to the parse the CSV in Go? was it calling out to some Perl subroutines to do the work? ;) Sam From toby.corkindale at strategicdata.com.au Mon May 17 23:49:49 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Tue, 18 May 2010 16:49:49 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <20100518061741.GB22182@nipl.net> References: <4BF20E43.3010503@strategicdata.com.au> <20100518061741.GB22182@nipl.net> Message-ID: <4BF2388D.2070100@strategicdata.com.au> On 18/05/10 16:17, Sam Watkins wrote: > On Tue, May 18, 2010 at 01:49:23PM +1000, Toby Corkindale wrote: >> After the last Perlmongers meeting I was curious to benchmark Perl vs Go >> vs Scala in more than just a trivial case. >> >> I setup a test to read in a large CSV file, performing some minor >> numeric and text manipulation upon each row, and outputing the results. >> >> I timed the results within the apps, so as to avoid the start-up time, >> and will report both below. >> >> >> Language : Process time : Total time >> Perl 5.10 : 1089 ms : 1153 ms >> Go : 1682 ms : 1687 ms >> Scala 2.8 : 2729 ms : 3114 ms > > Text::CSV use C code (XS) by default, but I think Go should still be faster, > as Text::CSV has to create all those perl data structures. If you did it in > pure Perl (Text::CSV_PP) it would be a lot slower. Perhaps Go is so much > slower than C, that the cost of setting up perl data structures doesn't matter. > > Can you provide your benchmark code? What did you use to the parse the CSV in > Go? was it calling out to some Perl subroutines to do the work? ;) http://github.com/TJC/PerfTesting Feel free to fork the repo and then send back improvements to the tests, or even tests written in other languages. (C, C++, Ruby, Python, Brace, etc) Cheers, Toby From daniel at rimspace.net Mon May 17 23:52:40 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Tue, 18 May 2010 16:52:40 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <4BF22DFF.6090809@strategicdata.com.au> (Toby Corkindale's message of "Tue, 18 May 2010 16:04:47 +1000") References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> Message-ID: <87mxvxhfbb.fsf@rimspace.net> Toby Corkindale writes: > On 18/05/10 15:28, Daniel Pittman wrote: >> Toby Corkindale writes: >> >>> After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs >>> Scala in more than just a trivial case. >>> >>> I setup a test to read in a large CSV file, performing some minor numeric and >>> text manipulation upon each row, and outputing the results. >> >> When you say "large", do you mean 1MB, 10MB, 1GB, 1TB? > > Uh, 10,000 lines + header. > >> I suspect that you will find the figures change relationship a little as you >> go up the first few orders of magnitude. > > Repeating test with 100,000 lines + header, we get: > > Perl - 10.962 sec > Scala - 13.31 sec > Go - 16.77 sec > > How curious.. Perl and Go scaled linearly with the size of the dataset, but > Scala didn't. Now repeating the test with 1,000,000 rows. I bet that you finally pushed the GC hard enough that Scala started to feel the pain and suffering of that... > Also including resident memory size this time.. > > Perl - 111.3 sec (~6 MB) > Scala - 115.5 sec (~73 MB) > Go - 154.3 sec (~2 MB) > > How curious. It looks like Scala would overtake Perl if we upped the magnitude > one more time! ...and now you are finally starting to hit the point that some of the long running process optimizations and other bits of JVM magic are starting to kick in. I am not entirely shocked by the result moving that way; benchmarking is a nasty black art, and small tests often show different results to large tests. Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From alec.clews at gmail.com Tue May 18 17:23:23 2010 From: alec.clews at gmail.com (Alec Clews) Date: Wed, 19 May 2010 10:23:23 +1000 Subject: [Melbourne-pm] n00b question: how to exclude test code from production Message-ID: <4BF32F7B.1050403@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have a Perl script that I want to embed some tests in (Using Test::More). What is the best practice way to ignore the test code in production? Thanks - -- Alec Clews Personal Melbourne, Australia. Jabber: alecclews at jabber.org.au PGPKey ID: 0x9BBBFC7C Blog http://alecthegeek.wordpress.com/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvzL3sACgkQP+e5j5u7/HxaJgCdGibMLZIouyiBTXpsYOJWDIDg W1oAoMQVHOsWArig96NaWc5nsskb7GLP =gev5 -----END PGP SIGNATURE----- From toby.corkindale at strategicdata.com.au Tue May 18 17:40:29 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 19 May 2010 10:40:29 +1000 Subject: [Melbourne-pm] n00b question: how to exclude test code from production In-Reply-To: <4BF32F7B.1050403@gmail.com> References: <4BF32F7B.1050403@gmail.com> Message-ID: <4BF3337D.7050205@strategicdata.com.au> On 19/05/10 10:23, Alec Clews wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I have a Perl script that I want to embed some tests in (Using > Test::More). What is the best practice way to ignore the test code in > production? Put your production code in lib/MyModule.pm and script/myscript.pl (Where myscript.pl parses the command-line flags and then call MyModule) Then put your tests into t/*.t One test can "use MyModule" and run a variety of tests upon it. Another test can create a fake MyModule for myscript.pl to hit, and then call myscript.pl with a variety of parameters to check how they are handled. From sam at nipl.net Tue May 18 19:01:54 2010 From: sam at nipl.net (Sam Watkins) Date: Wed, 19 May 2010 02:01:54 +0000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <87mxvxhfbb.fsf@rimspace.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> Message-ID: <20100519020154.GB22403@nipl.net> On Tue, May 18, 2010 at 04:52:40PM +1000, Daniel Pittman wrote: > Toby Corkindale writes: > > On 18/05/10 15:28, Daniel Pittman wrote: > >> Toby Corkindale writes: > >> > >>> After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs > >>> Scala in more than just a trivial case. > >>> > >>> I setup a test to read in a large CSV file, performing some minor numeric and > >>> text manipulation upon each row, and outputing the results. > >> > >> When you say "large", do you mean 1MB, 10MB, 1GB, 1TB? > > > > Uh, 10,000 lines + header. How many megabytes is that, I mean a line could have 1 field or 10,000 fields, and a field could be 1 bytes or 1MB! I have a C (actually brace) program which loads and indexes a 22,000 line 630KB file, a Tagalog - English dictionary, in 0.020 seconds on my wimpy little VPS. It's not TSV it's records of key: value pairs (like mail headers) but if anything it's harder to parse than CSV because I have to look up the keys. Almost half of the 0.020 seconds is the time to fork and exec! ("hello world" takes ~ 0.008s) and it's not a very efficent implementation. Even in pure perl I can do something similar in under 0.1 seconds. So I'm guessing your CSV file is a lot bigger than that? I tried my program with a larger file, it can load and index a 63MB file in 1.5 seconds (when Linux has cached the file). Sam From toby.corkindale at strategicdata.com.au Tue May 18 19:11:52 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 19 May 2010 12:11:52 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <20100519020154.GB22403@nipl.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> Message-ID: <4BF348E8.7020303@strategicdata.com.au> On 19/05/10 12:01, Sam Watkins wrote: > On Tue, May 18, 2010 at 04:52:40PM +1000, Daniel Pittman wrote: >> Toby Corkindale writes: >>> On 18/05/10 15:28, Daniel Pittman wrote: >>>> Toby Corkindale writes: >>>> >>>>> After the last Perlmongers meeting I was curious to benchmark Perl vs Go vs >>>>> Scala in more than just a trivial case. >>>>> >>>>> I setup a test to read in a large CSV file, performing some minor numeric and >>>>> text manipulation upon each row, and outputing the results. >>>> >>>> When you say "large", do you mean 1MB, 10MB, 1GB, 1TB? >>> >>> Uh, 10,000 lines + header. > > How many megabytes is that, I mean a line could have 1 field or 10,000 fields, > and a field could be 1 bytes or 1MB! Actually, I was wrong, I was using 100,000 lines per "small" file, 1,000,000 lines per "medium" file and 10m lines per "big" file. The big file is 260 MByte. I was also using a poor CSV implementation in the Scala version, and I've now replaced it with a Java CSV engine I found. (Doesn't seem to be any pure-Scala implementations yet.) The updated times are: small file: Perl - 1.089 secs Scala - 1.857 secs Go - 1.682 secs big file: Perl - 111.3 s Scala - 89.05 s Go - 154.3 s > I have a C (actually brace) program which loads and indexes a 22,000 line 630KB > file, a Tagalog - English dictionary, in 0.020 seconds on my wimpy little VPS. > It's not TSV it's records of key: value pairs (like mail headers) but if > anything it's harder to parse than CSV because I have to look up the keys. > > Almost half of the 0.020 seconds is the time to fork and exec! > ("hello world" takes ~ 0.008s) and it's not a very efficent implementation. > Even in pure perl I can do something similar in under 0.1 seconds. So I'm > guessing your CSV file is a lot bigger than that? Would you like to submit a brace version of my simplistic benchmark? Then we can compare apples with apples. It'll probably eat the others alive; so would a C implementation, I'm guessing - but then again, maybe not - the Scala version is looking pretty snappy now. From daniel at rimspace.net Tue May 18 19:26:58 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Wed, 19 May 2010 12:26:58 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <4BF348E8.7020303@strategicdata.com.au> (Toby Corkindale's message of "Wed, 19 May 2010 12:11:52 +1000") References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> Message-ID: <878w7gwrrh.fsf@rimspace.net> Toby Corkindale writes: > On 19/05/10 12:01, Sam Watkins wrote: >> On Tue, May 18, 2010 at 04:52:40PM +1000, Daniel Pittman wrote: >>> Toby Corkindale writes: >>>> On 18/05/10 15:28, Daniel Pittman wrote: >>>>> Toby Corkindale writes: [...] >> I have a C (actually brace) program which loads and indexes a 22,000 line 630KB >> file, a Tagalog - English dictionary, in 0.020 seconds on my wimpy little VPS. >> It's not TSV it's records of key: value pairs (like mail headers) but if >> anything it's harder to parse than CSV because I have to look up the keys. >> >> Almost half of the 0.020 seconds is the time to fork and exec! >> ("hello world" takes ~ 0.008s) and it's not a very efficent implementation. >> Even in pure perl I can do something similar in under 0.1 seconds. So I'm >> guessing your CSV file is a lot bigger than that? > > Would you like to submit a brace version of my simplistic benchmark? Then we > can compare apples with apples. > It'll probably eat the others alive; so would a C implementation, I'm guessing > - but then again, maybe not - the Scala version is looking pretty snappy > now. Interestingly, it looks like the JVM (and LLVM) is sometimes starting to live up to the initial promise of JIT-style development: that your optimizing compiler can do *better* than static compilation, because it has the real world to watch.[1] I don't know it will do better, but I suspect that as the files grow larger the Scala / JVM version will manage to mostly maintain parity with the C (or Brace) version of the code.[2] On a Perl related note, I was interested to see that someone is looking to wire the LLVM engine to the Perl5 VM. Initial results suggested little gain when Perl itself was compiled with GCC and the Perl bytecodes emitted to LLVM. Theoretically, though, if Perl was LLVM-compiled and retained the IL version you could see LLVM machine-compile with the C-level functions inlined. Daniel Footnotes: [1] See also the fashion for trace-compilation of JavaScript. [2] ...unless the C version cheats and skips all the memory management overhead, by not freeing anything 'til the end of the process or something exciting like that. :) -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From sam at nipl.net Tue May 18 20:52:25 2010 From: sam at nipl.net (Sam Watkins) Date: Wed, 19 May 2010 03:52:25 +0000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <878w7gwrrh.fsf@rimspace.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> Message-ID: <20100519035225.GD22403@nipl.net> On Wed, May 19, 2010 at 12:26:58PM +1000, Daniel Pittman wrote: > [2] ...unless the C version cheats and skips all the memory management > overhead, by not freeing anything 'til the end of the process or > something exciting like that. :) My test program keeps all the data in memory, maybe this is different from what you have in mind. If I wanted to read each record then discard it, I could use resizable buffers to hold each record or field, and avoid (or 99.9% avoid) having to call malloc or free during the loop. For a C version that keeps all data in memory, it could allocate slabs of memory, there's no need to allocate each string separately. That said, my test program I mentioned earlier does call strdup (malloc) for each string it keeps. I don't think it calls free at all, at the moment. If an app wants to keep everything in memory, I would be inclined to allocate space for the entire file, read it all in, break it up and index it. That would require only one or two alloc/free pairs; this (or mmap private) is how I would write a C program if I was optimizing it for speed, and this is how I allocate memory in the next version of my brace translator, which is very fast. The only problem with doing this is that allocating large chunks can supposedly lead to memory fragmentation, or rather memory fragmentation can lead to malloc failures for large chunks. In practise I have never observed this problem, it might affect small embedded systems, but such systems probably don't have the resources to load the whole file at once anyway, and should use a streaming model. It is also less of a problem with computers / operation systems, having virtual memory, such as every PC and my phone. I regard fragmentation / large blocks problem as a problem with the malloc model of allocation: it should be possible for the system to automatically relocate objects if needed. This would entail using indexes and relative pointers, I believe it would not be a significant performance hit for most apps, given that the allocator can only move stuff when you invoke it. Sam From daniel at rimspace.net Tue May 18 22:20:28 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Wed, 19 May 2010 15:20:28 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <20100519035225.GD22403@nipl.net> (Sam Watkins's message of "Wed, 19 May 2010 03:52:25 +0000") References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> Message-ID: <871vd8v55v.fsf@rimspace.net> Sam Watkins writes: > On Wed, May 19, 2010 at 12:26:58PM +1000, Daniel Pittman wrote: > >> [2] ...unless the C version cheats and skips all the memory management >> overhead, by not freeing anything 'til the end of the process or >> something exciting like that. :) > > My test program keeps all the data in memory, maybe this is different from > what you have in mind. Well, not entirely: I just think it will make the performance curve of your application look poor once it gets asked to process a 4GB CSV file, while languages that do free unused memory will do a little better. :) [...] > That said, my test program I mentioned earlier does call strdup (malloc) for > each string it keeps. I don't think it calls free at all, at the moment. That is what I meant by "cheats", because you are now comparing JVM-with-free to C-without-free, which is not apples-to-apples. For small data sets this is a good trade-off, and it stops paying off as data sizes grow. [...] > I regard fragmentation / large blocks problem as a problem with the malloc > model of allocation: it should be possible for the system to automatically > relocate objects if needed. This would entail using indexes and relative > pointers, I believe it would not be a significant performance hit for most > apps, given that the allocator can only move stuff when you invoke it. It only requires indirect pointers, like the "handle" model that various platforms have used over the years. The cost is smaller than you might imagine. Alternately, use a tagged object implementation and update the source references when you promote objects, or use a generational GC and get the same sort of benefits without the cost of relocation at all. Daniel Ideally, structure your language so you can use *any* of those techniques, or even "never free anything", depending on the dynamic behaviour of the application. Which, not coincidentally, VM systems can do better than static code systems. :) -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From sam at nipl.net Wed May 19 19:36:15 2010 From: sam at nipl.net (Sam Watkins) Date: Thu, 20 May 2010 02:36:15 +0000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <871vd8v55v.fsf@rimspace.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> Message-ID: <20100520023615.GA5321@nipl.net> On Wed, May 19, 2010 at 03:20:28PM +1000, Daniel Pittman wrote: > Well, not entirely: I just think it will make the performance curve of your > application look poor once it gets asked to process a 4GB CSV file, while > languages that do free unused memory will do a little better. :) Well it depends whether you actually need to keep all the data in memory. If the task is to "load" a data file, such as my dictionary, for subsequent random access, I want to keep it all in memory. Perhaps storage can be saved by "interning" common strings or some fast compression scheme. If you really need to keep 4GB of data in RAM for fast random access, you basically need more than 4GB of RAM. Stream processing is a different thing. Nicely written C performs much better than any interpreted language or bytecode/JIT language for both stream processing and loading. I don't know of any compiled language that performs better than well-written C for many cases, because C is close to the metal, and allows the programmer to do things their own way. For example with stream processing, I commonly use a single resizable buffer for each line, so there is no need to free anything inside the loop. You said that I am cheating by not freeing anything, but if I write my C (or brace) code well, there is no need to malloc or free anything except perhaps a single line buffer, and maybe realloc it a couple times if it started off too small. Going nuts with malloc and free for every field in a file is almost always a bad way to program in C. In Java, perl, python, etc., you really have little or no choice about that, there's no nice way to do it as efficiently as in C, you need to use an object / buffer for each string. So I suspect your Java / perl program will run maybe 10 times slower than a well-written C, brace or go program. Sam From toby.corkindale at strategicdata.com.au Wed May 19 20:06:44 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Thu, 20 May 2010 13:06:44 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <20100520023615.GA5321@nipl.net> References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> Message-ID: <4BF4A744.2080407@strategicdata.com.au> On 20/05/10 12:36, Sam Watkins wrote: > On Wed, May 19, 2010 at 03:20:28PM +1000, Daniel Pittman wrote: >> Well, not entirely: I just think it will make the performance curve of your >> application look poor once it gets asked to process a 4GB CSV file, while >> languages that do free unused memory will do a little better. :) > > Well it depends whether you actually need to keep all the data in memory. > If the task is to "load" a data file, such as my dictionary, for subsequent > random access, I want to keep it all in memory. Perhaps storage can be saved > by "interning" common strings or some fast compression scheme. If you really > need to keep 4GB of data in RAM for fast random access, you basically need more > than 4GB of RAM. > > Stream processing is a different thing. Nicely written C performs much better > than any interpreted language or bytecode/JIT language for both stream > processing and loading. I don't know of any compiled language that performs > better than well-written C for many cases, because C is close to the metal, and > allows the programmer to do things their own way. > > For example with stream processing, I commonly use a single resizable buffer > for each line, so there is no need to free anything inside the loop. You said > that I am cheating by not freeing anything, but if I write my C (or brace) code > well, there is no need to malloc or free anything except perhaps a single line > buffer, and maybe realloc it a couple times if it started off too small. Going > nuts with malloc and free for every field in a file is almost always a bad way > to program in C. In Java, perl, python, etc., you really have little or no > choice about that, there's no nice way to do it as efficiently as in C, you > need to use an object / buffer for each string. So I suspect your Java / perl > program will run maybe 10 times slower than a well-written C, brace or go > program. So, are you going to provide us with a well-written Brace program that performs the same task as the Go/Scala/Perl implementations then? It's very simple, should only take you 5 minutes if you're familiar with your language. Only requirements are: You perform "good" practice as far as closing files and freeing memory goes - by which I mean that you do eventually close the file and free memory, and your program should not grow to infinite memory size if the input file keeps growing. So you can use a static buffer for every row of the CSV if you like. Given the same input file, your output should match the Perl output. Good luck, Toby From daniel at rimspace.net Wed May 19 21:32:32 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Thu, 20 May 2010 14:32:32 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <20100520023615.GA5321@nipl.net> (Sam Watkins's message of "Thu, 20 May 2010 02:36:15 +0000") References: <4BF20E43.3010503@strategicdata.com.au> <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> Message-ID: <87fx1n8a73.fsf@rimspace.net> Sam Watkins writes: > On Wed, May 19, 2010 at 03:20:28PM +1000, Daniel Pittman wrote: > >> Well, not entirely: I just think it will make the performance curve of your >> application look poor once it gets asked to process a 4GB CSV file, while >> languages that do free unused memory will do a little better. :) > > Well it depends whether you actually need to keep all the data in memory. That is a fair point. :) [...] > Stream processing is a different thing. Nicely written C performs much > better than any interpreted language or bytecode/JIT language for both > stream processing and loading. Well, generally. Hand-tuned assembly code often beats the portable assembly of C at the same job, too. I don't know I would stick with the statement forever, though, especially if the stream processing was complex and had many decision points. > I don't know of any compiled language that performs better than well-written > C for many cases, because C is close to the metal, and allows the programmer > to do things their own way. ...and, just to check here, you are only counting "C that is written well", not "C in the hands of a random programmer", right? > For example with stream processing, I commonly use a single resizable buffer > for each line, so there is no need to free anything inside the loop. You > said that I am cheating by not freeing anything, but if I write my C (or > brace) code well, there is no need to malloc or free anything except perhaps > a single line buffer, and maybe realloc it a couple times if it started off > too small. Yeah, and your VM language is going to do the same thing: use a static buffer, get static buffer performance. Witness, for example, Perl. :) > Going nuts with malloc and free for every field in a file is almost always a > bad way to program in C. In Java, perl, python, etc., you really have > little or no choice about that, there's no nice way to do it as efficiently > as in C, you need to use an object / buffer for each string. Nonsense: my $x = '.' x 10000; # preallocate sysread(FH, $x, 5000); # ...and use that storage Feel free to verify my claims, if you wish. :) Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From alfiejohn at gmail.com Thu May 20 05:26:47 2010 From: alfiejohn at gmail.com (Alfie John) Date: Thu, 20 May 2010 22:26:47 +1000 Subject: [Melbourne-pm] Still performing well.. In-Reply-To: <87fx1n8a73.fsf@rimspace.net> References: <4BF20E43.3010503@strategicdata.com.au> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <87fx1n8a73.fsf@rimspace.net> Message-ID: On Thu, May 20, 2010 at 2:32 PM, Daniel Pittman wrote: > my $x = '.' x 10000; # preallocate > sysread(FH, $x, 5000); # ...and use that storage > C-C-C-Combo Breaker. As you can see from above, always know what your system is doing. In Abrash's Zen of Code Optimization, he shows that simple C can outperform simple assembly. I hear you say that's impossible because assembly is *closer* to the bare metal than C. In fact it *is* the bare metal! Later in the book, after you realise what the CPU and memory bus is doing, what Daniel said comes closer to the truth: you (almost) can't beat hand-tuned assembly. But again, it all depends on what you're doing. For equivalent code paths of different languages, if nothing is firing off in the background like GC or you're not thrashing your language's malloc etc then is all O(1), so who cares. Once performance comparisons get to O(1) for different languages, then I would rather optimise my typing and comprehension (i.e. easiest language for the job). Once again. It all depends on what you're doing. If you run your program on different architectures like an i7, amd64, P4, core2duo, then you'll probably see huge differences on where the performance spectrum languages sit. You'd be surprised what nasties might bite. >I don't know of any compiled language that performs better than well-written > C for many cases, because C is close to the metal, and allows the programmer > to do things their own way. And as for that one: FORTRAN. Alfie -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam at nipl.net Thu May 20 21:05:10 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 04:05:10 +0000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <4BF4A744.2080407@strategicdata.com.au> References: <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> Message-ID: <20100521040510.GF32217@nipl.net> hi Toby, > So, are you going to provide us with a well-written Brace program that > performs the same task as the Go/Scala/Perl implementations then? Ok, I did write a brace / C program that does the same thing. It's not terribly well-written but it doesn't allocate memory in its inner loop. Brace translates directly to C, so it performs the same as an equivalent C program using the same techniques. > It's very simple, should only take you 5 minutes if you're familiar with > your language. lol, I wrote the CSV parser too so it took a little longer than that! it might be useful for me in future though. > Only requirements are: > You perform "good" practice as far as closing files and freeing memory > goes - by which I mean that you do eventually close the file and free > memory, and your program should not grow to infinite memory size if the > input file keeps growing. Ok, in this case I don't need to do that explicitly in brace. Two of the block structures I used take care of it for me. 'F_in' opens and closes a file. 'eachline' reads all the lines from a file using a single buffer, which it frees at the end. > So you can use a static buffer for every row of the CSV if you like. 'eachline' does use a single resizable buffer. > Given the same input file, your output should match the Perl output. There were a few discrepancies originally because I had used float rather than double. I changed it to use double (num), and now it does match exactly. Here are the times on my VPS: $ ./read.pl data.csv >/dev/null Code took 2.23198 wallclock secs ( 2.19 usr + 0.00 sys = 2.19 CPU) @ 0.46/s (n=1) $ ./read.b data.csv >/dev/null Code took: 0.227998 So my brace / C version is nearly 10 times faster than perl, even though the perl version uses Text::CSV_XS, which is written in C! The difficulty for Text::CSV_XS is that it must allocate and use perl data structures. I don't doubt that Go would perform similarly to C or brace, if the CSV parser were written in the same way. I repeated the test with Text::CSV_PP, which uses pure perl rather than C / XS to parse the CSV, and might be a better comparison of the two langauges for the stated task. I get this result: $ ./read-pp.pl data.csv >/dev/null Code took 12.1239 wallclock secs (11.86 usr + 0.01 sys = 11.87 CPU) @ 0.08/s (n=1) I'm surprised that the pure-perl version is only about 5 times slower than the XS version, that's a real credit to perl. However the brace / C version is 53 times faster than the perl version. My brace program is not low-level programming. It uses my library libb and is shorter and more high-level than the other programs shown. It uses medium-level generic data structures: resizable string buffers, and vectors. If I avoided these and used low-level C arrays, it could run much faster. But it's already twice as fast as GNU wc on the same data. Actually that is because it's using UTF-8, in the C locale it's 6 times faster than my brace program. If the brace program / my libs were optimized for speed I think it could approach that speed for parsing CSV. For myself, I prefer to use TSV format with C-style escapes \t \n \\. Unlike CSV, that works nicely with awk, cut and other unix tools. I'll add the CSV parser to my brace library so I can use it again. Sam Watkins Here is the main program in brace, as you can see it's about half the size of the perl version and 1/4 the size of the scala version. Including the CSV parser it's about the same size as the scala version! brace is work in progress, so this program could be improved as I improve the library. (Accessing objects and vector elements is a bit ugly at the moment.) I can provide the stand-alone C translation of the brace version if you'd like to test it. #!/usr/bin/env bx use b cstr usage[] = "input_file", NULL Main(): bm_block("Code took"): args(cstr, file) new(v, vec, cstr, 3) F_in(file): rl() eachline(l): vec_clear(v) split_csv(v, l) cstr name = *(cstr*)v(v, 0) int i = atoi(*(cstr*)v(v, 1)) num f = atof(*(cstr*)v(v, 2)) sf("%s is %.02f", name, i*f) Here is the CSV parser, this will go in my library libb: def split_csv(v, l): split_csv(v, l, my(p), my(f), my(c), my(o)) def split_csv(v, l, p, f, c, o): char *p = l char *f = p repeat: char c = *p if c == '\0': vec_push(v, f) break eif c == ',': *p++ = '\0' vec_push(v, f) f = p eif c == '"': f = ++p csv_read_quoted(p, c, o) else: ++p def csv_read_quoted(p, c, o): repeat: char c = *p if c == '\0': error("csv: mismatched quotes") eif c == '"': if p[1] == '"': char *o = p+1 p += 2 csv_read_quoted_copy(o, p, c) break eif among(p[1], ',', '\0'): *p++ = '\0' break else: error("csv: quote inside field must be followed by quote, comma or newline") else: ++p def csv_read_quoted_copy(o, i, c): repeat: char c = *i if c == '\0': error("csv: mismatched quotes") eif c == '"': if i[1] == '"': ++i eif among(i[1], ',', '\0'): *o++ = '\0' ++i break else: error("csv: quote inside field must be followed by quote, comma or newline") else: *o++ = *i++ From toby.corkindale at strategicdata.com.au Thu May 20 21:39:21 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 21 May 2010 14:39:21 +1000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <20100521040510.GF32217@nipl.net> References: <87r5l9hj8b.fsf@rimspace.net> <4BF22DFF.6090809@strategicdata.com.au> <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> Message-ID: <4BF60E79.6010307@strategicdata.com.au> On 21/05/10 14:05, Sam Watkins wrote: > hi Toby, > >> So, are you going to provide us with a well-written Brace program that >> performs the same task as the Go/Scala/Perl implementations then? > > Ok, I did write a brace / C program that does the same thing. It's not > terribly well-written but it doesn't allocate memory in its inner loop. Brace > translates directly to C, so it performs the same as an equivalent C program > using the same techniques. > >> It's very simple, should only take you 5 minutes if you're familiar with >> your language. > > lol, I wrote the CSV parser too so it took a little longer than that! > it might be useful for me in future though. > >> Only requirements are: >> You perform "good" practice as far as closing files and freeing memory >> goes - by which I mean that you do eventually close the file and free >> memory, and your program should not grow to infinite memory size if the >> input file keeps growing. > > Ok, in this case I don't need to do that explicitly in brace. Two of the block > structures I used take care of it for me. 'F_in' opens and closes a file. > 'eachline' reads all the lines from a file using a single buffer, which it > frees at the end. > >> So you can use a static buffer for every row of the CSV if you like. > > 'eachline' does use a single resizable buffer. > >> Given the same input file, your output should match the Perl output. > > There were a few discrepancies originally because I had used float rather than > double. I changed it to use double (num), and now it does match exactly. > > Here are the times on my VPS: > > $ ./read.pl data.csv>/dev/null > Code took 2.23198 wallclock secs ( 2.19 usr + 0.00 sys = 2.19 CPU) @ 0.46/s (n=1) > > $ ./read.b data.csv>/dev/null > Code took: 0.227998 > > So my brace / C version is nearly 10 times faster than perl, even though the > perl version uses Text::CSV_XS, which is written in C! The difficulty for > Text::CSV_XS is that it must allocate and use perl data structures. > > I don't doubt that Go would perform similarly to C or brace, if the CSV parser > were written in the same way. > > I repeated the test with Text::CSV_PP, which uses pure perl rather than C / XS > to parse the CSV, and might be a better comparison of the two langauges for the > stated task. I get this result: > > $ ./read-pp.pl data.csv>/dev/null > Code took 12.1239 wallclock secs (11.86 usr + 0.01 sys = 11.87 CPU) @ 0.08/s (n=1) > > I'm surprised that the pure-perl version is only about 5 times slower than the > XS version, that's a real credit to perl. However the brace / C version is 53 > times faster than the perl version. > > My brace program is not low-level programming. It uses my library libb and is > shorter and more high-level than the other programs shown. It uses > medium-level generic data structures: resizable string buffers, and vectors. > If I avoided these and used low-level C arrays, it could run much faster. But > it's already twice as fast as GNU wc on the same data. Actually that is > because it's using UTF-8, in the C locale it's 6 times faster than my brace > program. If the brace program / my libs were optimized for speed I think it > could approach that speed for parsing CSV. > > For myself, I prefer to use TSV format with C-style escapes \t \n \\. > Unlike CSV, that works nicely with awk, cut and other unix tools. > I'll add the CSV parser to my brace library so I can use it again. > > Sam Watkins > > > > Here is the main program in brace, as you can see it's about half the size of > the perl version and 1/4 the size of the scala version. Including the CSV > parser it's about the same size as the scala version! brace is work in > progress, so this program could be improved as I improve the library. > (Accessing objects and vector elements is a bit ugly at the moment.) > > I can provide the stand-alone C translation of the brace version if you'd like > to test it. .. that might be neccessary, as I just tried compiling Brace on my computer, and it fails to build: .all.c: In function ?gr__mitshm_fault_h?: .all.c:16876: error: ?X_ShmAttach? undeclared (first use in this function) .all.c:16876: error: (Each undeclared identifier is reported only once .all.c:16876: error: for each function it appears in.) That thing is declared in /usr/include/X11/extensions/shmproto.h so maybe it's just not being included in your code somewhere? From sam at nipl.net Thu May 20 23:42:18 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 06:42:18 +0000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <4BF60E79.6010307@strategicdata.com.au> References: <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> Message-ID: <20100521064218.GG32217@nipl.net> > .. that might be neccessary, as I just tried compiling Brace on my > computer, and it fails to build: What OS / version computer that? It has built on various systems at various times including Windows, *bsd and GNU hurd, but the only platforms that definitely work at any given time are Debian and Ubuntu Linux, because that's what I use. Thanks for trying it anyway! I think I'll just write a faster C version instead of sending generated C code, the C code generated by brace can be a bit ugly. I hope to improve that in future! Sam From toby.corkindale at strategicdata.com.au Thu May 20 23:44:55 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 21 May 2010 16:44:55 +1000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <20100521064218.GG32217@nipl.net> References: <87mxvxhfbb.fsf@rimspace.net> <20100519020154.GB22403@nipl.net> <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> Message-ID: <4BF62BE7.9080009@strategicdata.com.au> On 21/05/10 16:42, Sam Watkins wrote: >> .. that might be neccessary, as I just tried compiling Brace on my >> computer, and it fails to build: > > What OS / version computer that? It has built on various systems at various > times including Windows, *bsd and GNU hurd, but the only platforms that > definitely work at any given time are Debian and Ubuntu Linux, because that's > what I use. Thanks for trying it anyway! Ubuntu's current release - 10.04 (Lucid Lynx). That's GCC 4.4, although the error didn't seem to be a gcc error, just a lack of including the right headers I think. > I think I'll just write a faster C version instead of sending generated C code, > the C code generated by brace can be a bit ugly. I hope to improve that in > future! OK, and there might even be some existing CSV libraries you can use this time :) From sam at nipl.net Thu May 20 23:52:41 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 06:52:41 +0000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <4BF62BE7.9080009@strategicdata.com.au> References: <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> Message-ID: <20100521065241.GH32217@nipl.net> >> What OS / version computer that?> > Ubuntu's current release - 10.04 (Lucid Lynx). That's odd, I use that too, not sure if I've built brace on it yet though. Perhaps you don't have all the X headers installed. There is a debian/control file there which mentions the packages you need. I'll get back to you. >> I think I'll just write a faster C version instead of sending generated C code, >> the C code generated by brace can be a bit ugly. I hope to improve that in >> future! > > OK, and there might even be some existing CSV libraries you can use this > time :) I can use existing CSV libraries for C with brace too, but I don't know how well they'll perform. Sam From toby.corkindale at strategicdata.com.au Thu May 20 23:54:01 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 21 May 2010 16:54:01 +1000 Subject: [Melbourne-pm] brace / C CSV reader, Re: Still performing well.. In-Reply-To: <20100521065241.GH32217@nipl.net> References: <4BF348E8.7020303@strategicdata.com.au> <878w7gwrrh.fsf@rimspace.net> <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> Message-ID: <4BF62E09.4010000@strategicdata.com.au> On 21/05/10 16:52, Sam Watkins wrote: >>> What OS / version computer that?> >> Ubuntu's current release - 10.04 (Lucid Lynx). > > That's odd, I use that too, not sure if I've built brace on it yet though. > Perhaps you don't have all the X headers installed. There is a debian/control > file there which mentions the packages you need. I'll get back to you. OK. I do have the right packages installed, I'm pretty sure of that - as I said, I can see the include files in /usr/include. From sam at nipl.net Fri May 21 01:23:26 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 08:23:26 +0000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <4BF62E09.4010000@strategicdata.com.au> References: <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> Message-ID: <20100521082326.GI32217@nipl.net> results first! Here's the leaderboard of CSV readers in various languages, compared to C, for a 100,000 line CSV file: C 1.00 brace 1.16 perl XS 11.33 (bad) go 17.50 scala 19.32 perl 62.51 I wrote a C version, based on my brace version (the 1.00 above). It's 14% faster than the brace version, because its not using fancy "vectors" or "buffers", just arrays. It's less flexible though: has a limited line length and number of fields, and does not check for overflow at the moment. I could fix that with a minimal performance hit, but that's what those brace data structures are for anyway, I can't be bothered doing it all again with realloc right now. $ ./read-c data.csv >/dev/null Code took 0.195998 $ ./read.b data.csv > /dev/null Code took: 0.227998 $ ./read.pl data.csv >/dev/null Code took 2.21998 wallclock secs ( 2.17 usr + 0.01 sys = 2.18 CPU) @ 0.46/s (n=1) $ ./read-pp.pl data.csv >/dev/null Code took 12.2519 wallclock secs (11.99 usr + 0.00 sys = 11.99 CPU) @ 0.08/s (n=1) People can add languages / programs to this table just by comparing speed to the perl or C version running on their test machine, no need to run every version. (I didn't try to run the scala or go versions yet.) How about a parallel version that runs on the GPU? ;) I'm not saying "go" is bad, I'm saying the go code used must have been bad because go is definitely not that much slower than C. I'd like to see what JavaScript + V8 and JavaScript + Tracemonkey could do here, and how Java + android javac would do. I guessing these three have better performance than perl XS but at least 5 times slower than C. I'd also like to see how a "malloc and free everything" C version would compare, I'm guessing about 3 or 4 times slower. I should fix the C / brace version to use fread not fgets for better correctness (allowing \n in quoted fields) and maybe to go a little faster. The printf output, even going to /dev/null, took more than half the time for the C code; so if we are testing just CSV reading, C is actually "more faster" than my figures indicate. $ ./read-quiet-c data.csv Code took 0.080000 The perl output apparently takes a large chunk of the time for perl XS too: $ ./read-quiet.pl data.csv Code took 1.48799 wallclock secs ( 1.47 usr + 0.00 sys = 1.47 CPU) @ 0.68/s (n=1) $ ./read-pp-quiet.pl data.csv Code took 11.3239 wallclock secs (11.13 usr + 0.00 sys = 11.13 CPU) @ 0.09/s (n=1) Still, for our task of CSV reading, excluding the calculation and output, a rough bit of C code is 18.60 times faster than Perl XS (Text::CSV_XS) and 141.55 times faster than pure Perl (Text::CSV_PP). Perl+XS may be able to compete with Scala but neither can compete with C (for speed). sed|awk performs pretty well: $ time < data.csv sed 's/,/ /g; s/""/"/g; s/"//g;' | awk '{print $1" is "$2*$3}' >/dev/null real 0m1.228s It's also a lot shorter (although not quite correct, silly CSV!). Using clean TSV instead of silly CSV, gawk achieves 0.428s and mawk achieves 0.268s which is getting pretty close to C. Toby Corkindale wrote: > OK. > I do have the right packages installed, I'm pretty sure of that - as I > said, I can see the include files in /usr/include. I'll try building brace at home later, if my laptop works, it's acting a bit flakey, I think a RAM module is loose or something. Sam From sam at nipl.net Fri May 21 01:28:01 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 08:28:01 +0000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <20100521082326.GI32217@nipl.net> References: <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> Message-ID: <20100521082801.GA30257@nipl.net> As usual, I forgot to attach the code. read.c is attached here! Sam On Fri, May 21, 2010 at 08:23:26AM +0000, Sam Watkins wrote: > results first! > > Here's the leaderboard of CSV readers in various languages, compared to C, > for a 100,000 line CSV file: > > C 1.00 > brace 1.16 > perl XS 11.33 > (bad) go 17.50 > scala 19.32 > perl 62.51 > > > I wrote a C version, based on my brace version (the 1.00 above). It's 14% > faster than the brace version -------------- next part -------------- A non-text attachment was scrubbed... Name: read.c Type: text/x-csrc Size: 2028 bytes Desc: not available URL: From toby.corkindale at strategicdata.com.au Fri May 21 01:38:24 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 21 May 2010 18:38:24 +1000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <20100521082326.GI32217@nipl.net> References: <20100519035225.GD22403@nipl.net> <871vd8v55v.fsf@rimspace.net> <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> Message-ID: <4BF64680.4040709@strategicdata.com.au> On 21/05/10 18:23, Sam Watkins wrote: > Here's the leaderboard of CSV readers in various languages, compared to C, > for a 100,000 line CSV file: > > C 1.00 > brace 1.16 > perl XS 11.33 > (bad) go 17.50 > scala 19.32 > perl 62.51 Umm, I think you're comparing the wrong results there; for the 100k file, Perl only takes 1.1 seconds for me. (However the C version only takes 0.11 seconds on that file!) For the "big" 10m row file, the C version takes 7.90 seconds on my testbed, which definitely takes it into the lead, by far! (The next fastest contender is over a minute, and perl takes 108 seconds) > I should fix the C / brace version to use fread not fgets for better > correctness (allowing \n in quoted fields) and maybe to go a little faster. > > The printf output, even going to /dev/null, took more than half the time for > the C code; so if we are testing just CSV reading, C is actually "more faster" > than my figures indicate. *nods* Someone else recently pointed this out -- the printf() is consuming the majority of the time, apparently due to it's flushing. I'm going to modify the tests and give them another shot once I've eliminated the buffer flushing.. From sam at nipl.net Fri May 21 01:54:37 2010 From: sam at nipl.net (Sam Watkins) Date: Fri, 21 May 2010 08:54:37 +0000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <4BF64680.4040709@strategicdata.com.au> References: <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> <4BF64680.4040709@strategicdata.com.au> Message-ID: <20100521085437.GB30543@nipl.net> On Fri, May 21, 2010 at 06:38:24PM +1000, Toby Corkindale wrote: > On 21/05/10 18:23, Sam Watkins wrote: > > Here's the leaderboard of CSV readers in various languages, compared > to C, > > for a 100,000 line CSV file: > > > > C 1.00 > > brace 1.16 > > perl XS 11.33 > > (bad) go 17.50 > > scala 19.32 > > perl 62.51 > > > Umm, I think you're comparing the wrong results there; for the 100k > file, Perl only takes 1.1 seconds for me. > (However the C version only takes 0.11 seconds on that file!) Yes, those figures were relative to the C version, that's why C is 1.00. 1.1 / 0.11 = 10 means perl XS gets a score of about 10 on your machine - it's 10 times slower than C. By the way, I compiled it something like this: gcc -pedantic -std=gnu99 -Wall -Wextra -O2 -o read-c read.c > For the "big" 10m row file, the C version takes 7.90 seconds on my > testbed, which definitely takes it into the lead, by far! (The next > fastest contender is over a minute, and perl takes 108 seconds) Yes, the C version is about 10 times faster than the next fastest (excluding brace, which is essentially just C anyway). > I'm going to modify the tests and give them another shot once I've > eliminated the buffer flushing.. ok, cool. When I just commented out the printf I'm not sure how much other stuff the C compiler might have been discarding... "hey, he's not using this, no need to calculate it!" Sam From toby.corkindale at strategicdata.com.au Fri May 21 02:21:16 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Fri, 21 May 2010 19:21:16 +1000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <20100521085437.GB30543@nipl.net> References: <20100520023615.GA5321@nipl.net> <4BF4A744.2080407@strategicdata.com.au> <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> <4BF64680.4040709@strategicdata.com.au> <20100521085437.GB30543@nipl.net> Message-ID: <4BF6508C.50404@strategicdata.com.au> On 21/05/10 18:54, Sam Watkins wrote: > On Fri, May 21, 2010 at 06:38:24PM +1000, Toby Corkindale wrote: >> On 21/05/10 18:23, Sam Watkins wrote: >>> Here's the leaderboard of CSV readers in various languages, compared >> to C, >>> for a 100,000 line CSV file: >>> >>> C 1.00 >>> brace 1.16 >>> perl XS 11.33 >>> (bad) go 17.50 >>> scala 19.32 >>> perl 62.51 >> >> >> Umm, I think you're comparing the wrong results there; for the 100k >> file, Perl only takes 1.1 seconds for me. >> (However the C version only takes 0.11 seconds on that file!) > > Yes, those figures were relative to the C version, that's why C is 1.00. > > 1.1 / 0.11 = 10 means perl XS gets a score of about 10 on your machine > - it's 10 times slower than C. > > By the way, I compiled it something like this: > > gcc -pedantic -std=gnu99 -Wall -Wextra -O2 -o read-c read.c > >> For the "big" 10m row file, the C version takes 7.90 seconds on my >> testbed, which definitely takes it into the lead, by far! (The next >> fastest contender is over a minute, and perl takes 108 seconds) > > Yes, the C version is about 10 times faster than the next fastest > (excluding brace, which is essentially just C anyway). > >> I'm going to modify the tests and give them another shot once I've >> eliminated the buffer flushing.. > > ok, cool. When I just commented out the printf I'm not sure how much other > stuff the C compiler might have been discarding... "hey, he's not using this, > no need to calculate it!" Yeah! I had a bit of a flamewar on another list about that.. A guy insisted that I should comment out the printf() in the scala version, and then compare it's performance.. Uh.. but surely the intelligent JVM will optimise out /heaps/ of stuff if I did that.. and there's no way to compare the output either. The guy in question was yelling at me about how apparently I didn't know what I was doing, and was crap at testing performance. I didn't think it was that unreasonable to want all the tested programs to produce identical output! Sheesh. curiously, I've made the Perl version use buffered IO and it seems to be (very very slightly) slower than the original, not faster! How odd. I'm doing: my $output = IO::Handle->new->fdopen(fileno(STDOUT), 'w'); $output->autoflush(0); ... $outout->printf(...) ... $output->close; Does that seem right to everyone else? Swapping to unbuffered I/O for Scala brought the time down from 89 to 67 seconds on the biggest file.. and then using some performance improvements suggested by someone else, got it down to a tiny 26 seconds! (And under 1 seconds for the small file) Poor old Perl is looking very sorry for itself now, at 111 seconds :( From alfiejohn at gmail.com Fri May 21 03:39:14 2010 From: alfiejohn at gmail.com (Alfie John) Date: Fri, 21 May 2010 20:39:14 +1000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <4BF6508C.50404@strategicdata.com.au> References: <20100520023615.GA5321@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> <4BF64680.4040709@strategicdata.com.au> <20100521085437.GB30543@nipl.net> <4BF6508C.50404@strategicdata.com.au> Message-ID: On Fri, May 21, 2010 at 7:21 PM, Toby Corkindale < toby.corkindale at strategicdata.com.au> wrote: > curiously, I've made the Perl version use buffered IO and it seems to be > (very very slightly) slower than the original, not faster! How odd. > > I'm doing: > my $output = IO::Handle->new->fdopen(fileno(STDOUT), 'w'); > $output->autoflush(0); > ... > $outout->printf(...) > ... > $output->close; > > Does that seem right to everyone else? > Could it be because you're using the OO version? Strip out IO::Handle and autoflush by hand. Alfie -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam at nipl.net Sun May 23 16:23:40 2010 From: sam at nipl.net (Sam Watkins) Date: Sun, 23 May 2010 23:23:40 +0000 Subject: [Melbourne-pm] and the winner is C! (so far anyway, no big surprise) In-Reply-To: <4BF6508C.50404@strategicdata.com.au> References: <20100521040510.GF32217@nipl.net> <4BF60E79.6010307@strategicdata.com.au> <20100521064218.GG32217@nipl.net> <4BF62BE7.9080009@strategicdata.com.au> <20100521065241.GH32217@nipl.net> <4BF62E09.4010000@strategicdata.com.au> <20100521082326.GI32217@nipl.net> <4BF64680.4040709@strategicdata.com.au> <20100521085437.GB30543@nipl.net> <4BF6508C.50404@strategicdata.com.au> Message-ID: <20100523232340.GA7678@nipl.net> On Fri, May 21, 2010 at 07:21:16PM +1000, Toby Corkindale wrote: > curiously, I've made the Perl version use buffered IO and it seems to be > (very very slightly) slower than the original, not faster! How odd. Perl uses buffered IO by default. It doesn't seem to make that much difference though: $ ./read.pl data.csv >/dev/null Code took 2.16798 wallclock secs ( 2.14 usr + 0.01 sys = 2.15 CPU) @ 0.47/s (n=1) $ ./read-unbuf.pl data.csv >/dev/null Code took 2.43998 wallclock secs ( 2.28 usr + 0.14 sys = 2.42 CPU) @ 0.41/s (n=1) That one had a $|=1 in it, so it does a write after each line output. Sam From toby.corkindale at strategicdata.com.au Sun May 23 22:29:23 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 24 May 2010 15:29:23 +1000 Subject: [Melbourne-pm] Not performing so well.. Message-ID: <4BFA0EB3.100@strategicdata.com.au> So, I received some more optimisations for the Scala version.. After a while they descended into hideous, unmanageable code.. so I ignored though, and only kept things which kept the code seeming like that which a reasonable person might write. I also made some tweaks to the Perl version. With this, the results are now: Small file: Perl 0.744 seconds Scala 0.842 seconds Go: 1.55 seconds C: 0.083 seconds Medium file: Perl: 7.12 seconds Scala: 3.28 seconds Go: 15.1 seconds C: 0.780 seconds Big file: Perl: 71.2 seconds Scala: 23.9 seconds Go: 153 seconds C: 7.83 seconds I've improved the performance of the Perl test by doing stuff in a slightly less readable way, but hey, the Scala test is doing that now too.. However despite the large gains for the Perl version (114 down to 71 seconds), the Scala version has leaped ahead, from 115 to 24 seconds. Of possibly interesting note is the memory sizes: Scala: 214 MB Perl: 6 MB Go: 2 MB C: 0.5 MB Now, when is someone going to submit a Perl6 version? From daniel at rimspace.net Sun May 23 22:58:00 2010 From: daniel at rimspace.net (Daniel Pittman) Date: Mon, 24 May 2010 15:58:00 +1000 Subject: [Melbourne-pm] Not performing so well.. In-Reply-To: <4BFA0EB3.100@strategicdata.com.au> (Toby Corkindale's message of "Mon, 24 May 2010 15:29:23 +1000") References: <4BFA0EB3.100@strategicdata.com.au> Message-ID: <878w79deon.fsf@rimspace.net> Toby Corkindale writes: > So, I received some more optimisations for the Scala version.. [...] > Of possibly interesting note is the memory sizes: > Scala: 214 MB What happens to Scala performance with -Xmx128M, to force it to actually GC the garbage being produced? Daniel -- ? Daniel Pittman ? daniel at rimspace.net ? +61 401 155 707 ? made with 100 percent post-consumer electrons From toby.corkindale at strategicdata.com.au Sun May 23 23:15:18 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Mon, 24 May 2010 16:15:18 +1000 Subject: [Melbourne-pm] Not performing so well.. In-Reply-To: <878w79deon.fsf@rimspace.net> References: <4BFA0EB3.100@strategicdata.com.au> <878w79deon.fsf@rimspace.net> Message-ID: <4BFA1976.1000000@strategicdata.com.au> On 24/05/10 15:58, Daniel Pittman wrote: > Toby Corkindale writes: > >> So, I received some more optimisations for the Scala version.. > > [...] > >> Of possibly interesting note is the memory sizes: >> Scala: 214 MB > > What happens to Scala performance with -Xmx128M, to force it to actually GC > the garbage being produced? I just tried with 128M, 64M, 512M and without any options. In every case, it used 114M of ram, and took about the same length of time. I think I may have made a typo when i said 214M before. Also, that the -Xmx512M option isn't being passed through. Also that I should have written EXPORT in the script. Oh dear. Running that all again.. Now it's matching the sizes up, and there's maybe a fraction of a second difference.. Like 250ms or so.. it's hard to tell, since that's about the error margin for the tests anyway, but running them over and over I think the average is a tiny bit higher. That was with the 64M option, since the 128m was actually higher than it was using, just.. From jarich at perltraining.com.au Mon May 24 22:20:29 2010 From: jarich at perltraining.com.au (Jacinta Richardson) Date: Tue, 25 May 2010 15:20:29 +1000 Subject: [Melbourne-pm] The 2010 Perl Survey Message-ID: <4BFB5E1D.9020401@perltraining.com.au> The Perl Survey 2010 is now live. Its purpose is to better understand > the demographics and opinions of the Perl community. You can complete > the survey at http://survey.perlfoundation.org - it should take about > 10 to 15 minutes. Once you've done that, please let your relevant > friends and colleagues know about the survey so they can coplete it as > well. My aim is to get a response of over 1000 individuals, and to > run the survey (lightly adapted) every two or three years so we can > see how the community changes over time. The official announcement of > the survey is here: > http://news.perlfoundation.org/2010/05/grant-update-the-perl-survey-1.html From toby.corkindale at strategicdata.com.au Wed May 26 00:47:47 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Wed, 26 May 2010 17:47:47 +1000 Subject: [Melbourne-pm] Perl 5.12.1 hidden Message-ID: <4BFCD223.6030007@strategicdata.com.au> Anyone know why Perl.org talks about 5.12.0 as the latest version, on both the front page and www.perl.org/get.html ? From shlomif at iglu.org.il Wed May 26 01:03:27 2010 From: shlomif at iglu.org.il (Shlomi Fish) Date: Wed, 26 May 2010 11:03:27 +0300 Subject: [Melbourne-pm] Perl 5.12.1 hidden In-Reply-To: <4BFCD223.6030007@strategicdata.com.au> References: <4BFCD223.6030007@strategicdata.com.au> Message-ID: <201005261103.28268.shlomif@iglu.org.il> On Wednesday 26 May 2010 10:47:47 Toby Corkindale wrote: > Anyone know why Perl.org talks about 5.12.0 as the latest version, on > both the front page and www.perl.org/get.html ? I presume this is the case because it wasn't updated yet. See: http://www.perl.org/siteinfo.html I'm CCing this message to the Perl.org webmasters as specified on that page. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Interview with Ben Collins-Sussman - http://shlom.in/sussman God considered inflicting XSLT as the tenth plague of Egypt, but then decided against it because he thought it would be too evil. Please reply to list if it's a mailing list post - http://shlom.in/reply . From jarich at perltraining.com.au Wed May 26 22:45:55 2010 From: jarich at perltraining.com.au (Jacinta Richardson) Date: Thu, 27 May 2010 15:45:55 +1000 Subject: [Melbourne-pm] PyCon Australia 2010 program announced Message-ID: <4BFE0713.2040405@perltraining.com.au> For those multi-language programmers amongst us: --- Hi all, The program for PyCon Australia 2010, to be held at the Sydney Masonic Center over the weekend of June 26 and 27, has been posted. View the full list of presentations and the schedule at: http://pycon-au.org/2010/conference/ Register here: http://pycon-au.org/reg Richard Jones PyCon Australia 2010 Program Chair From toby.corkindale at strategicdata.com.au Wed May 26 22:49:25 2010 From: toby.corkindale at strategicdata.com.au (Toby Corkindale) Date: Thu, 27 May 2010 15:49:25 +1000 Subject: [Melbourne-pm] Talks for next meeting In-Reply-To: <4BEB99CA.3050304@strategicdata.com.au> References: <4BEB99CA.3050304@strategicdata.com.au> Message-ID: <4BFE07E5.4080407@strategicdata.com.au> Hi everyone, A couple of things about the upcoming meeting.. Can I get a confirmation from whoever would like to volunteer to give a talk on Padre? Otherwise I'll try and get myself up to speed on it soon.. For another suggested talk - how about "What's new in Perl 5.12"? Not only has 5.12.0 come out, but even 5.12.1 and we still haven't covered it, so it must be time! I'll volunteer to do this talk, unless someone else would like it? Cheers, Toby On 13/05/10 16:18, Toby Corkindale wrote: > Hi everyone, > As we discussed at the meeting last night, Paul and Jacinta will be away > for a few months, and thus organisation of the meetings falls to the > rest of us mere mortals for the next three meetings. > > Jacinta and Paul, thank you for always managing these meetings for as > long as I've subscribed to the Melbourne.pm list. > > > The next meeting will be on the 9th of June, and will be hosted by > Strategic Data: > > Strategic Data > Level 2 > 51-55 Johnston Street > Fitzroy 3065 > > They have kindly offered to provide some snacks and pizza! > > > Talks: > I think we had a volunteer last night for a talk demonstrating Padre, > the development environment for Perl, in Perl? > > I'm sure we'll get a couple of lightning talks again.. > > Are there any other suggestions or volunteers for talks? Mail me any > time in the next few weeks if you come up with anything, or I'll find > something else to ramble on about again :) > > Cheers, > Toby > > > _______________________________________________ > Melbourne-pm mailing list > Melbourne-pm at pm.org > http://mail.pm.org/mailman/listinfo/melbourne-pm -- Strategic Data Pty Ltd Ph: 03 9340 9000