From tom at eborcom.com  Wed Jul  2 13:32:52 2008
From: tom at eborcom.com (Tom Hukins)
Date: Wed, 2 Jul 2008 21:32:52 +0100
Subject: Processing Data Partitions in Perl
Message-ID: <20080702203251.GA13126@eborcom.com>

Hi all,

Recently I've found myself working with various SQL result sets of the
following form:
  id  name     location
  --  ----     --------
   1  Cuba     NB
   2  New Inn  NB
   3  Spoons   CMK
   4  Vaults   Stony
   5  Crown    Stony

I want to work with these result sets as groups of records with the same
location.

Chapter 19 of SQL For Smarties (a wonderful book for anyone doing
anything non-trivial with SQL databases) tells me that I should refer to
such groups of records as partitions.

Nosing through CPAN led me to the partition implementation in
List::MoreUtils.  I quickly put something together along the following
lines:
  my @pub = $sth->fetchall_arrayref({});
  my $i = 0;
  my $prev;
  my @pub_by_location = part {
      ! defined $prev || $_->{location} ne $prev->{location} && $i++;
      $prev = $_;
      $i;
  } @pub;

And this gave me something like:
  [
    { id => 1, name => 'Cuba', location => 'NB' },
    { id => 2, name => 'New Inn', location => 'NB' },
  ],
  [
    { id => 3, name => 'Spoons', location => 'CMK' },
  ],
  [
    { id => 4, name => 'Vaults', location => 'Stony' },
    { id => 5, name => 'Crown', location => 'Stony' },
  ]

So I was happy.  But I was less happy about the idea of slurping
everything from the database into one array.  It doesn't matter much for
the five records in my contrived example, but it would for larger
datasets.

For large sets, I would like to retrieve all the records in
each partition, perform some operation on them, then forget about that
partition and move on to the next one.

So I wrote myself a little subroutine to do just this:
  sub make_partition_sub {
      my $record_sub = shift;
      my $field = shift;

      my (@group, $sub);

      $sub = {
          my $cur = $record_sub->();
	  return if (! defined $cur && ! scalar @group);

	  if (! defined $cur || $cur->{$field} ne $group[0]{$field}) {
	      my @old_group = @group;
	      @group = defined $cur ? $cur : ();
	      return [ @old_group ];
	  }
	  else {
	      push @group, $cur;
	      $sub->();
	  }
      };

      return $sub;
  }

I'm not too happy that it calls itself and returns from various places
within itself, but my attempts to make it more readable failed.

Anyway, when I call it as follows I get the results I want:
  my $return_record = sub { $sth->fetchrow_hashref() };
  my $get_partition = make_partition_sub($return_record, 'location');
  while (my $partition_ref = $get_partition->()) {
      my @partition = @$partition_ref;
      # Do something with @partition
  }

So now I'm happy that I have a generic partition builder that happens to
work with DBI statement handles, but also works with anything that
returns hash references of fields and values.  I've gone and written
something generic that seems useful that I can't already find on CPAN.

I guess I could also pull out the "if (!defined $cur..." line and make
the second argument a code reference instead of a field name.  This
would give more flexibility, but I worry that would make the code
gratuitously more generic:  I can't think of a common use case.

I thought I should run it past you lot to get feedback, check I haven't
reinvented rounder wheels and to see whether you reckon I've written
something generally useful that warrants packaging up and releasing,
despite its brevity.

Finally, what should I call such a thing?

Tom

From robbiebow at gmail.com  Thu Jul  3 07:20:51 2008
From: robbiebow at gmail.com (Robbie Bow)
Date: Thu, 3 Jul 2008 15:20:51 +0100
Subject: Processing Data Partitions in Perl
In-Reply-To: <20080702203251.GA13126@eborcom.com>
References: <20080702203251.GA13126@eborcom.com>
Message-ID: <973296c20807030720x120e4d2cre7015af46f41a845@mail.gmail.com>

On Wed, Jul 2, 2008 at 9:32 PM, Tom Hukins <tom at eborcom.com> wrote:

> Hi all,
>
> I want to work with these result sets as groups of records with the same
> location.


<snip>


> I thought I should run it past you lot to get feedback, check I haven't
> reinvented rounder wheels and to see whether you reckon I've written
> something generally useful that warrants packaging up and releasing,
> despite its brevity.
>

Sounds useful to me. It's like a better implemented select/fetchall_hashrefs
method.


>
> Finally, what should I call such a thing?


Hashes::Sort maybe?


>
> Tom
> _______________________________________________
> MiltonKeynes-pm mailing list
> MiltonKeynes-pm at pm.org
> http://mail.pm.org/mailman/listinfo/miltonkeynes-pm
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/miltonkeynes-pm/attachments/20080703/14a55408/attachment.html>

From ziya at suzen.net  Fri Jul  4 03:54:35 2008
From: ziya at suzen.net (Ziya Suzen)
Date: Fri, 4 Jul 2008 11:54:35 +0100
Subject: Processing Data Partitions in Perl
In-Reply-To: <20080702203251.GA13126@eborcom.com>
References: <20080702203251.GA13126@eborcom.com>
Message-ID: <7e7471f80807040354i214205av6d8685646f5e679d@mail.gmail.com>

On Wed, Jul 2, 2008 at 9:32 PM, Tom Hukins <tom at eborcom.com> wrote:
[..]
>   1  Cuba     NB
>   2  New Inn  NB
>   3  Spoons   CMK
>   4  Vaults   Stony
>   5  Crown    Stony

Assuming ordered by location I suppose?

> Chapter 19 of SQL For Smarties (a wonderful book for anyone doing
> anything non-trivial with SQL databases) tells me that I should refer to
> such groups of records as partitions.

Didn't know that. Thanks.

[..]
>
> So I wrote myself a little subroutine to do just this:
>  sub make_partition_sub {
>      my $record_sub = shift;
>      my $field = shift;
>
>      my (@group, $sub);
>
>      $sub = {
>          my $cur = $record_sub->();
>          return if (! defined $cur && ! scalar @group);
>
>          if (! defined $cur || $cur->{$field} ne $group[0]{$field}) {
>              my @old_group = @group;
>              @group = defined $cur ? $cur : ();
>              return [ @old_group ];
>          }
>          else {
>              push @group, $cur;
>              $sub->();
>          }
>      };
>
>      return $sub;
>  }
>

This hurts my brain... Too many code refs and recursion :-(
But hey, if it works I don't want to know what's under the bonnet.

>
> I'm not too happy that it calls itself and returns from various places
> within itself, but my attempts to make it more readable failed.
>
> Anyway, when I call it as follows I get the results I want:
>  my $return_record = sub { $sth->fetchrow_hashref() };
>  my $get_partition = make_partition_sub($return_record, 'location');
>  while (my $partition_ref = $get_partition->()) {
>      my @partition = @$partition_ref;
>      # Do something with @partition
>  }
>

I think more object oriented sort of API might be easier to understand and use.
Something like:

my $partitioner = Data::Partitioner->new( provider => sub { .. },
column_name => 'location' );

while (my $partition = $partitioner->get_partition()) { # returns
Data::Partition object
    while (my $rec = $partition->get_record()) {
        # Do something ...
    }
}

>
> I thought I should run it past you lot to get feedback, check I haven't
> reinvented rounder wheels and to see whether you reckon I've written
> something generally useful that warrants packaging up and releasing,
> despite its brevity.
>

I haven't seen anything similar on CPAN. I think it deserves to be CPANed.

>
> Finally, what should I call such a thing?
>

Hmm.. I guess I've already answered that ;-)

Cheers

--Ziya

From peter at dragonstaff.com  Fri Jul  4 09:00:00 2008
From: peter at dragonstaff.com (peter at dragonstaff.com)
Date: Fri, 04 Jul 2008 17:00:00 +0100
Subject: Perl jobs at BBC
Message-ID: <20080704170000.xswvy46n5wo48c0c@webmail.dragonstaff.com>

Perl Software Engineer posts at the BBC:
http://jobs.bbc.co.uk/fe/tpl_bbc01.asp?newms=jj&id=22622

Regards, Peter
http://perl.dragonstaff.co.uk



From tom at eborcom.com  Mon Jul  7 05:13:13 2008
From: tom at eborcom.com (Tom Hukins)
Date: Mon, 7 Jul 2008 13:13:13 +0100
Subject: Technical Meeting: Tuesday 8th July 2008
In-Reply-To: <20080619085820.GA18612@eborcom.com>
References: <20080619085820.GA18612@eborcom.com>
Message-ID: <20080707121312.GA73744@eborcom.com>

Reminder:  the meeting takes place tomorrow.

The OU is a bit out of the way, so if you need a lift or can offer a
lift, please shout.  I know Oliver needs a lift from Milton Keynes
railway station, so if you have a car and would like to see his talk,
please be generous.

Speakers:  Please mail me your slides as soon as possible to reduce
the risk of disaster.  Please test your laptops with the projector
when you arrive.

The original meeting announcement lives here:
http://mail.pm.org/pipermail/miltonkeynes-pm/2008-June/000440.html

If you have any questions about the meeting, ask away.

See you tomorrow,
Tom

From andyfrommk at googlemail.com  Mon Jul  7 11:46:10 2008
From: andyfrommk at googlemail.com (Andy Selby)
Date: Mon, 7 Jul 2008 19:46:10 +0100
Subject: Technical Meeting: Tuesday 8th July 2008
In-Reply-To: <20080707121312.GA73744@eborcom.com>
References: <20080619085820.GA18612@eborcom.com>
	<20080707121312.GA73744@eborcom.com>
Message-ID: <27183a390807071146j3476b42bt58f6203045e9fe16@mail.gmail.com>

> The OU is a bit out of the way, so if you need a lift or can offer a
> lift, please shout.  I know Oliver needs a lift from Milton Keynes
> railway station, so if you have a car and would like to see his talk,
> please be generous.

Since I'm the nearest to the station I'd be happy to give Oliver a lift.
Mail me off-list for my phone number Oliver.

From peter at dragonstaff.com  Mon Jul  7 17:09:47 2008
From: peter at dragonstaff.com (Peter Edwards)
Date: Tue, 8 Jul 2008 01:09:47 +0100
Subject: Technical Meeting: Tuesday 8th July 2008
In-Reply-To: <20080707121312.GA73744@eborcom.com>
References: <20080619085820.GA18612@eborcom.com>
	<20080707121312.GA73744@eborcom.com>
Message-ID: <00e201c8e08f$012c3a40$6401a8c0@DRAGON1>

I'm coming up by train to MK central railway station and then on to the
talk.
If you still need a lift Oliver, give me a shout by email or at the number
below.

Regards, Peter

Skype: Peter.Edwards  Tel/Fax: 0845 0537418

-----Original Message-----
From: miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org
[mailto:miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org] On Behalf Of
Tom Hukins
Sent: 07 July 2008 13:13
To: Milton Keynes Perl Mongers
Subject: Re: Technical Meeting: Tuesday 8th July 2008

Reminder:  the meeting takes place tomorrow.

The OU is a bit out of the way, so if you need a lift or can offer a lift,
please shout.  I know Oliver needs a lift from Milton Keynes railway
station, so if you have a car and would like to see his talk, please be
generous.

Speakers:  Please mail me your slides as soon as possible to reduce the risk
of disaster.  Please test your laptops with the projector when you arrive.

The original meeting announcement lives here:
http://mail.pm.org/pipermail/miltonkeynes-pm/2008-June/000440.html

If you have any questions about the meeting, ask away.

See you tomorrow,
Tom
_______________________________________________
MiltonKeynes-pm mailing list
MiltonKeynes-pm at pm.org
http://mail.pm.org/mailman/listinfo/miltonkeynes-pm


From oliver.gorwits at oucs.ox.ac.uk  Mon Jul  7 22:26:15 2008
From: oliver.gorwits at oucs.ox.ac.uk (Oliver Gorwits)
Date: Tue, 08 Jul 2008 06:26:15 +0100
Subject: Technical Meeting: Tuesday 8th July 2008
In-Reply-To: <20080707121312.GA73744@eborcom.com>
References: <20080619085820.GA18612@eborcom.com>
	<20080707121312.GA73744@eborcom.com>
Message-ID: <4872FA77.9020404@oucs.ox.ac.uk>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Hukins wrote:
| I know Oliver needs a lift from
| Milton Keynes railway station, so if you have a car and would
| like to see his talk, please be generous.

Many thanks to everyone who replied (on and off list) !

All sorted now :-)

- --
Oliver Gorwits, Network and Telecommunications Group,
Oxford University Computing Services
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIcvp32NPq7pwWBt4RAo64AKDD1Cak8RrO6VaL9n/hKo+7CYvI1ACgo6Qh
Yfxlh5ZEJLOx804zXtwyBPs=
=mjnZ
-----END PGP SIGNATURE-----

From tom at eborcom.com  Tue Jul  8 00:32:12 2008
From: tom at eborcom.com (Tom Hukins)
Date: Tue, 8 Jul 2008 08:32:12 +0100
Subject: Processing Data Partitions in Perl
In-Reply-To: <7e7471f80807040354i214205av6d8685646f5e679d@mail.gmail.com>
References: <20080702203251.GA13126@eborcom.com>
	<7e7471f80807040354i214205av6d8685646f5e679d@mail.gmail.com>
Message-ID: <20080708073211.GB77988@eborcom.com>

On Fri, Jul 04, 2008 at 11:54:35AM +0100, Ziya Suzen wrote:
> On Wed, Jul 2, 2008 at 9:32 PM, Tom Hukins <tom at eborcom.com> wrote:
> [..]
> >   1  Cuba     NB
> >   2  New Inn  NB
> >   3  Spoons   CMK
> >   4  Vaults   Stony
> >   5  Crown    Stony
> 
> Assuming ordered by location I suppose?

Not necessarily ordered, although items with the same location must
appear consecutively.  Number 6 must not have the location 'NB' or
'CMK', but it might have any other value.

> This hurts my brain... Too many code refs and recursion :-(

Yeah, same here.  I posted it to the list in the hope that someone
would tell me how I could write it better.  Everything I thought of
just made it look more complicated.

> But hey, if it works I don't want to know what's under the bonnet.

Exactly.  I can think of a few CPAN modules that I use although their
internals look scary.  I prefer to use clean, comprehensible code when
possible.  I can also think of scary code that I try to avoid, but
that's another story..

> I think more object oriented sort of API might be easier to
> understand and use.

I like this idea.  I like the 'column_name' argument for iterators
that return hash-based records, such as DBI's fetchrow_hashref().  The
constructor could also have a 'column_offset' argument for array-based
records, such as those returned by DBI's fetchrow_arrayref().

Thanks for the helpful comments,
Tom

From robbiebow at gmail.com  Wed Jul  9 01:37:39 2008
From: robbiebow at gmail.com (Robbie Bow)
Date: Wed, 9 Jul 2008 09:37:39 +0100
Subject: Temporary contract
Message-ID: <973296c20807090137m3cbf7555p3f8b4eccd8c928ca@mail.gmail.com>

Hi all

I'm about to do some freelance work and will be using an umbrella company to
do the processing / pay roll &c. What I'm looking for is a contract template
for the company that will be commissioning the work. Any ideas where I can
find such?

Cheers



Robbie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/miltonkeynes-pm/attachments/20080709/f03fca15/attachment.html>

From peter at dragonstaff.com  Wed Jul  9 07:57:07 2008
From: peter at dragonstaff.com (peter at dragonstaff.com)
Date: Wed, 09 Jul 2008 15:57:07 +0100
Subject: Temporary contract
In-Reply-To: <973296c20807090137m3cbf7555p3f8b4eccd8c928ca@mail.gmail.com>
References: <973296c20807090137m3cbf7555p3f8b4eccd8c928ca@mail.gmail.com>
Message-ID: <20080709155707.3z988k884oo4g404@webmail.dragonstaff.com>

Quoting Robbie Bow <robbiebow at gmail.com>:
> I'm about to do some freelance work and will be using an umbrella company to
> do the processing / pay roll &c. What I'm looking for is a contract template
> for the company that will be commissioning the work. Any ideas where I can
> find such?

Is the contract between your umbrella company and the client?
If so they probably have some carefully worded IR35 compliant proformas.

If not, I've got some sample contracts you could use.

Regards, Peter



From robbiebow at gmail.com  Wed Jul  9 08:38:22 2008
From: robbiebow at gmail.com (Robbie Bow)
Date: Wed, 9 Jul 2008 16:38:22 +0100
Subject: Temporary contract
In-Reply-To: <20080709155707.3z988k884oo4g404@webmail.dragonstaff.com>
References: <973296c20807090137m3cbf7555p3f8b4eccd8c928ca@mail.gmail.com>
	<20080709155707.3z988k884oo4g404@webmail.dragonstaff.com>
Message-ID: <973296c20807090838y7ac21d6lacb1ba1bf87c7c3e@mail.gmail.com>

On Wed, Jul 9, 2008 at 3:57 PM, <peter at dragonstaff.com> wrote:

> Quoting Robbie Bow <robbiebow at gmail.com>:
>
>> I'm about to do some freelance work and will be using an umbrella company
>> to
>> do the processing / pay roll &c. What I'm looking for is a contract
>> template
>> for the company that will be commissioning the work. Any ideas where I can
>> find such?
>>
>
> Is the contract between your umbrella company and the client?
> If so they probably have some carefully worded IR35 compliant proformas.



It is and I think they do and are supplying the client with a bog contract
which should be fine, but thanks a lot anyway.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/miltonkeynes-pm/attachments/20080709/02d00700/attachment.html>

From tom at eborcom.com  Tue Jul 15 15:53:49 2008
From: tom at eborcom.com (Tom Hukins)
Date: Tue, 15 Jul 2008 23:53:49 +0100
Subject: Technical Meeting: Tuesday 8th July 2008
In-Reply-To: <20080619085820.GA18612@eborcom.com>
References: <20080619085820.GA18612@eborcom.com>
Message-ID: <20080715225349.GA91958@eborcom.com>

On Thu, Jun 19, 2008 at 09:58:21AM +0100, Tom Hukins wrote:
> Milton Keynes Perl Mongers will hold our first technical meeting of
> 2008 on Tuesday 8th July at the Open University in the same room as
> last time.

I've already mentioned this on IRC, but all the slides have been
online for a few days now.  Get them while they're still hot from
http://miltonkeynes.pm.org/ .

I've also mentioned that our next set of talks takes place on Monday
8th of September in the same place, and for those of you who showed up
on time at the same time.  We have Barbie, JJ and Dave Cross confirmed
as speakers.  So we have a spare slot for another speaker or maybe
some lightning talks.  Any of the locals want to volunteer?

I had intended to write about our last meeting on use.perl but I
haven't got round to it yet, and as I'm about to escape for a long
weekend with some of your colleagues and ex-colleagues (if you work at
Spira or the OU) I guess I won't get round to it.  If any of you have
write ups, photos or other things, please share them.

See you on the 29th for our regular meeting in the pub.

Tom

From pm at gavinwestwood.co.uk  Fri Jul 25 05:11:52 2008
From: pm at gavinwestwood.co.uk (Gavin Westwood)
Date: Fri, 25 Jul 2008 13:11:52 +0100
Subject: Meeting: Tuesday 29th July
Message-ID: <4889C308.3010702@gavinwestwood.co.uk>

It's that time again!

The MKLUG and Milton Keynes Perl Mongers social meet up will be held at
the Wetherspoons pub, near the railway station (not the one in the snow
dome), next door to Chiquitos:

http://miltonkeynes.openguides.org/?J.D_Wetherspoon%2C_Central_Milton_Keynes

Starting from 8pm, and going on till the last people stumble off home.
Feel free to bring your laptop/Linux device along if you want a hand/to
show off, but be aware we've had problems with using their Wifi on a 
couple of occasions...

Have fun.  I won't be there due to my holiday, but I'll try to remember 
to raise "ein bier" to you all.

Gavin


From ziya at suzen.net  Mon Jul 28 05:38:49 2008
From: ziya at suzen.net (Ziya Suzen)
Date: Mon, 28 Jul 2008 13:38:49 +0100
Subject: Line numbers - good for filters/eval?
Message-ID: <7e7471f80807280538x4df1a396ua342ad9d2691d7f9@mail.gmail.com>

Hi there

During Tony's presentation about source filters we shortly talked
about issues with line numbers - if you can remember at all :-)
I came across this, though might be helpful:

http://perldoc.perl.org/perlsyn.html#Plain-Old-Comments-(Not!)

Cheers

--Ziya