From tom at eborcom.com Thu Sep 7 14:19:26 2006 From: tom at eborcom.com (Tom Hukins) Date: Thu, 7 Sep 2006 22:19:26 +0100 Subject: Join us on IRC Message-ID: <20060907211926.GB37163@eborcom.com> Please forgive this brief advertisement: We have an IRC channel mentioned on our Web page. It's mostly quiet, but it's picked up a little recently. So I thought I'd invite you all to pop by. Quite a few of you have tried it out at some point, but not at the same time, so it didn't work too well. I thought it would be interesting to persuade you all to pop by soon (now?) so we can get a decent number of people on there simultaneously. For those who don't know, IRC is an Internet chat protocol. You can leave it running in the background and get involved when it looks interesting, or when you want a quick break. I hope to see some of you in #miltonkeynes.pm on irc.perl.org soon. If you have any previous bad experience of IRC, please put it aside and join in. Ask questions, throw ideas around and let's see what happens. If you have any trouble using IRC, post to the list and we'll help out. Oh, and our next meeting is on the 25th. See you soon, Tom From tom at eborcom.com Thu Sep 7 15:17:10 2006 From: tom at eborcom.com (Tom Hukins) Date: Thu, 7 Sep 2006 23:17:10 +0100 Subject: Join us on IRC In-Reply-To: <20060907211926.GB37163@eborcom.com> References: <20060907211926.GB37163@eborcom.com> Message-ID: <20060907221710.GB38724@eborcom.com> On Thu, Sep 07, 2006 at 10:19:26PM +0100, Tom Hukins wrote: > I hope to see some of you in #miltonkeynes.pm on irc.perl.org soon. Wow, it worked. Already. But don't let that stop the rest of you. Tom From robbie at robbiebow.co.uk Fri Sep 8 15:06:23 2006 From: robbie at robbiebow.co.uk (Robbie Bow) Date: Fri, 08 Sep 2006 23:06:23 +0100 Subject: A little puzzle Message-ID: <4501E95F.6000108@robbiebow.co.uk> Here's the scenario: I am playing a game of Diplomacy with online friends. One guy (who isn't a player) is the games master; receives our orders by email and when the deadline for that turn is up, processes the orders and posts the results on a forum for us all to read. Now it happens that the GM is off on holiday for 2 weeks, so we have a problem: how to process the orders without a GM. I was thinking along the lines of having a script that we could all submit orders which it duly encrypts. The tricky bit (for me) is how to make a system such that to decrypt the data it would require all 4 players to process it in some way (thus none of us can peek at anyone else's orders before everyone is satisfied with their orders) - this also means the system needs to be able to replace orders from any player up until the point we all do our bit to decrypt it, and seeing as I will probably be hosting the script, will need to be able to set the other players up with their keys such that I cannot easily find what those keys are. Hmm. Any tips gratefully received. From ant at badmagic.com Fri Sep 8 16:51:18 2006 From: ant at badmagic.com (Ant Mitchell) Date: Sat, 9 Sep 2006 00:51:18 +0100 Subject: A little puzzle In-Reply-To: <4501E95F.6000108@robbiebow.co.uk> References: <4501E95F.6000108@robbiebow.co.uk> Message-ID: <351CDC19-B439-4779-AD73-EB304C291548@badmagic.com> Well, its late, and I might be misunderstanding - but why not just send each player an MD5 hash of your order. All players do the same. When the time comes to show your order, its easy to verify that the orders match the published md5s. If they don't match - someone is cheating !. You don't have to disclose the order to anyone until you all agree, so there is no possibility of artificial discovery. MD5 hash programs are easy to run on a text file. Not sure what the orders are but the need to be more than just a few numbers to ensure someone does not attempt to guess the MD5. If they are 'english' and do not follow a specific format, they should be pretty hard to guess. enjoy Ant On 8 Sep 2006, at 23:06, Robbie Bow wrote: > Here's the scenario: I am playing a game of Diplomacy with online > friends. One guy (who isn't a player) is the games master; receives > our > orders by email and when the deadline for that turn is up, > processes the > orders and posts the results on a forum for us all to read. Now it > happens that the GM is off on holiday for 2 weeks, so we have a > problem: > how to process the orders without a GM. > > I was thinking along the lines of having a script that we could all > submit orders which it duly encrypts. The tricky bit (for me) is > how to > make a system such that to decrypt the data it would require all 4 > players to process it in some way (thus none of us can peek at anyone > else's orders before everyone is satisfied with their orders) - this > also means the system needs to be able to replace orders from any > player > up until the point we all do our bit to decrypt it, and seeing as I > will > probably be hosting the script, will need to be able to set the other > players up with their keys such that I cannot easily find what those > keys are. > > Hmm. Any tips gratefully received. > > > > > _______________________________________________ > MiltonKeynes-pm mailing list > MiltonKeynes-pm at pm.org > http://mail.pm.org/mailman/listinfo/miltonkeynes-pm From leelarcombe at hotmail.com Sat Sep 9 13:10:53 2006 From: leelarcombe at hotmail.com (Lee Larcombe) Date: Sat, 09 Sep 2006 21:10:53 +0100 Subject: Perl question - thread/fork? Message-ID: Hi all I am trying to work out how I could call a program from within a perl script, and display some sort of indication to the user that it is running which stops when it is finished. So I have eg: Print "Starting analysis. Please wait...\n"; System "programThatTakesAges"; The 'programThatTakesAges' analyses some files and can take 10,20,30 mins or so (could be longer), then the perl script carries on and does things with those files. I would like the user to have some feedback that something is happening after the 'Please wait...' - a spinny cursor would be nice :-) Anyway, I can't work out how to get the script to do the system call and then do something whilst it waits for it to finish. I've been looking at using a thread or a fork - I'm sure it must be one of those - but I can't really work it out. Anyone got any tips or example code that would help? Thanks Lee ============================ Lee Larcombe www.larcombesplace.org.uk msn: leelarcombe at hotmail.com From peter at dragonstaff.com Mon Sep 11 00:40:59 2006 From: peter at dragonstaff.com (Peter Edwards) Date: Mon, 11 Sep 2006 08:40:59 +0100 Subject: Perl question - thread/fork? In-Reply-To: Message-ID: <012401c6d575$a186e520$6c01a8c0@metis> Hi Lee, what I've done before in this situation is write a module to wrap the spawning, monitoring and output tracking for a child process. Parent: call routine to spawn and detach child after redirecting input/output streams, note process id and stick it in a "process running" SQL table; get run id back and continue with something else Child: register an exit handler, go off and run its code Parent: (later) call routine to check whether given run id is still active; call routine to gain access to the standard output and standard error log files (I use this to do the equivalent of a "tail -f" on a web screen to check batch progress) Child: exits; the exit handler logs the run to an a "processes complete" SQL table and deletes the "process running" entry Parent: (later still) calls check routine and finds child has finished; the check routine incorporates a reaper to remove "process running" entries where the given process ID no longer exists on the system There's stuff in the "Perl Cookbook" which you'll find helpful. http://www.unix.org.ua/orelly/perl/cookbook/index.htm http://pleac.sourceforge.net/pleac_perl/processmanagementetc.html and CPAN http://search.cpan.org/author/GARROW/Proc-Spawn-1.03/Spawn.pm http://search.cpan.org/author/JDB/libwin32-0.26/Job/Job.pm http://search.cpan.org/~aristotle/Proc-Fork-0.3/lib/Proc/Fork.pm Here's an excerpt to fork a detached child under Unix: sub start_child { # argument, locking and permission code removed for brevity ... # start child process to run command $SIG{CHLD} = 'IGNORE'; my $mode = ($overwrite ? '>' : '>>' ); # overwrite or append to log my $kidpid = fork(); unless (defined $kidpid) { die("can't fork: $!"); } if ($kidpid == 0 ) # child { if (exists $ENV{MOD_PERL}) { Apache->request->cleanup_for_exec(); # untie the socket } setsid or warn "setsid cannot start a new session: $!"; open STDIN, "/dev/null" or die "Can't read /dev/null: $!"; open STDOUT, "$mode$logpath" or die "Can't write stdout to $ logpath: $!"; open STDERR, "$mode$errpath" or die "Can't write stderr to $errpath: $!"; local $| = 1; print STDOUT "Started\n"; # run command exec($cmdpath); # should stop here, if not, failed to exec warn "could not run $cmdpath: $!"; print "Failed could not run $cmdpath: $!\n"; CORE::exit(0); } # parent $SIG{CHLD} = 'DEFAULT'; # catch immediate child exits, which imply the command run is invalid select(undef,undef,undef,1.0); # wait 1 second POSIX::waitpid(-1, POSIX::WNOHANG()); # clean up kid if its defunct unless (kill(0,$kidpid)) # see if kid has stopped { my $msg = "Child process exited quickly, possibly due to an error in command \"$cmdpath\""; warn $msg; return ($msg, $kidpid, 0); } my $runid = sql_note_process_started($kidpid, $cmdpath, $logpath, $errpath); return ("", $kidpid, $runid); } Regards, Peter Dragonstaff Limited http://www.dragonstaff.com Business IT Consultancy -----Original Message----- From: miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org [mailto:miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org] On Behalf Of Lee Larcombe Sent: 09 September 2006 21:11 To: miltonkeynes-pm at pm.org Subject: Perl question - thread/fork? Hi all I am trying to work out how I could call a program from within a perl script, and display some sort of indication to the user that it is running which stops when it is finished. So I have eg: Print "Starting analysis. Please wait...\n"; System "programThatTakesAges"; The 'programThatTakesAges' analyses some files and can take 10,20,30 mins or so (could be longer), then the perl script carries on and does things with those files. I would like the user to have some feedback that something is happening after the 'Please wait...' - a spinny cursor would be nice :-) Anyway, I can't work out how to get the script to do the system call and then do something whilst it waits for it to finish. I've been looking at using a thread or a fork - I'm sure it must be one of those - but I can't really work it out. Anyone got any tips or example code that would help? Thanks Lee ============================ Lee Larcombe www.larcombesplace.org.uk msn: leelarcombe at hotmail.com _______________________________________________ MiltonKeynes-pm mailing list MiltonKeynes-pm at pm.org http://mail.pm.org/mailman/listinfo/miltonkeynes-pm From peter at dragonstaff.com Mon Sep 11 01:48:51 2006 From: peter at dragonstaff.com (Peter Edwards) Date: Mon, 11 Sep 2006 09:48:51 +0100 Subject: Perl question - thread/fork? In-Reply-To: <012401c6d575$a186e520$6c01a8c0@metis> Message-ID: <013301c6d57f$1ca92d90$6c01a8c0@metis> I meant to mention - if you want fine-grained control rather than a batch mechanism you can use POE: http://search.cpan.org/search?query=poe&mode=all http://search.cpan.org/author/RCAPUTO/POE-0.37/lib/POE/Wheel/Run.pm lets you fork off an asynchronous child process and have non-blocking communication with it using stdin/stdout. Regards, Peter From nik at ngo.org.uk Mon Sep 11 04:59:26 2006 From: nik at ngo.org.uk (Nik Clayton) Date: Mon, 11 Sep 2006 12:59:26 +0100 Subject: Perl question - thread/fork? In-Reply-To: References: Message-ID: <45054F9E.3050904@ngo.org.uk> Lee Larcombe wrote: > I am trying to work out how I could call a program from within a perl > script, and display some sort of indication to the user that it is running > which stops when it is finished. > > So I have eg: > > Print "Starting analysis. Please wait...\n"; > > System "programThatTakesAges"; > > The 'programThatTakesAges' analyses some files and can take 10,20,30 mins or > so (could be longer), then the perl script carries on and does things with > those files. I would like the user to have some feedback that something is > happening after the 'Please wait...' - a spinny cursor would be nice :-) > > Anyway, I can't work out how to get the script to do the system call and > then do something whilst it waits for it to finish. I've been looking at > using a thread or a fork - I'm sure it must be one of those - but I can't > really work it out. > > Anyone got any tips or example code that would help? You have two problems here. The first is how to start a new process in the background, and wait for it to finish. The second is how to draw a spinner. You can handily solve both these problems with Term::Twiddle, which takes the opposite approach -- namely, it handles updating the spinner in the background for you, and assumes that the long process will happen in the foreground. So your code looks like this: use Term::Twiddle; my $spin = Term::Twiddle->new(); print "Starting analysis. Please wait... "; $spin->start(); system 'programThatTakesAges'; $spin->stop(); Term::Twiddle's on CPAN, and that should do everything you asked for. Extra credit: If you know how long the process is going to take, or how far through the process the work is (or you can determine that, e.g., by examining the size of a file that's being created) then your users will probably prefer a progress bar. Term::ProgressBar or Term::Activity can handle the nitty gritty of actually drawing the progress bar for you. To start your long running process in the background you can either do the fork/exec yourself, or use a module. If you're not comfortable with the semantics behind fork/exec then Proc::Background or Proc::Fork (again, on CPAN) are probably your best bet, since the abstract away most of the tricky stuff. For example, if you know that your long running process is always going to create a 10MB output file (and your program can't do anything until the output file is complete) then something like this might work (note: untested code, may have typos) use strict; use warnings; use Readonly; use Proc::Background; use Term::ProgressBar; Readonly my $TEN_MB => 1024 * 1024 * 10; my $progress = Term::ProgressBar->new($TEN_MB); my $process = Proc::Background->new("programThatTakesAges"); do { # Get the file size, update the progress bar, which will # scale the result to fit in the bar my $file_size = (stat('/path/to/output'))[7]; $progress->update($file_size); # or: $progress->update((stat('/path/to/output'))[7]); # but that's less readable, and only saves you a variable # Sleep, to avoid issuing millions of stat() calls. Also means # that the progress bar is updated once per second sleep(1); } while($process->alive()); # There's a chance that the progress bar is not at 100% here. For # example, suppose we stat'd the file, and it was 9.8MB in size, # so the progress bar was updated to 98%. Then we slept for a # second, and in the intervening time the process finished, so the # $process->alive() call returned false. So processing is complete, # but the bar still says 98%. # # So, explicitly set the progress bar to 100% (i.e., the value we # passed to the constructor) so that the display looks clean. You # don't have to do this, but it avoids confusing users. $progress->update($TEN_MB); # Print some information about the process my $ec = $process->wait(); # To collect the exit code / avoid zombies print "Process complete:\n"; print " pid was: ", $process->pid(), "\n"; print " exit code was: ", $ec / 256, "\n"; print " elapsed (secs): ", $process->end_time - $process->start_time; N From tom at eborcom.com Tue Sep 19 12:48:40 2006 From: tom at eborcom.com (Tom Hukins) Date: Tue, 19 Sep 2006 20:48:40 +0100 Subject: Meeting: Tuesday 26th September 2006 Message-ID: <20060919194840.GA39178@eborcom.com> The summer's almost over. We've had our holidays, had our fun. Some of us went to Brum. In a week's time we will gather together to share stories, dream schemes and pretend we do stuff with something called Perl. We usually start out by talking about what we've been up to recently, ask a few questions that we'll forget the answers to and drift off into the obscure. Try it, you might like it. As usual, we'll meet in Wetherspoon's near the railway station (not the one in the snow dome). I'll show up some time between 7 and 7.30, but feel free to arrive whenever suits you. If you've not been along before and you would like my mobile number to help find us please ask off list. See the Web site for more details: http://miltonkeynes.pm.org/ In other news: expect a technical meeting with the random variety you have got accustomed to on 10th October. I'll send the announcement soon. Also, Birmingham.pm are looking for speakers for their forthcoming technical meetings: anyone up for a journey further afield? See you soon, Tom From tom at eborcom.com Tue Sep 26 15:58:12 2006 From: tom at eborcom.com (Tom Hukins) Date: Tue, 26 Sep 2006 23:58:12 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: <20060919194840.GA39178@eborcom.com> References: <20060919194840.GA39178@eborcom.com> Message-ID: <20060926225812.GA9639@eborcom.com> Hello, as usual I enjoyed chatting to you all in the pub. It was really good to see a couple of new faces too. I promised I would mention Amazon's distributed computing service to the list: http://aws.amazon.com/ec2 In fact, their Web services offerings in general are worth looking into if you haven't heard about them yet: http://aws.amazon.com/ (I particularly like the idea of Mechanical Turk and S3: Simple Storage Service) I look forward to seeing you all on the 10th October for the technical meet. I'll send an announcement once I've stopped stressing about moving house. Hopefully soon. ;) Tom From peter at dragonstaff.com Tue Sep 26 16:17:53 2006 From: peter at dragonstaff.com (Peter Edwards) Date: Wed, 27 Sep 2006 00:17:53 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: <20060926225812.GA9639@eborcom.com> Message-ID: <001701c6e1c1$ffa26b90$6c01a8c0@metis> Good to see you all tonight. I looked at Amazon Cloud a while ago. Interesting but I remain to be convinced :) I guess I had too many techie questions about exactly what the platform was and what I could do with it. They must be aiming it at corporate suits but nonetheless I can see that Google et al. web services are headed in this direction. Regards, Peter -----Original Message----- From: miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org [mailto:miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org] On Behalf Of Tom Hukins Sent: 26 September 2006 23:58 To: Milton Keynes Perl Mongers Subject: Re: Meeting: Tuesday 26th September 2006 Hello, as usual I enjoyed chatting to you all in the pub. It was really good to see a couple of new faces too. I promised I would mention Amazon's distributed computing service to the list: http://aws.amazon.com/ec2 In fact, their Web services offerings in general are worth looking into if you haven't heard about them yet: http://aws.amazon.com/ (I particularly like the idea of Mechanical Turk and S3: Simple Storage Service) I look forward to seeing you all on the 10th October for the technical meet. I'll send an announcement once I've stopped stressing about moving house. Hopefully soon. ;) Tom _______________________________________________ MiltonKeynes-pm mailing list MiltonKeynes-pm at pm.org http://mail.pm.org/mailman/listinfo/miltonkeynes-pm From tom at eborcom.com Wed Sep 27 02:36:17 2006 From: tom at eborcom.com (Tom Hukins) Date: Wed, 27 Sep 2006 10:36:17 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: <001701c6e1c1$ffa26b90$6c01a8c0@metis> References: <20060926225812.GA9639@eborcom.com> <001701c6e1c1$ffa26b90$6c01a8c0@metis> Message-ID: <20060927093617.GA13435@eborcom.com> On Wed, Sep 27, 2006 at 12:17:53AM +0100, Peter Edwards wrote: > I looked at Amazon Cloud a while ago. Interesting but I remain to be > convinced :) I guess I had too many techie questions about exactly > what the platform was and what I could do with it. As I understand it, you upload a Xen virtual machine guest image (they call this an AMI) which runs on their servers. The have detailed technical documentation at http://xrl.us/rw5d (Warning, PDF). I doubt I'd want to bet a business on this yet but it looks really interesting, particularly for organisations that can't afford lots of servers or only need serious computational power infrequently. Tom From pjm at sanger.ac.uk Wed Sep 27 02:58:23 2006 From: pjm at sanger.ac.uk (Paul Mooney) Date: Wed, 27 Sep 2006 10:58:23 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: <001701c6e1c1$ffa26b90$6c01a8c0@metis> References: <001701c6e1c1$ffa26b90$6c01a8c0@metis> Message-ID: On 27 Sep 2006, at 00:17, Peter Edwards wrote: > Good to see you all tonight. > I looked at Amazon Cloud a while ago. Interesting but I remain to be > convinced :) I guess I had too many techie questions about exactly > what the > platform was and what I could do with it. They must be aiming it at > corporate suits but nonetheless I can see that Google et al. web > services > are headed in this direction. Thanks for the link Tom, very interesting :) Peter: Any idea if the platform could be used to do some compute intensive stuff? It doesn't say it can't but like you I have lots of questions. Some of the stuff I do is embarrassingly parallel and the idea of renting 20 odd machines for 4 hours is quite appealing. There would be very little web front end, just some hot machines at the back end :) > > Regards, Peter > -----Original Message----- > From: miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org > [mailto:miltonkeynes-pm-bounces+peter=dragonstaff.com at pm.org] On > Behalf Of > Tom Hukins > Sent: 26 September 2006 23:58 > To: Milton Keynes Perl Mongers > Subject: Re: Meeting: Tuesday 26th September 2006 > > Hello, as usual I enjoyed chatting to you all in the pub. It was > really good to see a couple of new faces too. > > I promised I would mention Amazon's distributed computing service to > the list: > http://aws.amazon.com/ec2 > > In fact, their Web services offerings in general are worth looking > into if you haven't heard about them yet: > http://aws.amazon.com/ > > (I particularly like the idea of Mechanical Turk and S3: Simple > Storage Service) > > I look forward to seeing you all on the 10th October for the technical > meet. I'll send an announcement once I've stopped stressing about > moving house. Hopefully soon. ;) > > Tom > _______________________________________________ > MiltonKeynes-pm mailing list > MiltonKeynes-pm at pm.org > http://mail.pm.org/mailman/listinfo/miltonkeynes-pm > > > _______________________________________________ > MiltonKeynes-pm mailing list > MiltonKeynes-pm at pm.org > http://mail.pm.org/mailman/listinfo/miltonkeynes-pm From tom at eborcom.com Wed Sep 27 03:46:00 2006 From: tom at eborcom.com (Tom Hukins) Date: Wed, 27 Sep 2006 11:46:00 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: References: <001701c6e1c1$ffa26b90$6c01a8c0@metis> Message-ID: <20060927104600.GA13942@eborcom.com> On Wed, Sep 27, 2006 at 10:58:23AM +0100, Paul Mooney wrote: > Some of the stuff I do is embarrassingly parallel and the idea of > renting 20 odd machines for 4 hours is quite appealing. Think of the Web 2.0 mash-up potential: "People with DNA like yours also purchased..." ;) Tom From robbie at robbiebow.co.uk Wed Sep 27 11:42:12 2006 From: robbie at robbiebow.co.uk (Robbie Bow) Date: Wed, 27 Sep 2006 19:42:12 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: References: <001701c6e1c1$ffa26b90$6c01a8c0@metis> Message-ID: <451AC604.3000000@robbiebow.co.uk> > It doesn't say it can't but like you I have lots of > questions. Some of the stuff I do is embarrassingly parallel and the > idea of renting 20 odd machines for 4 hours is quite appealing. There > would be very little web front end, just some hot machines at the back > end :) Surely you guys should be looking at BOINC, or is your stuff too private for that? From pjm at sanger.ac.uk Thu Sep 28 01:41:31 2006 From: pjm at sanger.ac.uk (Paul Mooney) Date: Thu, 28 Sep 2006 09:41:31 +0100 Subject: Meeting: Tuesday 26th September 2006 In-Reply-To: <451AC604.3000000@robbiebow.co.uk> References: <001701c6e1c1$ffa26b90$6c01a8c0@metis> <451AC604.3000000@robbiebow.co.uk> Message-ID: On 27 Sep 2006, at 19:42, Robbie Bow wrote: >> It doesn't say it can't but like you I have lots of >> questions. Some of the stuff I do is embarrassingly parallel and the >> idea of renting 20 odd machines for 4 hours is quite appealing. There >> would be very little web front end, just some hot machines at the back >> end :) > > Surely you guys should be looking at BOINC, or is your stuff too > private for that? I just looked up what BOINC is... The SETI type stuff doesn't play nice in my field of work as we need CPU, memory /and/ bandwidth :( Each worker program can access a database quite a lot so working over a home users 1Mb broadband connection with its relatively high latency isn't ideal. SETI stuff is more about small data set, small memory, high CPU work. The EC2 instances "predictably provides" 1.75Gb of RAM which is pretty good but not enough for some of the things I run. Also if I had a central MySQL server running on an EC2 instance its physical storage is somewhere else in S3 world so access would be slow. If you're DB fits within the "free" 250Gb disk space you could copy the DB back and forth before and after the work is done I suppose :/ But... you really can borrow super computer like resources for any purpose. World Domination is within my reach!!