From lanas at securenet.net Thu Oct 1 03:24:52 2009 From: lanas at securenet.net (lanas) Date: Thu, 1 Oct 2009 06:24:52 -0400 Subject: [tpm] libtap (Perl related) Message-ID: <20091001062452.5114efc4@mistral.stie> Hello all, This is about a great gift from Perl (if it works) to C. I have to make something in C and since I took with Perl the devious habit of having tests developed at the same time as code, I thought of doing the same in C. And there's libtap, which brings TAP to C: http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap Do you know of any support for this project ? I've tried several examples (some from O'Reilly also) and the examples, even the most simple one, never exits. Maybe there's a gcc problem of some sort with the latest version, or Something. I've also seen that there are a few other such test frameworks around, but libtap would carry on familiarity. Cheers, Al From shlomif at iglu.org.il Thu Oct 1 03:56:59 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Thu, 01 Oct 2009 12:56:59 +0200 Subject: [tpm] libtap (Perl related) In-Reply-To: <20091001062452.5114efc4@mistral.stie> References: <20091001062452.5114efc4@mistral.stie> Message-ID: <200910011256.59897.shlomif@iglu.org.il> On Thursday 01 Oct 2009 12:24:52 lanas wrote: > Hello all, > > This is about a great gift from Perl (if it works) to C. I have to > make something in C and since I took with Perl the devious habit of > having tests developed at the same time as code, I thought of doing the > same in C. And there's libtap, which brings TAP to C: > > http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap > > Do you know of any support for this project ? I've tried several > examples (some from O'Reilly also) and the examples, even the > most simple one, never exits. Maybe there's a gcc problem of some > sort with the latest version, or Something. > The vanilla libtap has been broken for quite some time. Mandriva compiles it by using a specialised build process. I give a workaround in the HACKING.txt file in: https://svn.berlios.de/svnroot/repos/fc-solve/trunk/fc-solve/source/ {{{{{{{{{{ 3. Install the development headers of libtap ( http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap ). + * On Mandriva: + # urpmi libtap-devel + * On Debian: + ------------------------------- tar -xvf /home/shlomif/Desktop/tap-1.01.tar.gz cd tap-1.01 ./configure --prefix="$HOME/apps/libtap" make CFLAGS+=-UHAVE_LIBPTHREAD make install # For gcc finding tap.h in the includes echo 'export CPATH="$HOME/apps/libtap/include:$CPATH"' >> ~/.bashrc # For CMake finding libtap. echo 'export CMAKE_PREFIX_PATH="$HOME/apps/libtap:$CMAKE_PREFIX_PATH"' >> ~/.bashrc ------------------------------ }}}}}}}}}} I've been thinking of forking or rewriting it. I'd like to make it thread-safe and capable of being instantiated, etc. Regards, Shlomi Fish > I've also seen that there are a few other such test frameworks > around, but libtap would carry on familiarity. > > Cheers, > > Al > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Rethinking CPAN - http://shlom.in/rethinking-cpan Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Thu Oct 1 03:56:59 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Thu, 01 Oct 2009 12:56:59 +0200 Subject: [tpm] libtap (Perl related) In-Reply-To: <20091001062452.5114efc4@mistral.stie> References: <20091001062452.5114efc4@mistral.stie> Message-ID: <200910011256.59897.shlomif@iglu.org.il> On Thursday 01 Oct 2009 12:24:52 lanas wrote: > Hello all, > > This is about a great gift from Perl (if it works) to C. I have to > make something in C and since I took with Perl the devious habit of > having tests developed at the same time as code, I thought of doing the > same in C. And there's libtap, which brings TAP to C: > > http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap > > Do you know of any support for this project ? I've tried several > examples (some from O'Reilly also) and the examples, even the > most simple one, never exits. Maybe there's a gcc problem of some > sort with the latest version, or Something. > The vanilla libtap has been broken for quite some time. Mandriva compiles it by using a specialised build process. I give a workaround in the HACKING.txt file in: https://svn.berlios.de/svnroot/repos/fc-solve/trunk/fc-solve/source/ {{{{{{{{{{ 3. Install the development headers of libtap ( http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap ). + * On Mandriva: + # urpmi libtap-devel + * On Debian: + ------------------------------- tar -xvf /home/shlomif/Desktop/tap-1.01.tar.gz cd tap-1.01 ./configure --prefix="$HOME/apps/libtap" make CFLAGS+=-UHAVE_LIBPTHREAD make install # For gcc finding tap.h in the includes echo 'export CPATH="$HOME/apps/libtap/include:$CPATH"' >> ~/.bashrc # For CMake finding libtap. echo 'export CMAKE_PREFIX_PATH="$HOME/apps/libtap:$CMAKE_PREFIX_PATH"' >> ~/.bashrc ------------------------------ }}}}}}}}}} I've been thinking of forking or rewriting it. I'd like to make it thread-safe and capable of being instantiated, etc. Regards, Shlomi Fish > I've also seen that there are a few other such test frameworks > around, but libtap would carry on familiarity. > > Cheers, > > Al > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Rethinking CPAN - http://shlom.in/rethinking-cpan Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From jbl at jbldata.com Thu Oct 1 08:49:56 2009 From: jbl at jbldata.com (J. Bobby Lopez) Date: Thu, 1 Oct 2009 11:49:56 -0400 Subject: [tpm] Tracking memory usage in perl - how much memory used by a perl variable/hash/array? Message-ID: Looking around on CPAN to find a way to dump the amount of memory used by any given variable - anyone familiar with how to do this? -- J. Bobby Lopez Web: http://jbldata.com/ Twitter: http://www.twitter.com/jbobbylopez -------------- next part -------------- An HTML attachment was scrubbed... URL: From talexb at gmail.com Thu Oct 1 08:57:45 2009 From: talexb at gmail.com (Alex Beamish) Date: Thu, 1 Oct 2009 11:57:45 -0400 Subject: [tpm] Tracking memory usage in perl - how much memory used by a perl variable/hash/array? In-Reply-To: References: Message-ID: On Thu, Oct 1, 2009 at 11:49 AM, J. Bobby Lopez wrote: > Looking around on CPAN to find a way to dump the amount of memory used by > any given variable - anyone familiar with how to do this? How about this? http://search.cpan.org/~tels/Devel-Size-0.71/lib/Devel/Size.pm I searched CPAN for Memory and this was on the third page. -- Alex Beamish Toronto, Ontario aka talexb From jbl at jbldata.com Thu Oct 1 09:00:49 2009 From: jbl at jbldata.com (J. Bobby Lopez) Date: Thu, 1 Oct 2009 12:00:49 -0400 Subject: [tpm] Tracking memory usage in perl - how much memory used by a perl variable/hash/array? In-Reply-To: References: Message-ID: Heh, you beat me to it, http://search.cpan.org/~tels/Devel-Size-0.71/lib/Devel/Size.pm Looks like the ticket, though it says that increases the memory overhead of the program significantly, so I'll have to be careful implementing it on a memory constrained system. On Thu, Oct 1, 2009 at 11:57 AM, Alex Beamish wrote: > On Thu, Oct 1, 2009 at 11:49 AM, J. Bobby Lopez wrote: > > Looking around on CPAN to find a way to dump the amount of memory used by > > any given variable - anyone familiar with how to do this? > > How about this? > > http://search.cpan.org/~tels/Devel-Size-0.71/lib/Devel/Size.pm > > I searched CPAN for Memory and this was on the third page. > > -- > Alex Beamish > Toronto, Ontario > aka talexb > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -- J. Bobby Lopez Web: http://jbldata.com/ Twitter: http://www.twitter.com/jbobbylopez -------------- next part -------------- An HTML attachment was scrubbed... URL: From linux at alteeve.com Thu Oct 1 08:44:54 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:44:54 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <200909301817.57674.shlomif@iglu.org.il> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> Message-ID: <4AC4CE76.3050001@alteeve.com> Shlomi Fish wrote: > On Wednesday 30 Sep 2009 17:39:14 Madison Kelly wrote: >> Hi all, >> >> Thus far, I've opened a file, read it in, shifted/ignored the first X >> number of line and then wrote out the resulting file. This works, but >> strikes me as horribly inefficient. Doubly so when the file could be >> larger than RAM. >> >> So then, is there a more efficient/faster way of saying "Delete the >> first X number of lines from file foo"? >> > > What I would do in pure-Perl is: (untested) > > <<<<<<<<<<<<<<< > my $num_lines_to_del = shift(@ARGV); > my $filename = shift(@ARGV); > # Maybe use File::Temp here > my $temp_fn = $filename.".new"; > > open my $in_fh, "<", $filename > or die "Could not open '$filename' - $! !"; > > > open my $temp_out_fh, ">", $temp_fn > or die "Could not open temp filename - $!"; > > foreach my $i (1 .. $num_lines_to_del) > { > # Read one line. > scalar(<$in_fh>); > } > > my $buf_len = 1024 * 16; > > my $buffer; > while (read($in_fh, $buffer, $buf_len)) > { > print {$temp_out_fh} $buffer; > } > close($temp_out_fh); > close($in_fh); > > rename($temp_fn, $filename); > > Like I said - untested, but I hope you get the idea. > > Regards, > > Shlomi Fish If I read this right, the entire file will be copied, sans initial lines, and never use more that 1024*16 bytes of memory. That would certainly ensure I never eat up all the RAM, regardless of the file size being messed with. Awesome. Madi From linux at alteeve.com Thu Oct 1 08:40:04 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:40:04 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: References: <4AC37BA2.90608@alteeve.com> Message-ID: <4AC4CD54.3000804@alteeve.com> Antonio Sun wrote: > > > On Wed, Sep 30, 2009 at 11:57 AM, Alex Beamish > wrote: > > > So then, is there a more efficient/faster way of saying "Delete > the first X > > number of lines from file foo"? > > I would expect you could fashion something workable like this > > # Get the number of lines in the file .. > wc -l file > > # tail the desired number of lines to a new file .. > tail -n $numberOfLinesInFile-$numberOfLinesYouDontWant file >newfile > > # and move the files around. > rm file; mv newfile file > > > If using shell script instead of Perl is an option, then sed is the > best tool > > ToDel=20 > sed -i "1,${ToDel}d" file > > will delete the first 20 lines from the file. > > HTH > > Antonio Hrm, that looks interesting, and is only one system call. Maybe I'll add it as an optional way to cut the first few lines off for people on systems with 'sed'. Regardless, I will test it on some large files to see how much faster it is. Thanks Madi From linux at alteeve.com Thu Oct 1 08:36:31 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:36:31 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: References: <4AC37BA2.90608@alteeve.com> Message-ID: <4AC4CC7F.3080502@alteeve.com> Alex Beamish wrote: > On Wed, Sep 30, 2009 at 11:39 AM, Madison Kelly wrote: >> Hi all, >> >> Thus far, I've opened a file, read it in, shifted/ignored the first X >> number of line and then wrote out the resulting file. This works, but >> strikes me as horribly inefficient. Doubly so when the file could be larger >> than RAM. >> >> So then, is there a more efficient/faster way of saying "Delete the first X >> number of lines from file foo"? > > I would expect you could fashion something workable like this > > # Get the number of lines in the file .. > wc -l file > > # tail the desired number of lines to a new file .. > tail -n $numberOfLinesInFile-$numberOfLinesYouDontWant file >newfile > > # and move the files around. > rm file; mv newfile file That would work, but I am trying to/hoping to come up with a pure-perl solution. I'm hoping to make the module I am using this in as portable as possible, so I'd like to stick to core modules and functions. Perhaps for now, what I am doing is the only real option. Madi From linux at alteeve.com Thu Oct 1 08:42:14 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:42:14 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <19139.41890.796217.428615@hetman.ua> References: <4AC37BA2.90608@alteeve.com> <19139.41890.796217.428615@hetman.ua> Message-ID: <4AC4CDD6.7050901@alteeve.com> Viktor Pavlenko wrote: >>>>>> "AS" == Antonio Sun writes: > > AS> [...] > > AS> If using shell script instead of Perl is an option, then sed is the best > AS> tool > > AS> ToDel=20 > AS> sed -i "1,${ToDel}d" file > > AS> will delete the first 20 lines from the file. > > perl -i -ne 'print if $. > 20' file > > (my two cents in Perl) Where 'if' is the input file handle? I've not played with '$.'... If this works it'd be the pure-perl way I was hoping for. Do you know if this still reads the whole file into memory at once? My biggest concern, even more than speed, is that this may need to be done on files larger than available RAM. Thanks! Madi From linux at alteeve.com Thu Oct 1 08:44:54 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:44:54 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <200909301817.57674.shlomif@iglu.org.il> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> Message-ID: <4AC4CE76.3050001@alteeve.com> Shlomi Fish wrote: > On Wednesday 30 Sep 2009 17:39:14 Madison Kelly wrote: >> Hi all, >> >> Thus far, I've opened a file, read it in, shifted/ignored the first X >> number of line and then wrote out the resulting file. This works, but >> strikes me as horribly inefficient. Doubly so when the file could be >> larger than RAM. >> >> So then, is there a more efficient/faster way of saying "Delete the >> first X number of lines from file foo"? >> > > What I would do in pure-Perl is: (untested) > > <<<<<<<<<<<<<<< > my $num_lines_to_del = shift(@ARGV); > my $filename = shift(@ARGV); > # Maybe use File::Temp here > my $temp_fn = $filename.".new"; > > open my $in_fh, "<", $filename > or die "Could not open '$filename' - $! !"; > > > open my $temp_out_fh, ">", $temp_fn > or die "Could not open temp filename - $!"; > > foreach my $i (1 .. $num_lines_to_del) > { > # Read one line. > scalar(<$in_fh>); > } > > my $buf_len = 1024 * 16; > > my $buffer; > while (read($in_fh, $buffer, $buf_len)) > { > print {$temp_out_fh} $buffer; > } > close($temp_out_fh); > close($in_fh); > > rename($temp_fn, $filename); > > Like I said - untested, but I hope you get the idea. > > Regards, > > Shlomi Fish If I read this right, the entire file will be copied, sans initial lines, and never use more that 1024*16 bytes of memory. That would certainly ensure I never eat up all the RAM, regardless of the file size being messed with. Awesome. Madi From linux at alteeve.com Thu Oct 1 08:47:38 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:47:38 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <1254327392.11743.0.camel@hatsya.starnix.com> References: <4AC37BA2.90608@alteeve.com> <1254327392.11743.0.camel@hatsya.starnix.com> Message-ID: <4AC4CF1A.40903@alteeve.com> G. Matthew Rice wrote: > On Wed, 2009-09-30 at 11:39 -0400, Madison Kelly wrote: >> Thus far, I've opened a file, read it in, shifted/ignored the first X >> number of line and then wrote out the resulting file. This works, but >> strikes me as horribly inefficient. Doubly so when the file could be >> larger than RAM. >> >> So then, is there a more efficient/faster way of saying "Delete the >> first X number of lines from file foo"? > > Do you have to do it in Perl? You could do: > > tail -n +N out.txt > > where: N = X + 1 > > Regards, > --matt I would much rather do it in perl. The module set this method is in is designed to be a pure-perl module. However, if the other methods don't work, it's another option to experiment with. As I mentioned to Antonio though, if yours or his methods shows a significant performance increase, I make make one of these two methods an optional way of pruning the file for people who know their program will be on a system with sed/tail. Thanks! Madi From linux at alteeve.com Thu Oct 1 08:53:46 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:53:46 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <87tyykv36x.fsf@quad.sysarch.com> References: <4AC37BA2.90608@alteeve.com> <87tyykv36x.fsf@quad.sysarch.com> Message-ID: <4AC4D08A.7040103@alteeve.com> Uri Guttman wrote: >>>>>> "MK" == Madison Kelly writes: > > MK> Hi all, > MK> Thus far, I've opened a file, read it in, shifted/ignored the first > MK> X number of line and then wrote out the resulting file. This works, > MK> but strikes me as horribly inefficient. Doubly so when the file could > MK> be larger than RAM. > > MK> So then, is there a more efficient/faster way of saying "Delete the > MK> first X number of lines from file foo"? > > this is a file system problem, not a perl one. the file system (other > than on things like vms) doesn't allow deleting from the front of the > file. it has to be copied in and out with the edits being done on the > fly. there are many perlish ways to do this including tie::file, reading > line by line and printing but ignoring the early lines using $. and > slurping in the whole file (file::slurp::read_file) into an array, > splicing off the lines and printing it out again (with > file::slurp::write_file). > > uri Hrm, I wasn't thinking about this at the file system level. I wonder then if shell tools will in fact be any faster than pure-perl implementations if they all need to read in -> write out the full file behind the scenes. Thanks for the datapoint! Madi From linux at alteeve.com Thu Oct 1 08:58:24 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 11:58:24 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <1254345601.7509.35.camel@desktop.barefootcomputing.com> References: <4AC37BA2.90608@alteeve.com> <87tyykv36x.fsf@quad.sysarch.com> <1254345601.7509.35.camel@desktop.barefootcomputing.com> Message-ID: <4AC4D1A0.9020507@alteeve.com> Liam R E Quin wrote: > On Wed, 2009-09-30 at 11:53 -0400, Uri Guttman wrote: >>>>>>> "MK" == Madison Kelly writes: >> MK> Hi all, >> MK> Thus far, I've opened a file, read it in, shifted/ignored the first >> MK> X number of line and then wrote out the resulting file. This works, >> MK> but strikes me as horribly inefficient. Doubly so when the file could >> MK> be larger than RAM. > > > In this case I'd consider a short C program that used mmap() and bcopy() > and ftrunc(). The point is to avoid per-line processing, and to avoid > having the whole file in memory (mmap() maps the file on disk into > memory locations without actually reading it). > > In perl, you can use sysread and syswrite to avoid per-line processing. I will look into these, thanks. > Watch out, if something is writing to the file while you do this, you > can end up with corruption, though. You can use file locking, Does perl support file locking? If so, how does an active file handle know when another perl process has locked a file? Will it pause and wait for the file to be unlocked? This is a good point I'd not thought of. My programs do handle forking and so could have multiple file handles writing to the same file. If perl doesn't, then I could use my own lock file that each thread checks before trying to write to the file and goes into a sleep look until the lock file is removed. Another good data point! > However, Liam's Rule of Optimization :-) is that the fastest way to do > something is not to do it at all. For example, what if you had a > separate file for every X lines? Then you'd just delete the oldest file. > > Or, buy more memory so it all fits :-) > > Liam I am hoping that my modules will be one day used by other people, so I want to keep memory requirements as modest as possible. Besides, I've never been fond of the "buy your way out of your problem" approach... If I did, I'd be writing this email on a Mac. ;) Madi From uri at StemSystems.com Thu Oct 1 09:31:02 2009 From: uri at StemSystems.com (Uri Guttman) Date: Thu, 01 Oct 2009 12:31:02 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <4AC4D08A.7040103@alteeve.com> (Madison Kelly's message of "Thu\, 01 Oct 2009 11\:53\:46 -0400") References: <4AC37BA2.90608@alteeve.com> <87tyykv36x.fsf@quad.sysarch.com> <4AC4D08A.7040103@alteeve.com> Message-ID: <874oqjjcu1.fsf@quad.sysarch.com> >>>>> "MK" == Madison Kelly writes: MK> Uri Guttman wrote: >> this is a file system problem, not a perl one. the file system (other >> than on things like vms) doesn't allow deleting from the front of the >> file. it has to be copied in and out with the edits being done on the >> fly. there are many perlish ways to do this including tie::file, reading >> line by line and printing but ignoring the early lines using $. and >> slurping in the whole file (file::slurp::read_file) into an array, >> splicing off the lines and printing it out again (with >> file::slurp::write_file). MK> Hrm, I wasn't thinking about this at the file system level. I wonder MK> then if shell tools will in fact be any faster than pure-perl MK> implementations if they all need to read in -> write out the full file MK> behind the scenes. specialized unix tools written in c will almost always be much faster than perl if they are doing what they do best. that is always the difference between a general purpose language vs a focused tool. uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From stuart at morungos.com Thu Oct 1 09:39:27 2009 From: stuart at morungos.com (Stuart Watt) Date: Thu, 01 Oct 2009 12:39:27 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <4AC4D1A0.9020507@alteeve.com> References: <4AC37BA2.90608@alteeve.com> <87tyykv36x.fsf@quad.sysarch.com> <1254345601.7509.35.camel@desktop.barefootcomputing.com> <4AC4D1A0.9020507@alteeve.com> Message-ID: <4AC4DB3F.3020001@morungos.com> Madison Kelly wrote: > Does perl support file locking? If so, how does an active file handle > know when another perl process has locked a file? Will it pause and > wait for the file to be unlocked? Kind of. The flock() function that does the main magic is usually available, but it does delegate a lot to the operating system or C runtime, and these vary a lot. The answer to the "how does it know" bit is at the OS level. Under a Perl handle is an OS handle, and that is what is used to ensure exclusive access. Generally Perl can't find the other process, but the OS prevents any other process from using the file. If you're on a UNIX-style system, flock() is usually OK (except on shared file systems), but on Windows, I have had problems with the emulated versions. Windows has its own API access to its own mutex systems, which are reliable but generally unpleasant to look at. I now tend to use a SQLite database, which provides exclusive transactions, as a way of synchronizing between separate processes, and I share data between processes that way too. I have learnt to avoid depending too much on Perl file system access on Windows. I have no idea how SQLite does what it does internally, but it seems a very simple solution to code, and one which is very reliable. I've also used MLDBM::Sync in the past, but with less success. I prefer using higher level modules to using flock() directly, as they often add additional logic that boosts portability. My experiences with direct flock() coding were not positive or rewarding. All the best Stuart -- Stuart Watt ARM Product Developer Information Balance -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomif at iglu.org.il Thu Oct 1 09:52:31 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Thu, 01 Oct 2009 18:52:31 +0200 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <4AC4CE76.3050001@alteeve.com> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> <4AC4CE76.3050001@alteeve.com> Message-ID: <200910011852.32117.shlomif@iglu.org.il> On Thursday 01 Oct 2009 17:44:54 Madison Kelly wrote: > Shlomi Fish wrote: > > On Wednesday 30 Sep 2009 17:39:14 Madison Kelly wrote: > >> Hi all, > >> > >> Thus far, I've opened a file, read it in, shifted/ignored the first X > >> number of line and then wrote out the resulting file. This works, but > >> strikes me as horribly inefficient. Doubly so when the file could be > >> larger than RAM. > >> > >> So then, is there a more efficient/faster way of saying "Delete the > >> first X number of lines from file foo"? > > > > What I would do in pure-Perl is: (untested) > > > > <<<<<<<<<<<<<<< > > my $num_lines_to_del = shift(@ARGV); > > my $filename = shift(@ARGV); > > # Maybe use File::Temp here > > my $temp_fn = $filename.".new"; > > > > open my $in_fh, "<", $filename > > or die "Could not open '$filename' - $! !"; > > > > > > open my $temp_out_fh, ">", $temp_fn > > or die "Could not open temp filename - $!"; > > > > foreach my $i (1 .. $num_lines_to_del) > > { > > # Read one line. > > scalar(<$in_fh>); > > } > > > > my $buf_len = 1024 * 16; > > > > my $buffer; > > while (read($in_fh, $buffer, $buf_len)) > > { > > print {$temp_out_fh} $buffer; > > } > > close($temp_out_fh); > > close($in_fh); > > > > rename($temp_fn, $filename); > > > > Like I said - untested, but I hope you get the idea. > > > > Regards, > > > > Shlomi Fish > > If I read this right, the entire file will be copied, sans initial > lines, and never use more that 1024*16 bytes of memory. That would > certainly ensure I never eat up all the RAM, regardless of the file size > being messed with. Awesome. That's mostly right. The perl 5 interpreter overhead is much more than 16KB of RAM, and it also has some overhead for every variable. Furthermore, if the first $num_lines_to_del lines are very long that it is possible that skipping them using "perldoc -f scalar" will consume much more memory. I'm not sure how smart is <$filehandle> is, but maybe it's better in void context. But you are right that this is a pure-Perl solution that mostly fulfills these constraints. Like I said - it's untested so all warnings apply. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Stop Using MSIE - http://www.shlomifish.org/no-ie/ Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Thu Oct 1 09:52:31 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Thu, 01 Oct 2009 18:52:31 +0200 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <4AC4CE76.3050001@alteeve.com> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> <4AC4CE76.3050001@alteeve.com> Message-ID: <200910011852.32117.shlomif@iglu.org.il> On Thursday 01 Oct 2009 17:44:54 Madison Kelly wrote: > Shlomi Fish wrote: > > On Wednesday 30 Sep 2009 17:39:14 Madison Kelly wrote: > >> Hi all, > >> > >> Thus far, I've opened a file, read it in, shifted/ignored the first X > >> number of line and then wrote out the resulting file. This works, but > >> strikes me as horribly inefficient. Doubly so when the file could be > >> larger than RAM. > >> > >> So then, is there a more efficient/faster way of saying "Delete the > >> first X number of lines from file foo"? > > > > What I would do in pure-Perl is: (untested) > > > > <<<<<<<<<<<<<<< > > my $num_lines_to_del = shift(@ARGV); > > my $filename = shift(@ARGV); > > # Maybe use File::Temp here > > my $temp_fn = $filename.".new"; > > > > open my $in_fh, "<", $filename > > or die "Could not open '$filename' - $! !"; > > > > > > open my $temp_out_fh, ">", $temp_fn > > or die "Could not open temp filename - $!"; > > > > foreach my $i (1 .. $num_lines_to_del) > > { > > # Read one line. > > scalar(<$in_fh>); > > } > > > > my $buf_len = 1024 * 16; > > > > my $buffer; > > while (read($in_fh, $buffer, $buf_len)) > > { > > print {$temp_out_fh} $buffer; > > } > > close($temp_out_fh); > > close($in_fh); > > > > rename($temp_fn, $filename); > > > > Like I said - untested, but I hope you get the idea. > > > > Regards, > > > > Shlomi Fish > > If I read this right, the entire file will be copied, sans initial > lines, and never use more that 1024*16 bytes of memory. That would > certainly ensure I never eat up all the RAM, regardless of the file size > being messed with. Awesome. That's mostly right. The perl 5 interpreter overhead is much more than 16KB of RAM, and it also has some overhead for every variable. Furthermore, if the first $num_lines_to_del lines are very long that it is possible that skipping them using "perldoc -f scalar" will consume much more memory. I'm not sure how smart is <$filehandle> is, but maybe it's better in void context. But you are right that this is a pure-Perl solution that mostly fulfills these constraints. Like I said - it's untested so all warnings apply. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Stop Using MSIE - http://www.shlomifish.org/no-ie/ Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From linux at alteeve.com Thu Oct 1 18:07:16 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 21:07:16 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <200909301817.57674.shlomif@iglu.org.il> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> Message-ID: <4AC55244.8030204@alteeve.com> Shlomi Fish wrote: > my $num_lines_to_del = shift(@ARGV); > my $filename = shift(@ARGV); > # Maybe use File::Temp here > my $temp_fn = $filename.".new"; > > open my $in_fh, "<", $filename > or die "Could not open '$filename' - $! !"; > > > open my $temp_out_fh, ">", $temp_fn > or die "Could not open temp filename - $!"; > > foreach my $i (1 .. $num_lines_to_del) > { > # Read one line. > scalar(<$in_fh>); > } > > my $buf_len = 1024 * 16; > > my $buffer; > while (read($in_fh, $buffer, $buf_len)) > { > print {$temp_out_fh} $buffer; > } > close($temp_out_fh); > close($in_fh); > > rename($temp_fn, $filename); > > Like I said - untested, but I hope you get the idea. > > Regards, > > Shlomi Fish I've decided to try this solution, and it works great! I'm still waiting to test it against very large files (> 1-2GB), but so far it seems very quick. Thank you again! Madi From linux at alteeve.com Thu Oct 1 18:07:16 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 01 Oct 2009 21:07:16 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <200909301817.57674.shlomif@iglu.org.il> References: <4AC37BA2.90608@alteeve.com> <200909301817.57674.shlomif@iglu.org.il> Message-ID: <4AC55244.8030204@alteeve.com> Shlomi Fish wrote: > my $num_lines_to_del = shift(@ARGV); > my $filename = shift(@ARGV); > # Maybe use File::Temp here > my $temp_fn = $filename.".new"; > > open my $in_fh, "<", $filename > or die "Could not open '$filename' - $! !"; > > > open my $temp_out_fh, ">", $temp_fn > or die "Could not open temp filename - $!"; > > foreach my $i (1 .. $num_lines_to_del) > { > # Read one line. > scalar(<$in_fh>); > } > > my $buf_len = 1024 * 16; > > my $buffer; > while (read($in_fh, $buffer, $buf_len)) > { > print {$temp_out_fh} $buffer; > } > close($temp_out_fh); > close($in_fh); > > rename($temp_fn, $filename); > > Like I said - untested, but I hope you get the idea. > > Regards, > > Shlomi Fish I've decided to try this solution, and it works great! I'm still waiting to test it against very large files (> 1-2GB), but so far it seems very quick. Thank you again! Madi From vvp at cogeco.ca Thu Oct 1 18:53:52 2009 From: vvp at cogeco.ca (Viktor Pavlenko) Date: Thu, 1 Oct 2009 21:53:52 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <4AC4CDD6.7050901@alteeve.com> References: <4AC37BA2.90608@alteeve.com> <19139.41890.796217.428615@hetman.ua> <4AC4CDD6.7050901@alteeve.com> Message-ID: <19141.23856.562266.113877@hetman.ua> >>>>> "MK" == Madison Kelly writes: >> perl -i -ne 'print if $. > 20' file MK> Where 'if' is the input file handle? I've not played with MK> '$.'... $. is the line number, starting from 1. `if' is just an `if', you can write it traditional way: if ($. > 20) { print }. It says to print $_ (omitted) if the line number is greater than 20. It's a perl one-liner, it allows you to read one line at a time and put it in $_, -n means don't print it, and -i tells perl to edit the file in place (all tmp file business is taken care of). What follows -e is the script to execute. MK> 'If this works it'd be the pure-perl way I was hoping MK> for. Do you know if this still reads the whole file into MK> memory at once? My biggest concern, even more than speed, is MK> that this may need to be done on files larger than available MK> RAM. I'm sure it reads a line at a time and then throws it away, so no worries about memory. Your point about inefficiency of reading & copying remains valid. HTH -- Viktor From legrady at gmail.com Thu Oct 1 19:20:36 2009 From: legrady at gmail.com (Tom Legrady) Date: Thu, 1 Oct 2009 22:20:36 -0400 Subject: [tpm] Fast(er) way of deleting X lines from start of a file In-Reply-To: <19141.23856.562266.113877@hetman.ua> References: <4AC37BA2.90608@alteeve.com> <19139.41890.796217.428615@hetman.ua> <4AC4CDD6.7050901@alteeve.com> <19141.23856.562266.113877@hetman.ua> Message-ID: <3c9af5830910011920w59cc666eh2276a976523b0c0a@mail.gmail.com> Use the perl documentation installed on your machine with the binary. perldoc perlrun describes all the command line options, including -p, -n, -i. You'll see the file is processed line by line. Of course the system buffers are the controlling factor for memory usager, if your lines are an ordinary length of 80 or so characters, since the buffers are a few KB in size On Thu, Oct 1, 2009 at 9:53 PM, Viktor Pavlenko wrote: > >>>>> "MK" == Madison Kelly writes: > > >> perl -i -ne 'print if $. > 20' file > > MK> Where 'if' is the input file handle? I've not played with > MK> '$.'... > > $. is the line number, starting from 1. `if' is just an `if', you can > write it traditional way: if ($. > 20) { print }. It says to print $_ > (omitted) if the line number is greater than 20. It's a perl > one-liner, it allows you to read one line at a time and put it in $_, > -n means don't print it, and -i tells perl to edit the file in place > (all tmp file business is taken care of). What follows -e is the > script to execute. > > MK> 'If this works it'd be the pure-perl way I was hoping > MK> for. Do you know if this still reads the whole file into > MK> memory at once? My biggest concern, even more than speed, is > MK> that this may need to be done on files larger than available > MK> RAM. > > I'm sure it reads a line at a time and then throws it away, so no > worries about memory. Your point about inefficiency of reading & > copying remains valid. > > HTH > > -- > Viktor > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thakore.kartik at gmail.com Fri Oct 2 14:37:42 2009 From: thakore.kartik at gmail.com (Kartik Thakore) Date: Fri, 2 Oct 2009 17:37:42 -0400 Subject: [tpm] Flexibly scheduling scripts In-Reply-To: <19139.65328.124377.206442@hetman.ua> References: <9B64CF03-E583-43C9-946F-9549EC50F796@gmail.com> <4AC3CE7E.5020008@morungos.com> <19139.59040.727446.596327@hetman.ua> <19139.62802.278169.457533@hetman.ua> <19139.65328.124377.206442@hetman.ua> Message-ID: <9A160D77-940A-45A9-9E86-0D508297451E@gmail.com> Hi guys, So this worked out really well!! I am no longer tied to my computer desk, and in less then a 150 lines! If any one would like to see the code for this I am going to put it on cpan. Kartik Thakore On 30-Sep-09, at 9:00 PM, Viktor Pavlenko wrote: >>>>>> "KT" == Kartik Thakore writes: > >>> How large can they be? Our most complicated schedules are 120 >>> jobs. > > KT> I have 12 scripts but due to args several different combos. I > KT> have 200+ entries. But I think I can cut them as extensions of > KT> scripts. > > Put some sleep in your loop and you will be fine. You don't change > your scripts every second, do you? The standard *NIX approach is to > send HUP signal when your config file changes. > > -- > Viktor > From linux at alteeve.com Fri Oct 2 17:37:02 2009 From: linux at alteeve.com (Madison Kelly) Date: Fri, 02 Oct 2009 20:37:02 -0400 Subject: [tpm] ftruncate (was: Fast(er) way of deleting X lines from start of a file) In-Reply-To: <4AC37BA2.90608@alteeve.com> References: <4AC37BA2.90608@alteeve.com> Message-ID: <4AC69CAE.4000801@alteeve.com> Madison Kelly wrote: > Hi all, > > Thus far, I've opened a file, read it in, shifted/ignored the first X > number of line and then wrote out the resulting file. This works, but > strikes me as horribly inefficient. Doubly so when the file could be > larger than RAM. > > So then, is there a more efficient/faster way of saying "Delete the > first X number of lines from file foo"? > > Thanks all! > > Madi While talking about this with my boss and colleague, one mentioned "ftruncate" as a possible solution. There was debate about whether this was implemented in perl though. I did a google on the topic, but it thought it was a typo for 'truncate'. Does such a beast exist? Madi From linux at alteeve.com Fri Oct 2 17:44:45 2009 From: linux at alteeve.com (Madison Kelly) Date: Fri, 02 Oct 2009 20:44:45 -0400 Subject: [tpm] Renaming a file with multiple scripts having opened filehandles on it Message-ID: <4AC69E7D.40202@alteeve.com> Hi all, Second question of the evening! My log method can also archive the log file once it reaches a certain size. Traditionally, I simply closed the file handle, renamed it to "log.1" and opened a new file handle to the old file name. However, Today while testing this, I noticed that if a second program/thread had an open file handle on the log file being 'rename'ed, it would follow the log file to the new name. This makes sense, now that I think of it (I am guessing it tracks the inode?). What I want to do though is have some way for other threads recognize that the file name has changed. That way I would be able to close() the existing file handle and reconnect to the new file with the old name. I use IO::Handle... Is there a way to do this? Thanks again! Madi From legrady at gmail.com Fri Oct 2 17:54:31 2009 From: legrady at gmail.com (Tom Legrady) Date: Fri, 2 Oct 2009 20:54:31 -0400 Subject: [tpm] ftruncate (was: Fast(er) way of deleting X lines from start of a file) In-Reply-To: <4AC69CAE.4000801@alteeve.com> References: <4AC37BA2.90608@alteeve.com> <4AC69CAE.4000801@alteeve.com> Message-ID: <3c9af5830910021754i5bf7b054x2596b338fd7dba2@mail.gmail.com> You probably made a typo typing 'ftruncate', googling it worked for me ... http://linux.about.com/library/cmd/blcmdl2_ftruncate.htm But it truncates a file, trimming away stuff from the end. There's 'head', there's 'tail', there's 'ftruncate', just not what you want .. I'd call it rhead, 'reverse head', or maybe 'allbut' Tom O, n Fri, Oct 2, 2009 at 8:37 PM, Madison Kelly wrote: > Madison Kelly wrote: > >> Hi all, >> >> Thus far, I've opened a file, read it in, shifted/ignored the first X >> number of line and then wrote out the resulting file. This works, but >> strikes me as horribly inefficient. Doubly so when the file could be larger >> than RAM. >> >> So then, is there a more efficient/faster way of saying "Delete the first >> X number of lines from file foo"? >> >> Thanks all! >> >> Madi >> > > While talking about this with my boss and colleague, one mentioned > "ftruncate" as a possible solution. There was debate about whether this was > implemented in perl though. I did a google on the topic, but it thought it > was a typo for 'truncate'. Does such a beast exist? > > Madi > > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From liam at holoweb.net Fri Oct 2 18:45:16 2009 From: liam at holoweb.net (Liam R E Quin) Date: Fri, 02 Oct 2009 21:45:16 -0400 Subject: [tpm] Renaming a file with multiple scripts having opened filehandles on it In-Reply-To: <4AC69E7D.40202@alteeve.com> References: <4AC69E7D.40202@alteeve.com> Message-ID: <1254534316.20063.81.camel@desktop.barefootcomputing.com> On Fri, 2009-10-02 at 20:44 -0400, Madison Kelly wrote: > [...] I noticed that if a second program/thread > had an open file handle on the log file being 'rename'ed, it would > follow the log file to the new name. This makes sense, now that I think > of it (I am guessing it tracks the inode?). This is platform specific, but yes, on Unix-like systems, if you rename a file, something writing to that file will continue to write to it under the new name. On a Unix (and Linux) file system, the directory entries (e.g. for "/etc/group" I mean the "group" entry in the "/etc" directory) just point to where the data is stored - this is why when you use "ln group fizz", there are now two names for the same data, and if you remove either one, the other remains behind. This is different from symbolic links, ln -s, of course. > > What I want to do though is have some way for other threads recognize > that the file name has changed. You'd have to notify them, e.g. with a shared variable or a signal of some sort. Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From uri at StemSystems.com Fri Oct 2 20:47:06 2009 From: uri at StemSystems.com (Uri Guttman) Date: Fri, 02 Oct 2009 23:47:06 -0400 Subject: [tpm] Renaming a file with multiple scripts having opened filehandles on it In-Reply-To: <1254534316.20063.81.camel@desktop.barefootcomputing.com> (Liam R. E. Quin's message of "Fri\, 02 Oct 2009 21\:45\:16 -0400") References: <4AC69E7D.40202@alteeve.com> <1254534316.20063.81.camel@desktop.barefootcomputing.com> Message-ID: <87tyyhyw91.fsf@quad.sysarch.com> >>>>> "LREQ" == Liam R E Quin writes: LREQ> On Fri, 2009-10-02 at 20:44 -0400, Madison Kelly wrote: >> [...] I noticed that if a second program/thread >> had an open file handle on the log file being 'rename'ed, it would >> follow the log file to the new name. This makes sense, now that I think >> of it (I am guessing it tracks the inode?). LREQ> This is platform specific, but yes, on Unix-like systems, if you LREQ> rename a file, something writing to that file will continue to write LREQ> to it under the new name. On a Unix (and Linux) file system, LREQ> the directory entries (e.g. for "/etc/group" I mean the "group" LREQ> entry in the "/etc" directory) just point to where the data is LREQ> stored - this is why when you use "ln group fizz", there are LREQ> now two names for the same data, and if you remove either one, LREQ> the other remains behind. This is different from symbolic links, LREQ> ln -s, of course. >> >> What I want to do though is have some way for other threads recognize >> that the file name has changed. LREQ> You'd have to notify them, e.g. with a shared variable or a signal of LREQ> some sort. this is also why the best loggers use a server to do the logging and the programs send log message to it. then you only have one program opening and writing to the log files and you can manage/rotate them cleanly. having multiple processes logging to the same file is a poor design. one other way around it besides a shared variable or such is to mark the log dir with a flag file. this can be checked each time a process wants to write to the log file. it will slow things down but allow for the current design to work with log rotation. uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From linux at alteeve.com Fri Oct 2 20:53:48 2009 From: linux at alteeve.com (Madison Kelly) Date: Fri, 02 Oct 2009 23:53:48 -0400 Subject: [tpm] Renaming a file with multiple scripts having opened filehandles on it In-Reply-To: <87tyyhyw91.fsf@quad.sysarch.com> References: <4AC69E7D.40202@alteeve.com> <1254534316.20063.81.camel@desktop.barefootcomputing.com> <87tyyhyw91.fsf@quad.sysarch.com> Message-ID: <4AC6CACC.3030902@alteeve.com> Uri Guttman wrote: >>>>>> "LREQ" == Liam R E Quin writes: > > LREQ> On Fri, 2009-10-02 at 20:44 -0400, Madison Kelly wrote: > >> [...] I noticed that if a second program/thread > >> had an open file handle on the log file being 'rename'ed, it would > >> follow the log file to the new name. This makes sense, now that I think > >> of it (I am guessing it tracks the inode?). > > > LREQ> This is platform specific, but yes, on Unix-like systems, if you > LREQ> rename a file, something writing to that file will continue to write > LREQ> to it under the new name. On a Unix (and Linux) file system, > LREQ> the directory entries (e.g. for "/etc/group" I mean the "group" > LREQ> entry in the "/etc" directory) just point to where the data is > LREQ> stored - this is why when you use "ln group fizz", there are > LREQ> now two names for the same data, and if you remove either one, > LREQ> the other remains behind. This is different from symbolic links, > LREQ> ln -s, of course. > > >> > >> What I want to do though is have some way for other threads recognize > >> that the file name has changed. > > LREQ> You'd have to notify them, e.g. with a shared variable or a signal of > LREQ> some sort. > > this is also why the best loggers use a server to do the logging and the > programs send log message to it. then you only have one program opening > and writing to the log files and you can manage/rotate them > cleanly. having multiple processes logging to the same file is a poor > design. one other way around it besides a shared variable or such is to > mark the log dir with a flag file. this can be checked each time a > process wants to write to the log file. it will slow things down but > allow for the current design to work with log rotation. > > uri > Thank you both. I think I might try setting up a DBus-based server that does nothing more that listen for messages and triggers cycle'ing/truncating/whatever. Madi From arocker at vex.net Sat Oct 3 13:26:25 2009 From: arocker at vex.net (arocker at vex.net) Date: Sat, 3 Oct 2009 16:26:25 -0400 Subject: [tpm] ftruncate (was: Fast(er) way of deleting X lines from start of a file) Message-ID: <2e8a6c4b72e2900cbe2f045d02938c55.squirrel@webmail.vex.net> > Madison Kelly wrote: > This works, but strikes me as horribly inefficient. Beware of assumptions about performance; measure first, then decide if optimisation is justified. What's your time worth, compared to the system's? If it takes ten seconds rather than 5 to do a daily job, who cares? The same numbers for every login might be a different matter. From indy at indigostar.com Tue Oct 6 08:59:51 2009 From: indy at indigostar.com (Indy Singh) Date: Tue, 6 Oct 2009 11:59:51 -0400 Subject: [tpm] Apache permission question References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> Message-ID: <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> Hi All, I have a Perl CGI script that creates log files, and a second Perl CGI script that is used to view and delete the log files. However the second script is not able to delete the files. The first script is located in the Apache cgi-bin directory, the second is located in a subdirectory below htdocs. This allows the second script to be password protected with a .htaccess file. The configuration of this Apache server runs files in the cgi-bin directory as cgi scripts and any perl script below htdocs run as mod-perl. I am happy with this aspect of the configuration. The problem is that the second script does not have permission to delete the log files. I have found that this is because the two scripts run under a different used id. The cgi-bin scirpt runs as me, the second script runs as 'Apache'. Anyone have any ideas on how to resolve this issue. Preferrably without mucking around with the Apache configuration. Indy Singh IndigoSTAR Software -- www.indigostar.com From antoniosun at lavabit.com Tue Oct 6 09:21:44 2009 From: antoniosun at lavabit.com (Antonio Sun) Date: Tue, 6 Oct 2009 12:21:44 -0400 Subject: [tpm] Apache permission question In-Reply-To: <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> Message-ID: On Tue, Oct 6, 2009 at 11:59 AM, Indy Singh wrote: > The problem is that the second script does not have permission to delete > the log files. I have found that this is because the two scripts run under > a different used id. The cgi-bin scirpt runs as me, the second script runs > as 'Apache'. Please give the 'ls -l' of the log file. I suspect that it's only the unix's file permission problem. Also keep in mind that if the 2nd script might not be able to delete the log file unless you do some log rotates. cheers -------------- next part -------------- An HTML attachment was scrubbed... URL: From ejanev at gmail.com Tue Oct 6 09:28:35 2009 From: ejanev at gmail.com (Emil Janev) Date: Tue, 6 Oct 2009 12:28:35 -0400 Subject: [tpm] Apache permission question In-Reply-To: <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> Message-ID: Hi Indy, It seems that changing the file and/or directory permissions ( making them less restrictive ) would fix the particular problem. Regards, Emil On Tue, Oct 6, 2009 at 11:59 AM, Indy Singh wrote: > Hi All, > > I have a Perl CGI script that creates log files, and a second Perl CGI > script that is used to view and delete the log files. However the second > script is not able to delete the files. > > The first script is located in the Apache cgi-bin directory, the second is > located in a subdirectory below htdocs. This allows the second script to be > password protected with a .htaccess file. The configuration of this Apache > server runs files in the cgi-bin directory as cgi scripts and any perl > script below htdocs run as mod-perl. I am happy with this aspect of the > configuration. > > The problem is that the second script does not have permission to delete > the log files. I have found that this is because the two scripts run under > a different used id. The cgi-bin scirpt runs as me, the second script runs > as 'Apache'. > > Anyone have any ideas on how to resolve this issue. Preferrably without > mucking around with the Apache configuration. > > Indy Singh > IndigoSTAR Software -- www.indigostar.com > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -- Emil Janev -------------- next part -------------- An HTML attachment was scrubbed... URL: From indy at indigostar.com Tue Oct 6 09:29:17 2009 From: indy at indigostar.com (Indy Singh) Date: Tue, 6 Oct 2009 12:29:17 -0400 Subject: [tpm] Apache permission question References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> Message-ID: <7C10603D8C554570A29887A8AACA705E@ROADKILL> ls -l returns: -rw-r--r-- 1 indigo psacln 1061738 Aug 31 23:57 /var/www/vhosts/indigostar.com/httpdocs/private/logs/200908-index.log whoami returns 'apache' in this script and returns 'indigo' in the other script. It looks like a user id issue. I don't really want to give the file world write permission. Although group write pemission might work. I am looking for ideas on a 'best practice' solution to this. Indy Singh IndigoSTAR Software -- www.indigostar.com ----- Original Message ----- From: Antonio Sun To: Indy Singh Cc: tpm at to.pm.org Sent: Tuesday, October 06, 2009 12:21 PM Subject: Re: [tpm] Apache permission question On Tue, Oct 6, 2009 at 11:59 AM, Indy Singh wrote: The problem is that the second script does not have permission to delete the log files. I have found that this is because the two scripts run under a different used id. The cgi-bin scirpt runs as me, the second script runs as 'Apache'. Please give the 'ls -l' of the log file. I suspect that it's only the unix's file permission problem. Also keep in mind that if the 2nd script might not be able to delete the log file unless you do some log rotates. cheers From talexb at gmail.com Tue Oct 6 13:25:17 2009 From: talexb at gmail.com (Alex Beamish) Date: Tue, 6 Oct 2009 16:25:17 -0400 Subject: [tpm] Apache permission question In-Reply-To: <7C10603D8C554570A29887A8AACA705E@ROADKILL> References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> <7C10603D8C554570A29887A8AACA705E@ROADKILL> Message-ID: Hi Indy, Typically you'd just have 'apache' and 'indigo' in the same group, and make the log file group writable. That way you avoid having the file world writable, which is undesirable. Alex On 10/6/09, Indy Singh wrote: > ls -l returns: > -rw-r--r-- 1 indigo psacln 1061738 Aug 31 23:57 > /var/www/vhosts/indigostar.com/httpdocs/private/logs/200908-index.log > > whoami returns 'apache' in this script and returns 'indigo' in the other > script. > > It looks like a user id issue. I don't really want to give the file > world write permission. Although group write pemission might work. > > I am looking for ideas on a 'best practice' solution to this. > > Indy Singh > IndigoSTAR Software -- www.indigostar.com > > > ----- Original Message ----- > From: Antonio Sun > To: Indy Singh > Cc: tpm at to.pm.org > Sent: Tuesday, October 06, 2009 12:21 PM > Subject: Re: [tpm] Apache permission question > > > > > > On Tue, Oct 6, 2009 at 11:59 AM, Indy Singh wrote: > > The problem is that the second script does not have permission to delete > the log files. I have found that this is because the two scripts run > under a different used id. The cgi-bin scirpt runs as me, the second > script runs as 'Apache'. > > Please give the 'ls -l' of the log file. I suspect that it's only the > unix's file permission problem. > > Also keep in mind that if the 2nd script might not be able to delete the > log file unless you do some log rotates. > > cheers > > > > > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -- Alex Beamish Toronto, Ontario aka talexb From arocker at vex.net Tue Oct 6 15:19:50 2009 From: arocker at vex.net (arocker at vex.net) Date: Tue, 6 Oct 2009 18:19:50 -0400 Subject: [tpm] Apache permission question In-Reply-To: <7C10603D8C554570A29887A8AACA705E@ROADKILL> References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca> <6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL> <7C10603D8C554570A29887A8AACA705E@ROADKILL> Message-ID: > > It looks like a user id issue. I don't really want to give the file > world write permission. To delete the file, the user has to have write permission on the directory containing the file, not the file itself. Make the directory's permissions rwxrwxr-x (775) and add Apache to the group owning the directory. That will allow a script running as Apache to delete files. (I think it would work even if the "world" permissions were as tight as ---) From indy at indigostar.com Tue Oct 6 16:59:20 2009 From: indy at indigostar.com (Indy Singh) Date: Tue, 6 Oct 2009 19:59:20 -0400 Subject: [tpm] Apache permission question References: <3CFC3BB2-FD05-4E05-AC1E-0B523F0DBBAD@isx.ca><6BD14C8FB79F40128EBADDAAF165D8CC@ROADKILL><7C10603D8C554570A29887A8AACA705E@ROADKILL> Message-ID: > To delete the file, the user has to have write permission on the > directory > containing the file, not the file itself. Aha, that works. Simple solution for my needs. Thanks. Thanks everyone for your suggestions. Indy Singh IndigoSTAR Software -- www.indigostar.com ----- Original Message ----- From: To: Sent: Tuesday, October 06, 2009 6:19 PM Subject: Re: [tpm] Apache permission question > > >> It looks like a user id issue. I don't really want to give the file >> world write permission. > > To delete the file, the user has to have write permission on the > directory > containing the file, not the file itself. > > Make the directory's permissions rwxrwxr-x (775) and add Apache to > the > group owning the directory. That will allow a script running as Apache > to > delete files. (I think it would work even if the "world" permissions > were > as tight as ---) > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm From thakore.kartik at gmail.com Fri Oct 9 04:57:08 2009 From: thakore.kartik at gmail.com (Kartik Thakore) Date: Fri, 9 Oct 2009 07:57:08 -0400 Subject: [tpm] YAPC::NA 2010 Message-ID: <5CD719D3-3D40-43B0-A235-DDF72A0FA0E3@gmail.com> Where and when is the venue for YAPC::NA? Is there a site somewhere? Kartik Thakore From dave.s.doyle at gmail.com Fri Oct 9 05:08:48 2009 From: dave.s.doyle at gmail.com (Dave Doyle) Date: Fri, 9 Oct 2009 08:08:48 -0400 Subject: [tpm] YAPC::NA 2010 In-Reply-To: <5CD719D3-3D40-43B0-A235-DDF72A0FA0E3@gmail.com> References: <5CD719D3-3D40-43B0-A235-DDF72A0FA0E3@gmail.com> Message-ID: Indeed there is: http://yapc2010.com/ Columbus, Ohio The date is not picked quite yet. Typically it's held in the early twenties of June though their proposal document lists two possible times; June 7 - 9 or June 21 - 23. I don't know if the venue is for sure but it looks like it will be Ohio State University. I took a charter bus to Columbus once for a Toronto FC game and it took about 8 hours. I'm sure the drive would take less time if you did it yourself. Google Maps suggests just under 7 hours. Dave -- dave.s.doyle at gmail.com On Fri, Oct 9, 2009 at 7:57 AM, Kartik Thakore wrote: > Where and when is the venue for YAPC::NA? Is there a site somewhere? > > Kartik Thakore > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at starnix.com Fri Oct 9 09:55:40 2009 From: matt at starnix.com (G. Matthew Rice) Date: Fri, 09 Oct 2009 12:55:40 -0400 Subject: [tpm] YAPC::NA 2010 In-Reply-To: References: <5CD719D3-3D40-43B0-A235-DDF72A0FA0E3@gmail.com> Message-ID: <1255107340.11509.0.camel@hatsya.starnix.com> On Fri, 2009-10-09 at 08:08 -0400, Dave Doyle wrote: > I took a charter bus to Columbus once for a Toronto FC game and it > took about 8 hours. I'm sure the drive would take less time if you did > it yourself. Google Maps suggests just under 7 hours. Is it too early for a carpooling call? :) --matt From quantum.mechanic.1964 at gmail.com Sat Oct 10 19:53:28 2009 From: quantum.mechanic.1964 at gmail.com (Quantum Mechanic) Date: Sat, 10 Oct 2009 22:53:28 -0400 Subject: [tpm] Larry was only joking Message-ID: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> While wasting time today, I visited the Wikiquote page for Larry Wall. Among the numerous gems was this one: Hey, I had to let awk be better at *something*... :-) > The awk in question looked like this: : awk is much better than perl at one-line summations: > : awk '{t += $1} END {print t}' > Which was followed up by this: : With perl you can say > : perl -ane '$t += $F[0]' > : but you have no way of tacking on an END clause. > Now, that was a long time ago, so it may have been true then. But certainly not anymore, and not for a long time: perl -lane '$t += $F[0] }{ print $t' Not that I'm trying to golf it, but that seems reasonably simple (especially compared with the attempts in the original thread). I first saw this idea in a Usenet post by Abigail on how to write wc in Perl: perl -pe '}{$_=$.' [The '}{' seems to be called the "Eskimo Greeting".] -- -QM Quantum Mechanics: The dreams stuff is made of -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomif at iglu.org.il Sun Oct 11 06:09:02 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Sun, 11 Oct 2009 15:09:02 +0200 Subject: [tpm] Larry was only joking In-Reply-To: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> References: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> Message-ID: <200910111509.03287.shlomif@iglu.org.il> On Sunday 11 Oct 2009 04:53:28 Quantum Mechanic wrote: > While wasting time today, I visited the Wikiquote page for Larry > Wall. > Among the numerous gems was this one: > > Hey, I had to let awk be better at *something*... > > :-) > > The awk in question looked like this: > : awk is much better than perl at one-line summations: > > : awk '{t += $1} END {print t}' > > Which was followed up by this: > : With perl you can say > : > > : perl -ane '$t += $F[0]' > > : but you have no way of tacking on an END clause. > > Now, that was a long time ago, so it may have been true then. But certainly > not anymore, and not for a long time: > > perl -lane '$t += $F[0] }{ print $t' > Heh nice. Golf put aside, you can still use an explicit END {... } clause in a -p or -n program. BTW, I talked with Larry Wall on IRC and from what he recalled perl's -i flag was his innovation. Later on, the GNU people implemented a similar "-i" flag in GNU sed. The POSIX sed standard still doesn't have it: http://www.opengroup.org/onlinepubs/000095399/utilities/sed.html It's possible that the sed implementations of the BSDs (FreeBSD, NetBSD, OpenBSD, etc.) have implemented a similar flag, but it may still be absent from the seds of older and or less actively maintained distributions (e.g: HP/UX , AIX , Win32's Services For Unix , maybe Solaris , etc.) Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ "The Human Hacking Field Guide" - http://shlom.in/hhfg Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Sun Oct 11 06:09:02 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Sun, 11 Oct 2009 15:09:02 +0200 Subject: [tpm] Larry was only joking In-Reply-To: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> References: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> Message-ID: <200910111509.03287.shlomif@iglu.org.il> On Sunday 11 Oct 2009 04:53:28 Quantum Mechanic wrote: > While wasting time today, I visited the Wikiquote page for Larry > Wall. > Among the numerous gems was this one: > > Hey, I had to let awk be better at *something*... > > :-) > > The awk in question looked like this: > : awk is much better than perl at one-line summations: > > : awk '{t += $1} END {print t}' > > Which was followed up by this: > : With perl you can say > : > > : perl -ane '$t += $F[0]' > > : but you have no way of tacking on an END clause. > > Now, that was a long time ago, so it may have been true then. But certainly > not anymore, and not for a long time: > > perl -lane '$t += $F[0] }{ print $t' > Heh nice. Golf put aside, you can still use an explicit END {... } clause in a -p or -n program. BTW, I talked with Larry Wall on IRC and from what he recalled perl's -i flag was his innovation. Later on, the GNU people implemented a similar "-i" flag in GNU sed. The POSIX sed standard still doesn't have it: http://www.opengroup.org/onlinepubs/000095399/utilities/sed.html It's possible that the sed implementations of the BSDs (FreeBSD, NetBSD, OpenBSD, etc.) have implemented a similar flag, but it may still be absent from the seds of older and or less actively maintained distributions (e.g: HP/UX , AIX , Win32's Services For Unix , maybe Solaris , etc.) Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ "The Human Hacking Field Guide" - http://shlom.in/hhfg Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From quantum.mechanic.1964 at gmail.com Sun Oct 11 12:09:03 2009 From: quantum.mechanic.1964 at gmail.com (Quantum Mechanic) Date: Sun, 11 Oct 2009 15:09:03 -0400 Subject: [tpm] Larry was only joking In-Reply-To: <200910111509.03287.shlomif@iglu.org.il> References: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> <200910111509.03287.shlomif@iglu.org.il> Message-ID: <77f3972e0910111209x1975c0bemf018f040f3828fd8@mail.gmail.com> On Sun, Oct 11, 2009 at 9:09 AM, Shlomi Fish wrote: > On Sunday 11 Oct 2009 04:53:28 Quantum Mechanic wrote: > [snip] > > The awk in question looked like this: > > : awk is much better than perl at one-line summations: > > > : awk '{t += $1} END {print t}' > > > > Which was followed up by this: > > : With perl you can say > > : > > > : perl -ane '$t += $F[0]' > > > : but you have no way of tacking on an END clause. > > > > Now, that was a long time ago, so it may have been true then. But > certainly > > not anymore, and not for a long time: > > > > perl -lane '$t += $F[0] }{ print $t' > > > > Heh nice. Golf put aside, you can still use an explicit END {... } clause > in a > -p or -n program. > Yes, that's true. I guess it bugs me that an END is inside the loop, even though it's only registered once. I guess I was also toying with the golf aspect, and the Eskimo Greeting. Probably too much obfu, when an END is much clearer: perl -lane '$t += $F[0]; END {print $t}' BTW, I talked with Larry Wall on IRC and from what he recalled perl's -i > flag > was his innovation. Later on, the GNU people implemented a similar "-i" > flag > in GNU sed. The POSIX sed standard still doesn't have it: > > http://www.opengroup.org/onlinepubs/000095399/utilities/sed.html > > It's possible that the sed implementations of the BSDs (FreeBSD, NetBSD, > OpenBSD, etc.) have implemented a similar flag, but it may still be absent > from the seds of older and or less actively maintained distributions (e.g: > HP/UX , AIX , Win32's Services For Unix , maybe Solaris , etc.) > Sometimes good ideas get loose, and everyone wants one. =) -- -QM Quantum Mechanics: The dreams stuff is made of -------------- next part -------------- An HTML attachment was scrubbed... URL: From shlomif at iglu.org.il Sat Oct 17 18:03:25 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Sun, 18 Oct 2009 03:03:25 +0200 Subject: [tpm] Larry was only joking In-Reply-To: <77f3972e0910111209x1975c0bemf018f040f3828fd8@mail.gmail.com> References: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> <200910111509.03287.shlomif@iglu.org.il> <77f3972e0910111209x1975c0bemf018f040f3828fd8@mail.gmail.com> Message-ID: <200910180303.25989.shlomif@iglu.org.il> Hi Quantum Mechanic! Sorry for the late-ish response. On Sunday 11 Oct 2009 21:09:03 Quantum Mechanic wrote: > On Sun, Oct 11, 2009 at 9:09 AM, Shlomi Fish wrote: > > On Sunday 11 Oct 2009 04:53:28 Quantum Mechanic wrote: > > [snip] > > > > The awk in question looked like this: > > > : awk is much better than perl at one-line summations: > > > > : awk '{t += $1} END {print t}' > > > > > > Which was followed up by this: > > > : With perl you can say > > > : > > > > : perl -ane '$t += $F[0]' > > > > : but you have no way of tacking on an END clause. > > > > > > Now, that was a long time ago, so it may have been true then. But > > > > certainly > > > > > not anymore, and not for a long time: > > > > > > perl -lane '$t += $F[0] }{ print $t' > > > > Heh nice. Golf put aside, you can still use an explicit END {... } clause > > in a > > -p or -n program. > > Yes, that's true. :-) > I guess it bugs me that an END is inside the loop, even > though it's only registered once. I guess I was also toying with the golf > aspect, and the Eskimo Greeting. Probably too much obfu, when an END is > much clearer: > > perl -lane '$t += $F[0]; END {print $t}' > Yeah well, this may be a colour of the bike shed argument: * http://bikeshed.com/ (try hitting reload a few times - ;-) ) * http://en.wikipedia.org/wiki/Parkinson%27s_Law_of_Triviality Or it's a matter of style and preference. I think the meaning of the second program is clearer, and it's most probably not substantially slower, because it only has a small overhead of compilation. "Eskimo greeting" - heh. Reminds me of "the spaceship operator", "the Orcish manuever", "the goatse operator", and probably a few other things I don't readily recall for Perl's syntactical paradigms that some people claim is line-noise. I happen to think that when they are properly formatted, they actually provide very useful visual aids that allow to see the meaning of the code very quickly. See for example: http://www.shlomifish.org/humour/fortunes/shlomif.html#scary-perl-expertise > > BTW, I talked with Larry Wall on IRC and from what he recalled perl's -i > > flag > > was his innovation. Later on, the GNU people implemented a similar "-i" > > flag > > in GNU sed. The POSIX sed standard still doesn't have it: > > > > http://www.opengroup.org/onlinepubs/000095399/utilities/sed.html > > > > It's possible that the sed implementations of the BSDs (FreeBSD, NetBSD, > > OpenBSD, etc.) have implemented a similar flag, but it may still be > > absent from the seds of older and or less actively maintained > > distributions (e.g: HP/UX , AIX , Win32's Services For Unix , maybe > > Solaris , etc.) > > Sometimes good ideas get loose, and everyone wants one. =) > Yeah. I've been thinking that in software (and in many other fields too) you get credit for results, not for originality. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ First stop for Perl beginners - http://perl-begin.org/ Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Sat Oct 17 18:03:25 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Sun, 18 Oct 2009 03:03:25 +0200 Subject: [tpm] Larry was only joking In-Reply-To: <77f3972e0910111209x1975c0bemf018f040f3828fd8@mail.gmail.com> References: <77f3972e0910101953n5d235816k40cc6d0038658500@mail.gmail.com> <200910111509.03287.shlomif@iglu.org.il> <77f3972e0910111209x1975c0bemf018f040f3828fd8@mail.gmail.com> Message-ID: <200910180303.25989.shlomif@iglu.org.il> Hi Quantum Mechanic! Sorry for the late-ish response. On Sunday 11 Oct 2009 21:09:03 Quantum Mechanic wrote: > On Sun, Oct 11, 2009 at 9:09 AM, Shlomi Fish wrote: > > On Sunday 11 Oct 2009 04:53:28 Quantum Mechanic wrote: > > [snip] > > > > The awk in question looked like this: > > > : awk is much better than perl at one-line summations: > > > > : awk '{t += $1} END {print t}' > > > > > > Which was followed up by this: > > > : With perl you can say > > > : > > > > : perl -ane '$t += $F[0]' > > > > : but you have no way of tacking on an END clause. > > > > > > Now, that was a long time ago, so it may have been true then. But > > > > certainly > > > > > not anymore, and not for a long time: > > > > > > perl -lane '$t += $F[0] }{ print $t' > > > > Heh nice. Golf put aside, you can still use an explicit END {... } clause > > in a > > -p or -n program. > > Yes, that's true. :-) > I guess it bugs me that an END is inside the loop, even > though it's only registered once. I guess I was also toying with the golf > aspect, and the Eskimo Greeting. Probably too much obfu, when an END is > much clearer: > > perl -lane '$t += $F[0]; END {print $t}' > Yeah well, this may be a colour of the bike shed argument: * http://bikeshed.com/ (try hitting reload a few times - ;-) ) * http://en.wikipedia.org/wiki/Parkinson%27s_Law_of_Triviality Or it's a matter of style and preference. I think the meaning of the second program is clearer, and it's most probably not substantially slower, because it only has a small overhead of compilation. "Eskimo greeting" - heh. Reminds me of "the spaceship operator", "the Orcish manuever", "the goatse operator", and probably a few other things I don't readily recall for Perl's syntactical paradigms that some people claim is line-noise. I happen to think that when they are properly formatted, they actually provide very useful visual aids that allow to see the meaning of the code very quickly. See for example: http://www.shlomifish.org/humour/fortunes/shlomif.html#scary-perl-expertise > > BTW, I talked with Larry Wall on IRC and from what he recalled perl's -i > > flag > > was his innovation. Later on, the GNU people implemented a similar "-i" > > flag > > in GNU sed. The POSIX sed standard still doesn't have it: > > > > http://www.opengroup.org/onlinepubs/000095399/utilities/sed.html > > > > It's possible that the sed implementations of the BSDs (FreeBSD, NetBSD, > > OpenBSD, etc.) have implemented a similar flag, but it may still be > > absent from the seds of older and or less actively maintained > > distributions (e.g: HP/UX , AIX , Win32's Services For Unix , maybe > > Solaris , etc.) > > Sometimes good ideas get loose, and everyone wants one. =) > Yeah. I've been thinking that in software (and in many other fields too) you get credit for results, not for originality. Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ First stop for Perl beginners - http://perl-begin.org/ Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From linux at alteeve.com Wed Oct 21 13:01:09 2009 From: linux at alteeve.com (Madison Kelly) Date: Wed, 21 Oct 2009 16:01:09 -0400 Subject: [tpm] Projector and print outs. Message-ID: <4ADF6885.8070708@alteeve.com> Hi all, Will a projector be available for next Thursday's talk? If so, would I be able to stop in this week to test it out? I've not yet used a projector with the laptop and though it *should* work, I'd hate to find out I was wrong half an hour before the talk. :) Also, I'd like to print out or make available as a PDF the contents of the talk before hand so that people can follow along. Mainly because I won't have the time to go into as much detail as I would like and the paper is more complete. So then, how many people would like a print out? Or more specifically, how many people should I plan on having attend? If I remember from last time, it was about 15 people. Should I just play it safe and run off 15 copies? If you are coming, and have a laptop, I can give you a copy of the PDF before starting. This would save some trees. :) Madi From liam at holoweb.net Wed Oct 21 13:16:53 2009 From: liam at holoweb.net (Liam R E Quin) Date: Wed, 21 Oct 2009 16:16:53 -0400 Subject: [tpm] Projector and print outs. In-Reply-To: <4ADF6885.8070708@alteeve.com> References: <4ADF6885.8070708@alteeve.com> Message-ID: <1256156213.23589.98.camel@desktop.barefootcomputing.com> On Wed, 2009-10-21 at 16:01 -0400, Madison Kelly wrote: > Hi all, > > Will a projector be available for next Thursday's talk? If so, would > I be able to stop in this week to test it out? I've not yet used a > projector with the laptop and though it *should* work, I'd hate to find > out I was wrong half an hour before the talk. :) If with Linux, the laptop function key (e.g. fn-f2 or fn-f5 is common) should work, but if not, I have a desktop gnome launcher that runs, xrandr --output VGA-0 --mode 1024x768 and that will work except for very old projectors that need 800x600. It's always worth testing beforehand, though. Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From legrady at gmail.com Wed Oct 21 15:22:43 2009 From: legrady at gmail.com (Tom Legrady) Date: Wed, 21 Oct 2009 18:22:43 -0400 Subject: [tpm] Projector and print outs. In-Reply-To: <1256156213.23589.98.camel@desktop.barefootcomputing.com> References: <4ADF6885.8070708@alteeve.com> <1256156213.23589.98.camel@desktop.barefootcomputing.com> Message-ID: <3c9af5830910211522p18680aber4acd0403d092efe6@mail.gmail.com> If you have the PDF on a spare USB key, people can pass it around and make themselves a copy. You might even get it back at the end! On Wed, Oct 21, 2009 at 4:16 PM, Liam R E Quin wrote: > On Wed, 2009-10-21 at 16:01 -0400, Madison Kelly wrote: > > Hi all, > > > > Will a projector be available for next Thursday's talk? If so, would > > I be able to stop in this week to test it out? I've not yet used a > > projector with the laptop and though it *should* work, I'd hate to find > > out I was wrong half an hour before the talk. :) > > If with Linux, the laptop function key (e.g. fn-f2 or fn-f5 is common) > should work, but if not, I have a desktop gnome launcher that runs, > xrandr --output VGA-0 --mode 1024x768 > and that will work except for very old projectors that need 800x600. > > It's always worth testing beforehand, though. > > Liam > > > -- > Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ > Pictures from old books: http://fromoldbooks.org/ > Ankh: irc.sorcery.net irc.gnome.org www.advogato.org > > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > http://mail.pm.org/mailman/listinfo/toronto-pm > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sfryer at sourcery.ca Thu Oct 22 11:38:23 2009 From: sfryer at sourcery.ca (Shaun Fryer) Date: Thu, 22 Oct 2009 14:38:23 -0400 Subject: [tpm] Available for contracting Message-ID: <982579710910221138r57bbbb6dy34f5af21f13bc97c@mail.gmail.com> Hello again folks, Hope life in TO has been treating you all well. I'm still available for contract work should anyone have any opportunities to share. I'll be based out of the city until mid-November, following which I may, depending on what happens in the mean time, be available for full-time work. My preference is for telecommute, and flexible hours, but I'm open minded to what other options are presented. All the best! -- Shaun Fryer 1-866-920-9209 From arocker at vex.net Fri Oct 23 09:53:33 2009 From: arocker at vex.net (arocker at vex.net) Date: Fri, 23 Oct 2009 12:53:33 -0400 Subject: [tpm] Glob problem Message-ID: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> I'm having a problem with a glob statement. It is perfectly valid, and returns expected results when applied to a local (Windows) directory, but fails to match similar filenames in a remotely connected directory. Does this sound familiar to anyone? (I haven't had time to determine the precise mechanism of the remote connection, but I believe it is supposed to look like a drive.) From mfowle at navicominc.com Fri Oct 23 10:27:00 2009 From: mfowle at navicominc.com (Mark Fowle) Date: Fri, 23 Oct 2009 13:27:00 -0400 Subject: [tpm] Glob problem In-Reply-To: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> Message-ID: <759E3F14A23281479A85A082BBCFA542765B4E@sbsa.NavicomInc.local> If I remember correctly. Globing is done by the shell. Some remote file systems are mapped by the OS, like SMB mounts, others are little more than glorified FTP connections and are not mapped. I think the general solution is to read the directory into a perl data structure and act on it there instead of trusting the OS to handle globing as expected. -----Original Message----- From: toronto-pm-bounces+mfowle=navicominc.com at pm.org [mailto:toronto-pm-bounces+mfowle=navicominc.com at pm.org] On Behalf Of arocker at vex.net Sent: Friday, October 23, 2009 12:54 PM To: Toronto PerlMongers Subject: [tpm] Glob problem I'm having a problem with a glob statement. It is perfectly valid, and returns expected results when applied to a local (Windows) directory, but fails to match similar filenames in a remotely connected directory. Does this sound familiar to anyone? (I haven't had time to determine the precise mechanism of the remote connection, but I believe it is supposed to look like a drive.) _______________________________________________ toronto-pm mailing list toronto-pm at pm.org http://mail.pm.org/mailman/listinfo/toronto-pm From liam at holoweb.net Fri Oct 23 11:02:10 2009 From: liam at holoweb.net (Liam R E Quin) Date: Fri, 23 Oct 2009 14:02:10 -0400 Subject: [tpm] Glob problem In-Reply-To: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> Message-ID: <1256320930.16405.20.camel@desktop.barefootcomputing.com> On Fri, 2009-10-23 at 12:53 -0400, arocker at vex.net wrote: > I'm having a problem with a glob statement. > > It is perfectly valid, and returns expected results when applied to a > local (Windows) directory, but fails to match similar filenames in a > remotely connected directory. A common problem with using glob on a network drive is that the remote file system may be case sensitive. File::Find might work better, as you can use /i on matches... Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From arocker at vex.net Fri Oct 23 12:27:03 2009 From: arocker at vex.net (arocker at vex.net) Date: Fri, 23 Oct 2009 15:27:03 -0400 Subject: [tpm] Glob problem In-Reply-To: <1256320930.16405.20.camel@desktop.barefootcomputing.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> Message-ID: <7070a507dd1e1b60456c7be897592718.squirrel@webmail.vex.net> > > A common problem with using glob on a network drive is that the > remote file system may be case sensitive. > It's mounted as a Windows directory, so it shouldn't be. > File::Find might work better, as you can use /i on matches... > That's an interesting suggestion, thanks. I may try that. From shlomif at iglu.org.il Fri Oct 23 13:18:19 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Fri, 23 Oct 2009 22:18:19 +0200 Subject: [tpm] Glob problem In-Reply-To: <1256320930.16405.20.camel@desktop.barefootcomputing.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> Message-ID: <200910232218.19950.shlomif@iglu.org.il> Hi Liam! (and all.) On Friday 23 Oct 2009 20:02:10 Liam R E Quin wrote: > On Fri, 2009-10-23 at 12:53 -0400, arocker at vex.net wrote: > > I'm having a problem with a glob statement. > > > > It is perfectly valid, and returns expected results when applied to a > > local (Windows) directory, but fails to match similar filenames in a > > remotely connected directory. > > A common problem with using glob on a network drive is that the > remote file system may be case sensitive. > > File::Find might work better, as you can use /i on matches... > The problem with File::Find is that it will recurse to all directories and sub-directories until you tell it to stop or prune it. You can probably do something similar with perldoc -f opendir , perldoc -f readdir and some File::Spec hackery (see the File::Spec->no_upwards() there). I usually prefer to avoid using glob() in even moderately important code, because it tends to break a lot, due to the fact it has shell-like behaviour. Just use readdir(). Regards, Shlomi Fish > Liam > -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Optimising Code for Speed - http://shlom.in/optimise Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Fri Oct 23 13:18:19 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Fri, 23 Oct 2009 22:18:19 +0200 Subject: [tpm] Glob problem In-Reply-To: <1256320930.16405.20.camel@desktop.barefootcomputing.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> Message-ID: <200910232218.19950.shlomif@iglu.org.il> Hi Liam! (and all.) On Friday 23 Oct 2009 20:02:10 Liam R E Quin wrote: > On Fri, 2009-10-23 at 12:53 -0400, arocker at vex.net wrote: > > I'm having a problem with a glob statement. > > > > It is perfectly valid, and returns expected results when applied to a > > local (Windows) directory, but fails to match similar filenames in a > > remotely connected directory. > > A common problem with using glob on a network drive is that the > remote file system may be case sensitive. > > File::Find might work better, as you can use /i on matches... > The problem with File::Find is that it will recurse to all directories and sub-directories until you tell it to stop or prune it. You can probably do something similar with perldoc -f opendir , perldoc -f readdir and some File::Spec hackery (see the File::Spec->no_upwards() there). I usually prefer to avoid using glob() in even moderately important code, because it tends to break a lot, due to the fact it has shell-like behaviour. Just use readdir(). Regards, Shlomi Fish > Liam > -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ Optimising Code for Speed - http://shlom.in/optimise Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From arocker at vex.net Mon Oct 26 15:53:36 2009 From: arocker at vex.net (arocker at vex.net) Date: Mon, 26 Oct 2009 18:53:36 -0400 Subject: [tpm] Glob problem In-Reply-To: <1256320930.16405.20.camel@desktop.barefootcomputing.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> Message-ID: Thanks everyone for the input on this, especially Liam. File::Find did the trick. (As the application uses only a single-level directory, recursion, &c. wasn't an issue.) From uri at StemSystems.com Mon Oct 26 17:59:52 2009 From: uri at StemSystems.com (Uri Guttman) Date: Mon, 26 Oct 2009 20:59:52 -0400 Subject: [tpm] Glob problem In-Reply-To: (arocker@vex.net's message of "Mon\, 26 Oct 2009 18\:53\:36 -0400") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> Message-ID: <87fx95odmf.fsf@quad.sysarch.com> >>>>> "a" == arocker writes: a> Thanks everyone for the input on this, especially Liam. File::Find did the a> trick. a> (As the application uses only a single-level directory, recursion, &c. a> wasn't an issue.) if so, then why the need for file::find? you can read a single dir and grep for the files you want with opendir and readdir. or use File::Slurp's read_dir for a simpler api. file::find is when you want deeper searching as it is overkill for a single level. it needs a callback sub, you have to collect entries into a global, etc. a simple read_dir/grep is one line. uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From liam at holoweb.net Mon Oct 26 18:55:25 2009 From: liam at holoweb.net (Liam R E Quin) Date: Mon, 26 Oct 2009 21:55:25 -0400 Subject: [tpm] Glob problem In-Reply-To: <87fx95odmf.fsf@quad.sysarch.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> Message-ID: <1256608525.16405.135.camel@desktop.barefootcomputing.com> On Mon, 2009-10-26 at 20:59 -0400, Uri Guttman wrote: > >>>>> "a" == arocker writes: [...] > a> (As the application uses only a single-level directory, recursion, &c. > a> wasn't an issue.) > > if so, then why the need for file::find? "There's more than one way to do it" - if it works, it's fine, and maybe it'll be easier to extend to subdirectories later ;) Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From uri at StemSystems.com Mon Oct 26 20:21:12 2009 From: uri at StemSystems.com (Uri Guttman) Date: Mon, 26 Oct 2009 23:21:12 -0400 Subject: [tpm] Glob problem In-Reply-To: <1256608525.16405.135.camel@desktop.barefootcomputing.com> (Liam R. E. Quin's message of "Mon\, 26 Oct 2009 21\:55\:25 -0400") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> Message-ID: <87ws2hmsif.fsf@quad.sysarch.com> >>>>> "LREQ" == Liam R E Quin writes: LREQ> On Mon, 2009-10-26 at 20:59 -0400, Uri Guttman wrote: >> >>>>> "a" == arocker writes: LREQ> [...] a> (As the application uses only a single-level directory, recursion, &c. a> wasn't an issue.) >> >> if so, then why the need for file::find? LREQ> "There's more than one way to do it" - if it works, it's fine, LREQ> and maybe it'll be easier to extend to subdirectories later ;) but some ways are MORE equal than others! just because you can bubble sort and it 'works' doesn't make it a good solution for most sorting problems. the amount of code and relative complexity of file::find vs a simple readdir/grep is an issue (or at least it is to me). simpler code is easier to read, understand and debug and it usually runs faster too. the OP may keep the code as is but should know about alternative and probably better solutions for the next time or during a refactoring. i have seen too much overly complicated code because the existing solution worked and kept 'working' until it became unworkable code. you get this way from 36 years of coding. :/ and if i had known that the dir scan was only one level deep (a critical fact missing from the 'problem specification') i would have mentioned readdir/grep earlier vs glob. i thought there was a special reason for using glob and he was having issues with it and it was not just a general shallow dir search. i never recommend glob anyhow if i can help it vs readdir/grep. uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From arocker at vex.net Tue Oct 27 07:24:53 2009 From: arocker at vex.net (arocker at vex.net) Date: Tue, 27 Oct 2009 10:24:53 -0400 Subject: [tpm] Glob problem In-Reply-To: <87ws2hmsif.fsf@quad.sysarch.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> Message-ID: To clarify Uri's points, I will try to give a clear definition of the particular problem: "given the first part of a filename and a regex for the rest of it, to search a single directory for a file matching that specification, and return the complete filename, (for a transfer routine that requires the exact filename)". > the amount of code and relative complexity of file::find vs a > simple readdir/grep is an issue (or at least it is to me). The original glob, (which worked perfectly for local directories), was: my ($file_name) = glob $from_dir . "\\" . $job_no ."XX-*.txt"; The File::Find solution required "use File::Find;" and my ($file_name); my $wanted = $job_no . "XX-.*txt"; find sub { $file_name = $File::Find:name if /^$wanted/ ; }, $from_dir; If I had been golfing seriously, I could probably have cut that down, but I went for clarity. Given that most of the keystrokes are specifying the input and output, I don't see a significant difference in code or complexity. From uri at StemSystems.com Tue Oct 27 08:21:58 2009 From: uri at StemSystems.com (Uri Guttman) Date: Tue, 27 Oct 2009 11:21:58 -0400 Subject: [tpm] Glob problem In-Reply-To: (arocker@vex.net's message of "Tue\, 27 Oct 2009 10\:24\:53 -0400") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> Message-ID: <87eioolv55.fsf@quad.sysarch.com> >>>>> "a" == arocker writes: a> To clarify Uri's points, I will try to give a clear definition of the a> particular problem: "given the first part of a filename and a regex for a> the rest of it, to search a single directory for a file matching that a> specification, and return the complete filename, (for a transfer routine a> that requires the exact filename)". >> the amount of code and relative complexity of file::find vs a >> simple readdir/grep is an issue (or at least it is to me). a> The original glob, (which worked perfectly for local directories), was: a> my ($file_name) = glob $from_dir . "\\" . $job_no ."XX-*.txt"; use File::Slurp ; my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, read_dir $from_dir ; one day i will add the options to read_dir to grep and also prefix the dir to the entries. it would then look something like this: my ($file_name) = read_dir {prefix_dir => 1}, qr/${job_no}XX-.*\.txt/, $from_dir ; uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From liam at holoweb.net Tue Oct 27 08:40:50 2009 From: liam at holoweb.net (Liam R E Quin) Date: Tue, 27 Oct 2009 11:40:50 -0400 Subject: [tpm] Glob problem In-Reply-To: <87eioolv55.fsf@quad.sysarch.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> <87eioolv55.fsf@quad.sysarch.com> Message-ID: <1256658050.16405.163.camel@desktop.barefootcomputing.com> On Tue, 2009-10-27 at 11:21 -0400, Uri Guttman wrote: > use File::Slurp ; > > my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, > read_dir $from_dir ; Oh, that read_dir, I thought you meant readdir() and opendir(), the built-in posix functions. File::Find is included with perl-base here and File::Slurp isn't installed here, but otherwise I don't have strong feelings. Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From uri at StemSystems.com Tue Oct 27 09:07:32 2009 From: uri at StemSystems.com (Uri Guttman) Date: Tue, 27 Oct 2009 12:07:32 -0400 Subject: [tpm] Glob problem In-Reply-To: <1256658050.16405.163.camel@desktop.barefootcomputing.com> (Liam R. E. Quin's message of "Tue\, 27 Oct 2009 11\:40\:50 -0400") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> <87eioolv55.fsf@quad.sysarch.com> <1256658050.16405.163.camel@desktop.barefootcomputing.com> Message-ID: <87ws2gizwb.fsf@quad.sysarch.com> >>>>> "LREQ" == Liam R E Quin writes: LREQ> On Tue, 2009-10-27 at 11:21 -0400, Uri Guttman wrote: >> use File::Slurp ; >> >> my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, >> read_dir $from_dir ; LREQ> Oh, that read_dir, I thought you meant readdir() and opendir(), the LREQ> built-in posix functions. either one. slurp's read_dir saves you the open/close call and the dirhandle variable so it is somewhat cleaner. and it already greps out . and .. for you (not needed in the example i showed). if/when i add those new options, it would be even better. the prefix_dir option is interesting as that is a common bug when reading dirs (but not always with globbing) as dirs only contain the file name and not the path to that dir. so you need to prefix the dir when opening the file (unless you are chdir'ed there already). the option would remove the need for the extra map call to do the prefixing (easy enough but redundant in the use of the dir path). one issue is needing to use file::spec to merge the parts vs a simple join on '/'. file::find has a global with the full path to the file that you can use for this. LREQ> File::Find is included with perl-base here and File::Slurp isn't LREQ> installed here, but otherwise I don't have strong feelings. slurp is on cpan. some think it should be in the core but p5p doesn't. lobby for it to be in core if you can! :) uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From mike at stok.ca Tue Oct 27 10:00:40 2009 From: mike at stok.ca (Mike Stok) Date: Tue, 27 Oct 2009 13:00:40 -0400 Subject: [tpm] 29 October 2009 Meeting Message-ID: <1737D7FF-4935-446F-848A-ACE5C436FA10@stok.ca> October's meeting will be this Thursday, and have Madison Kelly talking on "An Introduction to Perl Modules - Writing, Documenting and Testing Object-Oriented Perl Modules". I will send out another email and update http://to.pm.org/ when I know which room we'll be using. Mike -- Mike Stok http://www.stok.ca/~mike/ The "`Stok' disclaimers" apply. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at stok.ca Tue Oct 27 10:20:38 2009 From: mike at stok.ca (Mike Stok) Date: Tue, 27 Oct 2009 13:20:38 -0400 Subject: [tpm] 29 October 2009 Meeting - Room Details Message-ID: <482CB024-3694-4A5D-835B-A8DE2257316B@stok.ca> October's meeting will be this Thursday, and have Madison Kelly talking on "An Introduction to Perl Modules - Writing, Documenting and Testing Object-Oriented Perl Modules". Location: 2 Bloor Street West, classroom 18 on the 8th floor Time: 6:45pm See http://to.pm.org/ for more details. Mike -- Mike Stok http://www.stok.ca/~mike/ The "`Stok' disclaimers" apply. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arocker at vex.net Tue Oct 27 10:21:04 2009 From: arocker at vex.net (arocker at vex.net) Date: Tue, 27 Oct 2009 13:21:04 -0400 Subject: [tpm] Glob problem In-Reply-To: <87eioolv55.fsf@quad.sysarch.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> <87eioolv55.fsf@quad.sysarch.com> Message-ID: <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> > > use File::Slurp ; > A constraint I didn't specify was "use only stuff in the standard distribution, where possible". My powers don't extend to installing from CPAN, and the person who can is busy doing other things I need done. > my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, > read_dir $from_dir ; > Frankly, I don't see that as any great improvement. (I put the regex value in a variable so I could make changes for testing purposes, without disturbing the selection code.) From uri at StemSystems.com Tue Oct 27 10:48:44 2009 From: uri at StemSystems.com (Uri Guttman) Date: Tue, 27 Oct 2009 13:48:44 -0400 Subject: [tpm] Glob problem In-Reply-To: <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> (arocker@vex.net's message of "Tue\, 27 Oct 2009 13\:21\:04 -0400") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> Message-ID: <87tyxkhgn7.fsf@quad.sysarch.com> >>>>> "a" == arocker writes: >> use File::Slurp ; >> a> A constraint I didn't specify was "use only stuff in the standard a> distribution, where possible". My powers don't extend to installing from a> CPAN, and the person who can is busy doing other things I need done. that is always a lousy constraint. seen it too often in some places. >> my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, >> read_dir $from_dir ; >> a> Frankly, I don't see that as any great improvement. (I put the regex value a> in a variable so I could make changes for testing purposes, without a> disturbing the selection code.) using a var is fine. the main win is perl regexes vs globbing which is very limited in matching power. globs can keep the prefix dir for you which is nice. if i ever get the prefix option to read_dir i could even make it a new function like read_dir_prefix and then the code would be like: my ($file_name) = read_dir_prefix qr/${job_no}XX-.*\.txt/, $from_dir ; and that reads pretty well and is about as short as you can get. still cpan but lobby and it could be in core one day! :) uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From shlomif at iglu.org.il Tue Oct 27 11:15:39 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Tue, 27 Oct 2009 20:15:39 +0200 Subject: [tpm] Glob problem In-Reply-To: <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> Message-ID: <200910272015.40216.shlomif@iglu.org.il> Hi arocker! See below for my responses. On Tuesday 27 Oct 2009 19:21:04 arocker at vex.net wrote: > > use File::Slurp ; > > A constraint I didn't specify was "use only stuff in the standard > distribution, where possible". My powers don't extend to installing from > CPAN, and the person who can is busy doing other things I need done. > First of all, you should know that many people on the Perl blogosphere, the Perl IRC world, etc. have little patience for people who say "We cannot install anything from CPAN." See, for example, what Matt S. Trout wrote about it here: http://www.shadowcat.co.uk/blog/matt-s-trout/but-i-cant-use-cpan/ Namely, you can use local::lib or whatever, etc. That put aside, File::Slurp is a simple, self-contained wrapper for some of the functions in perldoc perlfunc and others in perldoc File::Spec and other core modules. I'm also quite fond of Ingy's IO::All which is more comprehensive, and has a lot of syntactic sugar, but also potentially more buggy and quirky. See: http://search.cpan.org/dist/IO-All/ What I'm saying is that you can do all that using only core stuff. See: * perldoc -f readdir * perldoc -f opendir * perldoc File::Spec * perldoc -f grep * perldoc -f map > > my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, > > read_dir $from_dir ; > > Frankly, I don't see that as any great improvement. (I put the regex value > in a variable so I could make changes for testing purposes, without > disturbing the selection code.) The improvement here is conveying the meaning precisely - if you use File::Find people may think you want to traverse a directory recursively. You shouldn't pick the shortest way (you're not doing golf in production, I hope), but the one that is more semantically correct and/or performs better (runs faster, consumes less resources, etc.). See for example: http://en.wikipedia.org/wiki/KISS_principle Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ What Makes Software Apps High Quality - http://shlom.in/sw-quality Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From shlomif at iglu.org.il Tue Oct 27 11:15:39 2009 From: shlomif at iglu.org.il (Shlomi Fish) Date: Tue, 27 Oct 2009 20:15:39 +0200 Subject: [tpm] Glob problem In-Reply-To: <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> Message-ID: <200910272015.40216.shlomif@iglu.org.il> Hi arocker! See below for my responses. On Tuesday 27 Oct 2009 19:21:04 arocker at vex.net wrote: > > use File::Slurp ; > > A constraint I didn't specify was "use only stuff in the standard > distribution, where possible". My powers don't extend to installing from > CPAN, and the person who can is busy doing other things I need done. > First of all, you should know that many people on the Perl blogosphere, the Perl IRC world, etc. have little patience for people who say "We cannot install anything from CPAN." See, for example, what Matt S. Trout wrote about it here: http://www.shadowcat.co.uk/blog/matt-s-trout/but-i-cant-use-cpan/ Namely, you can use local::lib or whatever, etc. That put aside, File::Slurp is a simple, self-contained wrapper for some of the functions in perldoc perlfunc and others in perldoc File::Spec and other core modules. I'm also quite fond of Ingy's IO::All which is more comprehensive, and has a lot of syntactic sugar, but also potentially more buggy and quirky. See: http://search.cpan.org/dist/IO-All/ What I'm saying is that you can do all that using only core stuff. See: * perldoc -f readdir * perldoc -f opendir * perldoc File::Spec * perldoc -f grep * perldoc -f map > > my ($file_name) = map "$from_dir\\$_", grep /${job_no}XX-.*\.txt/, > > read_dir $from_dir ; > > Frankly, I don't see that as any great improvement. (I put the regex value > in a variable so I could make changes for testing purposes, without > disturbing the selection code.) The improvement here is conveying the meaning precisely - if you use File::Find people may think you want to traverse a directory recursively. You shouldn't pick the shortest way (you're not doing golf in production, I hope), but the one that is more semantically correct and/or performs better (runs faster, consumes less resources, etc.). See for example: http://en.wikipedia.org/wiki/KISS_principle Regards, Shlomi Fish -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ What Makes Software Apps High Quality - http://shlom.in/sw-quality Chuck Norris read the entire English Wikipedia in 24 hours. Twice. From liam at holoweb.net Tue Oct 27 11:20:04 2009 From: liam at holoweb.net (Liam R E Quin) Date: Tue, 27 Oct 2009 14:20:04 -0400 Subject: [tpm] Glob problem In-Reply-To: <87tyxkhgn7.fsf@quad.sysarch.com> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <1256320930.16405.20.camel@desktop.barefootcomputing.com> <87fx95odmf.fsf@quad.sysarch.com> <1256608525.16405.135.camel@desktop.barefootcomputing.com> <87ws2hmsif.fsf@quad.sysarch.com> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <87tyxkhgn7.fsf@quad.sysarch.com> Message-ID: <1256667604.16405.173.camel@desktop.barefootcomputing.com> On Tue, 2009-10-27 at 13:48 -0400, Uri Guttman wrote: > > my ($file_name) = read_dir_prefix qr/${job_no}XX-.*\.txt/, $from_dir ; Maybe this should be called, pathnames_from_dir_matching $dir, $pattern since the "prefix" is really the path (relative or full), and then people could understand the code without having to read the documentation (something I consider an important principle but that's not universally agreed upon, a sort of locality principle) Best, Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From liam at holoweb.net Tue Oct 27 11:34:13 2009 From: liam at holoweb.net (Liam R E Quin) Date: Tue, 27 Oct 2009 14:34:13 -0400 Subject: [tpm] Glob problem In-Reply-To: <200910272015.40216.shlomif@iglu.org.il> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <200910272015.40216.shlomif@iglu.org.il> Message-ID: <1256668453.16405.177.camel@desktop.barefootcomputing.com> On Tue, 2009-10-27 at 20:15 +0200, Shlomi Fish wrote: > First of all, you should know that many people on the Perl blogosphere, the > Perl IRC world, etc. have little patience for people who say "We cannot > install anything from CPAN." See, for example, what Matt S. Trout wrote about > it here: I know that I've more than once worked in places that have a strict policy of not allowing consultants (or employees) to install software from the network without a close security review. And more than once been at such places where there's no direct internet connection, no Web browsing or ftp or ssh etc etc., on the production machines, and sometimes not on the development machines either. The danger of people riding high horses is it hurts more when they fall off ;-) Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From liam at holoweb.net Tue Oct 27 11:34:13 2009 From: liam at holoweb.net (Liam R E Quin) Date: Tue, 27 Oct 2009 14:34:13 -0400 Subject: [tpm] Glob problem In-Reply-To: <200910272015.40216.shlomif@iglu.org.il> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <200910272015.40216.shlomif@iglu.org.il> Message-ID: <1256668453.16405.177.camel@desktop.barefootcomputing.com> On Tue, 2009-10-27 at 20:15 +0200, Shlomi Fish wrote: > First of all, you should know that many people on the Perl blogosphere, the > Perl IRC world, etc. have little patience for people who say "We cannot > install anything from CPAN." See, for example, what Matt S. Trout wrote about > it here: I know that I've more than once worked in places that have a strict policy of not allowing consultants (or employees) to install software from the network without a close security review. And more than once been at such places where there's no direct internet connection, no Web browsing or ftp or ssh etc etc., on the production machines, and sometimes not on the development machines either. The danger of people riding high horses is it hurts more when they fall off ;-) Liam -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org From uri at StemSystems.com Tue Oct 27 11:38:31 2009 From: uri at StemSystems.com (Uri Guttman) Date: Tue, 27 Oct 2009 14:38:31 -0400 Subject: [tpm] Glob problem In-Reply-To: <200910272015.40216.shlomif@iglu.org.il> (Shlomi Fish's message of "Tue\, 27 Oct 2009 20\:15\:39 +0200") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <200910272015.40216.shlomif@iglu.org.il> Message-ID: <87skd4fzrs.fsf@quad.sysarch.com> >>>>> "SF" == Shlomi Fish writes: SF> That put aside, File::Slurp is a simple, self-contained wrapper SF> for some of the functions in perldoc perlfunc and others in SF> perldoc File::Spec and other core modules. I'm also quite fond of SF> Ingy's IO::All which is more comprehensive, and has a lot of SF> syntactic sugar, but also potentially more buggy and quirky. See: then you don't know file::slurp at all. it is far from a wrapper. it is the fastest way to slurp files (see the benchmark script included in the tarball). it also has many return variations and useful options that are not duplicated in file::spec or other modules. please learn about the module before you make such casual comments. SF> http://search.cpan.org/dist/IO-All/ and that uses file::slurp internally for its slurping and it loads tons of cpan for you (file::slurp has no dependencies). uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From uri at StemSystems.com Tue Oct 27 11:38:31 2009 From: uri at StemSystems.com (Uri Guttman) Date: Tue, 27 Oct 2009 14:38:31 -0400 Subject: [tpm] Glob problem In-Reply-To: <200910272015.40216.shlomif@iglu.org.il> (Shlomi Fish's message of "Tue\, 27 Oct 2009 20\:15\:39 +0200") References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <200910272015.40216.shlomif@iglu.org.il> Message-ID: <87skd4fzrs.fsf@quad.sysarch.com> >>>>> "SF" == Shlomi Fish writes: SF> That put aside, File::Slurp is a simple, self-contained wrapper SF> for some of the functions in perldoc perlfunc and others in SF> perldoc File::Spec and other core modules. I'm also quite fond of SF> Ingy's IO::All which is more comprehensive, and has a lot of SF> syntactic sugar, but also potentially more buggy and quirky. See: then you don't know file::slurp at all. it is far from a wrapper. it is the fastest way to slurp files (see the benchmark script included in the tarball). it also has many return variations and useful options that are not duplicated in file::spec or other modules. please learn about the module before you make such casual comments. SF> http://search.cpan.org/dist/IO-All/ and that uses file::slurp internally for its slurping and it loads tons of cpan for you (file::slurp has no dependencies). uri -- Uri Guttman ------ uri at stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- From arocker at vex.net Tue Oct 27 12:19:32 2009 From: arocker at vex.net (arocker at vex.net) Date: Tue, 27 Oct 2009 15:19:32 -0400 Subject: [tpm] Glob problem In-Reply-To: <200910272015.40216.shlomif@iglu.org.il> References: <8a40d4099104db108087756b6be4d543.squirrel@webmail.vex.net> <87eioolv55.fsf@quad.sysarch.com> <5dcf457b78ad309543189c6305e22ebf.squirrel@webmail.vex.net> <200910272015.40216.shlomif@iglu.org.il> Message-ID: <14ef44f4edd516473231b7466745cc62.squirrel@webmail.vex.net> > First of all, you should know that many people on the Perl blogosphere, > the > Perl IRC world, etc. have little patience for people who say "We cannot > install anything from CPAN." Well, they can argue with the client's security officer. I did specify it as a preference rather than a total inability. It's all very well insisting on CPAN if you're just playing with yourself, but if there are two ways of doing something, I'm going to pick the one that involves the least interaction, especially with a scarce resource who's doing something more important for me. > The improvement here is conveying the meaning precisely - Actually, glob expressed exactly what I wanted, shell-like filename expansion. > if you use File::Find people may think you want to traverse a directory > recursively. Not if they're reading the reference in "Perl in a Nutshell", which gives no real indication of depth traversal. If they started with perldoc, they might. > You shouldn't pick the shortest way (you're not doing golf in production, Not for the sake of golfing, but for clarity and simplicity. I don't believe in recreating OS functions in user-space if it can be avoided. (I see this happening a lot in modern programs, perhaps because programmers feel a compulsion to use the stuff they were forced to learn in CS.) > and/or performs better (runs faster, consumes less resources, etc.). I'm only going to worry about performance if it turns out to be noticeably egregious in practice. "Premature optimisation.. &c". From linux at alteeve.com Thu Oct 29 09:37:16 2009 From: linux at alteeve.com (Madison Kelly) Date: Thu, 29 Oct 2009 12:37:16 -0400 Subject: [tpm] Tonight's talk; Introduction to Writing Perl Modules - Eight copies Message-ID: <4AE9C4BC.9020001@alteeve.com> Hi all, I printed off eight copies of tonight's talk before running out of paper. Should that be enough? If not, I can get however many more I might need run off before the talk. I will have the PDF on a USB key for anyone who has a laptop, too. I hope to see you all this evening! Madi From arocker at vex.net Sat Oct 31 12:11:33 2009 From: arocker at vex.net (arocker at vex.net) Date: Sat, 31 Oct 2009 15:11:33 -0400 Subject: [tpm] Last Thursday's meeting Message-ID: <32dad1848fa47862a0ec5ef5a30e0cb4.squirrel@webmail.vex.net> Thanks to Madison for a very interesting talk, and a very thorough handout. The code is at http://wiki.alteeve.com/index.php/AN::Tools