From darren at DarrenDuncan.net Thu Sep 2 22:59:18 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Thu Sep 2 22:59:35 2004 Subject: [VPM] Solution for the MakeMaker question Message-ID: In case anyone was interested in the solution to the question I posted on August 30 of subject "MakeMaker question", I have it here. In order for 'make test' to see a perl module that is being stored under the t/ directory in a Perl module distribution, rather than the lib/ directory with the main modules, I found that adding a simple "use lib 't/';" (or in my case, "use lib 't/lib/';") to the top of each of my t/*.t files did the trick. No change to the Makefile.PL necessary. FYI: My having asked the question to you in the first place was due to some mistaken assumptions on my part as to what happens during the install process. For one thing, I had assumed that 'make' would be copying the test files to some special location known to it, somewhere under blib/, and so I would have no idea for where to make a "use lib ..." point to. Also, I had assumed that "use lib ..." only took absolute paths such as "/home/user/foo" (I only saw it used that way before) and hence I couldn't use it because I didn't know the layout of the end user's system. Taken together, I thought MakeMaker would have to calculate some things. However, now that I know that, 1, "use lib ..." can take relative paths, and 2, the current working directory *during* the tests remains the same as where you were before starting the install process, everything looks much simpler, and my bad assumptions are gone. So, all's well that end's well. Thanks to Tim Bunce for pointing me in the right direction. I have now applied the relevant change to all 4 of my distributions that used tests-only modules, and the newer distros have been uploaded to CPAN over the last few days; users are now saved some pollution. Have a good day. -- Darren Duncan From cconstan at csc.uvic.ca Tue Sep 7 10:36:39 2004 From: cconstan at csc.uvic.ca (Carl B. Constantine) Date: Tue Sep 7 10:36:41 2004 Subject: [VPM] alternative to perl's Open? Message-ID: <20040907153639.GC29898@csc> A recent hack here at UVic caused no end of grief for system staff. We managed to shut the cracker down, but not before quite a bit of damage was done to 75 web pages. The exploit took advantage of a perl CGI script that used the Open command. The perl script is supposed to take a file, upload it to the site and run the file (used for testing people's course code). The cracker took advantage of that and included a standard pipe '|' in the command to wget and the rest is, as they say, history. So my question is, how do you code around that? If you need to do something like this, what should you do? Thanks in advance. -- Carl B. Constantine University of Victoria Programmer Analyst http://www.csc.uvic.ca UNIX System Administrator Victoria, BC, Canada cconstan@csc.uvic.ca ELW B206, 721-8766 From Peter at PSDT.com Tue Sep 7 10:42:36 2004 From: Peter at PSDT.com (Peter Scott) Date: Tue Sep 7 10:42:23 2004 Subject: [VPM] Solution for the MakeMaker question In-Reply-To: References: Message-ID: >In case anyone was interested in the solution to the question I >posted on August 30 of subject "MakeMaker question", I have it here. > >In order for 'make test' to see a perl module that is being stored >under the t/ directory in a Perl module distribution, rather than >the lib/ directory with the main modules, I found that adding a >simple "use lib 't/';" (or in my case, "use lib 't/lib/';") to the >top of each of my t/*.t files did the trick. No change to the >Makefile.PL necessary. Been busy with travel, just got to this. I have had the same problem and I believe I've solved it the same way. The only thing I do differently is that I want to be able to execute my tests individually, and not be constrained to running them from the parent directory. So I do either a use lib qw(t/lib lib); or use FindBin depending on whim. Then I can cd t ./77foo.t with impunity. >FYI: My having asked the question to you in the first place was due >to some mistaken assumptions on my part as to what happens during >the install process. For one thing, I had assumed that 'make' would >be copying the test files to some special location known to it, >somewhere under blib/, and so I would have no idea for where to make >a "use lib ..." point to. Also, I had assumed that "use lib ..." >only took absolute paths such as "/home/user/foo" (I only saw it >used that way before) and hence I couldn't use it because I didn't >know the layout of the end user's system. Taken together, I thought >MakeMaker would have to calculate some things. However, now that I >know that, 1, "use lib ..." can take relative paths, and 2, the >current working directory *during* the tests remains the same as >where you were before starting the install process, everything looks >much simpler, and my bad assumptions are gone. > >So, all's well that end's well. Thanks to Tim Bunce for pointing me >in the right direction. From abez at abez.ca Tue Sep 7 11:24:55 2004 From: abez at abez.ca (abez) Date: Tue Sep 7 11:25:37 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <20040907153639.GC29898@csc> Message-ID: >From CGI.pm my $query = CGI->new; $filename = $query->param('uploaded_file'); while(<$filename>) { print; } The file is saved to a tmp dir and then opened. $filename is the file handle. It doesn't matter what the user named their file. If you are running perl code that other people supply you really can't stop much. For instance they could have just forked a telnet daemon. I'd suggest running the perl scripts under a user who didn't have privileges to anything. chroot can also help you. So make a little mini installation of perl. When you run a script chroot to the sandbox and setuid to something very weak. abram On Tue, 7 Sep 2004, Carl B. Constantine wrote: > A recent hack here at UVic caused no end of grief for system staff. We > managed to shut the cracker down, but not before quite a bit of damage > was done to 75 web pages. > > The exploit took advantage of a perl CGI script that used the Open > command. The perl script is supposed to take a file, upload it to the > site and run the file (used for testing people's course code). The > cracker took advantage of that and included a standard pipe '|' in the > command to wget and the rest is, as they say, history. > > So my question is, how do you code around that? If you need to do > something like this, what should you do? > > Thanks in advance. > > -- abez ------------------------------------------ http://www.abez.ca/ Abram Hindle (abez@abez.ca) ------------------------------------------ abez From cconstan at csc.uvic.ca Tue Sep 7 11:30:31 2004 From: cconstan at csc.uvic.ca (Carl B. Constantine) Date: Tue Sep 7 11:30:34 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: References: <20040907153639.GC29898@csc> Message-ID: <20040907163031.GD29898@csc> *On Tue Sep 07, 2004 at 09:24:55AM -0700, abez (abez@abez.ca) wrote: > > >From CGI.pm > my $query = CGI->new; > $filename = $query->param('uploaded_file'); > while(<$filename>) { print; } > > The file is saved to a tmp dir and then opened. $filename is the file > handle. It doesn't matter what the user named their file. > > If you are running perl code that other people supply you really can't > stop much. For instance they could have just forked a telnet daemon. > > I'd suggest running the perl scripts under a user who didn't have > privileges to anything. They did just that. It was a user CGI (we use suExec) and they used a pipe command to wget to get their stuff and run a daemon program backdoor for entry into the box. It was quite nasty. > chroot can also help you. > > So make a little mini installation of perl. When you run a script chroot > to the sandbox and setuid to something very weak. Not sure that's doable in this situation, but I'll look into it. -- Carl B. Constantine University of Victoria Programmer Analyst http://www.csc.uvic.ca UNIX System Administrator Victoria, BC, Canada cconstan@csc.uvic.ca ELW B206, 721-8766 From yf110 at victoria.tc.ca Tue Sep 7 11:36:25 2004 From: yf110 at victoria.tc.ca (Malcolm Dew-Jones) Date: Tue Sep 7 11:36:33 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <20040907153639.GC29898@csc> References: <20040907153639.GC29898@csc> Message-ID: On Tue, 7 Sep 2004, Carl B. Constantine wrote: > A recent hack here at UVic caused no end of grief for system staff. We > managed to shut the cracker down, but not before quite a bit of damage > was done to 75 web pages. > > The exploit took advantage of a perl CGI script that used the Open > command. 1. perl -T (taint mode) should always be used for cgi scripts 2. from perdoc -f open Use 3-argument form to open a file with arbitrary weird characters in it, open(FOO, '<', $file); 3. sysopen 4. When validating file names (for taint mode) it is probably better to determine what is allowed and check the name is 100% valid, because it is usually easier to know ahead of time what is good than what might be bad. Then, if paraniod, also check for illegal things in the name. From abez at abez.ca Tue Sep 7 11:38:55 2004 From: abez at abez.ca (abez) Date: Tue Sep 7 11:39:55 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <20040907163031.GD29898@csc> Message-ID: You must have an older version of CGI.pm because the source code only opens a temporary file they created. sub read_multipart { my($self,$boundary,$length) = @_; my($buffer) = new MultipartBuffer($boundary,$length); my(%header,$body); while (!$buffer->eof) { %header = $buffer->readHeader; # In beta1 it was "Content-disposition". In beta2 it's # "Content-Disposition" # Sheesh. my($key) = $header{'Content-disposition'} ? 'Content-disposition' : 'Content-Disposition'; my($param) = $header{$key}=~/ name="(.*?)"/; my($filename) = $header{$key}=~/ filename="(.*?)"/; # add this parameter to our list $self->add_parameter($param); # If no filename specified, then just read the data and assign # it # to our parameter list. unless ($filename) { my($value) = $buffer->readBody; push(@{$self->{$param}},$value); next; } # If we get here, then we are dealing with a potentially large # uploaded form. Save the data to a temporary file, then open # the file for reading. my($tmpfile) = new TempFile; open (OUT,">$tmpfile") || die "CGI open of $tmpfile: $!\n"; chmod 0666,$tmpfile; # make sure anyone can delete it. my $data; while ($data = $buffer->read) { print OUT $data; } close OUT; # Now create a new filehandle in the caller's namespace. # The name of this filehandle just happens to be identical # to the original filename (NOT the name of the temporary # file, which is hidden!) my($frame)=1; my($cp); do { $cp = caller($frame++); } until $cp!~/^CGI/; my($filehandle) = "$cp\:\:$filename"; open($filehandle,$tmpfile) || die "CGI open of $tmpfile: $!\n"; push(@{$self->{$param}},$filename); # Under Unix, it would be safe to let the temporary file # be deleted immediately. However, I fear that other operating # systems are not so forgiving. Therefore we save a reference # to the temporary file in the CGI object so that the file # isn't unlinked until the CGI object itself goes out of # scope. This is a bit hacky. $self->{"$tmpfile"}=$tmpfile; } } On Tue, 7 Sep 2004, Carl B. Constantine wrote: > *On Tue Sep 07, 2004 at 09:24:55AM -0700, abez (abez@abez.ca) wrote: > > > > >From CGI.pm > > my $query = CGI->new; > > $filename = $query->param('uploaded_file'); > > while(<$filename>) { print; } > > > > The file is saved to a tmp dir and then opened. $filename is the file > > handle. It doesn't matter what the user named their file. > > > > If you are running perl code that other people supply you really can't > > stop much. For instance they could have just forked a telnet daemon. > > > > I'd suggest running the perl scripts under a user who didn't have > > privileges to anything. > > They did just that. It was a user CGI (we use suExec) and they used a > pipe command to wget to get their stuff and run a daemon program > backdoor for entry into the box. > > It was quite nasty. > > > chroot can also help you. > > > > So make a little mini installation of perl. When you run a script chroot > > to the sandbox and setuid to something very weak. > > Not sure that's doable in this situation, but I'll look into it. > > -- abez ------------------------------------------ http://www.abez.ca/ Abram Hindle (abez@abez.ca) ------------------------------------------ abez From Peter at PSDT.com Tue Sep 7 11:39:19 2004 From: Peter at PSDT.com (Peter Scott) Date: Tue Sep 7 12:01:19 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <20040907153639.GC29898@csc> References: <20040907153639.GC29898@csc> Message-ID: <6.1.2.0.2.20040907093801.0278f470@shell2.webquarry.com> At 08:36 AM 9/7/2004, Carl B. Constantine wrote: >A recent hack here at UVic caused no end of grief for system staff. We >managed to shut the cracker down, but not before quite a bit of damage >was done to 75 web pages. > >The exploit took advantage of a perl CGI script that used the Open >command. The perl script is supposed to take a file, upload it to the >site and run the file (used for testing people's course code). The >cracker took advantage of that and included a standard pipe '|' in the >command to wget and the rest is, as they say, history. > >So my question is, how do you code around that? If you need to do >something like this, what should you do? Known exploit, known solutions. 3-arg open (perldoc -f open) makes the most sense in this case. Taint checking helps but I am rapidly becoming of the opinion that it is not the panacea it is generally made out to be. -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From Peter at PSDT.com Tue Sep 7 11:45:41 2004 From: Peter at PSDT.com (Peter Scott) Date: Tue Sep 7 12:26:21 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <20040907163031.GD29898@csc> References: <20040907153639.GC29898@csc> <20040907163031.GD29898@csc> Message-ID: <6.1.2.0.2.20040907094043.0278f470@shell2.webquarry.com> At 09:30 AM 9/7/2004, Carl B. Constantine wrote: >*On Tue Sep 07, 2004 at 09:24:55AM -0700, abez (abez@abez.ca) wrote: > > > > >From CGI.pm > > my $query = CGI->new; > > $filename = $query->param('uploaded_file'); > > while(<$filename>) { print; } > > > > The file is saved to a tmp dir and then opened. $filename is the file > > handle. It doesn't matter what the user named their file. > > > > If you are running perl code that other people supply you really can't > > stop much. For instance they could have just forked a telnet daemon. > > > > I'd suggest running the perl scripts under a user who didn't have > > privileges to anything. > >They did just that. It was a user CGI (we use suExec) and they used a >pipe command to wget to get their stuff and run a daemon program >backdoor for entry into the box. > >It was quite nasty. Just a sec. You asked for an alternative to perl's open(). But the exploit occurred through an unsafe argument being passed to wget. But it seems highly unlikely that wget was invoked with either input set to stdin or output set to stdout. So was open() involved at all? If it was just a matter of getting a url from the user into $url and then doing something like system("wget $url") then the answer is either to do regex validation of $url or to use the list form of system() to bypass the shell. -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From Peter at PSDT.com Tue Sep 7 11:57:08 2004 From: Peter at PSDT.com (Peter Scott) Date: Tue Sep 7 12:47:21 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <6.1.2.0.2.20040907094043.0278f470@shell2.webquarry.com> References: <20040907153639.GC29898@csc> <20040907163031.GD29898@csc> <6.1.2.0.2.20040907094043.0278f470@shell2.webquarry.com> Message-ID: <6.1.2.0.2.20040907095215.027cdc10@shell2.webquarry.com> At 09:45 AM 9/7/2004, I wrote: >At 09:30 AM 9/7/2004, Carl B. Constantine wrote: >>*On Tue Sep 07, 2004 at 09:24:55AM -0700, abez (abez@abez.ca) wrote: >> > >> > >From CGI.pm >> > my $query = CGI->new; >> > $filename = $query->param('uploaded_file'); >> > while(<$filename>) { print; } >> > >> > The file is saved to a tmp dir and then opened. $filename is the file >> > handle. It doesn't matter what the user named their file. >> > >> > If you are running perl code that other people supply you really can't >> > stop much. For instance they could have just forked a telnet daemon. >> > >> > I'd suggest running the perl scripts under a user who didn't have >> > privileges to anything. >> >>They did just that. It was a user CGI (we use suExec) and they used a >>pipe command to wget to get their stuff and run a daemon program >>backdoor for entry into the box. >> >>It was quite nasty. > >Just a sec. You asked for an alternative to perl's open(). But the >exploit occurred through an unsafe argument being passed to wget. But >it seems highly unlikely that wget was invoked with either input set >to stdin or output set to stdout. So was open() involved at all? If >it was just a matter of getting a url from the user into $url and then >doing something like > > system("wget $url") > >then the answer is either to do regex validation of $url or to use the >list form of system() to bypass the shell. I think I misinterpreted you. The wget command wasn't in your code. The user inserted it with a '|' as part of a filename argument that ended up in an open() statement in your program. Right? What we're having trouble understanding is how a file upload CGI could do this given how CGI.pm does file uploads. So was the exploit via an open() statement in CGI.pm or in customer code? If the latter, what does that open() statement look like? -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From cconstan at csc.uvic.ca Tue Sep 7 14:24:55 2004 From: cconstan at csc.uvic.ca (Carl B. Constantine) Date: Tue Sep 7 14:25:05 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <6.1.2.0.2.20040907095215.027cdc10@shell2.webquarry.com> References: <20040907153639.GC29898@csc> <20040907163031.GD29898@csc> <6.1.2.0.2.20040907094043.0278f470@shell2.webquarry.com> <6.1.2.0.2.20040907095215.027cdc10@shell2.webquarry.com> Message-ID: <20040907192455.GJ29898@csc> *On Tue Sep 07, 2004 at 09:57:08AM -0700, Peter Scott (Peter@PSDT.com) wrote: > >Just a sec. You asked for an alternative to perl's open(). But the > >exploit occurred through an unsafe argument being passed to wget. But > >it seems highly unlikely that wget was invoked with either input set > >to stdin or output set to stdout. So was open() involved at all? If > >it was just a matter of getting a url from the user into $url and then > >doing something like > > > > system("wget $url") > > > >then the answer is either to do regex validation of $url or to use the > >list form of system() to bypass the shell. > > I think I misinterpreted you. The wget command wasn't in your > code. The user inserted it with a '|' as part of a filename argument > that ended up in an open() statement in your program. Right? That is correct. > What we're having trouble understanding is how a file upload CGI could > do this given how CGI.pm does file uploads. So was the exploit via an > open() statement in CGI.pm or in customer code? If the latter, what > does that open() statement look like? It was an exploit in the user script, not in CGI.pm. I can't answer what the code looked like at present. I'll try to find out for you though. -- Carl B. Constantine University of Victoria Programmer Analyst http://www.csc.uvic.ca UNIX System Administrator Victoria, BC, Canada cconstan@csc.uvic.ca ELW B206, 721-8766 From yf110 at victoria.tc.ca Tue Sep 7 14:27:37 2004 From: yf110 at victoria.tc.ca (Malcolm Dew-Jones) Date: Tue Sep 7 14:27:45 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: <6.1.2.0.2.20040907093801.0278f470@shell2.webquarry.com> References: <20040907153639.GC29898@csc> <6.1.2.0.2.20040907093801.0278f470@shell2.webquarry.com> Message-ID: On Tue, 7 Sep 2004, Peter Scott wrote: > Taint > checking helps but I am rapidly becoming of the opinion that it is not > the panacea it is generally made out to be. Yes, it is important to realize that taint mode does not make a program safe. It simply identifies many possible weaknesses. It is particularly useful because it can identify weakness that the programmer may not otherwise recognize. However, it is still up to the skill of the programmer to understand the true implications of the identified weaknesses, and provide correct (and bug free) solutions. Interestingly, Javascript at one point also included a taint mode, which was dropped because it was considered a security "dead end" after some amount of experience with its use. (According to the o'reilly guide to javascript.) $0.02 From yf110 at victoria.tc.ca Tue Sep 7 15:30:39 2004 From: yf110 at victoria.tc.ca (Malcolm Dew-Jones) Date: Tue Sep 7 15:30:46 2004 Subject: [VPM] alternative to perl's Open? In-Reply-To: References: <20040907153639.GC29898@csc> <6.1.2.0.2.20040907093801.0278f470@shell2.webquarry.com> Message-ID: On Tue, 7 Sep 2004, Malcolm Dew-Jones wrote: > > Interestingly, Javascript at one point also included a taint mode, which > was dropped because it was considered a security "dead end" after some > amount of experience with its use. (According to the o'reilly guide to > javascript.) Of course as soon as I thought about this I realized it is irrelevent. Security in javascript is a problem because the person running a browser cannot trust the javascript code that was sent to them. Having that code taint check its input is pretty much irrelevent at that stage. In perl (proto-typically, and in this case) , the code (the cgi script) is trustworthy, (give or take bugs), and asking basically trusted code to check its input is entirely relevent. From Peter at PSDT.com Thu Sep 9 11:47:33 2004 From: Peter at PSDT.com (Peter Scott) Date: Thu Sep 9 11:57:11 2004 Subject: [VPM] Meeting? Message-ID: <6.1.2.0.2.20040909094538.0282f038@shell2.webquarry.com> Victoria.pm's next meeting will be Sep 21. Who wants to present, or has a request for a topic? Also, who missed Abram's Parse::RecDescent talk and would like to see it repeated? I was there and it was so good *I* would like to see it again. -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Thu Sep 9 12:44:54 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Thu Sep 9 12:45:06 2004 Subject: [VPM] Meeting? In-Reply-To: <6.1.2.0.2.20040909094538.0282f038@shell2.webquarry.com> References: <6.1.2.0.2.20040909094538.0282f038@shell2.webquarry.com> Message-ID: At 9:47 AM -0700 9/9/04, Peter Scott wrote: >Victoria.pm's next meeting will be Sep 21. Who wants to present, or >has a request for a topic? It's going to be close, but I *may* be able to give an updated version of my database talk from July, but this version would demonstrate my new database portability CPAN modules in use. More specifically, said talk would focus on introducing those modules, say how they work, and giving use examples. The problem is that they are pre-alpha at the moment, and while the infrastructure is there, I haven't actually tested them to do more than connect/disconnect a database. I plan to have table creation, table reverse engineering, and basic data manipulation done within the next couple weeks, the minimum needed for a proper demo, but it would be close. And since I'm focused on making the code, I don't have time to make a slide show. On the other hand, I could draw diagrams on the white/black board. Peter, re your demo machine, you may want to start now looking into getting multiple database engines installed, to avoid unknown problems later. The SQLite was easy, as you know, though I plan to use 3.0.x (curr 'beta', planned 'stable' within a few weeks) this time around. You could also try installing MySQL 4.1.4+ (curr 'gamma', probably stable any month now) and PostgreSQL 7.4.5+ (curr 'stable'; 8.0 is 'beta'). On my part, I haven't installed those on my system yet, but I plan to during the next week as that's what I'll use for my next big round of dev/testing following today's CPAN release. Do this just if you have time and a good backup. So if we don't come up with something else for Sept 21, then I'll talk. Note that if I don't talk Sept 21, I definitely will in October. >Also, who missed Abram's Parse::RecDescent talk and would like to >see it repeated? I was there and it was so good *I* would like to >see it again. I was at that talk, and something similar would be good for a repeat. However, doing the *same* thing 2 months in a row seems excessive. OTOH, something that might work is that, if I give the above talk in Sept, then I could collaborate with Abram for an October talk where I have actually put Parse::RecDescent to work in my module (that won't be in time for Sept), so his re-presentation could include an actual working example of moderate complexity (that parses VIEW/SELECT and PROCEDURE/FUNCTION create sql, for example). -- Darren Duncan From Peter at PSDT.com Thu Sep 9 18:26:09 2004 From: Peter at PSDT.com (Peter Scott) Date: Thu Sep 9 18:33:23 2004 Subject: [VPM] Meeting? In-Reply-To: References: <6.1.2.0.2.20040909094538.0282f038@shell2.webquarry.com> Message-ID: <6.1.2.0.2.20040909162253.0282f038@shell2.webquarry.com> Another possibility is a 'basics' talk (we're not just for experts). I just gave an extemporaneous 45-minute talk on Perl's command line flags to JPL.pm and it went down well. So I could whomp up something based on perldoc perlrun, or we could even split it up between several people. Reaction? -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Thu Sep 9 19:40:15 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Thu Sep 9 19:40:26 2004 Subject: [VPM] Meeting? In-Reply-To: <6.1.2.0.2.20040909162131.02804ae8@shell2.webquarry.com> References: <6.1.2.0.2.20040909094538.0282f038@shell2.webquarry.com> <6.1.2.0.2.20040909162131.02804ae8@shell2.webquarry.com> Message-ID: At 4:21 PM -0700 9/9/04, Peter Scott wrote: >At 10:44 AM 9/9/2004, you wrote: >>I have actually put Parse::RecDescent to work in my module (that >>won't be in time for Sept), so his re-presentation could include an >>actual working example of moderate complexity (that parses >>VIEW/SELECT and PROCEDURE/FUNCTION create sql, for example). > >SQL::Statement won't do? No. SQL::Statement is very much under-powered relative to my module, and is so different that I'm better off trying to start from scratch than use it. However, I can look at its source code and learn a few tidbits for how to do certain things. SQL::Statement was written at least 6 years ago, and while it does see updates, it simply can not handle these commonly used database features, from what I can see: multiple table joins, compound queries, group-by, sub-selects, a lot of functions, named views and stored procedures, several data types, etc. It can only handle the simplest kinds of selects and queries, and there is no support for any kind of relational database features. Even that simple Genealogy database demo that I made for July wouldn't be able to run through it, since it uses joined tables. SQL::Statement is designed to help implement simple database back-ends, such as DBD::CSV (a plain text file), and that's about it. -- Darren Duncan From Peter at PSDT.com Wed Sep 15 12:35:24 2004 From: Peter at PSDT.com (Peter Scott) Date: Wed Sep 15 12:35:31 2004 Subject: [VPM] Meeting Message-ID: <6.1.2.0.2.20040915103212.02728070@shell2.webquarry.com> Okay, if we had a consensus I must have missed it, so here's my suggestion for next week's meeting: I'll talk on Perl's command line flag (use of -l, -p, -F, that kind of thing) and ramifications thereof. I'll do most of it on a whiteboard since the supporting material is just perldoc perlrun. I'd prefer to keep it to under an hour and give someone else a chance to talk about something brief in the other half. How's that sound? -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Wed Sep 15 13:33:51 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Wed Sep 15 13:34:04 2004 Subject: [VPM] Meeting In-Reply-To: <6.1.2.0.2.20040915103212.02728070@shell2.webquarry.com> References: <6.1.2.0.2.20040915103212.02728070@shell2.webquarry.com> Message-ID: At 10:35 AM -0700 9/15/04, Peter Scott wrote: >Okay, if we had a consensus I must have missed it, so here's my >suggestion for next week's meeting: > >I'll talk on Perl's command line flag (use of -l, -p, -F, that kind >of thing) and ramifications thereof. I'll do most of it on a >whiteboard since the supporting material is just perldoc perlrun. >I'd prefer to keep it to under an hour and give someone else a >chance to talk about something brief in the other half. How's that >sound? That works for me. I could introduce my database portability modules in the second hour, mostly on the white board, and without running code or handouts. While I may have running code by then, I wouldn't count on it. Considering my progress rate, I certainly shouldn't take up the whole 2 hours. Of course, if someone else has something to discuss in the second hour, I'll put off my thing until October, when I can do a much better job of it (not that I can't do a lot of good even now). Has anyone in the group not talked yet? BTW, that "Tim Oreilly in a Nutshell" book is a great read. -- Darren Duncan From Peter at PSDT.com Thu Sep 16 19:01:36 2004 From: Peter at PSDT.com (Peter Scott) Date: Thu Sep 16 19:05:41 2004 Subject: [VPM] Victoria Perl Mongers Meeting September 21 Message-ID: <6.1.2.0.2.20040916154159.02750030@shell2.webquarry.com> Victoria.pm will meet at its regular date, time, and place on Tuesday, September 21, 7pm, at UVic. The room will be announced the day before. I will speak for half the time on Perl's command line flags (perldoc perlrun). This talk will be accessible to beginners as well as more advanced Perlers. There will be another topic presented in the rest of the time, possibly Darren Duncan on databases, but he's perfectly willing to yield to anyone else who wants to try their hand at filling a short time slot. Or I may just keep going :-) Other topics to be covered as time permits; make requests for anything particular. (Courtesy copy to VLUG members by permission of the list manager. Victoria.pm's home page is .) -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Thu Sep 16 20:14:15 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Thu Sep 16 20:14:25 2004 Subject: [VPM] Victoria Perl Mongers Meeting September 21 In-Reply-To: <6.1.2.0.2.20040916154159.02750030@shell2.webquarry.com> References: <6.1.2.0.2.20040916154159.02750030@shell2.webquarry.com> Message-ID: I just updated the web site, adding a slightly modified version of the message: September 21, 2004 Meeting The room will be announced the day before. Peter Scott will speak for half the time on Perl's command line flags (perldoc perlrun). This talk will be accessible to beginners as well as more advanced Perlers. There will be another topic presented in the rest of the time, possibly Darren Duncan on databases, but he's perfectly willing to yield to anyone else who wants to try their hand at filling a short time slot. Or Peter may just keep going :-) Other topics to be covered as time permits; make requests for anything particular. -- Darren Duncan At 5:01 PM -0700 9/16/04, Peter Scott wrote: >Victoria.pm will meet at its regular date, time, and place on >Tuesday, September 21, 7pm, at UVic. The room will be announced the >day before. > >I will speak for half the time on Perl's command line flags (perldoc >perlrun). This talk will be accessible to beginners as well as more >advanced Perlers. > >There will be another topic presented in the rest of the time, >possibly Darren Duncan on databases, but he's perfectly willing to >yield to anyone else who wants to try their hand at filling a short >time slot. Or I may just keep going :-) > >Other topics to be covered as time permits; make requests for >anything particular. > >(Courtesy copy to VLUG members by permission of the list manager. >Victoria.pm's home page is .) >-- >Peter Scott >Pacific Systems Design Technologies >http://www.perldebugged.com/ >*** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Sat Sep 18 02:04:52 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Sat Sep 18 02:05:16 2004 Subject: [VPM] [RFC] naming a module for SQL routines Message-ID: This is a request for comment with the intent of determining a better name for one of my CPAN modules prior to it being registered with the Perl 5 Module List. The module in question has been using the name "SQL::SyntaxModel" since about September 26th of 2003, following a previous RFC process which seemed to favor that name, but not unanimously. I have thought of some new ones, given below, and welcome feedback on them or further suggestions. Please send all replies to both the list(s) and directly to me. I had been satisfied with the current/old name so far, but this has changed during the last few days. For one thing, there has been resistence from the module list and alternate suggestions that I didn't like. For another thing, it seems that my module has evolved significantly since last year's RFC process, and so I am open to a new name that is adapted to the module's shifts of focus. Some constraints that the new name must follow: 1. It must be a noun, rather than a verb. The module is a rigorously structured data container, and doesn't do anything besides storing data within its own Perl variables. The name should represent what it *is* rather than what it does. It is external code which uses the module that does any "doing". 2. It must be a level-2 name, under the SQL::* name space, like "SQL::*". SQL is the problem domain which the module addresses, so it would fit best with the other SQL::* modules, that also address this problem domain, such as SQL::Statement. At the same time, this module stands alone, is not based on or designed explicitly to work with any other module, and is not one parallel member of a larger framework, so a level-3 name (SQL::*::*) is not appropriate. 3. The second-level portion of the name should be composed of only one or two words, like the other SQL::* modules are, and the old name is. FYI, this is a list of other CPAN modules that I have identified which exist in the same problem domain (there are probably more): SQL::Statement SQL::Translator SQL::YASP SQL::Generator SQL::Schema SQL::Abstract SQL::Snippet SQL::Catalog DB::Ent DBIx::Abstract DBIx::AnyDBD DBIx::DBSchema DBIx::Namespace DBIx::SearchBuilder TripleStore As usual, this new name should highlight what makes my module unique compared to the other CPAN offerings, including what its focus is and what it does best; hopefully that is one and the same of saying what it *is*. A few new ideas of my own: - SQL::SchemaModel (note for similarity that "SQL::Schema" is already taken by an existing framework that hasn't been updated in over 4 years) - SQL::RoutineModel (no similar names) - SQL::Routine (no similar names) - SQL::Routines (ditto) Under the circumstances, I am leaning towards the "Routine" set, partly because no name even similar exists on CPAN. The main reason concerns a revisiting of one of the module's main intended uses, which was to support the idea of any database-related activity being representable by a SQL routine or imitation thereof. An implementation of a SQL routine that my module describes/models/defines could either be stored in a database schema (eg: as a SQL stored procedure, function, or trigger), or it could be stored on the client/application side (eg: as a fusion of Perl code and DBI calls). But from the application's point of view, the routine simply exists in a locally addressable space and can be invoked more or less like a Perl routine (function or procedure), regardless of whether it is actually in the database or on the client. A routine is quite generic in scope and can hold complete instructions for any kind of database activity, including arbitrarily complex select queries, DML, schema creation or manipulation, user management, transactions, and connections. Therefore, saying that my module supports or models routines means that it supports and models all types of SQL. It was also designed in the hindsight of SQL-2003, and is not limited to SQL-1992. But while my module can represent SQL effectively, it is not exactly the same as SQL, and can be used with both databases or applications that don't want to talk SQL. Hence, calling it a "SyntaxModel" is somewhat archaic. The module is designed with a strong emphasis on portability, and it is expected that one should be able to use it effectively when porting an application between databases and/or porting a database schema (plus data) from one product to another, including advanced databases having multiple schemas per catalog, or schema objects for: domains, schema generators, tables, views, routines. All schema objects, as well as client-side SQL, is broken down to an atomic representation that can be easily understood by a computer, and effectively converted with whatever variances are required by different products. This not only means being converted into SQL, but also converted into Perl code when you want to emulate certain features that a database engine doesn't natively support. Even some SQL can emulate other missing SQL features, such as emulating SQL unions or intersections with SQL join operations. My module makes it a lot easier for external code to do this. I am resistent to using any names which describe too much about how the module is internally structured. For example, I don't want to use the word "tree" anywhere even though the module POD mentions that word constantly. So my alternate suggestions are: a. A focus on the fact it can represent complete schemas, both database-side and application-side, which include routines. b. A focus on the fact that it represents everything with routines. Barring any further suggestions, I'm leaning towards the latter. However, there is another matter to keep in mind when picking a name. For one thing, each primary object produced by my module can and does represent an entire set of SQL routines at once. In fact, the intention is that a typical application will only ever use a single "Container" object in which lives all of their database and application descriptions, whose definitions overlap. One thing that concerns me is that a name like "Routine" may suggest that each object just represents a single routine, rather than a related set of them; "RoutineModel" seems less likely to suggest this, though I could be wrong. So I welcome feedback on these matters. Eg: Am I worried about nothing with the one-vs-many implications? Which of "SQL::Routine" or "SQL::RoutineModel" is better? Or is "SQL::SyntaxModel" still best of all? And what other good suggestions are there, if any. Reasons are always helpful, but not required. At the very least, I would like to end up with a situation where multiple people agree that one choice is best, the more the better, and a consensus best of all. Thank you very much in advance for any help. I really appreciate it. -- Darren Duncan P.S. Some helpful urls, but the documentation focus is out of date relative to what I said above, as well as some structural issues: http://search.cpan.org/dist/SQL-SyntaxModel/ http://search.cpan.org/dist/SQL-SyntaxModel/lib/SQL/SyntaxModel.pm http://search.cpan.org/dist/SQL-SyntaxModel/lib/SQL/SyntaxModel/Language.pod P.P.S. The module is close to completion, so about 90% of what you see now in the module should stay that way for awhile. From Peter at PSDT.com Sat Sep 18 12:29:47 2004 From: Peter at PSDT.com (Peter Scott) Date: Sat Sep 18 12:32:57 2004 Subject: [VPM] [RFC] naming a module for SQL routines In-Reply-To: References: Message-ID: <6.1.2.0.2.20040918102735.02798c00@shell2.webquarry.com> At 12:04 AM 9/18/2004, Darren Duncan wrote: >The main reason concerns a revisiting of one of the module's main >intended uses, which was to support the idea of any database-related >activity being representable by a SQL routine or imitation thereof. An >implementation of a SQL routine that my module >describes/models/defines could either be stored in a database schema >(eg: as a SQL stored procedure, function, or trigger), or it could be >stored on the client/application side (eg: as a fusion of Perl code >and DBI calls). But from the application's point of view, the routine >simply exists in a locally addressable space and can be invoked more >or less like a Perl routine (function or procedure), regardless of >whether it is actually in the database or on the client. > >A routine is quite generic in scope and can hold complete instructions >for any kind of database activity, including arbitrarily complex >select queries, DML, schema creation or manipulation, user management, >transactions, and connections. Therefore, saying that my module >supports or models routines means that it supports and models all >types of SQL. It was also designed in the hindsight of SQL-2003, and >is not limited to SQL-1992. This is a bit off the wall, but after giving this considerable thought, what comes to mind is SQL::Mechanize, after WWW::Mechanize. >But while my module can represent SQL effectively, it is not exactly >the same as SQL, and can be used with both databases or applications >that don't want to talk SQL. Hence, calling it a "SyntaxModel" is >somewhat archaic. Given this, perhaps DBIx::Mechanize would be more appropriate. YMMV. -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Sat Sep 18 14:17:23 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Sat Sep 18 14:17:49 2004 Subject: [VPM] Re: [RFC] naming a module for SQL routines In-Reply-To: <6.1.2.0.2.20040918102735.02798c00@shell2.webquarry.com> References: <6.1.2.0.2.20040918102735.02798c00@shell2.webquarry.com> Message-ID: Thanks to Peter Scott for being the first to reply. Just to make clear, the module I'm asking to have named today is just a container that defines actions which can be done against a database, but it does not actually talk to any databases or otherwise execute the definitions. Therefore, "[dbms-related-thingy]::Mechanize" might be a good name for another module which uses mine and performs the actions, but not for my module itself. AFAIK, WWW::Mechanize actually "does" the stuff that it models. I also want to head-off any false impressions I may have caused when I said "it is not exactly SQL". That comment was more of a reference to fringe cases such as older or non-standard databases, or services that want to be used like databases, that don't use SQL. In fact I expect that most uses of my module will be in connection with databases that do understand SQL, and SQL will be generated from my module's objects to be their input. For the other cases, think of my module as a better attempt to bring SQL to those that know it not. I do not feel that the "DBIx" namespace is appropriate because that namespace is for modules that explicitly extend and/or depend on the DBI library, and this module of mine does not. It can be used with DBI, where external code provides the bridge, but it doesn't have to be. I also consider that DBIx modules are usually all "doer" modules. I appreciate your suggestions, though; they may get used elsewhere. For everyone's information, after I've given this more thought, I'm currently leaning towards one specific name, which is "SQL::Routine". I decided not to worry about the perception of "represents one vs many". People could think of that name as conceptually being "not just one SQL statement but a whole bunch". Do any of you think that "SQL::Routine" can work best, or do you favor something else? Thanks in advance. -- Darren Duncan At 10:29 AM -0700 9/18/04, Peter Scott wrote: >At 12:04 AM 9/18/2004, Darren Duncan wrote: >>The main reason concerns a revisiting of one of the module's main >>intended uses, which was to support the idea of any >>database-related activity being representable by a SQL routine or >>imitation thereof. An implementation of a SQL routine that my >>module describes/models/defines could either be stored in a >>database schema (eg: as a SQL stored procedure, function, or >>trigger), or it could be stored on the client/application side (eg: >>as a fusion of Perl code and DBI calls). But from the >>application's point of view, the routine simply exists in a locally >>addressable space and can be invoked more or less like a Perl >>routine (function or procedure), regardless of whether it is >>actually in the database or on the client. >> >>A routine is quite generic in scope and can hold complete >>instructions for any kind of database activity, including >>arbitrarily complex select queries, DML, schema creation or >>manipulation, user management, transactions, and connections. >>Therefore, saying that my module supports or models routines means >>that it supports and models all types of SQL. It was also designed >>in the hindsight of SQL-2003, and is not limited to SQL-1992. > >This is a bit off the wall, but after giving this considerable >thought, what comes to mind is SQL::Mechanize, after WWW::Mechanize. > >>But while my module can represent SQL effectively, it is not >>exactly the same as SQL, and can be used with both databases or >>applications that don't want to talk SQL. Hence, calling it a >>"SyntaxModel" is somewhat archaic. > >Given this, perhaps DBIx::Mechanize would be more appropriate. YMMV. From Peter at PSDT.com Mon Sep 20 05:49:00 2004 From: Peter at PSDT.com (Peter Scott) Date: Mon Sep 20 11:25:49 2004 Subject: [VPM] Victoria Perl Mongers Meeting tomorrow Message-ID: <6.1.2.0.2.20040917154822.02ae2090@shell2.webquarry.com> Victoria.pm will meet at its regular date, time, and place tomorrow, Tuesday, September 21, 7pm, at UVic. The location is Harry Hickman Building room 120 (the Building Formerly Known As the Centre for Innovative Teaching). See http://uvic.ca for maps if necessary. I will speak for half the time on Perl's command line flags (perldoc perlrun). This talk will be accessible to beginners as well as more advanced Perlers. There will be another topic presented in the rest of the time, possibly Darren Duncan on databases, but he's perfectly willing to yield to anyone else who wants to try their hand at filling a short time slot. Or I may just keep going :-) Other topics to be covered as time permits; make requests for anything particular. (Courtesy copy to VLUG members by permission of the list manager. Victoria.pm's home page is .) -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Mon Sep 20 12:16:13 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Mon Sep 20 12:16:21 2004 Subject: [VPM] Victoria Perl Mongers Meeting tomorrow In-Reply-To: <6.1.2.0.2.20040917154822.02ae2090@shell2.webquarry.com> References: <6.1.2.0.2.20040917154822.02ae2090@shell2.webquarry.com> Message-ID: At 3:49 AM -0700 9/20/04, Peter Scott wrote: >Victoria.pm will meet at its regular date, time, and place tomorrow, >Tuesday, September 21, 7pm, at UVic. The location is Harry Hickman >Building room 120 (the Building Formerly Known As the Centre for >Innovative Teaching). See http://uvic.ca for maps if necessary. I have updated the web site with the confirmed location, and the regular location now show's the building's new name. -- Darren Duncan From abez at abez.ca Tue Sep 28 18:05:33 2004 From: abez at abez.ca (abez) Date: Tue Sep 28 18:06:27 2004 Subject: [VPM] Possible Guest Lecture Message-ID: I think I can recruit someone to give an intro to Python to Perl Mongers in October. I know Perl Mongers is about perl but maybe we should open up Perl Mongers to similar languages or at least have some contrast. abram p.s. python is slow -- abez ------------------------------------------ http://www.abez.ca/ Abram Hindle (abez@abez.ca) ------------------------------------------ abez From Peter at PSDT.com Tue Sep 28 18:29:33 2004 From: Peter at PSDT.com (Peter Scott) Date: Tue Sep 28 18:30:42 2004 Subject: [VPM] Possible Guest Lecture In-Reply-To: References: Message-ID: <6.1.2.0.2.20040928161827.02f3d9a8@shell2.webquarry.com> At 04:05 PM 9/28/2004, abez wrote: >I think I can recruit someone to give an intro to Python to Perl Mongers >in October. Sounds great to me. Unfortunately I will not be there; I am moving that day. Please go ahead and make the arrangements. >I know Perl Mongers is about perl but maybe we should open >up Perl Mongers to similar languages or at least have some contrast. Totally agree. I would also like to see someone compare Ruby and SmallTalk to the Perl O-O model. >abram > >p.s. python is slow You do know about http://weblogs.mozillazine.org/zach/archives/006129.html, right? :-) -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/ From darren at DarrenDuncan.net Tue Sep 28 19:44:06 2004 From: darren at DarrenDuncan.net (Darren Duncan) Date: Tue Sep 28 19:44:15 2004 Subject: [VPM] Possible Guest Lecture In-Reply-To: References: Message-ID: At 4:05 PM -0700 9/28/04, abez wrote: >I think I can recruit someone to give an intro to Python to Perl Mongers >in October. I know Perl Mongers is about perl but maybe we should open >up Perl Mongers to similar languages or at least have some contrast. >abram >p.s. python is slow I say go for it! If we're going to have a talk about another language, what better than Perl's twin pillar in the Parrot project. Besides, I might learn something. I've heard about Python but never written it. -- Darren Duncan From Peter at PSDT.com Wed Sep 29 11:06:59 2004 From: Peter at PSDT.com (Peter Scott) Date: Wed Sep 29 11:06:59 2004 Subject: [VPM] Perl Module RPM generator Message-ID: <6.1.2.0.2.20040929090430.02a56b80@shell2.webquarry.com> This is actually on-topic for both VLUG and Victoria.pm; a new module was just uploaded to CPAN which -- well, you can read the description below. It's at http://search.cpan.org/~gyepi/Ovid-0.04/ovid. I am unfamiliar with the author. >This is the README file for Ovid, which recursively builds CPAN >modules and all dependent modules as >RPM files. > >Each RPM that Ovid generates lists all its required files so rpm tools >like apt-get (the rpm version), >urpm, yum and others can find and install them as dependencies. >Ovid puts an end to manual dependency chasing madness! > >INSTALLATION > >To install this module type the following: > > perl Makefile.PL > make > make test > make install > >Alternately, if you are on an RPM based system, type the following to >build an RPM file: > > rpmbuild -ta Ovid-.tar.gz > >For a version 0.01, you would type > > rpmbuild -ta Ovid-0.01.tar.gz > >After the RPM is built, type: > > rpm -i /path/to/rpm/file > > >DEPENDENCIES > >This module requires that the CPAN module be installed and configured. >It also uses the following standard perl modules: > > Exporter > File::Basename > POSIX > File::Copy > >COPYRIGHT AND LICENCE > >Copyright (C) 2004 Gyepi Sam > >This library is free software; you can redistribute it and/or modify >it under the same terms as Perl itself. -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com/ *** New! *** http://www.perlmedic.com/