From extasia at extasia.org Sat Jul 1 18:42:22 2006 From: extasia at extasia.org (David Alban) Date: Sat, 1 Jul 2006 18:42:22 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? Message-ID: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> Greetings, Say I want to do something like: $hosts => { foo => { source_tree => '/some/dir', rsync_target => "bat:$spool_dir/foo/", }, bar => { source_tree => '/some/other/dir', rsync_target => "bat:$spool_dir/bar/", }, . . . }; But I don't want to hardcode hostnames "foo" and "bar" in the rsync_target values. Is there a way to do something like: $hosts => { foo => { source_tree => '/some/dir', rsync_target => "bat:$spool_dir/" . __KEY__ . '/', }, bar => { source_tree => '/some/other/dir', rsync_target => "bat:$spool_dir/" . __KEY__ . '/', }, . . . }; That is, is there some magic string I can put in the source to indicate to the compiler to use the value of the current hash key? Of course, I just noticed that this ability would have to allow the coder to specify out to what level the magic string referred. 'Cause in my example, I wouldn't want the value substituted for __KEY__ to be the string 'rsync_target'. :-) Thanks, David -- Live in a world of your own, but always welcome visitors. From mbudash at sonic.net Sat Jul 1 20:27:11 2006 From: mbudash at sonic.net (Michael Budash) Date: Sat, 1 Jul 2006 20:27:11 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> References: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> Message-ID: On Jul 1, 2006, at 6:42 PM, David Alban wrote: > Greetings, > > Say I want to do something like: > > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/foo/", > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/bar/", > }, > . > . > . > }; > > But I don't want to hardcode hostnames "foo" and "bar" in the > rsync_target values. Is there a way to do something like: > > > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/" . __KEY__ . '/', > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/" . __KEY__ . '/', > }, > . > . > . > }; > > That is, is there some magic string I can put in the source to > indicate to the compiler to use the value of the current hash key? > > Of course, I just noticed that this ability would have to allow the > coder to specify out to what level the magic string referred. 'Cause > in my example, I wouldn't want the value substituted for __KEY__ to be > the string 'rsync_target'. :-) this doesn't answer your question about the existence of such a 'magic' string, but... you might want to explain why you can't simply handle this in the code that makes use of this data. of course, my question assumes that your example data describes your only need, and not something like this, where the key ref is floating: > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:foo/$spool_dir/", > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/bar/", > }, > boo => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/boo/boo/", > }, > . > . > . > }; -- Michael Budash Michael Budash Consulting michael at budashconsulting.com 707-252-7670 off 707-363-4262 cel From cataldo at boutell.com Sat Jul 1 20:27:40 2006 From: cataldo at boutell.com (cataldo) Date: Sat, 01 Jul 2006 20:27:40 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> References: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> Message-ID: <44A73D2C.7070601@boutell.com> Would it be easier to create a class that knows what you want? $target = $hosts->rsync_target($hostname); and quickly code in your own magic? -stephen Stephen Cataldo SpaceShare.com 6420 Colby St. Oakland, CA 94618 (510) 520-6175 Replacing Cars with Community: Interested in learning more about SpaceShare? Join our monthly newsletter list: www.spaceshare.com/email_newsletter but I suppose procrastination is False Laziness. David Alban wrote: >Greetings, > >Say I want to do something like: > > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/foo/", > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/bar/", > }, > . > . > . > }; > >But I don't want to hardcode hostnames "foo" and "bar" in the >rsync_target values. Is there a way to do something like: > > > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/" . __KEY__ . '/', > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/" . __KEY__ . '/', > }, > . > . > . > }; > >That is, is there some magic string I can put in the source to >indicate to the compiler to use the value of the current hash key? > >Of course, I just noticed that this ability would have to allow the >coder to specify out to what level the magic string referred. 'Cause >in my example, I wouldn't want the value substituted for __KEY__ to be >the string 'rsync_target'. :-) > >Thanks, >David > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.pm.org/pipermail/sanfrancisco-pm/attachments/20060701/1ebe30e8/attachment.html From extasia at extasia.org Sat Jul 1 20:47:41 2006 From: extasia at extasia.org (David Alban) Date: Sat, 1 Jul 2006 20:47:41 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: References: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> Message-ID: <4c714a9c0607012047q1ed70bf9ub43b8df9c33e9f48@mail.gmail.com> On 7/1/06, Michael Budash wrote: > you might want to explain why you can't simply handle this in the > code that makes use of this data. Yeah, that's what i finally did. :-) The more I thought about it, the more I became convinced that if there was a way to do this, it would probably be unnecessarily complicated. So I did: $hosts => { foo => { source_tree => '/some/dir', rsync_target => "bat:$spool_dir/::SOURCEHOST::/", }, bar => { source_tree => '/some/other/dir', rsync_target => "bat:$spool_dir/::SOURCEHOST::/", }, bat => { source_tree => '/yet/another/dir', rsync_target => "mumble:$raid_root/::SOURCEHOST::/", }, . . . }; Then no matter which machine I'm on, I can do: $rsync_target = $hosts->{ $hostname }{ rsync_target } =~ s#::SOURCEHOST::#$hostname#; What I *really* wanted was to be avoid copy errors if I added a new host in the future. That is, in the editor, copy and paste the foo data, rename the foo key with, say, baz, but then forget to change foo wherever it appears in the values of the new baz hash. Thanks. -- Live in a world of your own, but always welcome visitors. From extasia at extasia.org Sat Jul 1 20:51:07 2006 From: extasia at extasia.org (David Alban) Date: Sat, 1 Jul 2006 20:51:07 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <44A73D2C.7070601@boutell.com> References: <4c714a9c0607011842ob8907d1v62fd260e84f727bd@mail.gmail.com> <44A73D2C.7070601@boutell.com> Message-ID: <4c714a9c0607012051i6784ced3y9d4adc753a7bc3d8@mail.gmail.com> Perhaps? OOP is on my list of things to learn. :-) On 7/1/06, cataldo wrote: > Would it be easier to create a class that knows what you want? > > $target = $hosts->rsync_target($hostname); > and quickly code in your own magic? -- Live in a world of your own, but always welcome visitors. From gavin at fastperl.com Mon Jul 3 10:55:42 2006 From: gavin at fastperl.com (Gavin Jefferies) Date: Mon, 3 Jul 2006 10:55:42 -0700 (PDT) Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <4c714a9c0607012047q1ed70bf9ub43b8df9c33e9f48@mail.gmail.com> Message-ID: <20060703175542.80439.qmail@web31804.mail.mud.yahoo.com> Hi David, To avoid the copy error you could populate your hosts HASH via a subroutine. This also protects you against forgetting to do the substitution. my $hosts = {}; sub mkhosts { my $hostname = shift; my $parms = shift; die "no rsync_target specified for $hostname" unless $parms->{ rsync_target }; $parms->{ rsync_target } =~ s#::SOURCEHOST::#$hostname#; $hosts->{ $hostname } = $parms; } mkhosts(foo => { source_tree => '/some/dir', rsync_target => "bat:$spool_dir/::SOURCEHOST::/", }); mkhosts(bar => { source_tree => '/some/other/dir', rsync_target => "bat:$spool_dir/::SOURCEHOST::/", }); mkhosts(bat => { source_tree => '/yet/another/dir', rsync_target => "mumble:$raid_root/::SOURCEHOST::/", }); HTH, Gavin --- David Alban wrote: > On 7/1/06, Michael Budash wrote: > > you might want to explain why you can't simply handle > this in the > > code that makes use of this data. > > Yeah, that's what i finally did. :-) > > The more I thought about it, the more I became convinced > that if there > was a way to do this, it would probably be unnecessarily > complicated. > So I did: > > $hosts => { > foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/::SOURCEHOST::/", > }, > bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/::SOURCEHOST::/", > }, > bat => { > source_tree => '/yet/another/dir', > rsync_target => > "mumble:$raid_root/::SOURCEHOST::/", > }, > . > . > . > }; > > Then no matter which machine I'm on, I can do: > > $rsync_target > = $hosts->{ $hostname }{ rsync_target } =~ > s#::SOURCEHOST::#$hostname#; > > What I *really* wanted was to be avoid copy errors if I > added a new > host in the future. That is, in the editor, copy and > paste the foo > data, rename the foo key with, say, baz, but then forget > to change foo > wherever it appears in the values of the new baz hash. > > Thanks. > > -- > Live in a world of your own, but always welcome visitors. > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm > From extasia at extasia.org Wed Jul 5 16:15:04 2006 From: extasia at extasia.org (David Alban) Date: Wed, 5 Jul 2006 16:15:04 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <20060703175542.80439.qmail@web31804.mail.mud.yahoo.com> References: <4c714a9c0607012047q1ed70bf9ub43b8df9c33e9f48@mail.gmail.com> <20060703175542.80439.qmail@web31804.mail.mud.yahoo.com> Message-ID: <4c714a9c0607051615s5417e04fve2d52067079db9a9@mail.gmail.com> Thanks, Gavin. I like that idea! Unfortunately, almost no matter what I use, there's always the danger of "cut and paste" error. The data is going to be in the source (or in a config file) somewhere in some form. And the error will happen at "edittime", not runtime. At least with the token ("::SOURCEHOST::") I don't have to worry about that so much. On 7/3/06, Gavin Jefferies wrote: > mkhosts(foo => { > source_tree => '/some/dir', > rsync_target => "bat:$spool_dir/::SOURCEHOST::/", > }); > > mkhosts(bar => { > source_tree => '/some/other/dir', > rsync_target => "bat:$spool_dir/::SOURCEHOST::/", > }); > > mkhosts(bat => { > source_tree => '/yet/another/dir', > rsync_target => > "mumble:$raid_root/::SOURCEHOST::/", > }); -- Live in a world of your own, but always welcome visitors. From boss at gregerhaga.net Thu Jul 6 07:04:32 2006 From: boss at gregerhaga.net (boss at gregerhaga.net) Date: Thu, 6 Jul 2006 17:04:32 +0300 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <4c714a9c0607051615s5417e04fve2d52067079db9a9@mail.gmail.com> References: <4c714a9c0607012047q1ed70bf9ub43b8df9c33e9f48@mail.gmail.com> <20060703175542.80439.qmail@web31804.mail.mud.yahoo.com> <4c714a9c0607051615s5417e04fve2d52067079db9a9@mail.gmail.com> Message-ID: <20060706170432.c9dr0flqb4oc484c@webmail.gregerhaga.net> Quoting David Alban : > Thanks, Gavin. > > I like that idea! > > Unfortunately, almost no matter what I use, there's always the danger > of "cut and paste" error. The data is going to be in the source (or > in a config file) somewhere in some form. And the error will happen > at "edittime", not runtime. At least with the token > ("::SOURCEHOST::") I don't have to worry about that so much. how about putting the config in an xml file? It would allow you to use the exact same script for various xml-config files. You wuldn't need to change anything in the script itself when source and target changes. See my site for example of how I upload and download contents as I develop my site, http://www.gregerhaga.net/scripts.php ( ftp loader ) just an idea, Greger < > > On 7/3/06, Gavin Jefferies wrote: >> mkhosts(foo => { >> source_tree => '/some/dir', >> rsync_target => "bat:$spool_dir/::SOURCEHOST::/", >> }); >> >> mkhosts(bar => { >> source_tree => '/some/other/dir', >> rsync_target => "bat:$spool_dir/::SOURCEHOST::/", >> }); >> >> mkhosts(bat => { >> source_tree => '/yet/another/dir', >> rsync_target => >> "mumble:$raid_root/::SOURCEHOST::/", >> }); > > -- > Live in a world of your own, but always welcome visitors. > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm > From vlb at cfcl.com Fri Jul 7 06:39:44 2006 From: vlb at cfcl.com (Vicki Brown) Date: Fri, 7 Jul 2006 06:39:44 -0700 Subject: [sf-perl] JOB Seeker: Quality-oriented Internal Documentation Specialist Message-ID: Does your organization need someone with my particular combination of skills and experience? I am now looking for my next project, to start in September, in San Francisco or on the Peninsula. I'm currently completing week 4 of a 12-week internal documentation contract in Sunnyvale. I may be offered an extension, but that remains to be seen. Even so, I'd prefer to find an interesting project that doesn't require a daily commute into the South Bay. I am an experienced programmer, web weaver, and writer; however, I'm not seeking application development projects or jobs that entail creating customer-focused materials. Instead, I seek to apply my experience in software engineering and web technology, along with my skills at problem solving and technical writing, to enhance engineering productivity through improvements in communication. I have a passion for quality (assessment and assurance), process (creation and improvement), and documentation (of all forms). I enjoy solving problems. I enjoy writing. I love the Web. I'm great at reviewing (code and text) and fixing (code and text). I love working on document mechanization tools, automated text processing, and data filters. I am especially interested in creating and enhancing internal documentation in support of software engineering organizations. In particular, I excel at: * finding, gathering, and coalescing scattered bits of data, leading to improved data discovery, navigability, and ease of use. * design and implementation of reproducible processes; process review and improvement * writing, editing, re-writing, review, and proofreading; organization and maintenance * asking the right questions, interviewing domain experts, and converting random thoughts and meeting notes into meaningful and useful documentation * implementing and exploiting web technologies such as weblogs, wikis, PHP scripts, and more, to improve Intranet content management and accessibility See http://www.philtres.com/vlb for links to my resume, work history, writing samples and much more. What can I do for you? - Vicki -- Vicki Brown http://www.philtres.com/vlb Technical Documentation: Analysis, Composition, Editing, and Review Data Analysis and Dataflow programming; Web Content and Tools Internet or Intranet <> Contract or FTE <> San Francisco and Peninsula -- - Vicki ZZZ zzZ San Francisco Bay Area, CA z |\ _,,,---,,_ Books, Cats, Tech zz /,`.-'`' -. ;-;;,_ http://cfcl.com/vlb |,4- ) )-,_. ,\ ( `'-' http://cfcl.com/vlb/weblog '---''(_/--' `-'\_) http://vlb.typepad.com/commentary/ From qw at sf.pm.org Sat Jul 8 16:44:05 2006 From: qw at sf.pm.org (Quinn Weaver) Date: Sat, 8 Jul 2006 16:44:05 -0700 Subject: [sf-perl] OSCON update Message-ID: <20060708234405.GA1288@fu.funkspiel.org> I reported earlier that you could get a discount on OSCON registration. However, it turns out the code has changed. The new registration discount code, as just sent to me by O'Reilly, is os06dsug If you're registering, paste this into the appropriate form field to save 15% (off the early-registration price, which is evidently still in effect for PM members). PS: There's some OSCON Ruby event brewing that may be of interest to y'all: http://blog.pdxruby.org/pages/foscon2006 -- qw (Quinn Weaver); #President, San Francisco Perl Mongers =for information, visit http://sf.pm.org/weblog =cut From Peter.Loo at source.wolterskluwer.com Mon Jul 10 07:13:47 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 10 Jul 2006 07:13:47 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <20060708234405.GA1288@fu.funkspiel.org> Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A0195E99E@phxmail02.phx.ndchealth.com> Hi Quinn, What is OSCON? Apologies for my ignorance. Peter -----Original Message----- From: sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.org [mailto:sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.or g] On Behalf Of Quinn Weaver Sent: Saturday, July 08, 2006 4:44 PM To: sfpug at sf.pm.org Subject: [sf-perl] OSCON update I reported earlier that you could get a discount on OSCON registration. However, it turns out the code has changed. The new registration discount code, as just sent to me by O'Reilly, is os06dsug If you're registering, paste this into the appropriate form field to save 15% (off the early-registration price, which is evidently still in effect for PM members). PS: There's some OSCON Ruby event brewing that may be of interest to y'all: http://blog.pdxruby.org/pages/foscon2006 -- qw (Quinn Weaver); #President, San Francisco Perl Mongers =for information, visit http://sf.pm.org/weblog =cut _______________________________________________ SanFrancisco-pm mailing list SanFrancisco-pm at pm.org http://mail.pm.org/mailman/listinfo/sanfrancisco-pm This E-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail, and destroy all copies of the original message. From Peter.Loo at source.wolterskluwer.com Mon Jul 10 07:13:47 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 10 Jul 2006 07:13:47 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <20060708234405.GA1288@fu.funkspiel.org> Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A0195E99E@phxmail02.phx.ndchealth.com> Hi Quinn, What is OSCON? Apologies for my ignorance. Peter -----Original Message----- From: sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.org [mailto:sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.or g] On Behalf Of Quinn Weaver Sent: Saturday, July 08, 2006 4:44 PM To: sfpug at sf.pm.org Subject: [sf-perl] OSCON update I reported earlier that you could get a discount on OSCON registration. However, it turns out the code has changed. The new registration discount code, as just sent to me by O'Reilly, is os06dsug If you're registering, paste this into the appropriate form field to save 15% (off the early-registration price, which is evidently still in effect for PM members). PS: There's some OSCON Ruby event brewing that may be of interest to y'all: http://blog.pdxruby.org/pages/foscon2006 -- qw (Quinn Weaver); #President, San Francisco Perl Mongers =for information, visit http://sf.pm.org/weblog =cut _______________________________________________ SanFrancisco-pm mailing list SanFrancisco-pm at pm.org http://mail.pm.org/mailman/listinfo/sanfrancisco-pm This E-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail, and destroy all copies of the original message. From rdm at cfcl.com Mon Jul 10 07:37:26 2006 From: rdm at cfcl.com (Rich Morin) Date: Mon, 10 Jul 2006 07:37:26 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <8E3D502A002DA04FADBDED4CB4D94D3A0195E99E@phxmail02.phx.ndchealth.com> References: <8E3D502A002DA04FADBDED4CB4D94D3A0195E99E@phxmail02.phx.ndchealth.com> Message-ID: At 7:13 AM -0700 7/10/06, Loo, Peter # PHX wrote: > What is OSCON? Apologies for my ignorance. Google is your friend: http://www.google.com/search?ie=utf8&oe=utf8&q=OSCON -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm at cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development From Peter.Loo at source.wolterskluwer.com Mon Jul 10 09:40:46 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 10 Jul 2006 09:40:46 -0700 Subject: [sf-perl] Opening a pipe to sqlplus Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A0195EB06@phxmail02.phx.ndchealth.com> Hi All, I was just wondering if anyone knows how to code an "open" statement to "sqlplus" while concealing the userid and password. The problem that I am having is when the pipe is opened, one can view the userid and password using the Unix command "ps -ef". Thanks. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.pm.org/pipermail/sanfrancisco-pm/attachments/20060710/c9dffe66/attachment.html From daniel at electricrain.com Mon Jul 10 09:54:35 2006 From: daniel at electricrain.com (Dan Sully) Date: Mon, 10 Jul 2006 09:54:35 -0700 Subject: [sf-perl] Opening a pipe to sqlplus In-Reply-To: <8E3D502A002DA04FADBDED4CB4D94D3A0195EB06@phxmail02.phx.ndchealth.com> References: <8E3D502A002DA04FADBDED4CB4D94D3A0195EB06@phxmail02.phx.ndchealth.com> Message-ID: <20060710165435.GA525@electricrain.com> * Loo, Peter # PHX shaped the electrons to say... >I was just wondering if anyone knows how to code an "open" statement to >"sqlplus" while concealing the userid and password. The problem that I >am having is when the pipe is opened, one can view the userid and >password using the Unix command "ps -ef". Something like this: my $sqlplus = Expect->spawn('sqlplus', join('@', $username, $sid)); $sqlplus->expect(30, ['Enter password: ', sub { my $self = shift; $self->log_stdout(0); $self->send("$password\r"); $self->expect(0, '-re', "\r\n"); $self->log_stdout(1); }]); $sqlplus->interact; -D -- dd if=/dev/sarcasm of=/dev/clue From Peter.Loo at source.wolterskluwer.com Mon Jul 10 15:18:44 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 10 Jul 2006 15:18:44 -0700 Subject: [sf-perl] Opening a pipe to sqlplus In-Reply-To: <20060710165435.GA525@electricrain.com> Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A0195EEDA@phxmail02.phx.ndchealth.com> Thanks Dan, I am looking into Expect as we don't have it installed here. Peter -----Original Message----- From: sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.org [mailto:sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.or g] On Behalf Of Dan Sully Sent: Monday, July 10, 2006 9:55 AM To: San Francisco Perl Mongers User Group Subject: Re: [sf-perl] Opening a pipe to sqlplus * Loo, Peter # PHX shaped the electrons to say... >I was just wondering if anyone knows how to code an "open" statement to >"sqlplus" while concealing the userid and password. The problem that I >am having is when the pipe is opened, one can view the userid and >password using the Unix command "ps -ef". Something like this: my $sqlplus = Expect->spawn('sqlplus', join('@', $username, $sid)); $sqlplus->expect(30, ['Enter password: ', sub { my $self = shift; $self->log_stdout(0); $self->send("$password\r"); $self->expect(0, '-re', "\r\n"); $self->log_stdout(1); }]); $sqlplus->interact; -D -- dd if=/dev/sarcasm of=/dev/clue _______________________________________________ SanFrancisco-pm mailing list SanFrancisco-pm at pm.org http://mail.pm.org/mailman/listinfo/sanfrancisco-pm This E-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail, and destroy all copies of the original message. From vlb at cfcl.com Mon Jul 10 20:26:01 2006 From: vlb at cfcl.com (Vicki Brown) Date: Mon, 10 Jul 2006 20:26:01 -0700 Subject: [sf-perl] JOB: Perl Programmer Opportunity Message-ID: Please respond directly to Monica. --- Begin Forward --- > Subject: Perl Programmer Opportunity > Date: Fri, 7 Jul 2006 16:58:20 -0400 > From: "Latimore, Monica" > > > I have a Perl programming opportunity in Redwood City. > > Please review the following requirement and let me know your level of >interest. If you're not available perhaps you know someone who may be a fit. > > Thank you in advance for your response. > > Monica~ > 408 441-7144 x107 > > Title: Perl Programmer > > Location: Redwood City, CA > > Contact Recruiter: Monica Latimore - >monica.latimore at modisit.com or call me >at (408) 441-7144 x107 > > Special Note: Salary and title is commensurate with experience. This is a >hands on job and the candidate must be able to program. > > Contract or Direct Hire is available > > We are in search of a highly driven and motivated Senior Software >Developer/Technical Lead to develop and build the backend server side >systems that power our content and ad optimization technologies. These >technologies enable my client to determine the correct advertisement or >content to serve out of a potential choice of thousands in a manner of >milliseconds. The optimization technologies employed are world class and >will be deployed world wide. > > The Senior Software Developer will to work as a member of a small team to: > *Aid in the formulation of an architecture and design > *Write commercial quality code > *Draft clear, concise functional specs for QA > *Develop QA test tools if and when necessary > *Handle rapid deployment cycles > > Ideal Candidate: > *Will have 5+ years of hands on programming experience in developing web >technologies and extensive experience in Perl, Linux, and Apache. > *Will have worked with and developed applications utilizing mySQL and Oracle > *Should have team leadership experience and should be strong in software >design and architecture. > *Will have worked in a commercial quality software development environment >and developed systems for large scale web servers and infinitely scalable >systems. > *Understanding or experience with ad supported media business model a plus > > Requirements: > *Extensive experience in developing applications in mod-Perl/Apache on >Linux or UNIX > *Experience programming in C/C++, Java a plus > *Understanding of SQL > *BS in Computer Science or similar > *Excellent English language skills (both spoken and written) > *Team leadership - in a team environment the ability to help design >scalable, commercial quality software and the ability to coach and mentor >other programmers > *Thorough understanding of QA documentation process, construction of QA >tools, and version control systems > *Knowledge of basic internet protocols > *Experience using systems such as CVS for source control management > > > > > Monica > > Monica Latimore > Modis > 226 Airport Parkway, Suite 600 > San Jose, California, 95110 > (408) 441-7144 x107 > (408) 441-7177 Fax > monica.latimore at modisit.com > www.modisit.com --- End Forward --- -- - Vicki ZZZ zzZ San Francisco Bay Area, CA z |\ _,,,---,,_ Books, Cats, Tech zz /,`.-'`' -. ;-;;,_ http://cfcl.com/vlb |,4- ) )-,_. ,\ ( `'-' http://cfcl.com/vlb/weblog '---''(_/--' `-'\_) http://vlb.typepad.com/commentary/ From moseley at hank.org Wed Jul 12 13:44:26 2006 From: moseley at hank.org (Bill Moseley) Date: Wed, 12 Jul 2006 13:44:26 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <20060708234405.GA1288@fu.funkspiel.org> References: <20060708234405.GA1288@fu.funkspiel.org> Message-ID: <20060712204425.GA13216@hank.org> On Sat, Jul 08, 2006 at 04:44:05PM -0700, Quinn Weaver wrote: > I reported earlier that you could get a discount on OSCON > registration. However, it turns out the code has changed. The new > registration discount code, as just sent to me by O'Reilly, is > > os06dsug > > If you're registering, paste this into the appropriate form field to > save 15% (off the early-registration price, which is evidently still > in effect for PM members). That only shows a 15% off the standard price, not the early reg price. -- Bill Moseley moseley at hank.org From qw at sf.pm.org Wed Jul 12 14:31:51 2006 From: qw at sf.pm.org (Quinn Weaver) Date: Wed, 12 Jul 2006 14:31:51 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <20060712204425.GA13216@hank.org> References: <20060708234405.GA1288@fu.funkspiel.org> <20060712204425.GA13216@hank.org> Message-ID: <20060712213151.GA35252@fu.funkspiel.org> On Wed, Jul 12, 2006 at 01:44:26PM -0700, Bill Moseley wrote: > On Sat, Jul 08, 2006 at 04:44:05PM -0700, Quinn Weaver wrote: > > I reported earlier that you could get a discount on OSCON > > registration. However, it turns out the code has changed. The new > > registration discount code, as just sent to me by O'Reilly, is > > > > os06dsug > > > > If you're registering, paste this into the appropriate form field to > > save 15% (off the early-registration price, which is evidently still > > in effect for PM members). > > That only shows a 15% off the standard price, not the early reg price. :( The O'Reilly newsletter I received reads >> Use code "os06dsug" when you register, and receive 15% off >> the early registration price. I will check back with O'Reilly about this. I'm guessing it was a misprint, and the double discount was supposed to be available only during the normal early-registration period. Bummer. -- qw (Quinn Weaver); #President, San Francisco Perl Mongers =for information, visit http://sf.pm.org/weblog =cut From sigje at sigje.org Wed Jul 12 14:37:26 2006 From: sigje at sigje.org (Jennifer Davis) Date: Wed, 12 Jul 2006 14:37:26 -0700 (PDT) Subject: [sf-perl] OSCON update In-Reply-To: <20060712213151.GA35252@fu.funkspiel.org> References: <20060708234405.GA1288@fu.funkspiel.org> <20060712204425.GA13216@hank.org> <20060712213151.GA35252@fu.funkspiel.org> Message-ID: Generally if you push enough they can get you the discount :) (on the early reg price) > > I will check back with O'Reilly about this. I'm guessing it was a misprint, > and the double discount was supposed to be available only during the normal > early-registration period. Bummer. > > -- > qw (Quinn Weaver); #President, San Francisco Perl Mongers > =for information, visit http://sf.pm.org/weblog =cut > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm > > > !DSPAM:44b56a8e38995368819438! > > From rdm at cfcl.com Wed Jul 12 15:05:16 2006 From: rdm at cfcl.com (Rich Morin) Date: Wed, 12 Jul 2006 15:05:16 -0700 Subject: [sf-perl] OSCON update In-Reply-To: <20060712213151.GA35252@fu.funkspiel.org> References: <20060708234405.GA1288@fu.funkspiel.org> <20060712204425.GA13216@hank.org> <20060712213151.GA35252@fu.funkspiel.org> Message-ID: At 2:31 PM -0700 7/12/06, Quinn Weaver wrote: > :( The O'Reilly newsletter I received reads > >>> Use code "os06dsug" when you register, and receive 15% off >>> the early registration price. > > I will check back with O'Reilly about this. I'm guessing it was > a misprint, and the double discount was supposed to be available > only during the normal early-registration period. Bummer. Possibly it wasn't a misprint, nor even a mistake, just imprecise phrasing. My suspicion is that it translates to: >>> Use code "os06dsug" when you register, and receive 15% off >>> the prevailing registration price (lower now, while we're >>> in the early registration period). That said, O'Reilly et al generally maintain admirable levels of quality in their work and frequently look further than others in their consideration of the second-order effects of their choices. So, don't be too hard on them... -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm at cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development From qw at sf.pm.org Wed Jul 12 19:06:25 2006 From: qw at sf.pm.org (Quinn Weaver) Date: Wed, 12 Jul 2006 19:06:25 -0700 Subject: [sf-perl] OSCON update In-Reply-To: References: <20060708234405.GA1288@fu.funkspiel.org> <20060712204425.GA13216@hank.org> <20060712213151.GA35252@fu.funkspiel.org> Message-ID: <20060713020625.GA35989@fu.funkspiel.org> On Wed, Jul 12, 2006 at 03:05:16PM -0700, Rich Morin wrote: > At 2:31 PM -0700 7/12/06, Quinn Weaver wrote: > > :( The O'Reilly newsletter I received reads > > > >>> Use code "os06dsug" when you register, and receive 15% off > >>> the early registration price. > > > > I will check back with O'Reilly about this. I'm guessing it was > > a misprint, and the double discount was supposed to be available > > only during the normal early-registration period. Bummer. Yes, I received confirmation from O'Reilly that this is the case. Sorry! I guess it couldn't hurt to follow Jennifer's example, but the official word is no. > Possibly it wasn't a misprint, nor even a mistake, just imprecise > phrasing. My suspicion is that it translates to: > > >>> Use code "os06dsug" when you register, and receive 15% off > >>> the prevailing registration price (lower now, while we're > >>> in the early registration period). Yes, but the newsletter was sent out after the early registration period had already ended. Anyway, we know what the deal is now. > That said, O'Reilly et al generally maintain admirable levels of > quality in their work and frequently look further than others in > their consideration of the second-order effects of their choices. > So, don't be too hard on them... I agree, O'Reilly rocks. :) -- qw (Quinn Weaver); #President, San Francisco Perl Mongers =for information, visit http://sf.pm.org/weblog =cut From extasia at extasia.org Sat Jul 15 10:04:59 2006 From: extasia at extasia.org (David Alban) Date: Sat, 15 Jul 2006 10:04:59 -0700 Subject: [sf-perl] Make a hash key's value dependent on the key itself? In-Reply-To: <20060706170432.c9dr0flqb4oc484c@webmail.gregerhaga.net> References: <4c714a9c0607012047q1ed70bf9ub43b8df9c33e9f48@mail.gmail.com> <20060703175542.80439.qmail@web31804.mail.mud.yahoo.com> <4c714a9c0607051615s5417e04fve2d52067079db9a9@mail.gmail.com> <20060706170432.c9dr0flqb4oc484c@webmail.gregerhaga.net> Message-ID: <4c714a9c0607151004p62b8e1d6h3df6c28bb98ca20c@mail.gmail.com> Thanks Greger. xml is grand, but I think I would still be at risk of an edit-time copy error. The risk would simply move from the source file to the config file. I wanted a solution that is fairly cut-and-paste-error resistant. The ::SOURCEHOST:: token seems to be that solution. On 7/6/06, boss at gregerhaga.net wrote: > how about putting the config in an xml file? > It would allow you to use the exact same script for various xml-config > files. You wuldn't need to change anything in the script itself when > source and target changes. -- Live in a world of your own, but always welcome visitors. From mehryar at mehryar.com Tue Jul 18 11:08:32 2006 From: mehryar at mehryar.com (mehryar) Date: Tue, 18 Jul 2006 11:08:32 -0700 (PDT) Subject: [sf-perl] handling email messages in Perl Message-ID: Hi folks, Im trying to read and write back out mail messages in Perl. I know Simon Cozens started the Email::* project to get away from the Mail::* bloat. But Im having a little mental block here. Im trying to read and write back out a mail folder like so: use Email::Folder; my $folder = Email::Folder->new("test_folder"); my @messages = $folder->messages; print "Weee! I got ", scalar @messages," email messages\n"; now I'd like to write these messages back out to a file so that I can re-read it again using the above code. And I can't seem to be able to do it. Basically I can't figure out how to write the message back into a "folder" type file that Email::Folder will be able to read again. I've tried a few ways Email::LocalDelivery, plain writing it out, but Im just missing something. thanks, -mehryar From Peter.Loo at source.wolterskluwer.com Thu Jul 20 10:31:21 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Thu, 20 Jul 2006 10:31:21 -0700 Subject: [sf-perl] opening a pipe to a system command Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A01A32A48@phxmail02.phx.ndchealth.com> Hi, Would someone happen to know how I can trap an error return from the pipe? Here is what I am doing. unless(open(SYSCMD, "| someSystemCommand -d someFlag -u moreFlag1 -w moreFlag2")) { DO SOME ERROR HANDLING... } I tried eval {}; also and nothing was trapped. Hope someone has an answer for this. :) Thanks. Peter Loo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.pm.org/pipermail/sanfrancisco-pm/attachments/20060720/1e297f56/attachment.html From herbr at pfinders.com Thu Jul 20 11:31:58 2006 From: herbr at pfinders.com (Herb Rubin) Date: Thu, 20 Jul 2006 11:31:58 -0700 (PDT) Subject: [sf-perl] opening a pipe to a system command In-Reply-To: <8E3D502A002DA04FADBDED4CB4D94D3A01A32A48@phxmail02.phx.ndchealth.com> Message-ID: <25144070.2801153420318840.JavaMail.root@z01.pfinders.com> Peter, If I remember correctly, the system command sets the $? variable with the return code. So, you have access to it. I also remember that you have to do some bitwise operation to get the numeric value properly from the return code. Then there's the issue of Perl on Windows compatibility... Here's an interesting web page for you: http://aspn.activestate.com/ASPN/Mail/Message/perl-unix-users/2073142 Herb Rubin ----- Original Message ----- From: Peter # PHX Loo To: sfpug at sf.pm.org Sent: Thursday, July 20, 2006 10:31:21 AM GMT-0800 Subject: [sf-perl] opening a pipe to a system command Hi, Would someone happen to know how I can trap an error return from the pipe? Here is what I am doing. unless(open(SYSCMD, "| someSystemCommand -d someFlag -u moreFlag1 -w moreFlag2")) { DO SOME ERROR HANDLING... } I tried eval {}; also and nothing was trapped. Hope someone has an answer for this. :) Thanks. Peter Loo From rdm at cfcl.com Sun Jul 23 09:40:15 2006 From: rdm at cfcl.com (Rich Morin) Date: Sun, 23 Jul 2006 09:40:15 -0700 Subject: [sf-perl] Beer and Scripting SIG (BASS) - reminder Message-ID: The Beer & Scripting SIG (http://www.cfcl.com/rdm/bass) will take place Wednesday evening, 7/26. Be there or be elsewhere! In case you were wondering, the discussions at BASS gatherings are not, erm, scripted. Rather, they reflect the interests of whatever scripters happen to show up that evening. One recent gathering focused on databases (especially PostgreSQL); other gatherings have discussed the vagaries of doing scripting for a living, which languages folks like (and why), etc. Sometimes, folks bring in books that they have recently read, software that they want to show off, etc. The informal nature of the gatherings allows this, as long as it doesn't conflict with important things like ordering and eating food! Also note that the calendar of Bay Area Scripting Events is available and open to submissions (all your BASE are belong to us :-). * webcal://cfcl.com/pub_dav/BA_Scripting_Groups.ics * http://cfcl.com/pub_dav/BA_Scripting_Groups.ics and that the list of local scripting groups is online at SF Bay Area Scripting Groups http://www.cfcl.com/rdm/bass/groups.php -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm at cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development From andy at petdance.com Sun Jul 23 09:45:37 2006 From: andy at petdance.com (Andy Lester) Date: Sun, 23 Jul 2006 11:45:37 -0500 Subject: [sf-perl] In town for LinuxWorld Message-ID: I'll be in town for LinuxWorld in August (http:// www.linuxworldexpo.com/live/12/events/12SFO06A). Anything going on Mongerwise I should know? I'd love to come do my talk on technical debt if there's a meeting in that time span. Andy -- Andy Lester => andy at petdance.com => www.petdance.com => AIM:petdance From rdm at cfcl.com Fri Jul 28 08:31:59 2006 From: rdm at cfcl.com (Rich Morin) Date: Fri, 28 Jul 2006 08:31:59 -0700 Subject: [sf-perl] DB design question Message-ID: This is one of those questions where there is no shortage of answers, but some are likely to be a lot better (for various reasons) than the others. So, I'm hoping for suggestions, rationales, etc. Background * There are billions of entities (5-10, at present). * Each entity has a unique name, which could be 100+ characters in length. * Each entity has a collection of attributes, drawn from a much larger set (could also be in the billions). * The "signature" of an attribute might be 50 characters. * I'd like to keep the total storage constrained to (say) one KB per entity. * It's OK (but not necessary) to presume that entity and attribute ids can only be used once. Problem For each new entity that I encounter, I need to determine and record its "unusual" attributes and save this in a way that will allow me to (later) find other entities which have similar sets of unusual attributes. Discussion Without presuming any particular database for this project, I'll use MySQL syntax to sketch out a "straw man" design: CREATE TABLE Entities ( id INT NOT NULL, name CHAR(200) ) CREATE TABLE Attrs ( id INT NOT NULL, sig CHAR(50), count INT NOT NULL ) CREATE TABLE Links ( id_entity INT NOT NULL, id_attr INT NOT NULL ) Using this setup, I could start with an entity, pick one of its attributes that has a low count, and look for other entities that have the same attribute. Repeating this for several attributes, I could build up a "cluster" of entities that have similar, unusual attributes. Another setup would involve the use of a blob in the Entities table, containing (say) 250 id_attr values. This limits my ability to use SQL, however, and it also keeps me from doing cute lookups in the Links table. Anyway, I'm open to suggestions... -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm at cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development From josh at agliodbs.com Fri Jul 28 08:36:30 2006 From: josh at agliodbs.com (Josh Berkus) Date: Fri, 28 Jul 2006 08:36:30 -0700 Subject: [sf-perl] DB design question In-Reply-To: References: Message-ID: <44CA2EFE.9070907@agliodbs.com> Rich, > Anyway, I'm open to suggestions... I have a solution for this, but you'll have to wait until I get back from OSCON. --Josh From david at fetter.org Fri Jul 28 09:07:40 2006 From: david at fetter.org (David Fetter) Date: Fri, 28 Jul 2006 09:07:40 -0700 Subject: [sf-perl] DB design question In-Reply-To: References: Message-ID: <20060728160740.GB32643@fetter.org> On Fri, Jul 28, 2006 at 08:31:59AM -0700, Rich Morin wrote: > This is one of those questions where there is no shortage of > answers, but some are likely to be a lot better (for various > reasons) than the others. So, I'm hoping for suggestions, > rationales, etc. Just FYI, the "entity-attribute-value" storage system is a thoroughly documented mistake. http://joecelkothesqlapprentice.blogspot.com/2006/06/give-me-your-thoughts-storing-format.html There are much better ways to go about this, but we'd need more info on just exactly what's going on in order to construct a better data model. Cheers, D > > Background > > * There are billions of entities (5-10, at present). > > * Each entity has a unique name, which could be 100+ > characters in length. > > * Each entity has a collection of attributes, drawn from > a much larger set (could also be in the billions). > > * The "signature" of an attribute might be 50 characters. > > * I'd like to keep the total storage constrained to (say) > one KB per entity. > > * It's OK (but not necessary) to presume that entity and > attribute ids can only be used once. > > Problem > > For each new entity that I encounter, I need to determine and > record its "unusual" attributes and save this in a way that > will allow me to (later) find other entities which have similar > sets of unusual attributes. > > > Discussion > > Without presuming any particular database for this project, > I'll use MySQL syntax to sketch out a "straw man" design: > > CREATE TABLE Entities ( > id INT NOT NULL, > name CHAR(200) > ) > > CREATE TABLE Attrs ( > id INT NOT NULL, > sig CHAR(50), > count INT NOT NULL > ) > > CREATE TABLE Links ( > id_entity INT NOT NULL, > id_attr INT NOT NULL > ) > > Using this setup, I could start with an entity, pick one of > its attributes that has a low count, and look for other > entities that have the same attribute. Repeating this for > several attributes, I could build up a "cluster" of entities > that have similar, unusual attributes. > > Another setup would involve the use of a blob in the Entities > table, containing (say) 250 id_attr values. This limits my > ability to use SQL, however, and it also keeps me from doing > cute lookups in the Links table. > > Anyway, I'm open to suggestions... > > -r > -- > http://www.cfcl.com/rdm Rich Morin > http://www.cfcl.com/rdm/resume rdm at cfcl.com > http://www.cfcl.com/rdm/weblog +1 650-873-7841 > > Technical editing and writing, programming, and web development > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm -- David Fetter http://fetter.org/ phone: +1 415 235 3778 AIM: dfetter666 Skype: davidfetter Remember to vote! From rdm at cfcl.com Fri Jul 28 09:28:07 2006 From: rdm at cfcl.com (Rich Morin) Date: Fri, 28 Jul 2006 09:28:07 -0700 Subject: [sf-perl] DB design question In-Reply-To: <20060728160740.GB32643@fetter.org> References: <20060728160740.GB32643@fetter.org> Message-ID: At 9:07 AM -0700 7/28/06, David Fetter wrote: > There are much better ways to go about this, but we'd need > more info on just exactly what's going on in order to > construct a better data model. Unfortunately, I'm not at liberty to give out arbitrary amounts of information, nor would you want to read through a total dump of the problem. That's why I used an abstract description. If you have some specific questions (however general :-), I'll try to answer them. -r -- http://www.cfcl.com/rdm Rich Morin http://www.cfcl.com/rdm/resume rdm at cfcl.com http://www.cfcl.com/rdm/weblog +1 650-873-7841 Technical editing and writing, programming, and web development From josh at agliodbs.com Fri Jul 28 11:06:06 2006 From: josh at agliodbs.com (Josh Berkus) Date: Fri, 28 Jul 2006 11:06:06 -0700 Subject: [sf-perl] DB design question In-Reply-To: <20060728160740.GB32643@fetter.org> References: <20060728160740.GB32643@fetter.org> Message-ID: <44CA520E.7000003@agliodbs.com> > Just FYI, the "entity-attribute-value" storage system is a > thoroughly documented mistake. Just FYI, this is a controversial opinion that not everyone agrees with. ;-> --Josh From fred at redhotpenguin.com Fri Jul 28 11:43:02 2006 From: fred at redhotpenguin.com (Fred Moyer) Date: Fri, 28 Jul 2006 11:43:02 -0700 Subject: [sf-perl] DB design question References: <1154112184.1C8FB94C@dj11.dngr.org> Message-ID: <1154112185.1F4A8FD6@fb8.dngr.org> On Fri, 28 Jul 2006 11:18 am, Josh Berkus wrote: > >> Just FYI, the "entity-attribute-value" storage system is a >> thoroughly documented mistake. > > Just FYI, this is a controversial opinion that not everyone agrees > with. In my experience you end up trading performance for flexibility with this approach. I was faced with a similar problem managing semi structured voter data and ending up using xml/rdf to manage the data. There's an article on perl.com, search for massive data aggregation using Perl and xml. I'm not saying that's the best approach, but it ended up working well for our use. Fred From friedman at highwire.stanford.edu Fri Jul 28 20:31:01 2006 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Fri, 28 Jul 2006 20:31:01 -0700 Subject: [sf-perl] DB design question In-Reply-To: References: Message-ID: <4E89B7EC-D1CE-4DCF-BD2D-83F7376E9C77@highwire.stanford.edu> Rich, Upon reading the problem definition here, and reading your straw man solution, I think you are thinking of this problem differently from the way you wrote it. > Problem > > For each new entity that I encounter, I need to determine and > record its "unusual" attributes and save this in a way that > will allow me to (later) find other entities which have similar > sets of unusual attributes. To me, that says that you are tracking each entity for the *sole purpose* of clustering them based on similar attributes. (You say attributes instead of attribute values. I'm assuming you meant attribute values, but you could just make attributes have binary values if not.) However, the design proposed seems to me to be a storage medium for entities with varying attributes. That has a large number of problems, the least of which is that given a completely free-form attribute table, your SQL queries to find clusters of similar entities will be ugly and slow. (As was previously pointed out by others.) What I would suggest is a matrix instead of relational tables. You could implement it in tables, but some other way might be more efficient. For example, what you are looking for is clusters with similar attributes. Imagine an Excel spreadsheet with each entity as a single row and each attribute as a column. The value for each attribute goes into the proper cell for each entity. To find "every entity with attribute X between 10 and 20", you sort the matrix on the column for attribute X and read off everything with a value between 10 and 20. (Entities without that attribute at all would get undef.) Sure, it looks like a relational table, but you're also talking about billions of entities and billions of attributes, so that'll never work. :-) However, if each entity only has a manageable number of attributes (100? 10? "reasonable" depends on a lot of context here) then you could create a sparse hash for the matrix and store that in some fashion. (MLDBM, for a simple start; objects using the Flyweight pattern for both entities and attributes in some sort of repository in the longer term probably.) Using a hash (tied to whatever) you can then cluster by sorting the hash by $entity{$attribute} and then read out the range of items with values where you want. Optimizing this sort (or search) would be the most important part of making it run with billions of items, but also pretty straightforward. For example, you could walk the hash in sections and cluster each section separately. If you really are dealing with billions of entities (Genomic data?) then you could split it across multiple machines and write a little broker/server package that merges results together. (Uh... I know there's something that handles the hard work of that on CPAN. POE maybe it's called?) A further optimization would be classification. If several attributes tend to go together, you could wrap them up into a bigger attribute, thus reducing the actual number of attributes you need to code for. If attributes are degrees of one another, you could merge them into a single non-binary-valued attribute, to reduce the number. If a bunch of entities have the exact same attributes, ignore all but the first one and keep track of "entity groups" in a separate storage mechanism. Anything to reduce the number of cells in the matrix would help with execution speed. But again, that's all really only necessary once you hit the limits of your first try. Do The Simplest Thing That Might Possibly Work, then optimize later. This has gotten rather long, but without more context, I can only make vague guesses at an approach. Best of luck with this! -- Mike Friedman PS - This is basically "compute a vector in attribute space for each entity, but only store the non-zero values." If that description makes more sense, code to it. :-) PPS - Fulltext search engines do something like this in their Query By Example feature. If you can find an open source search engine with QBE, you might be able to snag some code or at least design that would help with this kind of problem. On Jul 28, 2006, at 8:31 AM, Rich Morin wrote: > This is one of those questions where there is no shortage of > answers, but some are likely to be a lot better (for various > reasons) than the others. So, I'm hoping for suggestions, > rationales, etc. > > Background > > * There are billions of entities (5-10, at present). > > * Each entity has a unique name, which could be 100+ > characters in length. > > * Each entity has a collection of attributes, drawn from > a much larger set (could also be in the billions). > > * The "signature" of an attribute might be 50 characters. > > * I'd like to keep the total storage constrained to (say) > one KB per entity. > > * It's OK (but not necessary) to presume that entity and > attribute ids can only be used once. > > Problem > > For each new entity that I encounter, I need to determine and > record its "unusual" attributes and save this in a way that > will allow me to (later) find other entities which have similar > sets of unusual attributes. > > > Discussion > > Without presuming any particular database for this project, > I'll use MySQL syntax to sketch out a "straw man" design: > > CREATE TABLE Entities ( > id INT NOT NULL, > name CHAR(200) > ) > > CREATE TABLE Attrs ( > id INT NOT NULL, > sig CHAR(50), > count INT NOT NULL > ) > > CREATE TABLE Links ( > id_entity INT NOT NULL, > id_attr INT NOT NULL > ) > > Using this setup, I could start with an entity, pick one of > its attributes that has a low count, and look for other > entities that have the same attribute. Repeating this for > several attributes, I could build up a "cluster" of entities > that have similar, unusual attributes. > > Another setup would involve the use of a blob in the Entities > table, containing (say) 250 id_attr values. This limits my > ability to use SQL, however, and it also keeps me from doing > cute lookups in the Links table. > > Anyway, I'm open to suggestions... > > -r > -- > http://www.cfcl.com/rdm Rich Morin > http://www.cfcl.com/rdm/resume rdm at cfcl.com > http://www.cfcl.com/rdm/weblog +1 650-873-7841 > > Technical editing and writing, programming, and web development > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm --------------------------------------------------------------------- Michael Friedman HighWire Press Phone: 650-725-1974 Stanford University FAX: 270-721-8034 --------------------------------------------------------------------- From josh at agliodbs.com Sat Jul 29 10:17:23 2006 From: josh at agliodbs.com (Josh Berkus) Date: Sat, 29 Jul 2006 10:17:23 -0700 Subject: [sf-perl] DB design question In-Reply-To: <4E89B7EC-D1CE-4DCF-BD2D-83F7376E9C77@highwire.stanford.edu> References: <4E89B7EC-D1CE-4DCF-BD2D-83F7376E9C77@highwire.stanford.edu> Message-ID: <44CB9823.2050208@agliodbs.com> Mike, > PPS - Fulltext search engines do something like this in their Query > By Example feature. If you can find an open source search engine with > QBE, you might be able to snag some code or at least design that > would help with this kind of problem. Actually, it just so happens .... http://pgfoundry.org/projects/qbe ... although personally I'd solve Rich's problem with a bitmap rather than QBE if there are a large number of yet-to-be-defined attributes. --Josh From Peter.Loo at source.wolterskluwer.com Mon Jul 31 14:22:22 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 31 Jul 2006 14:22:22 -0700 Subject: [sf-perl] Runtime libraries Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A01B410C3@phxmail02.phx.ndchealth.com> Hi, Can someone please tell me how I can stdout runtime libraries within a program? Thanks. Peter Loo From friedman at highwire.stanford.edu Mon Jul 31 14:26:19 2006 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Mon, 31 Jul 2006 14:26:19 -0700 Subject: [sf-perl] Runtime libraries In-Reply-To: <8E3D502A002DA04FADBDED4CB4D94D3A01B410C3@phxmail02.phx.ndchealth.com> References: <8E3D502A002DA04FADBDED4CB4D94D3A01B410C3@phxmail02.phx.ndchealth.com> Message-ID: <4CD21016-2006-4A24-89AD-856346C3B67D@highwire.stanford.edu> Do you mean print the list of libraries that the script can use? If so, print the contents of @INC and you'll see where it's getting its modules from. -- Mike On Jul 31, 2006, at 2:22 PM, Loo, Peter # PHX wrote: > > Hi, > > Can someone please tell me how I can stdout runtime libraries within a > program? > > Thanks. > > > Peter Loo > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm --------------------------------------------------------------------- Michael Friedman HighWire Press Phone: 650-725-1974 Stanford University FAX: 270-721-8034 --------------------------------------------------------------------- From friedman at highwire.stanford.edu Mon Jul 31 14:26:19 2006 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Mon, 31 Jul 2006 14:26:19 -0700 Subject: [sf-perl] Runtime libraries In-Reply-To: <8E3D502A002DA04FADBDED4CB4D94D3A01B410C3@phxmail02.phx.ndchealth.com> References: <8E3D502A002DA04FADBDED4CB4D94D3A01B410C3@phxmail02.phx.ndchealth.com> Message-ID: <4CD21016-2006-4A24-89AD-856346C3B67D@highwire.stanford.edu> Do you mean print the list of libraries that the script can use? If so, print the contents of @INC and you'll see where it's getting its modules from. -- Mike On Jul 31, 2006, at 2:22 PM, Loo, Peter # PHX wrote: > > Hi, > > Can someone please tell me how I can stdout runtime libraries within a > program? > > Thanks. > > > Peter Loo > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm --------------------------------------------------------------------- Michael Friedman HighWire Press Phone: 650-725-1974 Stanford University FAX: 270-721-8034 --------------------------------------------------------------------- From Peter.Loo at source.wolterskluwer.com Mon Jul 31 14:31:17 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 31 Jul 2006 14:31:17 -0700 Subject: [sf-perl] Runtime libraries In-Reply-To: <4CD21016-2006-4A24-89AD-856346C3B67D@highwire.stanford.edu> Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A01B410DA@phxmail02.phx.ndchealth.com> Oh yes, thanks Michael. For the life of me, I was unable to remember @INC. You have saved my day. :D Peter Loo -----Original Message----- From: sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.org [mailto:sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.or g] On Behalf Of Michael Friedman Sent: Monday, July 31, 2006 2:26 PM To: San Francisco Perl Mongers User Group Cc: sfpug at sf.pm.org Subject: Re: [sf-perl] Runtime libraries Do you mean print the list of libraries that the script can use? If so, print the contents of @INC and you'll see where it's getting its modules from. -- Mike On Jul 31, 2006, at 2:22 PM, Loo, Peter # PHX wrote: > > Hi, > > Can someone please tell me how I can stdout runtime libraries within a > program? > > Thanks. > > > Peter Loo > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm --------------------------------------------------------------------- Michael Friedman HighWire Press Phone: 650-725-1974 Stanford University FAX: 270-721-8034 --------------------------------------------------------------------- _______________________________________________ SanFrancisco-pm mailing list SanFrancisco-pm at pm.org http://mail.pm.org/mailman/listinfo/sanfrancisco-pm This E-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail, and destroy all copies of the original message. From Peter.Loo at source.wolterskluwer.com Mon Jul 31 14:31:17 2006 From: Peter.Loo at source.wolterskluwer.com (Loo, Peter # PHX) Date: Mon, 31 Jul 2006 14:31:17 -0700 Subject: [sf-perl] Runtime libraries In-Reply-To: <4CD21016-2006-4A24-89AD-856346C3B67D@highwire.stanford.edu> Message-ID: <8E3D502A002DA04FADBDED4CB4D94D3A01B410DA@phxmail02.phx.ndchealth.com> Oh yes, thanks Michael. For the life of me, I was unable to remember @INC. You have saved my day. :D Peter Loo -----Original Message----- From: sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.org [mailto:sanfrancisco-pm-bounces+peter.loo=source.wolterskluwer.com at pm.or g] On Behalf Of Michael Friedman Sent: Monday, July 31, 2006 2:26 PM To: San Francisco Perl Mongers User Group Cc: sfpug at sf.pm.org Subject: Re: [sf-perl] Runtime libraries Do you mean print the list of libraries that the script can use? If so, print the contents of @INC and you'll see where it's getting its modules from. -- Mike On Jul 31, 2006, at 2:22 PM, Loo, Peter # PHX wrote: > > Hi, > > Can someone please tell me how I can stdout runtime libraries within a > program? > > Thanks. > > > Peter Loo > _______________________________________________ > SanFrancisco-pm mailing list > SanFrancisco-pm at pm.org > http://mail.pm.org/mailman/listinfo/sanfrancisco-pm --------------------------------------------------------------------- Michael Friedman HighWire Press Phone: 650-725-1974 Stanford University FAX: 270-721-8034 --------------------------------------------------------------------- _______________________________________________ SanFrancisco-pm mailing list SanFrancisco-pm at pm.org http://mail.pm.org/mailman/listinfo/sanfrancisco-pm This E-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply E-mail, and destroy all copies of the original message. From vlb at cfcl.com Mon Jul 31 22:25:33 2006 From: vlb at cfcl.com (Vicki Brown) Date: Mon, 31 Jul 2006 22:25:33 -0700 Subject: [sf-perl] installing CPAN inside firewall Message-ID: My Current Job (tm) includes a FreeBSD box under the desk. Said box is running Perl 5.005 and is accessible via VPN. I can get out from that box over sftp. AFAIK, nothing else much works. When I attempt to run Perl -MCPAN -e shell Perl wants to set up and configure CPAN. it insists on trying lynx, followed by ncftpget (neither of which work). There appears to be no easy way to tell it "Please don't try those; just use sftp". I can't even substitute sftp for ftp because it wants to use a nontransferable -n flag. It would be kinda nice to upgrade Perl. I could just pull down the sources directly I suppose but... Please don't recommend that I use the Free BSD Ports Collection. It's not available. I think I'm lucky to have Perl 5.005... -- - Vicki ZZZ zzZ San Francisco Bay Area, CA z |\ _,,,---,,_ Books, Cats, Tech zz /,`.-'`' -. ;-;;,_ http://cfcl.com/vlb |,4- ) )-,_. ,\ ( `'-' http://cfcl.com/vlb/weblog '---''(_/--' `-'\_) http://vlb.typepad.com/commentary/ From joshua.mcadams at gmail.com Mon Jul 31 23:12:15 2006 From: joshua.mcadams at gmail.com (Joshua McAdams) Date: Tue, 1 Aug 2006 01:12:15 -0500 Subject: [sf-perl] installing CPAN inside firewall In-Reply-To: References: Message-ID: <49d805d70607312312vc0d6227gf2ebcd2faca79f8d@mail.gmail.com> > My Current Job (tm) includes a FreeBSD box under the desk. Said box is > running Perl 5.005 and is accessible via VPN. Is 'said box' the only box that you have access to within the system? Possibly you can figure out how to use CPAN::Mini or, my favorite, CPAN::Mini::Inject to create a mirror that your under-the-desk box can reach via FTP? That is how I used to circumvent security ;)