From doug at phoenixinternet.com Mon Dec 1 18:04:36 2003 From: doug at phoenixinternet.com (Douglas E. Miles) Date: Thu Aug 5 00:17:10 2004 Subject: Phoenix.pm: Meeting 12/04/2002 Message-ID: <3FCBD714.3050905@phoenixinternet.com> Holiday Party social meeting! We'll be having a Phoenix.pm meeting Thursday, December 4th at 7:00PM. It will be held at Bowne, which is located at 1500 N. Central Avenue, which is on the Southwest corner of Central and McDowell. The parking lot is gated, so just press the button on the intercom, and tell the receptionist that you are there for the Perl meeting. Park in the lot that is straight ahead from the entrance on the South side of McDowell. Park in any uncovered, non-reserved space. Proceed to the main lobby, which is on the Northeast side of the parking lot. We'll be having a "geek" white elephant gift exchange. Bring any hardware/computer paraphernalia (working or not) you'd like to unload on some hapless victim. Please wrap the gift (or at least put it in a box) to increase the suspense. :) If you don't want to participate, you don't have to. You just won't get to take home any new useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are so inclined, feel free to bring a snack. I'll bring some cookies. From doug at phoenixinternet.com Mon Dec 1 18:08:58 2003 From: doug at phoenixinternet.com (Douglas E. Miles) Date: Thu Aug 5 00:17:10 2004 Subject: Phoenix.pm: Meeting 12/04/2003 Message-ID: <3FCBD81A.6000402@phoenixinternet.com> That should be 2003. Sorry for the time warp. We now continue with your regularly scheduled space-time continuum, already in progress. Holiday Party social meeting! We'll be having a Phoenix.pm meeting Thursday, December 4th at 7:00PM. It will be held at Bowne, which is located at 1500 N. Central Avenue, which is on the Southwest corner of Central and McDowell. The parking lot is gated, so just press the button on the intercom, and tell the receptionist that you are there for the Perl meeting. Park in the lot that is straight ahead from the entrance on the South side of McDowell. Park in any uncovered, non-reserved space. Proceed to the main lobby, which is on the Northeast side of the parking lot. We'll be having a "geek" white elephant gift exchange. Bring any hardware/computer paraphernalia (working or not) you'd like to unload on some hapless victim. Please wrap the gift (or at least put it in a box) to increase the suspense. :) If you don't want to participate, you don't have to. You just won't get to take home any new useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are so inclined, feel free to bring a snack. I'll bring some cookies. From intertwingled at qwest.net Tue Dec 2 02:48:41 2003 From: intertwingled at qwest.net (intertwingled) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 In-Reply-To: <3FCBD81A.6000402@phoenixinternet.com> References: <3FCBD81A.6000402@phoenixinternet.com> Message-ID: <3FCC51E9.7000007@qwest.net> I was going to bring a working VAXstation, but since I don't attend phoenx.pm any more.... oh well. =) Tony Douglas E. Miles wrote: > That should be 2003. Sorry for the time warp. We now continue with > your regularly scheduled space-time continuum, already in progress. > > Holiday Party social meeting! > > We'll be having a Phoenix.pm meeting Thursday, December 4th at 7:00PM. > It will be held at Bowne, which is located at 1500 N. Central Avenue, > which is on the Southwest corner of Central and McDowell. The parking > lot is gated, so just press the button on the intercom, and tell the > receptionist that you are there for the Perl meeting. Park in the lot > that is straight ahead from the entrance on the South side of McDowell. > Park in any uncovered, non-reserved space. Proceed to the main lobby, > which is on the Northeast side of the parking lot. > > We'll be having a "geek" white elephant gift exchange. Bring any > hardware/computer paraphernalia (working or not) you'd like to unload on > some hapless victim. Please wrap the gift (or at least put it in a box) > to increase the suspense. :) If you don't want to participate, you > don't have to. You just won't get to take home any new > useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are so inclined, feel > free to bring a snack. I'll bring some cookies. > > > > > From tran_fors at yahoo.com Tue Dec 2 16:54:36 2003 From: tran_fors at yahoo.com (Tran Forsythe) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 In-Reply-To: <3FCBD81A.6000402@phoenixinternet.com> Message-ID: <20031202225436.73307.qmail@web60710.mail.yahoo.com> Bugger... one of the few times I get this email in time, and it's to be one of my favourite types of meetings, and I'm heading out of town on Thursday evening... :p Oh well... I'll think of 'yall shivering as I'm on my cruise in the S. Caribbean next week. Oh, the sacrifices I make... ;) -Kurt --- "Douglas E. Miles" wrote: > That should be 2003. Sorry for the time warp. We > now continue with > your regularly scheduled space-time continuum, > already in progress. > > Holiday Party social meeting! > > We'll be having a Phoenix.pm meeting Thursday, > December 4th at 7:00PM. > It will be held at Bowne, which is located at 1500 > N. Central Avenue, > which is on the Southwest corner of Central and > McDowell. The parking > lot is gated, so just press the button on the > intercom, and tell the > receptionist that you are there for the Perl > meeting. Park in the lot > that is straight ahead from the entrance on the > South side of McDowell. > Park in any uncovered, non-reserved space. Proceed > to the main lobby, > which is on the Northeast side of the parking lot. > > We'll be having a "geek" white elephant gift > exchange. Bring any > hardware/computer paraphernalia (working or not) > you'd like to unload on > some hapless victim. Please wrap the gift (or at > least put it in a box) > to increase the suspense. :) If you don't want to > participate, you > don't have to. You just won't get to take home any > new > useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are > so inclined, feel > free to bring a snack. I'll bring some cookies. > > > __________________________________ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree From scott at illogics.org Tue Dec 2 17:08:03 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 Message-ID: <20031202230803.GJ1904@illogics.org> If this is one of those infamous Geek Cruises, please, smuggle me in in your luggage! -scott On 0, Tran Forsythe wrote: > > > Bugger... one of the few times I get this email in > time, and it's to be one of my favourite types of > meetings, and I'm heading out of town on Thursday > evening... :p Oh well... I'll think of 'yall > shivering as I'm on my cruise in the S. Caribbean next > week. Oh, the sacrifices I make... ;) > -Kurt > > --- "Douglas E. Miles" > wrote: > > That should be 2003. Sorry for the time warp. We > > now continue with > > your regularly scheduled space-time continuum, > > already in progress. > > > > Holiday Party social meeting! > > > > We'll be having a Phoenix.pm meeting Thursday, > > December 4th at 7:00PM. > > It will be held at Bowne, which is located at 1500 > > N. Central Avenue, > > which is on the Southwest corner of Central and > > McDowell. The parking > > lot is gated, so just press the button on the > > intercom, and tell the > > receptionist that you are there for the Perl > > meeting. Park in the lot > > that is straight ahead from the entrance on the > > South side of McDowell. > > Park in any uncovered, non-reserved space. Proceed > > to the main lobby, > > which is on the Northeast side of the parking lot. > > > > We'll be having a "geek" white elephant gift > > exchange. Bring any > > hardware/computer paraphernalia (working or not) > > you'd like to unload on > > some hapless victim. Please wrap the gift (or at > > least put it in a box) > > to increase the suspense. :) If you don't want to > > participate, you > > don't have to. You just won't get to take home any > > new > > useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are > > so inclined, feel > > free to bring a snack. I'll bring some cookies. > > > > > > > > > __________________________________ > Do you Yahoo!? > Protect your identity with Yahoo! Mail AddressGuard > http://antispam.yahoo.com/whatsnewfree From doug at phoenixinternet.com Wed Dec 3 12:30:56 2003 From: doug at phoenixinternet.com (Douglas E. Miles) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 In-Reply-To: <20031202230803.GJ1904@illogics.org> References: <20031202230803.GJ1904@illogics.org> Message-ID: <3FCE2BE0.7000803@phoenixinternet.com> Scott Walters wrote: > If this is one of those infamous Geek Cruises, please, > smuggle me in in your luggage! Well, Kurt's going to be there. So I guess so. :P > -scott > > On 0, Tran Forsythe wrote: > >> >>Bugger... one of the few times I get this email in >>time, and it's to be one of my favourite types of >>meetings, and I'm heading out of town on Thursday >>evening... :p Oh well... I'll think of 'yall >>shivering as I'm on my cruise in the S. Caribbean next >>week. Oh, the sacrifices I make... ;) From scott at illogics.org Wed Dec 3 13:23:32 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 Message-ID: <20031203192331.GZ1904@illogics.org> Was this an RSVP meeting? Hard to tell sometimes, what with only one negitive RSVP and all. If so, then I expect to make it. Thanks, -scott On 0, "Douglas E. Miles" wrote: > > Scott Walters wrote: > > If this is one of those infamous Geek Cruises, please, > > smuggle me in in your luggage! > > Well, Kurt's going to be there. So I guess so. :P > > > -scott > > > > On 0, Tran Forsythe wrote: > > > >> > >>Bugger... one of the few times I get this email in > >>time, and it's to be one of my favourite types of > >>meetings, and I'm heading out of town on Thursday > >>evening... :p Oh well... I'll think of 'yall > >>shivering as I'm on my cruise in the S. Caribbean next > >>week. Oh, the sacrifices I make... ;) > > > > From doug at phoenixinternet.com Wed Dec 3 13:44:21 2003 From: doug at phoenixinternet.com (Douglas E. Miles) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 In-Reply-To: <20031203192331.GZ1904@illogics.org> References: <20031203192331.GZ1904@illogics.org> Message-ID: <3FCE3D15.1030504@phoenixinternet.com> Scott Walters wrote: > Was this an RSVP meeting? Hard to tell sometimes, what with only > one negitive RSVP and all. If so, then I expect to make it. Yeah. I usually send out the reminders on Wednesday. Consider this your reminder, and please RSVP! From friedman at highwire.stanford.edu Wed Dec 3 15:12:23 2003 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: Meeting 12/04/2003 In-Reply-To: <3FCBD81A.6000402@phoenixinternet.com> References: <3FCBD81A.6000402@phoenixinternet.com> Message-ID: <634BD4D0-25D5-11D8-893C-000A956CAA2C@highwire.stanford.edu> I'll be there, as soon as I comb my "physical archives" of office toys... -- Mike On Dec 1, 2003, at 5:08 PM, Douglas E. Miles wrote: > That should be 2003. Sorry for the time warp. We now continue with > your regularly scheduled space-time continuum, already in progress. > > Holiday Party social meeting! > > We'll be having a Phoenix.pm meeting Thursday, December 4th at 7:00PM. > It will be held at Bowne, which is located at 1500 N. Central Avenue, > which is on the Southwest corner of Central and McDowell. The parking > lot is gated, so just press the button on the intercom, and tell the > receptionist that you are there for the Perl meeting. Park in the lot > that is straight ahead from the entrance on the South side of McDowell. > Park in any uncovered, non-reserved space. Proceed to the main lobby, > which is on the Northeast side of the parking lot. > > We'll be having a "geek" white elephant gift exchange. Bring any > hardware/computer paraphernalia (working or not) you'd like to unload > on > some hapless victim. Please wrap the gift (or at least put it in a > box) > to increase the suspense. :) If you don't want to participate, you > don't have to. You just won't get to take home any new > useless^H^H^H^H^H^H^H^H cool stuff. ;) If you are so inclined, feel > free to bring a snack. I'll bring some cookies. > > > --------------------------------------------------------------------- Michael Friedman HighWire Press, Stanford Southwest Phone: 480-456-0880 Tempe, Arizona FAX: 270-721-8034 --------------------------------------------------------------------- From m at pdxlug.org Thu Dec 4 17:04:26 2003 From: m at pdxlug.org (Matt Alexander) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <52273.68.110.171.77.1070579066.squirrel@webmail.pdxlug.org> I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' to the constructor like I can with LWP::UserAgent or WWW::Mechanize. Does anyone have a suggestion for how to enable keep-alives with RobotUA? Thanks, ~M P.S. The new O'Reilly book "Spidering Hacks" is incredible. Definitely check it out. From scott at illogics.org Thu Dec 4 17:09:21 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <20031204230921.GG1904@illogics.org> Very generally speaking: Go under the hood and hack the feature on. I haven't used any of those modules more than a wee widdle bit so I don't know how it all fits together, but one object is likely creating instances of others and this argument is something that could be perpetuated. Sorry I don't have a better answer, short on time today =( Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under books =) -scott On 0, Matt Alexander wrote: > > I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' to > the constructor like I can with LWP::UserAgent or WWW::Mechanize. Does > anyone have a suggestion for how to enable keep-alives with RobotUA? > Thanks, > ~M > P.S. The new O'Reilly book "Spidering Hacks" is incredible. Definitely > check it out. From jpowers at ccbill.com Thu Dec 4 18:14:58 2003 From: jpowers at ccbill.com (Jacob Powers) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <6365F4E07E769F4C89826D1DE90D803702A84F@CCBill-Exchange.exchange.cwie.net> As Scott said you will need to break into RobotUA to add that. RobotUA in its default form is very weak and ridden with "friendly" rules. Here are a few things I have done to it. 1. Remove all robot rules. Just comment out the use of Robot Rules. 2. Remove all wait commands. In order to be more website friendly it has a sleep(1), sometimes up to a minute, inbetween each request. Very annoying if you are trying to pull down many pages/sites quickly. Just set use_sleep to 0 and delay to 0. 3. Set it up to use MySQL for the seen/used URLs instead of using a hash (this gets really big really fast.) What I do is in the addUrl function, in RobotUA, I comment out the part where it adds/makes a hash and instead I MD5 the URL and put it in a table that has nothing but a char(32) field. This speeds the Robot up a lot. You also have to add the where it checks the Url, once again in the addUrl function to read from the DB instead of the hash. Those are just my personal tweaks to the RobotUA, or as I have renamed it RobotDB. Make sure you do ample testing in various scenarios with your hooks in place. Jacob Powers -----Original Message----- From: Scott Walters [mailto:scott@illogics.org] Sent: Thursday, December 04, 2003 4:09 PM To: Matt Alexander Subject: Re: Phoenix.pm: keep-alives with RobotUA Very generally speaking: Go under the hood and hack the feature on. I haven't used any of those modules more than a wee widdle bit so I don't know how it all fits together, but one object is likely creating instances of others and this argument is something that could be perpetuated. Sorry I don't have a better answer, short on time today =( Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under books =) -scott On 0, Matt Alexander wrote: > > I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' to > the constructor like I can with LWP::UserAgent or WWW::Mechanize. Does > anyone have a suggestion for how to enable keep-alives with RobotUA? > Thanks, > ~M > P.S. The new O'Reilly book "Spidering Hacks" is incredible. Definitely > check it out. From jpowers at ccbill.com Thu Dec 4 18:24:55 2003 From: jpowers at ccbill.com (Jacob Powers) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <6365F4E07E769F4C89826D1DE90D80370ABEAC@CCBill-Exchange.exchange.cwie.net> Opps that didn't come out as pretty as it went in.... oh well. Jacob Powers -----Original Message----- From: Jacob Powers Sent: Thursday, December 04, 2003 5:15 PM To: phoenix-pm-list@happyfunball.pm.org Subject: RE: Phoenix.pm: keep-alives with RobotUA As Scott said you will need to break into RobotUA to add that. RobotUA in its default form is very weak and ridden with "friendly" rules. Here are a few things I have done to it. 1. Remove all robot rules. Just comment out the use of Robot Rules. 2. Remove all wait commands. In order to be more website friendly it has a sleep(1), sometimes up to a minute, inbetween each request. Very annoying if you are trying to pull down many pages/sites quickly. Just set use_sleep to 0 and delay to 0. 3. Set it up to use MySQL for the seen/used URLs instead of using a hash (this gets really big really fast.) What I do is in the addUrl function, in RobotUA, I comment out the part where it adds/makes a hash and instead I MD5 the URL and put it in a table that has nothing but a char(32) field. This speeds the Robot up a lot. You also have to add the where it checks the Url, once again in the addUrl function to read from the DB instead of the hash. Those are just my personal tweaks to the RobotUA, or as I have renamed it RobotDB. Make sure you do ample testing in various scenarios with your hooks in place. Jacob Powers -----Original Message----- From: Scott Walters [mailto:scott@illogics.org] Sent: Thursday, December 04, 2003 4:09 PM To: Matt Alexander Subject: Re: Phoenix.pm: keep-alives with RobotUA Very generally speaking: Go under the hood and hack the feature on. I haven't used any of those modules more than a wee widdle bit so I don't know how it all fits together, but one object is likely creating instances of others and this argument is something that could be perpetuated. Sorry I don't have a better answer, short on time today =( Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under books =) -scott On 0, Matt Alexander wrote: > > I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' to > the constructor like I can with LWP::UserAgent or WWW::Mechanize. Does > anyone have a suggestion for how to enable keep-alives with RobotUA? > Thanks, > ~M > P.S. The new O'Reilly book "Spidering Hacks" is incredible. Definitely > check it out. From billn at billn.net Thu Dec 4 19:18:31 2003 From: billn at billn.net (Bill Nash) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA In-Reply-To: <6365F4E07E769F4C89826D1DE90D80370ABEAC@CCBill-Exchange.exchange.cwie.net> Message-ID: Think I spotted your problem: X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 }=) - billn On Thu, 4 Dec 2003, Jacob Powers wrote: > Opps that didn't come out as pretty as it went in.... oh well. > > Jacob Powers > -----Original Message----- > From: Jacob Powers > Sent: Thursday, December 04, 2003 5:15 PM > To: phoenix-pm-list@happyfunball.pm.org > Subject: RE: Phoenix.pm: keep-alives with RobotUA > > As Scott said you will need to break into RobotUA to add that. RobotUA > in its default form is very weak and ridden with "friendly" rules. Here > are a few things I have done to it. > > 1. Remove all robot rules. Just comment out the use of > Robot Rules. > 2. Remove all wait commands. In order to be more website > friendly it has a sleep(1), sometimes up to a minute, > inbetween each request. Very annoying if you are trying to pull > down many pages/sites quickly. Just set use_sleep > to 0 and delay to 0. > 3. Set it up to use MySQL for the seen/used URLs instead of > using a hash (this gets really big really fast.) What I > do is in the addUrl function, in RobotUA, I comment out the > part where it adds/makes a hash and instead I MD5 the URL and > put it in a table that has nothing but a char(32) field. > This speeds the Robot up a lot. You also have to add the > where it checks the Url, once again in the addUrl function > to read from the DB instead of the hash. > > > Those are just my personal tweaks to the RobotUA, or as I have renamed > it RobotDB. Make sure you do ample testing in various scenarios with > your hooks in place. > > Jacob Powers > > -----Original Message----- > From: Scott Walters [mailto:scott@illogics.org] > Sent: Thursday, December 04, 2003 4:09 PM > To: Matt Alexander > Subject: Re: Phoenix.pm: keep-alives with RobotUA > > Very generally speaking: > > Go under the hood and hack the feature on. > > I haven't used any of those modules more than a wee widdle bit > so I don't know how it all fits together, but one object is > likely creating instances of others and this argument is something > that could be perpetuated. > > Sorry I don't have a better answer, short on time today =( > > Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under > books =) > > -scott > > > On 0, Matt Alexander wrote: > > > > I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' > to > > the constructor like I can with LWP::UserAgent or WWW::Mechanize. > Does > > anyone have a suggestion for how to enable keep-alives with RobotUA? > > Thanks, > > ~M > > P.S. The new O'Reilly book "Spidering Hacks" is incredible. > Definitely > > check it out. > From scott at illogics.org Thu Dec 4 20:27:02 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs Message-ID: <20031205022702.GA6089@illogics.org> Hi folks, We're here at PPM and there was a disucssion of how to get debug what is locking up. I suggested this: $SIG{INT} = sub { require Carp; Carp::croak(); }; Then when it locks up, just kill -INT it. >From http://perldesignpatterns.com/?DebuggingPattern. Kind of a little FAQ for the #perlhelp newbies mostly ;) -scott From friedman at highwire.stanford.edu Fri Dec 5 00:43:16 2003 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA In-Reply-To: <6365F4E07E769F4C89826D1DE90D803702A84F@CCBill-Exchange.exchange.cwie.net> References: <6365F4E07E769F4C89826D1DE90D803702A84F@CCBill-Exchange.exchange.cwie.net> Message-ID: <4E343734-26EE-11D8-8689-000A956CAA2C@highwire.stanford.edu> Jacob, It's great that you have been able to modify RobotUA to meet your needs. I'd like to speak up, though, in favor of robot rules and request delays. I work for HighWire Press. We run almost 400 high-traffic websites for medical and scientific journals, such as the New England Journal of Medicine, Science Magazine, and the Oxford English Dictionary. We have serious hardware, software, and bandwidth for these sites. And yet, one robot that ignores our robot rules and requests pages faster than 1 per second could bring down an entire site. Even very popular sites such as these generally only handle a dozen requests per second at their highest traffic peaks. More than that can saturate our net connections, overwhelm the application server, and in the worst cases, crash the machine. Now, we've taken steps to avoid such behavior. If you hit one of our sites that fast, we'll block you from getting any pages within a second. But smaller sites don't always have the ability to do that. By requesting tons of pages as fast as possible, you can end up costing the provider money (for "extra" bandwidth charges) or even bring down the site as effectively as any Denial of Service attack. So I urge caution when removing the safeguards that are built into robot spidering software. Please don't overwhelm a site, just because you want a local copy of it. You can usually set the delay to 1 second and it won't take *that* much longer and you don't keep others from using the site at the same time. My $.02, -- Mike On Dec 4, 2003, at 5:14 PM, Jacob Powers wrote: > As Scott said you will need to break into RobotUA to add that. RobotUA > in its default form is very weak and ridden with "friendly" rules. Here > are a few things I have done to it. > > 1. Remove all robot rules. Just comment out the use of > Robot Rules. > 2. Remove all wait commands. In order to be more website > friendly it has a sleep(1), sometimes up to a minute, > inbetween each request. Very annoying if you are trying to pull > down many pages/sites quickly. Just set use_sleep > to 0 and delay to 0. > 3. Set it up to use MySQL for the seen/used URLs instead of > using a hash (this gets really big really fast.) What I > do is in the addUrl function, in RobotUA, I comment out the > part where it adds/makes a hash and instead I MD5 the URL and > put it in a table that has nothing but a char(32) field. > This speeds the Robot up a lot. You also have to add the > where it checks the Url, once again in the addUrl function > to read from the DB instead of the hash. > > > Those are just my personal tweaks to the RobotUA, or as I have renamed > it RobotDB. Make sure you do ample testing in various scenarios with > your hooks in place. > > Jacob Powers > > -----Original Message----- > From: Scott Walters [mailto:scott@illogics.org] > Sent: Thursday, December 04, 2003 4:09 PM > To: Matt Alexander > Subject: Re: Phoenix.pm: keep-alives with RobotUA > > Very generally speaking: > > Go under the hood and hack the feature on. > > I haven't used any of those modules more than a wee widdle bit > so I don't know how it all fits together, but one object is > likely creating instances of others and this argument is something > that could be perpetuated. > > Sorry I don't have a better answer, short on time today =( > > Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under > books =) > > -scott > > > On 0, Matt Alexander wrote: >> >> I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' > to >> the constructor like I can with LWP::UserAgent or WWW::Mechanize. > Does >> anyone have a suggestion for how to enable keep-alives with RobotUA? >> Thanks, >> ~M >> P.S. The new O'Reilly book "Spidering Hacks" is incredible. > Definitely >> check it out. >> --------------------------------------------------------------------- Michael Friedman HighWire Press, Stanford Southwest Phone: 480-456-0880 Tempe, Arizona FAX: 270-721-8034 --------------------------------------------------------------------- From phx-pm-list at grueslayer.com Fri Dec 5 08:04:59 2003 From: phx-pm-list at grueslayer.com (David A. Sinck) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA References: <6365F4E07E769F4C89826D1DE90D803702A84F@CCBill-Exchange.exchange.cwie.net> <4E343734-26EE-11D8-8689-000A956CAA2C@highwire.stanford.edu> Message-ID: <16336.37003.450037.700857@magnitude.righthandgraphics.com> \_ SMTP quoth Michael Friedman on 12/4/2003 23:43 as having spake thusly: \_ \_ [..] More than that can saturate our net connections, \_ overwhelm the application server, and in the worst cases, crash the \_ machine. Back in the day I heard that a pentium 90 could saturate a t1 if it chose. Just think if it was requesting the other side to do something rather than just saturate the network. I'm guessing you're not using a pentium 90 too. :-) \_ Now, we've taken steps to avoid such behavior. If you hit one of our \_ sites that fast, we'll block you from getting any pages within a \_ second. This sounds like a juicy bit, if you care to illucidate? David From jpowers at ccbill.com Fri Dec 5 11:16:48 2003 From: jpowers at ccbill.com (Jacob Powers) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <6365F4E07E769F4C89826D1DE90D803702A850@CCBill-Exchange.exchange.cwie.net> Mike, I agree with you whole heartly about not D.O.S.ing a site or having a malicious spyder hitting any site you don't own or have permission to. However, leaving the delays and waits in does create a *big* bottleneck in the software (upwards of 100 times greater). You can do additional tweaks to your spyder so it does not DOS a site. Once for example is your type of traversal, breadth or depth. Depending on if you are hitting many sites or just one you can also spread out the load (hit one URL with a request then hit a different URL on the next request, and so on). The default parameters are too friendly in my opinion. If I recall correctly the RobotUA only allows 10 requests, with a second delay in between each, and then requires you to sleep for a minute. That's excessive if you ask me. I use the RobotDB to spyder my own sites daily to check for broken links or changed pages. It can handle a lot more then one hit per second (as most professional or well done sites can) so speed means more to me then being nice to my server. But I completely agree with Mike about being aware of what you are doing and who you may be affecting. As always, be courteous if the site is not yours. Oh ya and one last thing, if you just want a local copy of a site use wget instead of RobotUA, it will work much better for that. Jacob Powers -----Original Message----- From: Michael Friedman [mailto:friedman@highwire.stanford.edu] Sent: Thursday, December 04, 2003 11:43 PM To: phoenix-pm-list@happyfunball.pm.org Subject: Re: Phoenix.pm: keep-alives with RobotUA Jacob, It's great that you have been able to modify RobotUA to meet your needs. I'd like to speak up, though, in favor of robot rules and request delays. I work for HighWire Press. We run almost 400 high-traffic websites for medical and scientific journals, such as the New England Journal of Medicine, Science Magazine, and the Oxford English Dictionary. We have serious hardware, software, and bandwidth for these sites. And yet, one robot that ignores our robot rules and requests pages faster than 1 per second could bring down an entire site. Even very popular sites such as these generally only handle a dozen requests per second at their highest traffic peaks. More than that can saturate our net connections, overwhelm the application server, and in the worst cases, crash the machine. Now, we've taken steps to avoid such behavior. If you hit one of our sites that fast, we'll block you from getting any pages within a second. But smaller sites don't always have the ability to do that. By requesting tons of pages as fast as possible, you can end up costing the provider money (for "extra" bandwidth charges) or even bring down the site as effectively as any Denial of Service attack. So I urge caution when removing the safeguards that are built into robot spidering software. Please don't overwhelm a site, just because you want a local copy of it. You can usually set the delay to 1 second and it won't take *that* much longer and you don't keep others from using the site at the same time. My $.02, -- Mike On Dec 4, 2003, at 5:14 PM, Jacob Powers wrote: > As Scott said you will need to break into RobotUA to add that. RobotUA > in its default form is very weak and ridden with "friendly" rules. Here > are a few things I have done to it. > > 1. Remove all robot rules. Just comment out the use of > Robot Rules. > 2. Remove all wait commands. In order to be more website > friendly it has a sleep(1), sometimes up to a minute, > inbetween each request. Very annoying if you are trying to pull > down many pages/sites quickly. Just set use_sleep > to 0 and delay to 0. > 3. Set it up to use MySQL for the seen/used URLs instead of > using a hash (this gets really big really fast.) What I > do is in the addUrl function, in RobotUA, I comment out the > part where it adds/makes a hash and instead I MD5 the URL and > put it in a table that has nothing but a char(32) field. > This speeds the Robot up a lot. You also have to add the > where it checks the Url, once again in the addUrl function > to read from the DB instead of the hash. > > > Those are just my personal tweaks to the RobotUA, or as I have renamed > it RobotDB. Make sure you do ample testing in various scenarios with > your hooks in place. > > Jacob Powers > > -----Original Message----- > From: Scott Walters [mailto:scott@illogics.org] > Sent: Thursday, December 04, 2003 4:09 PM > To: Matt Alexander > Subject: Re: Phoenix.pm: keep-alives with RobotUA > > Very generally speaking: > > Go under the hood and hack the feature on. > > I haven't used any of those modules more than a wee widdle bit > so I don't know how it all fits together, but one object is > likely creating instances of others and this argument is something > that could be perpetuated. > > Sorry I don't have a better answer, short on time today =( > > Re: Spidering Hacks, post a mini-review on http://phoenix.pm.org under > books =) > > -scott > > > On 0, Matt Alexander wrote: >> >> I'm using LWP::RobotUA, but I apparently can't pass 'keep_alive => 1' > to >> the constructor like I can with LWP::UserAgent or WWW::Mechanize. > Does >> anyone have a suggestion for how to enable keep-alives with RobotUA? >> Thanks, >> ~M >> P.S. The new O'Reilly book "Spidering Hacks" is incredible. > Definitely >> check it out. >> --------------------------------------------------------------------- Michael Friedman HighWire Press, Stanford Southwest Phone: 480-456-0880 Tempe, Arizona FAX: 270-721-8034 --------------------------------------------------------------------- From doug at phoenixinternet.com Fri Dec 5 11:21:11 2003 From: doug at phoenixinternet.com (doug@phoenixinternet.com) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs Message-ID: <.24.221.96.25.1070644871.squirrel@www.phoenixinternet.com> Scott Walters wrote: > Hi folks, > > We're here at PPM and there was a disucssion of how to get debug > what is locking up. I suggested this: > > $SIG{INT} = sub { require Carp; Carp::croak(); }; > > Then when it locks up, just kill -INT it. > >>From http://perldesignpatterns.com/?DebuggingPattern. > > Kind of a little FAQ for the #perlhelp newbies mostly ;) > > -scott > > > . > Scott, Thanks for this. It looks like I can't use it right now though. I think that the CGI is completing, but the Apache process is locking up. My reasoning is that I don't see any of the CGI processes, but I do get errors in the Apache error log: [Fri Dec 05 10:03:22 2003] [warn] child process 4239 still did not exit, sending a SIGTERM when I restart Apache. Also, I did get the CGI running off of the command line, as Mike suggested, and it runs just fine. Has anybody else seen this in Apache 2.x? Thanks. From friedman at highwire.stanford.edu Fri Dec 5 11:28:16 2003 From: friedman at highwire.stanford.edu (Michael Friedman) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA In-Reply-To: <16336.37003.450037.700857@magnitude.righthandgraphics.com> References: <6365F4E07E769F4C89826D1DE90D803702A84F@CCBill-Exchange.exchange.cwie.net> <4E343734-26EE-11D8-8689-000A956CAA2C@highwire.stanford.edu> <16336.37003.450037.700857@magnitude.righthandgraphics.com> Message-ID: <697D0980-2748-11D8-83DA-000A956CAA2C@highwire.stanford.edu> David, On Dec 5, 2003, at 7:04 AM, David A. Sinck wrote: > I'm guessing you're not using a pentium 90 too. :-) Heh. No, not Pentium 90s. More like 20 Sun E450s plus a multi-terabyte disk server. We've looked into Linux, though, and it could actually handle our traffic pretty well, if we had enough standard Dell (or similar) machines to host everything. > \_ Now, we've taken steps to avoid such behavior. If you hit one of our > \_ sites that fast, we'll block you from getting any pages within a > \_ second. > > This sounds like a juicy bit, if you care to illucidate? We wrote an Apache module (in C, but the prototype was in mod_perl) that keeps a log of traffic by user. If the module sees too many requests too fast from a single machine, single browser, or single proxy server, it aborts the Apache delivery process and sends back a warning page instead. Then it keeps a list of people who have been blocked and slowly lets the blocks expire over time. Since the module is high up in Apache's chain, it takes less effort to block someone than to serve their requested page, thus reducing the load on our machine and (since the warning page is small with no graphics) reducing the network traffic. You can still flood the server if you try hard enough with a DDOS, but it takes more effort than we have enemies, if you know what I mean. -- Mike --------------------------------------------------------------------- Michael Friedman HighWire Press, Stanford Southwest Phone: 480-456-0880 Tempe, Arizona FAX: 270-721-8034 --------------------------------------------------------------------- From doug at phoenixinternet.com Fri Dec 5 12:00:54 2003 From: doug at phoenixinternet.com (doug@phoenixinternet.com) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs In-Reply-To: <.24.221.96.25.1070644871.squirrel@www.phoenixinternet.com> References: <.24.221.96.25.1070644871.squirrel@www.phoenixinternet.com> Message-ID: <.24.221.96.25.1070647254.squirrel@www.phoenixinternet.com> Talking to myself again... > > Scott, > > Thanks for this. It looks like I can't use it right now though. I think > that the CGI is completing, but the Apache process is locking up. My > reasoning is that I don't see any of the CGI processes, but I do get > errors in the Apache error log: > > [Fri Dec 05 10:03:22 2003] [warn] child process 4239 still did not exit, > sending a SIGTERM > > when I restart Apache. Also, I did get the CGI running off of the command > line, as Mike suggested, and it runs just fine. Has anybody else seen > this in Apache 2.x? Thanks. > I forgot to mention that the CGI works fine on RedHat 7.3 with Apache 1.3 something. From scott at illogics.org Fri Dec 5 12:38:26 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs Message-ID: <20031205183826.GC6089@illogics.org> Oooooh, Apache 2. Apache 2 is threaded. You wouldn't beleive the volume of traffic that the problems related to this are generating... When this happens, er, when this message appears in the log, is process 4239 still in the process table? Can you see it? Is it zombie status, or ready to run, or running (is it a multiple CPU system?), or what? In other words, is Apache on crack, or is the process actually hanging? Are the headers correct when you run it from the command line? -scott On 0, doug@phoenixinternet.com wrote: > > Scott Walters wrote: > > Hi folks, > > > > We're here at PPM and there was a disucssion of how to get debug > > what is locking up. I suggested this: > > > > $SIG{INT} = sub { require Carp; Carp::croak(); }; > > > > Then when it locks up, just kill -INT it. > > > >>From http://perldesignpatterns.com/?DebuggingPattern. > > > > Kind of a little FAQ for the #perlhelp newbies mostly ;) > > > > -scott > > > > > > . > > > > Scott, > > Thanks for this. It looks like I can't use it right now though. I think > that the CGI is completing, but the Apache process is locking up. My > reasoning is that I don't see any of the CGI processes, but I do get > errors in the Apache error log: > > [Fri Dec 05 10:03:22 2003] [warn] child process 4239 still did not exit, > sending a SIGTERM > > when I restart Apache. Also, I did get the CGI running off of the command > line, as Mike suggested, and it runs just fine. Has anybody else seen > this in Apache 2.x? Thanks. From scott at illogics.org Fri Dec 5 12:45:43 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: keep-alives with RobotUA Message-ID: <20031205184543.GD6089@illogics.org> Not to mention you've stopped giving them links to try and follow if they are just a niave robot. Re: Sun E450s, that's on the NetBSD supported hardware list, so there is a good chance that Linux supports it also (or that support could be added). Given a choice between running Solaris on E450s and running Linux on a Dell, I'd have to go with the E450s. Consumer hardware sucks indescribeably bad. I keep trying to replace slowass.net's 180mhz PowerPC machine but each machine dies. So far it has outlived a 300mhz K6II, a 450mhz K6II, a 200mhz PII, and now I've got a 1.2ghz Athlon that suddenly decided that it is going to report its clock speed at 1.2ghz odd but run about as fast as a 300 on the bytebench. Consumer hardware is nothing but bonepile fodder. -scott On 0, Michael Friedman wrote: > > David, > > On Dec 5, 2003, at 7:04 AM, David A. Sinck wrote: > > I'm guessing you're not using a pentium 90 too. :-) > > Heh. No, not Pentium 90s. More like 20 Sun E450s plus a multi-terabyte > disk server. We've looked into Linux, though, and it could actually > handle our traffic pretty well, if we had enough standard Dell (or > similar) machines to host everything. > > > \_ Now, we've taken steps to avoid such behavior. If you hit one of our > > \_ sites that fast, we'll block you from getting any pages within a > > \_ second. > > > > This sounds like a juicy bit, if you care to illucidate? > > We wrote an Apache module (in C, but the prototype was in mod_perl) > that keeps a log of traffic by user. If the module sees too many > requests too fast from a single machine, single browser, or single > proxy server, it aborts the Apache delivery process and sends back a > warning page instead. Then it keeps a list of people who have been > blocked and slowly lets the blocks expire over time. > > Since the module is high up in Apache's chain, it takes less effort to > block someone than to serve their requested page, thus reducing the > load on our machine and (since the warning page is small with no > graphics) reducing the network traffic. > > You can still flood the server if you try hard enough with a DDOS, but > it takes more effort than we have enemies, if you know what I mean. > > -- Mike > --------------------------------------------------------------------- > Michael Friedman HighWire Press, Stanford Southwest > Phone: 480-456-0880 Tempe, Arizona > FAX: 270-721-8034 > --------------------------------------------------------------------- > From scott at illogics.org Fri Dec 5 13:26:05 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: typesafety Message-ID: <20031205192605.GE6089@illogics.org> Hi, When I presented this (and did a horrible job presenting - the quickie pre-presentation after Doug's was much better) I think it was on version 0.02. It is 0.004 now - a lot has changed. It understands a lot more code constructs, both specific code constructs and general things. Previous versions didn't consider context, but context bits in ops make it clear whether an op is leaving something on the stack and thus generating a value. This version is almost threatening to be useful. The main thing that I consider to be missing is typing of hash keys individually, which is needed to create objects without using Object::Lexical (though I don't know why anyone would ever want to do that). O::L makes instance data merely lexical data so reference to other objects and such typed data is kept in scalars. The common way to write objects is to subscript $_[0] as a hash using fixed keys, where $self = shift; $self->{foo} is one field, $self->{bar} is another, and so on. Well, typesafety doesn't understand that these are seperate from each other and type them seperately right now. But it should. So, I'm mailing the list looking for feedback - what else do I need to do to make this really usable? >From working on it for too long, I've developed developer myopia, where I'm psychologically unable to see shortcomings of my own work because each time I have in the past I've been punished by having to do work. You could say that developers like myself become conditioned against seeing flaws in our own work. If anyone on this list knows Perl 5's VM well, I wouldn't mind a code review either =) http://search.cpan.org/~swalters/typesafety-0.04/typesafety.pm Thanks, -scott From doug at phoenixinternet.com Fri Dec 5 14:26:08 2003 From: doug at phoenixinternet.com (doug@phoenixinternet.com) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs In-Reply-To: <20031205183826.GC6089@illogics.org> References: <20031205183826.GC6089@illogics.org> Message-ID: <.24.221.96.25.1070655968.squirrel@www.phoenixinternet.com> > Oooooh, Apache 2. Apache 2 is threaded. You wouldn't beleive the volume > of traffic that the problems related to this are generating... > > When this happens, er, when this message appears in the log, is > process 4239 still in the process table? Can you see it? Is > it zombie status, or ready to run, or running (is it a multiple CPU > system?), or what? In other words, is Apache on crack, or is the > process actually hanging? Are the headers correct when you run it > from the command line? > > -scott Apache actually ends up killing the processes, but I have noticed after testing this repeatedly, that there are an abnormaly high number of Apache processes running. So, I think that the processes are locking up, Apache spawns new ones, and then gives me warnings about the locked up ones when it kills them before restart. That's my theory anyway. From scott at illogics.org Fri Dec 5 14:55:33 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: stack backtraces from lockedup programs Message-ID: <20031205205533.GF6089@illogics.org> User mode thread packages and some kernel mode thread pages implement threads as processes. Even though memory is shared between the processes, it shows up once in the process table for each thread, so Apache is losing its own threads there. Report it, wait for the next version, but like Linux 2.4, Apache 2 is going to have some serious growing pains before it matures. (Perhaps someone will bite onto this flamebait since they seem to be ignoring the rest...) Perhaps you can run FastCGI or Apache 1.3 on another port, and set Apache 2 to proxy requests for URLs which map to CGIs or your CGI? -scott On 0, doug@phoenixinternet.com wrote: > > > Oooooh, Apache 2. Apache 2 is threaded. You wouldn't beleive the volume > > of traffic that the problems related to this are generating... > > > > When this happens, er, when this message appears in the log, is > > process 4239 still in the process table? Can you see it? Is > > it zombie status, or ready to run, or running (is it a multiple CPU > > system?), or what? In other words, is Apache on crack, or is the > > process actually hanging? Are the headers correct when you run it > > from the command line? > > > > -scott > > Apache actually ends up killing the processes, but I have noticed after > testing this repeatedly, that there are an abnormaly high number of Apache > processes running. So, I think that the processes are locking up, Apache > spawns new ones, and then gives me warnings about the locked up ones when > it kills them before restart. That's my theory anyway. > > From m at pdxlug.org Wed Dec 10 09:47:29 2003 From: m at pdxlug.org (Matt Alexander) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: PerlMagick fails to compile Message-ID: <56178.68.110.171.77.1071071249.squirrel@webmail.pdxlug.org> I have a RedHat 9 box with Perl 5.8.2 and I'm trying to install Image::Magick. When I run "perl Makefile.PL" I get the following errors. Any suggestions on getting this to work? perl Makefile.PL Checking if your kit is complete... Looks good Note (probably harmless): No library found for -llcms Note (probably harmless): No library found for -lexif Writing Makefile for Image::Magick make Magick.xs:76:24: magick/api.h: No such file or directory Magick.xs:145: parse error before "Image" Magick.xs:145: warning: no semicolon at end of struct or union Magick.xs:156: parse error before '}' token Magick.xs:169: parse error before "ImageInfo" Magick.xs:169: warning: no semicolon at end of struct or union Magick.xs:173: warning: data definition has no type or storage class Magick.xs:176: parse error before '*' token . . . Magick.xs:535: parse error before ')' token Magick.xs:536: dereferencing pointer to incomplete type Magick.xs:537: dereferencing pointer to incomplete type Magick.xs:537: `DrawInfo' undeclared (first use in this function) Magick.xs:537: parse error before ')' token Magick.xs:538: dereferencing pointer to incomplete type Magick.xs:538: `QuantizeInfo' undeclared (first use in this function) Magick.xs:538: parse error before ')' token Magick.xs:541: dereferencing pointer to incomplete type . . . Magick.xs:1255: `clip_mask' undeclared (first use in this function) Magick.xs:1269: `DirectClass' undeclared (first use in this function) Magick.xs:1281: `pixel' undeclared (first use in this function) Magick.xs:1288: `Quantum' undeclared (first use in this function) Magick.xs:1289: `MaxRGB' undeclared (first use in this function) Magick.xs:1308: `ColorspaceType' undeclared (first use in this function) Magick.xs:1308: parse error before "sp" . . . Magick.xs:8148: dereferencing pointer to incomplete type Magick.xs:8155: dereferencing pointer to incomplete type Magick.xs:8158: dereferencing pointer to incomplete type Magick.xs:8161: dereferencing pointer to incomplete type Magick.xs:8164: dereferencing pointer to incomplete type make: *** [Magick.o] Error 1 From phx-pm-list at grueslayer.com Wed Dec 10 10:12:06 2003 From: phx-pm-list at grueslayer.com (David A. Sinck) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: PerlMagick fails to compile References: <56178.68.110.171.77.1071071249.squirrel@webmail.pdxlug.org> Message-ID: <16343.17878.785883.224457@magnitude.righthandgraphics.com> \_ SMTP quoth Matt Alexander on 12/10/2003 07:47 as having spake thusly: \_ \_ I have a RedHat 9 box with Perl 5.8.2 and I'm trying to install \_ Image::Magick. When I run "perl Makefile.PL" I get the following errors. \_ Any suggestions on getting this to work? Is ImageMagick installed? Is ImageMagick-devel installed? Take apart ImageMagick-perl distributed with rh9 and see how they build? David \_ \_ perl Makefile.PL \_ Checking if your kit is complete... \_ Looks good \_ Note (probably harmless): No library found for -llcms \_ Note (probably harmless): No library found for -lexif \_ Writing Makefile for Image::Magick \_ \_ make \_ Magick.xs:76:24: magick/api.h: No such file or directory From m at pdxlug.org Wed Dec 10 10:58:12 2003 From: m at pdxlug.org (Matt Alexander) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: PerlMagick fails to compile In-Reply-To: <16343.17878.785883.224457@magnitude.righthandgraphics.com> References: <56178.68.110.171.77.1071071249.squirrel@webmail.pdxlug.org> <16343.17878.785883.224457@magnitude.righthandgraphics.com> Message-ID: <34903.68.110.171.77.1071075492.squirrel@webmail.pdxlug.org> David A. Sinck said: > > > \_ SMTP quoth Matt Alexander on 12/10/2003 07:47 as having spake thusly: > \_ > \_ I have a RedHat 9 box with Perl 5.8.2 and I'm trying to install > \_ Image::Magick. When I run "perl Makefile.PL" I get the following > errors. > \_ Any suggestions on getting this to work? > > > Is ImageMagick installed? > Is ImageMagick-devel installed? Yes and yes. > Take apart ImageMagick-perl distributed with rh9 and see how they > build? I got it working by installing Perl from source in my home directory, then building ImageMagick from source and pointing it to my new Perl binary. I also had to install bzip2-devel. The "make test" for PerlMagick returned lots of errors regarding "undefined symbol: XDPSPixelsPerPoint", which I'll investigate further, but for now I'm able to accomplish what I wanted with Image::Magick in my Perl scripts. Thanks, ~M From scott at illogics.org Sat Dec 13 10:31:37 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: [perlbug-followup@perl.org: [perl #24652] tieing stashes] Message-ID: <20031213163137.GK641@illogics.org> Some times people submit really interesting bugs that trigger discussion of new features. No one wants to add entirely new features to Perl 5 but the Perl 5 Porters are always eager to make existing things work together in more ways. There is serious discussion right now of having stashes (symbol tables) be tieable so you can programaticly handle attempts to define subroutines, local variables, our variables, or assign to local or our variables. Heh. -scott ----- Forwarded message from "kstar @ verizon. net Tassilo v.Parseval (via RT)" ----- Originator: tassilo.parseval@post.rwth-aachen.de post: Archive: Received: from onion.perl.org (onion.develooper.com [63.251.223.166]) (qmail 43961 invoked by uid 1005); 13 Dec 2003 05:25:39 -0000 (qmail 43943 invoked by uid 76); 13 Dec 2003 05:25:39 -0000 Subject: [perl #24652] tieing stashes Spam-Is-Higher-Than: 5 Date: Sat, 13 Dec 2003 00:24:58 -0500 Fri Dec 12 14:51:09 2003 12 Dec 2003 19:50:04 -0000 To: mailing list perl5-porters@perl.org perl5-porters@perl.org perl5-porters@perl.org bugs-bitbucket@netlabs.develooper.com perl5-porters@perl.org perl5-porters@perl.org From: kstar@verizon.net "Tassilo v.Parseval (via RT)" # New Ticket Created by Tassilo v. Parseval # Please include the string: [perl #24652] # in the subject line of all future correspondence about this issue. # This is a bug report for perl from tassilo.parseval@post.rwth-aachen.de, generated with the help of perlbug 1.34 running under perl v5.8.2. ----------------------------------------------------------------- [Please enter your report here] I noticed that perl wont stop you from tieing a stash: ethan@ethan:~$ perl5.8.2 -MTie::Hash -MPOSIX=_exit -e 'tie %:: => "Tie::StdHash"; _exit(1)' ethan@ethan:~$ When a proper shutdown via END happens, a seemingly unrelated message appears: ethan@ethan:~$ perl5.8.2 -MTie::Hash -e 'tie %:: => "Tie::StdHash"' Can't upgrade that kind of scalar during global destruction. ethan@ethan:~$ Should these ties make perl croak when they happen? I can't yet see what they could be used for especially since various perl versions behave very differently here. Putting the tie() into a BEGIN block is a particularly interesting experiment since functions defined in the thusly tied namespace can't be hooked into the package any longer: ethan@ethan:~$ perl5.8.2 -MTie::Hash BEGIN { tie %:: => "Tie::StdHash"; } sub test {1} Can't upgrade that kind of scalar at - line 2. ^D Can't upgrade that kind of scalar during global destruction. Also not bad: ethan@ethan:~$ perl5.8.2 -MTie::Hash BEGIN { tie %:: => "Tie::StdHash"; } sub test; Runaway prototype at - line 2. ^D Can't upgrade that kind of scalar during global destruction. Tassilo [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl v5.8.2: Configured by root at Fri Nov 14 07:29:04 UTC 2003. Summary of my perl5 (revision 5.0 version 8 subversion 2) configuration: Platform: osname=linux, osvers=2.4.21, archname=i686-linux uname='linux ethan 2.4.21 #3 tue nov 11 09:22:00 utc 2003 i686 unknown ' config_args='' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O3 -march=athlon -fomit-frame-pointer -foptimize-sibling-calls -finline-functions -fstrength-reduce -frerun-loop-opt -fexpensive-optimizations -fprefetch-loop-arrays -fmove-all-movables', cppflags='-fno-strict-aliasing' ccversion='', gccversion='3.2', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=/lib/libc-2.2.5.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.2.5' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic' cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib' Locally applied patches: --- @INC for perl v5.8.2: /usr/opt/perl5.8.2/lib/5.8.2/i686-linux /usr/opt/perl5.8.2/lib/5.8.2 /usr/opt/perl5.8.2/lib/site_perl/5.8.2/i686-linux /usr/opt/perl5.8.2/lib/site_perl/5.8.2 /usr/opt/perl5.8.2/lib/site_perl . --- Environment for perl v5.8.2: HOME=/home/ethan LANG=C LANGUAGE (unset) LC_CTYPE=de_DE@euro LD_LIBRARY_PATH=:/usr/local/j2sdk1.4.0/jre/lib/i386:/usr/local/pwlib/lib:/usr/local/lib LOGDIR (unset) PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/local/j2re1.4.1/bin:/usr/local/mpich-1.2.5/ch_p4/bin PERLDOC_PAGER=/bin/less -isR PERL_BADLANG (unset) SHELL=/bin/bash ----- End forwarded message ----- From scott at illogics.org Mon Dec 22 02:03:26 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: [andreas.koenig@anima.de: Re: [perl #24704] oldest bug in current perl? (Regex replace looses chars)] Message-ID: <20031222080326.GH19546@illogics.org> Here's a regex bug that has existed since perl 4 and is in the latest version of 5 =) I *love* the perl-5-porters bug reports. I had no idea how buggy perl was until I started following this list. The average months brings a dozen incorrect results, half a dozen inconsistent behaviors, two portability/build issues, and a coredumper. -scott ----- Forwarded message from Andreas J Koenig ----- ID: <87ad5lffya.fsf_-_@franz.ak.mind.de> List: contact perl5-porters-help@perl.org; run by ezmlm Status: No, hits=-4.9 required=7.0 tests=CARRIAGE_RETURNS,IN_REP_TO,REFERENCES,RT_TAG,SPAM_PHRASE_00_01,USER_AGENT,USER_AGENT_GNUS_UA,X_AUTH_WARNING version=2.44 post: By: la.mx.develooper.com SMTPD: qpsmtpd/0.26, http://develooper.com/code/qpsmtpd/ Type: text/plain; charset=us-ascii Spam-Index: -6.7 Filter-Rule: None, default is delivery References: <20031221141124.7669.BQW10602@nifty.com> <20031221142305.766C.BQW10602@nifty.com> Archive: help: Agent: Gnus/5.1002 (Gnus v5.10.2) Emacs/21.3 (gnu/linux) Received: from onion.perl.org (onion.develooper.com [63.251.223.166]) (qmail 15213 invoked by uid 1005); 22 Dec 2003 07:51:52 -0000 (qmail 15194 invoked by uid 76); 22 Dec 2003 07:51:52 -0000 Warning: franz.ak.mind.de: k set sender to andreas.koenig@anima.de using -f Cc: perl5-porters@perl.org Subject: Re: [perl #24704] oldest bug in current perl? (Regex replace looses chars) unsubscribe: Spam-Is-Higher-Than: 5 Date: Mon, 22 Dec 2003 08:51:25 +0100 Version: 1.0 Precedence: bulk To: mailing list perl5-porters@perl.org perl5-porters@perl.org SADAHIRO Tomoyuki <20031221142305.766C.BQW10602@nifty.com> (SADAHIRO Tomoyuki's message of "Sun, 21 Dec 2003 14:30:39 +0900") From: Andreas J Koenig If I rewrite the test to not use chr() but something that existed in perl4, say $_="65x"; s/65/$_-65/e; $res1 = $_; $_="x65x"; s/x//; s/65/$_-65/e; $res2 = $_; print $res1 eq $res2 ? "ok\n" : "not ok res1[$res1]res2[$res2]\n"; then I see the same result in perl4 as in bleed: not ok res1[0x]res2[0] -- andreas ----- End forwarded message ----- From wlindley at wlindley.com Mon Dec 22 10:23:02 2003 From: wlindley at wlindley.com (Bill Lindley) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: oldest bug in current perl? (Regex replace looses chars) In-Reply-To: <20031222080326.GH19546@illogics.org> References: <20031222080326.GH19546@illogics.org> Message-ID: <3FE71A66.4090201@wlindley.com> Hmm... > $_="65x"; s/65/$_-65/e; $res1 = $_; > > Looks to me like this should change the "65" to "0" and produce "0x" > $_="x65x"; s/x//; s/65/$_-65/e; $res2 = $_; > > Looks to me like this should eliminate the 'x's, and produce "0" > print $res1 eq $res2 ? "ok\n" : "not ok res1[$res1]res2[$res2]\n"; > >then I see the same result in perl4 as in bleed: > > not ok res1[0x]res2[0] > > Why's it "not ok"? \\/ From scott at illogics.org Mon Dec 22 11:45:57 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: oldest bug in current perl? (Regex replace looses Message-ID: <20031222174557.GI19546@illogics.org> lightbright# perl $_="65x"; s/65/$_-65/e; $res1 = $_; $_="x65x"; s/x//; s/65/$_-65/e; $res2 = $_; print $res1 eq $res2 ? "ok\n" : "not ok res1[$res1]res2[$res2]\n"; not ok res1[0x]res2[0] That s/x/// should only remove one 'x'. There is no /g after it. And the bug (according to the original email) wasn't that it was global, but the internal machinery loses count of its position under some circumstances - hard to reproduce circumstances such as this. I think they had to tweak it slightly to get it to perform in Perl 4 but it is the same bug. -scott On 0, Bill Lindley wrote: > > Hmm... > > > $_="65x"; s/65/$_-65/e; $res1 = $_; > > > > > Looks to me like this should change the "65" to "0" and produce "0x" > > > $_="x65x"; s/x//; s/65/$_-65/e; $res2 = $_; > > > > > Looks to me like this should eliminate the 'x's, and produce "0" > > > print $res1 eq $res2 ? "ok\n" : "not ok res1[$res1]res2[$res2]\n"; > > > >then I see the same result in perl4 as in bleed: > > > > not ok res1[0x]res2[0] > > > > > Why's it "not ok"? > > \\/ > From wlindley at wlindley.com Mon Dec 22 15:03:09 2003 From: wlindley at wlindley.com (Bill Lindley) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: oldest bug in current perl? (Regex replace looses In-Reply-To: <20031222174557.GI19546@illogics.org> References: <20031222174557.GI19546@illogics.org> Message-ID: <3FE75C0D.3040002@wlindley.com> Of course, no 'g' ... hmm... shows what happens... $cup->drink(cream => 'lots') while ($coffee_count++ < 2) ; # before $code->consider(); From tran_fors at yahoo.com Mon Dec 22 19:36:23 2003 From: tran_fors at yahoo.com (Tran Forsythe) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: You can never tell with some people... Message-ID: <20031223013623.26342.qmail@web60708.mail.yahoo.com> http://www.malevole.com/mv/misc/killerquiz/ -Kurt ps: Say, inn't that you as #6 on one of your fuzzier days, Scott? __________________________________ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree From doug at phoenixinternet.com Tue Dec 23 11:13:35 2003 From: doug at phoenixinternet.com (doug@phoenixinternet.com) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: You can never tell with some people... In-Reply-To: <20031223013623.26342.qmail@web60708.mail.yahoo.com> References: <20031223013623.26342.qmail@web60708.mail.yahoo.com> Message-ID: <.24.221.96.25.1072199615.squirrel@www.phoenixinternet.com> > http://www.malevole.com/mv/misc/killerquiz/ > > -Kurt > > ps: Say, inn't that you as #6 on one of your fuzzier > days, Scott? Completely off topic... Hey Kurt! How was the cruise? From scott at illogics.org Tue Dec 23 14:39:26 2003 From: scott at illogics.org (Scott Walters) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: You can never tell with some people... Message-ID: <20031223203926.GM19546@illogics.org> I actually recognized some of those guys. Bwahaha! I had a SNOBOL fetish going for a while, and Manson and me where tight, yo, we were like 'dis! -scott On 0, Tran Forsythe wrote: > > http://www.malevole.com/mv/misc/killerquiz/ > > -Kurt > > ps: Say, inn't that you as #6 on one of your fuzzier > days, Scott? > > __________________________________ > Do you Yahoo!? > Protect your identity with Yahoo! Mail AddressGuard > http://antispam.yahoo.com/whatsnewfree From johngnub at cox.net Wed Dec 24 08:57:32 2003 From: johngnub at cox.net (johnb) Date: Thu Aug 5 00:17:11 2004 Subject: Phoenix.pm: You can never tell with some people... In-Reply-To: <20031223203926.GM19546@illogics.org> Message-ID: <804AFEB4-3621-11D8-A063-000A9585B970@cox.net> Just when it was safe to start reading the PM....Oh well... On Tuesday, December 23, 2003, at 01:39 PM, Scott Walters wrote: > I actually recognized some of those guys. Bwahaha! > I had a SNOBOL fetish going for a while, and Manson and me > where tight, yo, we were like 'dis! > > -scott > > On 0, Tran Forsythe wrote: >> >> http://www.malevole.com/mv/misc/killerquiz/ >> >> -Kurt >> >> ps: Say, inn't that you as #6 on one of your fuzzier >> days, Scott? >> >> __________________________________ >> Do you Yahoo!? >> Protect your identity with Yahoo! Mail AddressGuard >> http://antispam.yahoo.com/whatsnewfree >