From lists at insecurity.org Fri May 9 09:53:06 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:51 2004 Subject: [Jax.PM] [Admin Post] [NON-PERL] Off topic Message-ID: How is everyone? ??? _Sx____________________ ('> iudicium ferat //\ Have Computer - v_/_ Will Hack... \|/ ____ \|/ "@'/ .. \`@" /_| \__/ |_\ \__U_/ From lists at insecurity.org Wed May 14 07:49:30 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:51 2004 Subject: [Jax.PM] [Admin Post] Message-ID: <811FCA12-860A-11D7-89FE-0003930FDFC6@insecurity.org> Everyone welcome a New Member =) Tomas Karlsson! New to the Jacksonville area as well - so we all need to get together and take him "sand" bar hopping :) Not counting me or (Nate twice) we officially have 11 people who at least publicly admit to at least "seeing" something about Perl :) If anyone wants to manage their account, see - http://mail.pm.org/mailman/listinfo/jacksonville-pm -Bill- :] _Sx____________________________________________________ $_ = "Jacksonville Perl Monger"; while(/([Jacksonville Perl Monger])/g) { print join(" ", map { defined $_ ? $_ : "" } $`, $&, $', $+), "\n"; } From nate at campin.net Wed May 14 13:19:29 2003 From: nate at campin.net (Nate Campi) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] perl HTTP header insertion proxy Message-ID: <20030514181929.GA988@campin.net> I have a need for a proxy that sits between a cache acting as a HTTP accelerator (apache mod_proxy) and a backend SSI apache site that can add the Content-Length header. We already have the Expires header on the origin site and will still be lacking Last-Modified but I think we'll be ok without it. Does anyone know of such a beast, or something like it? Perhaps there's a module that could be easily adapted. I'd tend to shy away from a full-blown mod_perl instance between the other two layers, too big for a small need. It seems that you could bust out a simpler standalone perl script that's easier to debug and/or extend later. -- Nate Campi http://www.campin.net From lists at insecurity.org Wed May 14 13:38:29 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] perl HTTP header insertion proxy In-Reply-To: <20030514181929.GA988@campin.net> Message-ID: <41F601F0-863B-11D7-AEF9-0003930FDFC6@insecurity.org> On Wednesday, May 14, 2003, at 02:19 PM, Nate Campi wrote: > [a jax.PM member posting] > I have a need for a proxy that sits between a cache acting as a HTTP > accelerator (apache mod_proxy) and a backend SSI apache site that can > add the Content-Length header. We already have the Expires header on > the > origin site and will still be lacking Last-Modified but I think we'll > be > ok without it. On the same server? Which version of Apache? Why won't another mod_proxy instance help? > Does anyone know of such a beast, or something like it? Perhaps there's > a module that could be easily adapted. > > I'd tend to shy away from a full-blown mod_perl instance between the > other two layers, too big for a small need. It seems that you could > bust > out a simpler standalone perl script that's easier to debug and/or > extend later. You can set up a reverse mirroring proxy - which is what I am thinking you mean. See ProxyPassReverse (to stop the client from by-passing the first proxy) and ProxyRemote (to identify your new proxy to the caching proxy.) I doubt it is too much different than the caching server configs. Sort of like a double reverse. Maybe I am confused =) If you simply want to insert random headers into your HTTP stream, then see http://stein.cshl.org/~lstein/talks/WWW6/sniffer/ I have modified this to make up all sort of lies about HTTP data... -Sx- :) From jproctor at persons.marlboro.edu Wed May 14 15:31:39 2003 From: jproctor at persons.marlboro.edu (Jason Proctor) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] [Admin Post] In-Reply-To: <811FCA12-860A-11D7-89FE-0003930FDFC6@insecurity.org> References: <811FCA12-860A-11D7-89FE-0003930FDFC6@insecurity.org> Message-ID: > Everyone welcome a New Member =) Yay! Of course, all this public welcoming and stuff will probably scare him off. I wonder if that's what happened to the last batch we lost. :) j (yet another Jax.PM denizen who isn't actually in Jax) From lists at insecurity.org Wed May 14 15:53:03 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] [Admin Post] In-Reply-To: Message-ID: <0E58752B-864E-11D7-BBE4-0003930FDFC6@insecurity.org> On Wednesday, May 14, 2003, at 04:31 PM, Jason Proctor wrote: > Of course, all this public welcoming and stuff will probably scare him > off. I wonder if that's what happened to the last batch we lost. :) Nah, he's a self-proclaimed Webmaster/Perl geek "Extro-Dan-Air" --- nothing will scare him off =) That Internet thingy didn't... -Sx- From nate at campin.net Wed May 14 17:15:28 2003 From: nate at campin.net (Nate Campi) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] perl HTTP header insertion proxy In-Reply-To: <41F601F0-863B-11D7-AEF9-0003930FDFC6@insecurity.org> References: <20030514181929.GA988@campin.net> <41F601F0-863B-11D7-AEF9-0003930FDFC6@insecurity.org> Message-ID: <20030514221528.GB988@campin.net> On Wed, May 14, 2003 at 02:38:29PM -0400, WC -Sx- Jones wrote: > On Wednesday, May 14, 2003, at 02:19 PM, Nate Campi wrote: > >I have a need for a proxy that sits between a cache acting as a HTTP > >accelerator (apache mod_proxy) and a backend SSI apache site that can > >add the Content-Length header. We already have the Expires header on > >the origin site and will still be lacking Last-Modified but I think > >we'll be ok without it. > > On the same server? Which version of Apache? Version of apache doesn't matter, I've tried 2.0 with mod_cache and mod_proxy and 1.3 mod_proxy and neither caches the site properly, since SSI doesn't send the headers needed to make content cacheable. 2.0 has directives to cache even without Wheter or not it's on the same server doesn't really matter either, though I'd run at least two and load balance them. > Why won't another mod_proxy instance help? Because none of the caches add the headers the later caches need if they are to cache. > >Does anyone know of such a beast, or something like it? Perhaps there's > >a module that could be easily adapted. > > > >I'd tend to shy away from a full-blown mod_perl instance between the > >other two layers, too big for a small need. It seems that you could > >bust > >out a simpler standalone perl script that's easier to debug and/or > >extend later. > > You can set up a reverse mirroring proxy - which is what I am thinking > you mean. > > See ProxyPassReverse (to stop the client from by-passing the first > proxy) and ProxyRemote (to identify your new proxy to the caching > proxy.) I'm already doing that, but the content is never cached. > I doubt it is too much different than the caching server configs. Sort > of like a double reverse. Maybe I am confused =) It's just that I only gave a very short description. See the mod_perl list for my description there: > If you simply want to insert random headers into your HTTP stream, then > see > > http://stein.cshl.org/~lstein/talks/WWW6/sniffer/ Interesting. -- Nate Campi http://www.campin.net From webmaster at gamehippo.com Wed May 14 17:53:02 2003 From: webmaster at gamehippo.com (Tomas Karlsson) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] [Admin Post] In-Reply-To: <0E58752B-864E-11D7-BBE4-0003930FDFC6@insecurity.org> References: Message-ID: <3EC2908E.9745.89202@localhost> Hey all! Thanks for the welcome notes. I am here, just thinking about something intelligent to say to introduce myself. ;-) Tomas From lists at insecurity.org Wed May 14 23:16:34 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] Re: perl HTTP header insertion proxy In-Reply-To: <20030514221528.GB988@campin.net> Message-ID: <03706764-868C-11D7-AE5E-0003930FDFC6@insecurity.org> OK, now I am prolly totally brain dead =) 'Expires' is a function of the proxy'ing server itself (mod_proxy.) How about fixing SSI server to use: fsize - prints the size of the specified file; and flastmod - prints the last modification date. It would mean coding/touching the SSI pages again. Is the SSI server truly only SSI or is it a CGI or mod_perl environment? I still think yet another proxy to do HTTP Header insertion is unnecessary. Besides, how would THAT program know what information to insert? The CachedLastModifiedFactor should determine if the cache pages should be updated... Are you using the Apache Server Admin Handbook written by Kabir? It pretty much describes what you stated in the first posting; but it never mentions that SSI output is unusable by a proxy. I would think that would be a major issue - one I feel he would have noted - unless the caching server doesn't recognize the proxy server to start with. In other words - is the SSI server a proxy server too? ??? -Sx- :] From nate at campin.net Thu May 15 01:10:05 2003 From: nate at campin.net (Nate Campi) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] Re: perl HTTP header insertion proxy In-Reply-To: <03706764-868C-11D7-AE5E-0003930FDFC6@insecurity.org> References: <20030514221528.GB988@campin.net> <03706764-868C-11D7-AE5E-0003930FDFC6@insecurity.org> Message-ID: <20030515061005.GC988@campin.net> On Thu, May 15, 2003 at 12:16:34AM -0400, WC -Sx- Jones wrote: > [a jax.PM member posting] > OK, now I am prolly totally brain dead =) > > 'Expires' is a function of the proxy'ing server itself (mod_proxy.) > How about fixing SSI server to use: Only the origin can and should dictate when content expires. > Are you using the Apache Server Admin Handbook written by Kabir? It > pretty much describes what you stated in the first posting; but it > never mentions that SSI output is unusable by a proxy. I would think > that would be a major issue - one I feel he would have noted - unless > the caching server doesn't recognize the proxy server to start with. > In other words - is the SSI server a proxy server too? If this handbook leaves this out, I question it's quality. As for your question, no the SSI server generates and serves content and is not any kind of proxy. The mod_perl site talks in many places about headers needed for caching: http://perl.apache.org/docs/general/correct_headers/correct_headers.html#Why_Headers SSI can not and will not set the Last-Modified since many pages go into making one URL, it also cannot set Content-Length since it can't stat a single file. There are various ways to get SSI-based site to cache, a good one being the XBitHack, but we generate pages with third partry software that cannot set the executable bit. See: http://www.bowiesnyder.com/writings/caching_shtml.htm I cannot crawl the filesystem to add the executable bit since I have at least a million files in my docroot. Apache 2.0 should be able to work with directives to ignore missing content-length headers, but long story short it doesn't. Apache 1.3 mod_proxy needs headers that aren't there and that the origin server(s) cannot feasibly add. A proxy is the simplest link to add to this chain. Anything else either adds too much complexity or simply isn't worth the effort. A proxy is simple and easy to simply turn off if you have trouble with it. I know you know Apache well Bill, but I've done my homework. -- Nate Campi http://www.campin.net From lists at insecurity.org Thu May 15 01:35:15 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] Re: perl HTTP header insertion proxy In-Reply-To: <20030515061005.GC988@campin.net> Message-ID: <6386E21B-869F-11D7-9AC1-0003930FDFC6@insecurity.org> On Thursday, May 15, 2003, at 02:10 AM, Nate Campi wrote: > > I cannot crawl the filesystem to add the executable bit since I have at > least a million files in my docroot. Owie? :) How the heck are 'dem inodes doin' ? Are you running out of file handles on lookups? Is that a SUN box or Linux? We are waaaaayyyyyy off Perl here =) Seriously, you might could schedule a sleeper to crawl the file system during system idle prolly. At 2:30AM I prolly need to sleep now... > I know you know Apache well Bill, but I've done my homework. I can see that you have indeed :) You make an old ApacheCon speaker proud =) The Kabir book is old and in fact out of print; being published in '99 it had to have been written before '98 -- and likely it is time I moved on to a more recent edition (alas none is available nor planned that I can tell); but I hesitate to adopt any other because I see so many books about Apache leave out so much. Kabir's book, I feel, is still the standard for covering nuts-and-bolts of Apache. What I should do is see if the publisher/author will allow me access to rewrite and update that edition. I feel a "group project" emerging - what do you think Nate? Anyways, one last stab at the original issue - but from an Apache 2.0 stand-point: Apache 2.0 accepts and handles other protocols besides HTTP - as you prolly know - there is a SMTP module already out to show off this feature. Have you thought about modifying the 2.0 stuff to make it do what you want? Maybe there is another 'feature' or module interfering with the desired operation you are expecting? I'll get a newer Apache 2.x (there are CAN and CERT advisories out regarding other editions) and will look at it further :) _Sx____________________ ('> iudicium ferat //\ Have Computer - v_/_ Will Hack... \|/ ____ \|/ "@'/ .. \`@" /_| \__/ |_\ \__U_/ From nate at campin.net Thu May 15 01:49:46 2003 From: nate at campin.net (Nate Campi) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] Re: perl HTTP header insertion proxy In-Reply-To: <6386E21B-869F-11D7-9AC1-0003930FDFC6@insecurity.org> References: <20030515061005.GC988@campin.net> <6386E21B-869F-11D7-9AC1-0003930FDFC6@insecurity.org> Message-ID: <20030515064945.GF988@campin.net> On Thu, May 15, 2003 at 02:35:15AM -0400, WC -Sx- Jones wrote: > On Thursday, May 15, 2003, at 02:10 AM, Nate Campi wrote: > > > >I cannot crawl the filesystem to add the executable bit since I have at > >least a million files in my docroot. > > Owie? :) How the heck are 'dem inodes doin' ? Are you running out > of file handles on lookups? Is that a SUN box or Linux? We are > waaaaayyyyyy off Perl here =) This is Sun, inodes tend not to be a problem, big disks and cron jobs to clean scrap files seem to take care of things. > Seriously, you might could schedule a sleeper to crawl the file system > during system idle prolly. At 2:30AM I prolly need to sleep now... Yes, but I'd want the freshly generated files to have the correct perms immediately. They are most of what's being accessed (news site) so waiting until later would defeat the purpose. > I feel a "group project" emerging - what do you think Nate? Hmm, I'd be happy to write up my recent experiences with caching. Between myself and the main developer working on it we could probably share some good stuff. > Apache 2.0 accepts and handles other protocols besides HTTP - as you > prolly know - there is a SMTP module already out to show off this > feature. Have you thought about modifying the 2.0 stuff to make it do > what you want? Maybe there is another 'feature' or module interfering > with the desired operation you are expecting? I'll get a newer Apache > 2.x (there are CAN and CERT advisories out regarding other editions) > and will look at it further :) This is a good idea. I had our resident apache hacker look at what it would take to modify 1.3 (since it's a proven code base) and it's not easily doable. I pointed out to him that 2.0 has the code to deal with unknown content-lengths in the mod_disk_cache module and he said that's promising. The main problem there is that this would get really low priority over stuff that actually makes us money (increasing scalability and efficiency doesn't get dollars behind you where I work, they want to see immediate $$). If I could bust out some way with a proxy I could do it all on my own. Anyhoo, I don't think I mentioned perl once in this post :( I'm off fishun tonight until Monday, have a good weekend. -- Nate Campi http://www.campin.net From lists at insecurity.org Fri May 16 09:34:14 2003 From: lists at insecurity.org (WC -Sx- Jones) Date: Mon Aug 2 21:30:52 2004 Subject: [Jax.PM] [NON-PERL] Apache 2.0 Proxy Follow-up Message-ID: <775AE63C-87AB-11D7-B1C5-0003930FDFC6@insecurity.org> Nate: How did you build Apache 2? I used: ./configure --verbose --with-layout=Apache --prefix=/usr/local/apache \ --enable-module=most --enable-shared=max --enable-module=so \ --enable-cache \ --enable-disk-cache \ --enable-mem-cache \ --enable-example \ --enable-deflate \ --enable-logio \ --enable-mime-magic \ --enable-cern-meta \ --enable-expires \ --enable-headers \ --enable-usertrack \ --enable-unique-id \ --enable-proxy \ --enable-proxy-connect \ --enable-proxy-ftp \ --enable-proxy-http \ --enable-static-htpasswd \ --enable-static-htdigest \ --enable-static-rotatelogs \ --enable-static-logresolve \ --enable-static-checkgid \ --enable-http \ --enable-dav \ --enable-info \ --enable-cgi \ --enable-dav-fs \ --enable-vhost-alias \ --enable-speling \ --disable-userdir \ --enable-rewrite ??? _Sx____________________ ('> iudicium ferat //\ Have Computer - v_/_ Will Hack... \|/ ____ \|/ "@'/ .. \`@" /_| \__/ |_\ \__U_/