Conditional http get for dynamic pages

[Scott Penrose]

On Saturday, Nov 23, 2002, at 16:13 Australia/Melbourne, David Dick 
wrote:

> G'day all,
> This is slightly off topic, but i figure there is some interest in web 
> stuff in the perl-mongers.  I've been reading the http rfc over the 
> last week and realising just how cool conditional gets are for dynamic 
> pages. It seems to reduce network traffic by a good amount.  Is anyone 
> else using them for dynamic pages?  Any experiences on this?
> Uru
> -dave

Are you talking about standard IMS request (If Mod Since). If so that 
has been a part of the standard for many years and built into most 
proxy and web servers. For example every time you make a request 
through SQUID it will do an IMS request to the up stream server to only 
get the response if has been modified.

However it is of course extremely difficult to do with dynamic pages. 
My preference is along the lines of this. If it is a dynamic page that 
changes only occasionally then you are best off generating the page. A 
good example of this would be the slashdot home page. Generate the page 
off line and let the web server deliver it. This has many advantages 
the most of which is that you don't have to write special code.

However, if it is a dynamic page which changes depending on the user 
logged in etc, then you are not going to get any advantage out of an 
IMS header response. There are a few reasons for this. A proxy cache 
generally does not cache these pages (it can do but is almost pointless 
so generally don't). Another is that things change in too complicated a 
way. Theoretically you should be able to get cookies and stuff with 
your IMS, but all this overhead generally slows things down quite a bit 
:-)

To reduce network traffic for dynamically generated pages I recommend 
the following strategy:
	- Generate the pages and put them somewhere sensible if they can be 
cached. If they can't be cached (eg: aways changing) then you gain 
nothing from IMS anyway.
	- Use a reasonable expiry time
	- GZIP the pages. If you use the right settings in apache they only 
get compressed once so it is reasonably efficient. You can even go 
further and compress yourself and then NOT check if the file is 
changed, just always send the current GZ file if GZIP is supported.

Scott
-- 
Scott Penrose
Open source developer
http://linux.dd.com.au/
scottp at dd.com.au

Dismaimer: Open sauce usually ends up never coming out (of the bottle).