LWP::UserAgent and referer?

Robert L. Harris Robert.L.Harris at rdlg.net
Mon Nov 26 11:42:47 CST 2001



Hmm, I did this one.  Looks nice.  Problem is now I'm getting a 404.  I put
a few prints in.  If I go to the index.html and select "save as" it downloads
fine.  If I go directly to the link I'm trying to get I get a 403, denied.  When
I run this segment of code I get a 404.  If I check the URL I'm accessing in
"$url" below against my save as, they're identicle.

Thoughts?



Thus spake John Evans (evansj at kilnar.com):

> On Wed, 21 Nov 2001, Robert L. Harris wrote:
> 
> >
> >
> > I'm trying to use a perl script to mirror some patches.  The maker wishes
> > you to manually go and download each time, which gets a bit tedious.
> >
> > I've got it getting a list of patches, etc, but when I use "getstore"
> > from LWP::Simple I get an error message:
> >
> 
> Here's how I do it and an example:
> 
> Imagine that the page http://www.foo.com/patches.html has links to:
> http://www.foo.com/files/patch011119.tar.gz
> http://www.foo.com/files/patch011120.tar.gz
> http://www.foo.com/files/patch011121.tar.gz
> 
> You want the tar.gz files:
> 
> Try this script. It was written quickly and not tested:
> 
> #!/usr/local/bin/perl
> 
> use LWP::UserAgent;
> use HTTP::Request;
> use HTTP::Response;
> use URI::Heuristic;
> 
> ### These two lines make sure that the URL is properly formatted
> my $raw_url = shift or die "usage: $0 [URL to Fetch]\n";
> my $url = URI::Heuristic::uf_urlstr($raw_url);
> 
> ### Build a new web client that will make our requests.
> $client = LWP::UserAgent->new();
> 
> ### Make our perl script look like IE 5.5 under Windows 98
> $client->agent("Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)");
> 
> ### Build the request for the file we want.
> $request = HTTP::Request->new(GET => $url);
> 
> ### Make it look like we came from their site
> $request->referer("http://www.foo.com/patches.html");
> 
> $response = $client->request($request);
> 
> if ($response->is_error()) {
>   printf("%s\n", $response->status_line);
> }
> else {
>   my $file = $response->content();
> }
> 
> 
> -- 
> John Evans
> http://evansj.kilnar.com/
> http://www.foo.com/files/patch011121.tar.gz
> 
> -----BEGIN GEEK CODE BLOCK-----
> Version: 3.1
> GCS d- s++:- a- C+++>++++ ULSB++++$ P+++$ L++++$
> E--- W++ N+ o? K? w O- M V PS+ !PE Y+ PGP t(--) 5-- X++(+++)
> R+++ tv+ b+++(++++) DI+++ D++>+++ G+ e h--- r+++ y+++
> ------END GEEK CODE BLOCK------
> 
> 
> 



:wq!
---------------------------------------------------------------------------
Robert L. Harris                |  Micros~1 :  
Senior System Engineer          |    For when quality, reliability 
  at RnD Consulting             |      and security just aren't
                                \_       that important!
DISCLAIMER:
      These are MY OPINIONS ALONE.  I speak for no-one else.
FYI:
 perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'




More information about the Pikes-peak-pm mailing list