LWP::UserAgent and referer?

John Evans evansj at kilnar.com
Wed Nov 21 18:03:46 CST 2001


On Wed, 21 Nov 2001, Robert L. Harris wrote:

>
>
> I'm trying to use a perl script to mirror some patches.  The maker wishes
> you to manually go and download each time, which gets a bit tedious.
>
> I've got it getting a list of patches, etc, but when I use "getstore"
> from LWP::Simple I get an error message:
>

Here's how I do it and an example:

Imagine that the page http://www.foo.com/patches.html has links to:
http://www.foo.com/files/patch011119.tar.gz
http://www.foo.com/files/patch011120.tar.gz
http://www.foo.com/files/patch011121.tar.gz

You want the tar.gz files:

Try this script. It was written quickly and not tested:

#!/usr/local/bin/perl

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use URI::Heuristic;

### These two lines make sure that the URL is properly formatted
my $raw_url = shift or die "usage: $0 [URL to Fetch]\n";
my $url = URI::Heuristic::uf_urlstr($raw_url);

### Build a new web client that will make our requests.
$client = LWP::UserAgent->new();

### Make our perl script look like IE 5.5 under Windows 98
$client->agent("Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)");

### Build the request for the file we want.
$request = HTTP::Request->new(GET => $url);

### Make it look like we came from their site
$request->referer("http://www.foo.com/patches.html");

$response = $client->request($request);

if ($response->is_error()) {
  printf("%s\n", $response->status_line);
}
else {
  my $file = $response->content();
}


-- 
John Evans
http://evansj.kilnar.com/
http://www.foo.com/files/patch011121.tar.gz

-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS d- s++:- a- C+++>++++ ULSB++++$ P+++$ L++++$
E--- W++ N+ o? K? w O- M V PS+ !PE Y+ PGP t(--) 5-- X++(+++)
R+++ tv+ b+++(++++) DI+++ D++>+++ G+ e h--- r+++ y+++
------END GEEK CODE BLOCK------







More information about the Pikes-peak-pm mailing list