LWP::UserAgent and referer?
John Evans
evansj at kilnar.com
Wed Nov 21 18:03:46 CST 2001
On Wed, 21 Nov 2001, Robert L. Harris wrote:
>
>
> I'm trying to use a perl script to mirror some patches. The maker wishes
> you to manually go and download each time, which gets a bit tedious.
>
> I've got it getting a list of patches, etc, but when I use "getstore"
> from LWP::Simple I get an error message:
>
Here's how I do it and an example:
Imagine that the page http://www.foo.com/patches.html has links to:
http://www.foo.com/files/patch011119.tar.gz
http://www.foo.com/files/patch011120.tar.gz
http://www.foo.com/files/patch011121.tar.gz
You want the tar.gz files:
Try this script. It was written quickly and not tested:
#!/usr/local/bin/perl
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use URI::Heuristic;
### These two lines make sure that the URL is properly formatted
my $raw_url = shift or die "usage: $0 [URL to Fetch]\n";
my $url = URI::Heuristic::uf_urlstr($raw_url);
### Build a new web client that will make our requests.
$client = LWP::UserAgent->new();
### Make our perl script look like IE 5.5 under Windows 98
$client->agent("Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)");
### Build the request for the file we want.
$request = HTTP::Request->new(GET => $url);
### Make it look like we came from their site
$request->referer("http://www.foo.com/patches.html");
$response = $client->request($request);
if ($response->is_error()) {
printf("%s\n", $response->status_line);
}
else {
my $file = $response->content();
}
--
John Evans
http://evansj.kilnar.com/
http://www.foo.com/files/patch011121.tar.gz
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS d- s++:- a- C+++>++++ ULSB++++$ P+++$ L++++$
E--- W++ N+ o? K? w O- M V PS+ !PE Y+ PGP t(--) 5-- X++(+++)
R+++ tv+ b+++(++++) DI+++ D++>+++ G+ e h--- r+++ y+++
------END GEEK CODE BLOCK------
More information about the Pikes-peak-pm
mailing list