[San-Diego-pm] Photo download problem

[Chris Grau]

On Sun, Aug 22, 2010 at 04:37:25PM -0700, Joel Fentin wrote:
> Chris Grau wrote:
>>> http://fentin.com/cgi-bin/temp.pl
>> [snip]
>>
>>> Wikimedia permits the downloading of photos and hotlinking. Yet I
>>> can't get it to do anything with perl. The two URLs above (same photo
>>> in each case) work in the FF browser. The third example is from my own
>>> website as test. It downloads content.
>>
>> I seem to have the opposite results when running your test code.  I
>> receive an error attempting to download wikimedia's content (trimmed):
>>
>>     $ HEAD http://upload.wikimedia.org/wikipedia/commons/1/19/PacificSilverFir_7645.jpg
>>     403 Forbidden
>>     X-Squid-Error: ERR_ACCESS_DENIED 0
>>
>> They must be blocking LWP's user agent, because wget works fine.  I
>> didn't bother testing a modified user agent string in LWP.
>
> Are you saying that it works fine in the browser because the browser
> has a different user agent?

Yes.

> And if so, are you saying that I have to spoof a user agent to get the
> photos?

Again, yes.  This, for example, works:

    use LWP::UserAgent;

    my $ua = LWP::UserAgent->new;
    $ua->agent( 'Wget/1.10.2' );

    my $response = $ua->get(
        'http://upload.wikimedia.org/wikipedia/commons/1/19/PacificSilverFir_7645.jpg'
    );

    if ( $response->is_success ) {
        say $response->status_line;
        say $response->content_type;
    }

Resulting in:

    200 OK
    image/jpeg