[Qatar-pm] Batch testing websites for HTTP status codes

[Nigel Gourlay]

The Qtel filter returns a HTTP status code of "403 Forbidden" when it
receives a request for a blocked site:

  ng at laptop:~$ telnet taipai.pm.org 80
  Trying 208.69.32.139...
  Connected to taipai.pm.org.
  Escape character is '^]'.
  HEAD / HTTP/1.1
  Host: taipei.pm.org

  HTTP/1.1 403 Forbidden
  Cache-Control: no-cache
  Pragma: no-cache
  Content-Type: text/html; charset=utf-8
  Connection: Keep-Alive
  Content-Length: 515


  Connection closed by foreign host.

I wanted to determine the status codes returned for a list of URLs,
supplied one-per-line in a text file. The following script did the
job:

 #!/usr/bin/perl -w
 use warnings;
 use strict;
 use LWP::UserAgent;

 my $url_file = $ARGV[0];
 my $ua = LWP::UserAgent->new;
 $ua->timeout(30);
 $ua->env_proxy;

 open(URLS,"< $url_file") or die($!);
 while(my $url = <URLS>){
     my $result;
     chomp($url);
     $url =~ s/^(\S*).*?$/$1/;
     my $response = $ua->get($url);
     printf("%s\t%s\n",$url,$response->message());
 }
 close(URLS);