malformed url versus broken link in perl - perl

Question

I'm looking to distinguish between typos in urls and actual broken links. For example:
typo:
www.google/com
broken link:
www.thislinkpointstonothing.org
I would like my application to recognize the first URL as misformed and to indicate the second URL to returns a 404 not found when queried. Is there a perl module or a regex I can use in order to make this distinction?
I'm trying to distinguish between bad links due to typos or - if the link follows the RFC - if the bad link is simply due to the page no longer existing.

Answer 1

You are talking about two different types of "queries": DNS lookups and HTTP requests. HTTP requests make use of DNS lookups - but not always. for example a server can be located on a local network and you can request pages from it using IP address and name information from an /etc/hosts file. Link names may not always contain a host name portion since they can be relative (this is often a good practice for making a website easily able to be moved behind a reverse proxy or if the host name changes.
Taking account of that distinction the gist of your question - whether you can check a link for correctness of the URI versus a truly missing a a page (404) is sound - but a DNS query for google/com should snot succeed. Is your application being redirected by a proxy to a search page?
A rough approximate answer follows - this probably won't be much use as it is but you'll get the idea. For more useful approaches in perl, you might want to check if some of the more featureful perl frameworks (Catalyst Mojo) have methods for doing this. Also, if you are doing this for your front end UI (i.e. javascript on a web page) there may be well established approaches that are quicker or simpler. The following steps seem like what you want to do with each link:
1 Check if the link/URI is "normal"; if not print an error; if it is then :
2 Do a DNS lookup on the host part of the link/URI; if not print an error; if it succeeds then:
3 Try to fetch the web page and print any errors or if it succeeds say so
Measuring "success" is hard to automate though: should it be defined strictly as status "200 OK"? Perhaps you or another member can add that part and find an elegant way to read in the "links" (broken or otherwise).
This script makes sloppy use of our to stash things and won't run with use strict. Please somebody make it look nicer and use only CORE:: modules :-)
use Regexp::Common qw/URI/;
use Net::DNS;
use 5.10.0;
use LWP::UserAgent;
my $url = "http://www.google.com/adsfdsa" ;
my $lookup = Net::DNS::Resolver->new;
my $ua = LWP::UserAgent->new;
### Step 0. regexp the URI ##
if ($url =~ /$RE{URI}{HTTP}{-keep}/){
say "$url is a URI ";
our $hostpart = $3; # stash the host part as per man page
our $filepart = $5; # stash the path
}
### Step 1. do a DNS look up and if it succeeds then or else ... etc. ###
if ($lookup->query($hostpart)) { say "$hostpart is a valid host" }
else { say " but $hostpart is an invalid host" }
### Step 2. fetch the page and check the return code ###
my $request = HTTP::Request->new(GET => $url);
my $response = $ua->request($request);
if ($response->is_success || $response->is_redirect ) {
print $request->content;
}
else {
say "but $filepart is an invalid path";
}
Output:
http://www.google.com/adsfdsa is a URI
www.google.com is a valid host
but /adsfdsa is an invalid path
You could make something like the above (but shorter, more elegant and more efficient!) read from your input and return the appropriate message to your users i.e. from wherever the error happens: i.e. at step 0., 1, or 2.). Note taht there are likely faster and simpler ways to do this, but modules like Regexp::Common and LWP are well exercised and tessted.