Perl WWW::Mechanize Comparing the response header content length of different urls - perl

Question

I have a question I'm hoping you could help with?
I have two text files containing the following:
FILE1.txt
http://www.dog.com/
http://www.cat.com/
http://www.antelope.com/
FILE2.txt
1
2
Barry
The output I correctly achieve is as follows:
http://www.dog.com/1
http://www.dog.com/2
http://www.dog.com/Barry
http://www.cat.com/1
http://www.cat.com/2
http://www.cat.com/Barry
http://www.antelope.com/1
http://www.antelope.com/2
http://www.antelope.com/Barry
Code to do the above
open my $animalUrls, '<', 'FILE1.txt' or die "Can't open: $!";
open my $directory, '<', 'FILE2.txt' or die "Can't open: $!";
my #directory = <$directory>; #each line of the file into an array
close $directory or die "Can't close: $!";
while (my $line = <$animalUrls>) {
chomp $line;
print $line.$_ foreach (#directory);
push (#newListOfUrls, $line.$_) foreach (#directory); #put each new url into array
}
Now the problem I am having:
I need to get the Content Length of the original urls (File1.txt) and compare the Content-Length of each of the new urls with the corresponding original one to see if they are the same or different, for example:
I need to compare the Content-Length of http://www.dog.com/1 with the Content-Length of original url http://www.dog.com/ to see if there is a difference.
Then I need to compare the Content-Length of http://www.dog.com/2 with the Content-Length of the original url http://www.dog.com/ to see if there is a difference.
Then I need to compare the Content-Length of http://www.dog.com/Barry with the Content-Length of the original url http://www.dog.com/ to see if there is a difference.
Then I need to compare the Content-Length of http://www.cat.com/1 with the Content-Length of the original url http://www.cat.com/ to see if there is a difference.
Then I need to compare the Content-Length of http://www.cat.com/2 with the Content-Length of the original url http://www.cat.com/ to see if there is a difference.
And so on........
Code to get the Content-Length:
print $mech->response->header('Content-Length'); #returns the content length
What I am having trouble with is how to compare each new url with the correct corresponding original one? (i.e not accidently comparing the Content-Length of http://www.cat.com/Barry with the Content-Length of http://www.dog.com/) Should I use a hash maybe and how would I go about that?
Your help with this would be much appreciated, Many Thanks

Answer 1

You should use a hash for this. I'd change your input code to make a more complex data structure, as this makes the task easier.
open my $animalUrls, '<', 'FILE1.txt' or die "Can't open: $!";
open my $directory, '<', 'FILE2.txt' or die "Can't open: $!";
my #directory = <$directory>; #each line of the file into an array
close $directory or die "Can't close: $!";
my $newURLs;
while ( my $baseURL = <$animalUrls> ) {
chomp $baseURL;
SUBDIR: foreach my $subdir (#directory) {
chomp $subdir;
next SUBDIR if $subdir eq "";
# put each new url into arrayref
push( #{ $newURLs->{$baseURL} }, $baseURL . $subdir );
}
}
We can now use this to our advantage. Assuming we have already set up Mechanize:
foreach my $url ( keys %{$newURLs} ) {
# first get the base URL and save its content length
$mech->get($url);
my $content_length = $mech->response->header('Content-Length');
# now iterate all the 'child' URLs
foreach my $child_url ( #{ $newURLs->{$url} } ) {
# get the content
$mech->get($child_url);
# compare
if ( $mech->response->header('Content-Length') != $content_length ) {
print "$child_url: different content length: $content_length vs "
. $mech->response->header('Content-Length') . "!\n";
}
}
}
You could even do it without the second set of foreach loops by putting the code where you build up your data structure.
If you are unfamiliar with these references, take a look at perlreftut. What we have done here is make a hash with a key for each of the base URLs, and put an array of all the generated child URLs into that. If you use Data::Dumper to output the final $newURLs, it will look something like this:
$VAR1 = {
'http://www.dog.com/' => [
'http://www.dog.com/1',
'http://www.dog.com/2',
],
'http://www.cat.com/' => [
'http://www.cat.com/1',
'http://www.cat.com/2',
],
};
EDIT: I updated the code. I used these files to test it:
URLS:
http://www.stackoverflow.com/
http://www.superuser.com/
Dirs:
faq
questions
/

Answer 2

This code seems to do what you need. It stores all the URLs in #urls and prints the content lengths as it fetches each URL. I don't know what you need the length data for afterwards, but I have stored the lengths of each response in the hash %lengths to associate them with the URLs.
use 5.010;
use warnings;
use LWP::UserAgent;
STDOUT->autoflush;
my #urls;
open my $fh, '<', 'FILE1.txt' or die $!;
while (my $base = <$fh>) {
chomp $base;
push #urls, $base;
open my $fh, '<', 'FILE2.txt' or die $!;
while (my $path = <$fh>) {
chomp $path;
push #urls, $base.$path;
}
}
my $ua = LWP::UserAgent->new;
my %lengths;
for my $url (#urls) {
my $resp = $ua->get($url);
my $length = $resp->header('Content-Length');
$lengths{$url} = $length;
printf "%s -- %s\n", $url, $length // 'undef';
}
output
http://www.dog.com/ -- undef
http://www.dog.com/1 -- 56244
http://www.dog.com/2 -- 56244
http://www.dog.com/Barry -- 56249
http://www.cat.com/ -- 156
http://www.cat.com/1 -- 11088
http://www.cat.com/2 -- 11088
http://www.cat.com/Barry -- 11088
http://www.antelope.com/ -- undef
http://www.antelope.com/1 -- undef
http://www.antelope.com/2 -- undef
http://www.antelope.com/Barry -- undef