Perl: Downloading all .jpgs from a web directory - perl

Question

#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use FindBin qw($Bin);
print $Bin;
my $folder = "$Bin/Resources";
mkdir($folder, 0700) unless(-d $folder );
chdir($folder) or die "can't chdir $folder\n";
my $url = 'http://www.ukgamingcomputers.co.uk/images/zalmanz11plus.jpg';
my $local_file_name = 'pic.jpg';
my $mech = WWW::Mechanize->new;
$mech->get( $url, ":content_file" => $local_file_name );
I am currently using this code to download a .jpg and put it in a folder called Resources. I want to download all the .jpgs in the http://www.ukgamingcomputers.co.uk/images/ directory. I have absolutely no idea how I would achieve this. If you have a code solution, I would be grateful!

Answer 1

I'm afraid you can't do that. It is also unlikely that the web site owner would want you to.
There is no practical problem with downloading an image in that path, but to fetch them all you need to know what they are called, and there is no way to get a directory listing using HTTP.
You could crawl the site, fetch all the HTML pages from it, and find the names of all the image files those pages link to, but that would be awkward to do and even less likely to be acceptable to the site owner. It would also get you only the images used on the site, and not all the images in the directory.
Some HTTP servers are configured to return a listing of the directory in HTML if no specific file is specified in the URL and there is no default index.html file to send, but that is unusual nowadays as it represents a security breach.
If you think the site owner won't mind you helping yourself to his pictures, why not just send an email asking for a copy of them?

Answer 2

Sort of following your example, this pulls jpgs from that site you listed.
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use WWW::Mechanize::Link;
use Getopt::Long;
exit int main( parse_args() );
sub main {
my $opts = shift;
my $folder = $opts->{folder};
chdir($folder) or die "can't chdir $opts->{folder}\n";
my $mech = WWW::Mechanize->new;
$mech->get( $opts->{url} );
for my $link ( $mech->links() ) {
next unless $link->text() =~ /jpg$/;
$mech->get( $link->url() );
$mech->save_content( $link->text() );
}
}
sub parse_args {
my %opts = (
url => "http://www.ukgamingcomputers.co.uk/images/",
folder => "/home/kprice/tmp",
);
GetOptions( \%opts, 'url|u=s', 'folder|d=s', ) or die $!;
return \%opts;
}
If you're on linux, this would work, but pull everything from that link:
$ wget -r http://www.ukgamingcomputers.co.uk/images/
EDIT: I corrected it a little after that quick copy/paste.

Answer 3

Do you have to use WWW::Mechanize?
Here's an example with HTML::LinkExtor and LWP::Simple
EDIT: This actually pulls all images from given address.
#!/usr/bin/perl
use warnings;
use strict;
use LWP::Simple;
use HTML::LinkExtor;
use Data::Dumper;
$Data::Dumper::Indent=1;
die "usage: $0 url\n" if #ARGV != 1;
my $url = shift;
$|++;
if ( $url !~ /^http/ ) {
print "usage: url ( http(s)://www.example.com/ )\n";
exit(1);
}
my %images = ();
my $html = get($url)
or die "could not get $url\n";
my $parser = HTML::LinkExtor->new(undef, $url);
$parser->parse($html);
my #all_link_refs = $parser->links();
for my $link_ref ( #all_link_refs ) {
my ( $html_tag, $attr_name, $this_url ) = #$link_ref;
if ( ($html_tag eq 'img') ) {
my $image_name = (split("/", $this_url))[-1];
$images{$image_name}++;
if ( $images{$image_name} == 1 ) {
print "Downloading $this_url to $image_name...\n";
open my $PIC, ">", "$image_name";
my $image = get($this_url);
print $PIC $image;
}
}
}
OUPUT:
$ test.pl http://google.com
Downloading http://google.com/intl/en_ALL/images/srpr/logo1w.png to logo1w.png...