thumbnail screenshot of webpages with Perl :: Mechanize - perl

Question

i use WWW::Mechanize::Firefox to control a firefox instance and dump the rendered page with $mech->content_as_png.
New update: see at the end of the initial posting:
thanks to user1126070 we have a new solution - which i want to try out later the day [right now i am in office and not at home - in front of the machine with the programme ]
$mech->repl->repl->setup_client( { extra_client_args => { timeout => 5*60 } } );
i try out the version that put links to #list and use eval and do the following:
while (scalar(#list)) {
my $link = pop(#list);
print "trying $link\n";
eval{
$mech->get($link);
sleep (5);
my $png = $mech->content_as_png();
my $name = "$_";
$name =~s/^www\.//;
$name .= ".png";
open(OUTPUT, ">$name");
print OUTPUT $png;
close(OUTPUT);
}
if ($#){
print "link: $link failed\n";
push(#list,$link);#put the end of the list
next;
}
print "$link is done!\n";
}
BTW: user1126070 what with the trimming down the images to thumbnail-size. Should i use imager here. Can you suggest some solution thing here...!? That would be great.
end of Update
Here the problem-outline continues - as written at the very beginning of this Q & A
problem-outline: I have a list of 2500 websites and need to grab a thumbnail screenshot of them. How do I do that? I could try to parse the sites either with Perl.- Mechanize would be a good thing. Note: i only need the results as a thumbnails that are a maximum 240 pixels in the long dimension. At the moment i have a solution which is slow and does not give back thumbnails: How to make the script running faster with less overhead - spiting out the thumbnails
But i have to be aware that setting it up can pose quite a challenge, though.
If all works as expected, you can simply use a script like this to dump images of the desired websites, but you should start Firefox and resize it to the desired width manually (height doesn't matter, WWW::Mechanize::Firefox always dumps the whole page).
What i have done so far is alot - i work with mozrepl. At the moment i struggle with timeouts: Is there a way to specify Net::Telnet timeout with WWW::Mechanize::Firefox?
At the moment my internet connection is very slow and sometimes I get error
with $mech->get():
command timed-out at /usr/local/share/perl/5.12.3/MozRepl/Client.pm line 186
SEE THIS ONE:
> $mech->repl->repl->timeout(100000);
Unfortunatly it does not work: Can't locate object method "timeout" via package "MozRepl"
Documentation says this should:
$mech->repl->repl->setup_client( { extra_client_args => { timeout => 1 +80 } } );
What i have tried allready; here it is:
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize::Firefox;
my $mech = new WWW::Mechanize::Firefox();
open(INPUT, "<urls.txt") or die $!;
while (<INPUT>) {
chomp;
print "$_\n";
$mech->get($_);
my $png = $mech->content_as_png();
my $name = "$_";
$name =~s/^www\.//;
$name .= ".png";
open(OUTPUT, ">$name");
print OUTPUT $png;
sleep (5);
}
Well this does not care about the size: See the output commandline:
linux-vi17:/home/martin/perl # perl mecha_test_1.pl
www.google.com
www.cnn.com
www.msnbc.com
command timed-out at /usr/lib/perl5/site_perl/5.12.3/MozRepl/Client.pm line 186
linux-vi17:/home/martin/perl #
And here - this is my source: see a snippet-example of the sites i have in the url-list.
urls.txt - the list of sources
www.google.com
www.cnn.com
www.msnbc.com
news.bbc.co.uk
www.bing.com
www.yahoo.com and so on...
BTW: With that many url's we have to expect that some will fail and handle that. For example, we put the failed ones in an array or hash and retry them X times.
UTSL
well how is this one here...
sub content_as_png {
my ($self, $tab, $rect) = #_;
$tab ||= $self->tab;
$rect ||= {};
# Mostly taken from
# http://wiki.github.com/bard/mozrepl/interactor-screenshot-server
my $screenshot = $self->repl->declare(<<'JS');
function (tab,rect) {
var browser = tab.linkedBrowser;
var browserWindow = Components.classes['#mozilla.org/appshell/window-mediator;1']
.getService(Components.interfaces.nsIWindowMediator)
.getMostRecentWindow('navigator:browser');
var win = browser.contentWindow;
var body = win.document.body;
if(!body) {
return;
};
var canvas = browserWindow
.document
.createElementNS('http://www.w3.org/1999/xhtml', 'canvas');
var left = rect.left || 0;
var top = rect.top || 0;
var width = rect.width || body.clientWidth;
var height = rect.height || body.clientHeight;
canvas.width = width;
canvas.height = height;
var ctx = canvas.getContext('2d');
ctx.clearRect(0, 0, width, height);
ctx.save();
ctx.scale(1.0, 1.0);
ctx.drawWindow(win, left, top, width, height, 'rgb(255,255,255)');
ctx.restore();
//return atob(
return canvas
.toDataURL('image/png', '')
.split(',')[1]
// );
}
JS
my $scr = $screenshot->($tab, $rect);
return $scr ? decode_base64($scr) : undef
};
Love to hear from you!
greetings zero

Answer 1

Are you tried this out? It is working?
$mech->repl->repl->setup_client( { extra_client_args => { timeout => 5*60 } } );
put links to #list and use eval
while (scalar(#list)) {
my $link = pop(#list);
print "trying $link\n";
eval{
$mech->get($link);
sleep (5);
my $png = $mech->content_as_png();
my $name = "$_";
$name =~s/^www\.//;
$name .= ".png";
open(OUTPUT, ">$name");
print OUTPUT $png;
close(OUTPUT);
}
if ($#){
print "link: $link failed\n";
push(#list,$link);#put the end of the list
next;
}
print "$link is done!\n";
}