I've been frustrating myself with this for way too many hours. I think this should be simple but I obviously have something fundamental wrong. I've read:
I've read the cpan docs for :
WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod
libwww-perl-5.837/lib/LWP/UserAgent.pm
And every bit of sample code or article I could find on Google.
This is my first time looking for help on Stack Overflow. Thanks for your help in advance. Here is the code:
#!/usr/bin/perl
use WWW::Mechanize;
my $mech = WWW::Mechanize->new ( agent => "Mozilla/5.0" );
my $proxy = 'http://fetch4.me';
$mech->no_proxy('localhost');
$mech->proxy(['http', 'https', 'gopher'], $proxy) or die $!;
$mech->get('http://www.google.com');
print $mech->uri(),"\n";
print $mech->content(),"\n";
print $mech->text(),"\n";
print $mech->status(),"\n";
Here is the output:
http://www.google.com
<html>Apache is functioning normally</html>
Apache is functioning normally
200
I'm running out of ideas here. Does this code work for you? Does it produce the same results? What's wrong with it? >.<
Thank you for your time.
The problem seems to be in the server fetch4.me. Try, for instance, instead
my $proxy = 'http://124.207.162.87:80';
Does saying:
my $mech = WWW::Mechanize->new ( agent => "Mozilla/5.0", noproxy => 1 );
help?
The doc implies you need to do that to avoid an implicit call to LWP's env_proxy.
Related
i want to write a little script to grab and play around with the results of the autocomplete of an input field.
So I find a principal solution with Selenium::Firefox, that base on module Selenium::Remote::Driver, but the description of the methods is without any examples.
I have this basic example, that is able to open google and insert a search string.
Then you can see that a result list is suggested and i want to get this list.
But i have no idea how this can be obtained?
Here is my code so far:
#!/usr/bin/perl
use strict;
use warnings;
use Selenium::Firefox;
my $mech = Selenium::Firefox->new(
startup_timeout => 20,
firefox_binary => '/srv/bin/firefox.62.0/firefox',
binary => '/usr/local/bin/geckodriver',
marionette_enabled => 1
);
my $search = "perl";
my $url = "https://www.google.com/";
$mech->get($url);
$mech->find_element_by_name("q");
sleep(3);
my $result = $mech->get_active_element();
$result->send_keys($search);
sleep (10);
$mech->shutdown_binary;
exit 0;
I could find no examples to use this Perl module - and there are more questions for it.
As for instance: find_element
How can i turn on the warnings instead of killing the script?
Or how can i step through the objects of the wep page?
Is it possible to connect to an already opened browser?
The description of the module is not understandable for people who are not experts and the authors did not answer to questions so far.
But my hope is that experts here can give me a hint?
I'm trying to scrape some data from metacriti* website using mechanize, but I'm getting no output
Here's my code with a url example:
my $metaURL = "http://www.metacriti*.com/game/pc/dota-2";
my $mech = WWW::Mechanize->new();
$mech->get($metaURL) or die "unable to get $metaURL";
my $tree = HTML::TreeBuilder::XPath->new;
$tree->parse($mech->content);
my #nodes = $tree->findnodes(q{//*[#id="main"]//a[contains(./#href, "user-reviews")]/span[#class="score_value"]});
print $_->string_value, "\n" foreach(#nodes); # text
#nodes array seems to be empty, my xpath seems good and since i'm using the same syntax in another working script, I really couldnt figure out what is wrong with this one...
Also since this is just the begining, maybe you can suggest me another easy way to scrape/parse websites... If there's any better one :)
Thank you in advance
The HTML seems to be really bad, if you search for $tree->findnodes( '//div[#id="main"]')->[0]->as_HTML you get a very bare div:
<div class="col main_col" id="main"><div itemscope="itemscope" itemtype="http://schema.org/SoftwareApplication"></div></div>
this indeed does not contain any a, which explains the result you got.
I tried using tidy to pretty print the HTML, but it barfed on the file.
If you forget about the div and use q{//a[contains(./#href, "user-reviews")]/span[#class="score_value"]} you will get a result though, 7.9 in this case.
recently my hosting company updated their servers, and i believe my code's inefficiency is being brought to light.
i have this code:
#!/usr/bin/perl
use CGI qw(:standard);
use CGI::Session;
$cgi = new CGI;
$sid = $cgi->cookie("CGISESSID") || undef;
$session = new CGI::Session(undef, $sid, {Directory=>'/tmp'});
$cookie = $cgi->cookie(CGISESSID => $session->id);
$sSid = $session->id();
print $cgi->header(-cookie=>$cookie);
print << "EOF";
HTML CODE
EOF
this was fine until (it seems) the update - now i'm getting this error in my logs:
Can't call method "id" on an undefined value at /home/users/web/XXXX/my-file.cgi line 10.
i then edited this into line 10:
$cookie = $cgi->cookie(CGISESSID => $session->id) || undef;
figuring that it would take care of the issue, but it didn't.
it seems to be causing a brief 500 server error, and then the site comes back online, with the session id persisting through the error.
i'm completely lost as to what is happening. what's worse is it isn't happening every time the page loads - it seems so random but i'm sure i'm just not encountering the specific situation to bring that error.
i've been using the same setup for all my cgi's for the past 7 years. not sure why it's causing a brief error now, but that's besides the point.
$session is undef, so that means new returned undef. The docs have the following to say about that:
Returns new session object, or undef on failure. Error message is accessible through errstr() - class method.
So call CGI::Session->errstr to find out what the problem is.
What is the perl equivalent for this php code?
$hash = hash_hmac('sha256', $all , $secret);
I tried using the below code but in vain. The values are different.
use Digest::SHA;
$sha = Digest::SHA->new('sha256');
$sha->add($secret);
$sha->add($all);
$digest = $sha->hexdigest;
Regards,
Pavan
Since my question was getting more views than I expected, I decided to answer it to help others with the same problem. I found the equivalent for it in PHP.
use Digest::SHA qw(hmac_sha256_hex);
$digest=hmac_sha256_hex($all, $secret);
Hope it helps.
I have written a Perl script to put back some SNMP values, which works fine. I have now written a script on the remote server and used the extend function in SNMP to put the value from the script into SNMP.
If I run:
snmpget -v2c -c public 10.0.0.10 'NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."cc_power"'
I get the result:
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."cc_power" = STRING: 544
But when I try to use my script to get the information back it doesn't get it. Here is the script:
#!/usr/bin/perl
use strict;
use SNMP;
use RRDs;
my $rrd_db = "/storage/db/rrd/cc_power.rrd";
my $sess;
my $val;
my $error;
$sess = new SNMP::Session(DestHost => "10.0.0.10", Community => "public", Version => 2);
my $power = $sess->get('NET-SNMP-EXTEND-MIB::nsExtendOutput1Line.\"cc_power\"');
$error=RRDs::error;
die "ERROR while updating RRD: $error\n" if $error;
my $date=time;
print "Data Script has been run - Output: ${date}:${power}\n";
but nothing is returned, and I have no idea why... no errors or anything, have I missed something stupid?
Hope someone can help as this is driving me nuts :)
I assume that you used netsnmp snmpget. Well, it hides too many details from you, as it loads MIB documents in background and nicely translate OIDs and SNMP values to all kinds of user friendly formats.
So next time pay attention to what decoration it performs and simulate that in your own code to achieve the same effects.