Debian Wheezy (perl 5.14)
Work nice:
use LWP::Simple;
print get( 'http://ip6-localhost:80' );
Not working (1)
use LWP::Simple;
print get( 'http://[::1]:80' );
Not working (2) [Return: Bad hostname]
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Not working (3) [Return: Connection refused]
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Why I need it? Because ldirectord need it. :(
Any suggestion?
Another post suggested using INET6Glue
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
print get( 'http://[::1]:80' );
print get( 'http://ipv6.google.com' );
Related
I try to use WWW::Mechanize in order to automate a session with GeoServer.
GeoServer comes with a REST API, which can be used with curl. But at the moment, it is impossible to create a datastore for ImageMosaicJDBC with the REST API, so i would like to add the new raster data source with a Perl script. it is based on WWW::Mechanize.
but it fails, with this message :
your session has expired.
The script is just below...
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use HTML::TreeBuilder;
use HTML::Tree;
use Getopt::Long;
use HTTP::Cookies;
my %CONF = (
username => 'admin',
password => 'geoserver',
);
GetOptions( \%CONF, "username=s", "password=s" ) or die "Bad options";
my $netloc = "193.55.67.151:8080";
my $url = "http://$netloc/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.GeoServerLoginPage";
my $cookie_jar = HTTP::Cookies->new;
my $agent = WWW::Mechanize->new( cookie_jar => $cookie_jar );
$agent->agent('User-Agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0');
# auth
$agent->get($url);
die $agent->res->status_line unless $agent->success;
$agent->set_fields(%CONF);
$agent->submit;
die $agent->res->status_line unless $agent->success;
# adding data store
$url = "http://$netloc/geoserver/web?wicket:bookmarkablePage=:org.geoserver.web.data.store.NewDataPage";
my $content = $agent->get($url);
die $agent->res->status_line unless $agent->success;
my $tree = HTML::Tree->new();
$tree->parse($content);
print $agent->content;
# storeform
$url = "http://$netloc/geoserver/web/?wicket:interface=:5:storeForm::IFormSubmitListener::";
my $content = $agent->post($url);
die $agent->res->status_line unless $agent->success;
my $tree = HTML::Tree->new();
$tree->parse($content);
print $agent->content;
# newdatapage
$url = "http://$netloc/geoserver/web/?wicket:interface=:6::::";
my $ref = "http://$netloc/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.store.NewDataPage";
my $content = $agent->get( $url, referer => $ref);
die $agent->res->status_line unless $agent->success;
my $tree = HTML::Tree->new();
$tree->parse($content);
print $agent->content;
I cannot see where the problem comes from... In particular, i used WireShark to inspect the HTTP exchanges, but every thing was ok for me. The JSESSIONID cookie was for example correctly rescueing.
Try setting timeout parameter while declaring $agent
I want to print the redirected url in perl.
Input url : http://pricecheckindia.com/go/store/snapdeal/52517?ref=velusliv
output url : http://www.snapdeal.com/product/vox-2-in-1-camcorder/1154987704?utm_source=aff_prog&utm_campaign=afts&offer_id=17&aff_id=1298&source=pricecheckindia
use LWP::UserAgent qw();
use CGI qw(:all);
print header();
my ($url) = "http://pricecheckindia.com/go/store/snapdeal/52517?ref=velusliv";
my $ua = LWP::UserAgent->new;
my $req = new HTTP::Request(GET => $url);
my $res = $ua->request($req);
print $res->request;
How to get this done in perl?
You need to examine the HTTP response to find the URL. The documentation of HTTP::Response gives full details of how to do this, but to summarise, you should do the following:
use strict;
use warnings;
use feature ':5.10'; # enables "say"
use LWP::UserAgent;
my $url = "http://pricecheckindia.com/go/store/snapdeal/52517?ref=velusliv";
my $ua = LWP::UserAgent->new;
my $req = new HTTP::Request(GET => $url);
my $res = $ua->request($req);
# you should add a check to ensure the response was actually successful:
if (! $res->is_success) {
say "GET failed! " . $res->status_line;
}
# show the base URI for the response:
say "Base URI: " . $res->base;
You can view redirects using HTTP::Response's redirects method:
if ($res->redirects) { # are there any redirects?
my #redirects = $res->redirects;
say join(", ", #redirects);
}
else {
say "No redirects.";
}
In this case, the base URI is the same as $url, and if you examine the contents of the page, you can see why.
# print out the contents of the response:
say $res->decoded_contents;
Right near the bottom of the page, there is the following code:
$(window).load(function() {
window.setTimeout(function() {
window.location = "http://www.snapdeal.com/product/vox-2-in-1-camcorder/1154987704?utm_source=aff_prog&utm_campaign=afts&offer_id=17&aff_id=1298&source=pricecheckindia"
}, 300);
});
The redirect is handled by javascript, and so is not picked up by LWP::UserAgent. If you want to get this URL, you will need to extract it from the response contents (or use a different client that supports javascript).
On a different note, your script starts off like this:
use LWP::UserAgent qw();
The code following the module name, qw(), is used to import particular subroutines into your script so that you can use them by name (instead of having to refer to the module name and the subroutine name). If the qw() is empty, it's not doing anything, so you can just omit it.
To have LWP::UserAgent follow redirects, just set the max_redirects option:
use strict;
use warnings;
use LWP::UserAgent qw();
my $url = "http://pricecheckindia.com/go/store/snapdeal/52517?ref=velusliv";
my $ua = LWP::UserAgent->new( max_redirect => 5 );
my $res = $ua->get($url);
if ( $res->is_success ) {
print $res->decoded_content; # or whatever
} else {
die $res->status_line;
}
However, that website is using a JavaScript redirect.
$(window).load(function() {
window.setTimeout(function() {
window.location = "http://www.snapdeal.com/product/vox-2-in-1-camcorder/1154987704?utm_source=aff_prog&utm_campaign=afts&offer_id=17&aff_id=1298&source=pricecheckindia"
}, 300);
});
This will not work unless you use a framework that enables JavaScript, like WWW::Mechanize::Firefox.
It will throw you an error for the last line $res - > request since it is returning hash and content from the response. So below is the code:
use LWP::UserAgent qw();
use CGI qw(:all);
print header();
my ($url) = "http://pricecheckindia.com/go/store/snapdeal/52517?ref=velusliv";
my $ua = LWP::UserAgent->new;
my $req = new HTTP::Request(GET => $url);
my $res = $ua->request($req);
print $res->content;
In a certain script I tried to write this:
my $ua = LWP::UserAgent->new;
my $res = $ua->post($url, Content => $data);
and got "400 Bad Request".
After some reading I tried this:
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new( 'POST', $url );
$req->content( $data );
my $res = $ua->request( $req );
and it worked, but I thought these two should do the same. What am I missing here?
Am I misunderstanding something in the documentation of HTTP::Request and LWP::UserAgent?
Is there a way to ask LWP::UserAgent to print what it is doing?
Here's one way to do it:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
{
no strict "refs";
no warnings "redefine";
my $orig_sub = \&LWP::UserAgent::send_request;
*{"LWP::UserAgent::send_request"} = sub {
my ($self, $request) = #_;
print $request->as_string . "\n";
my $response = $orig_sub->(#_);
print $response->as_string . "\n";
return $response;
};
}
my $a = LWP::UserAgent->new;
my $response = $a->get("http://google.com");
It will print out all the requests and responses that LWP::UserAgent does.
My symptom is that I cannot use a proxy with HTTPS requests with LWP. This seems to be a common problem, and the hints on Google and even here all suggest a work-around for setting the HTTPS_PROXY environment variable for use by Crypt::SSLeay.
My specific problem appears to be that LWP::Protocol::https is loading IO::Socket::SSL rather than Crypt::SSLeay. How can I force Crypt::SSLeay's use instead?
My code:
#!/usr/bin/perl
use strict;
use warnings;
$ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
And it's output, showing that Crypt::SSLeay is not being used:
Net/SSLeay.pm
IO/Socket/SSL.pm
/usr/lib/perl5/auto/Net/SSLeay/autosplit.ix
/usr/lib/perl5/auto/Net/SSLeay/set_proxy.al
/usr/lib/perl5/auto/Net/SSLeay/randomize.al
Simply adding an explicit use Crypt::SSLeay to my script has proven ineffective. It loads the module, but it continues to load IO::Socket::SSL, and use it for the HTTPS requests.
Try this:
use strict;
use warnings;
use Net::SSL (); # From Crypt-SSLeay
BEGIN {
$Net::HTTPS::SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
$ENV{HTTPS_PROXY} = 'http://10.0.3.1:3128';
}
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://www.meritrustcu.org/');
my $res = $ua->request($req);
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
I don't have a suitable proxy, so I haven't tried it myself.
This is what I did to get LWP and SOAP::Lite to work with our proxy at GE. This was after much digging on CPAN, google etc. I finally figured it out after running the test script in the Crypt::SSLeay package called net_ssl_test and it was able to connect through the proxy. The key is forcing Crypt::SSLeay to use Net::SSL as was mentioned above. This is not documented very well on CPAN however.
use LWP::UserAgent;
# override HTTPS setting in LWP to work with a proxy
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY} = 'http-proxy.ae.ge.com:80';
$ENV{HTTPS_PROXY_USERNAME} = 'username';
$ENV{HTTPS_PROXY_PASSWORD} = 'password';
$ua = new LWP::UserAgent;
# make a https request
my $req = HTTP::Request->new(GET => 'https://mail.google.com/');
my $res = $ua->request($req);
print $res->as_string;
The following code ...
my $user_agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(GET => $url);
my $response = $user_agent->request($request);
if ($response->is_success) {
print "OK\n";
} else {
die($response->status_line);
}
.. will fail with ..
500 Can't connect to <hostname> (Bad hostname '<hostname>')
.. if the hostname in $url is an IPv6 only address (that is: presence of an AAAA record, but no A record).
My questions are:
How do I enable IPv6 support in LWP?
How do I configure LWP's settings for "prefer-IPv4-over-IPv6" (A vs. AAAA) / "prefer-IPv6-over-IPv4" (AAAA vs. A)?
It looks like you just need to use Net::INET6Glue::INET_is_INET6. To quote its example:
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
print get( 'http://[::1]:80' );
print get( 'http://ipv6.google.com' );
I believe you'll have to change the module to use the IPV6 net module. By default it does not have this enabled: http://eintr.blogspot.com/2009/03/bad-state-of-ipv6-in-perl.html. I don't believe there is something as simple as "prefer-ipv6"
Debian Wheezy (perl 5.14)
Work nice:
use LWP::Simple;
print get( 'http://ip6-localhost:80' );
Not working (1)
use LWP::Simple;
print get( 'http://[::1]:80' );
Not working (2) [Return: Bad hostname]
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Not working (3) [Return: Connection refused]
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Soo, if you don't need IPv6 address in http request, it's fine. :(