Why does LWP fail to connect with "500 SSL negotiation failed"? - perl

My Perl script sends some information to a remote server.
Below is a portion of the code
#!/var/hvmail/libexec/perl
use strict;
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use constant HANDLER_URL => "https://www.website.com/handler.php";
$ENV{HTTPS_DEBUG} = 1;
my $ua = LWP::UserAgent->new;
# Some DB stuff, not applicable
my $row; # This is a DB row ($sth->fetchrow_hashref())
my $req = POST ''.HANDLER_URL, [ %$row ];
my $res = $ua->request($req);
$res->is_success is false with $res->status_line being
500 SSL negotiation failed
We are running CentOS 6.4, Perl 5.10.1, OpenSSL 1.0.1e-fips.
Update
Here's the full output:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
Error: [ 500 SSL negotiation failed: ]
Requested Command Output
Can't locate Net/SSLeay.pm
Can't locate LWP/Protocol/https.pm

You seem to be relying on Crypt::SSLeay. You shouldn't. It's outdated and incomplete.
Install the latest LWP::Protocol::https which will upgrade your LWP and install the preferred SSL/TLS stack consisting of the IO::Socket::SSL and Net::SSLeay.
A web search shows there are CentOS6 repositories with RPM packages for LWP::Protocol::https.
The server has disabled SSLv3 support which means the negotiation fails.
Once you install the package, if you are still seeing the same error, make sure your script is not forcing the use of Crypt::SSLeay. That is, make sure none of the following appears anywhere in your script:
use Net::HTTPS;
$Net::HTTPS::SSL_SOCKET_CLASS = 'Net::SSL';
or
local $ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = 'Net::SSL';
or
use Net::SSL;
If you are still running into problems, make sure there is no PERL_NET_HTTPS_SSL_SOCKET_CLASS environment variable in the script's run time environment.
Also, try
$ /var/hvmail/libexec/perl -MNet::SSLeay -le 'print $Net::SSLeay::VERSION'
and
$ /var/hvmail/libexec/perl -MLWP::Protocol::https -le 'print $LWP::Protocol::https::VERSION`'
and report the output.
I suspect the issue is that the new packages were installed for the system's perl whereas it seems you may have a separate perl.
If that is the case, you should install each package individually using /var/hvmail/libexec/perl. For example:
$ curl -O https://cpan.metacpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-1.7043.tar.gz
$ tar xvf App-cpanminus-1.7043.tar.gz
$ cd App-cpanminus-1.7043
$ /var/hvmail/libexec/perl Makefile.PL
$ make install
Figure out where cpanm was installed. I am hoping /var/hvmail/libexec.
$ /var/hvmail/libexec/cpanm LWP::Protocol::https
See also Updating all outdated Perl modules, but that may be risky on a production. Still, installing App::cpanoutdated, and seeing how outdated your Perl modules are might be useful
Now, keep in mind that tinkering with a production install like this is risky. Make sure you have a way to undo changes in case something goes wrong.
Finally, note that OpenSSL 1.0.1 versions are no longer supported:
With regards to current and future releases the OpenSSL project has adopted the following policy:
Version 1.1.0 will be supported until 2018-08-31.
Version 1.0.2 will be supported until 2019-12-31 (LTS).
Version 1.0.1 is no longer supported.
Version 1.0.0 is no longer supported.
Version 0.9.8 is no longer supported.

There is no need for ''.HANDLER_URL. It looks ugly, and HANDLER_URL is fine
You don't explain what is in $row or what the POST call requires, but it looks like this
my $req = POST ''.HANDLER_URL, [ %$row ];
my $res = $ua->request($req);
should be
my $res = $ua->post(HANDLER_URL, $row);

Related

Perl and Selenium::Remote::Driver

EDITED AGAIN
I have a server on AWS somewhere in Northern Virginia and this is my monitoring server. I ssh into this Ubuntu server from another State to do system administration. I want to do web automation tests on this server which will test a web application on the Internet hitting a URL and verify that I can selenium test a login and authenticate successfully. This server is on an AWS cloud I'm not quite sure which Perl module to use since I'm accessing it remotely.
There are two CPAN modules: Selenium::Remote::Driver and WWW::Selenium. I have tried both and they are giving me issues. And I really don't know which is appropriate for my scenario. When I use Selenium::Remote::Driver, I get the following error:
Selenium server did not return proper status at /usr/local/share/perl/5.18.2/Selenium/Remote/Driver.pm line 401.
When I use WWW::Selenium, I get this error:
Failed to start new browser session: org.openqa.selenium.server.RemoteCommandException: Error while launching browser
I was able to launch firefox manually from the AWS monitoring server by exporting the DISPLAY but it was really slow. I have heard that I can use a headless browser but I would have to export the DISPLAY by:
export DISPLAY=:5
But remember, I'm sshing into this AWS/Selenium server from my desktop so I'm assuming I use the above command on the AWS/Selenium Server while I'm ssh into it from my desktop? Actually, at this point, I'm not sure I'm doing here. Can somebody help?
The problem in this type of questions is that the variety of configurations and binaries in your setup might be so broad that the it is hard to actually provide a straight and correct answer for YOUR SETUP.
This answer has the following assumptions:
you have downloaded the selenium-server-standalone.jar into /usr/lib/
you have jdk 1.8 ( run the java -version in the shell
you have installed and configured the xvfb-run ( it is a fight on it's own )
So :
```
# ssh to your server , obs the -X !
ssh -X user-name#server-name
# start the selenium-server-standalone on the server
xvfb-run -e /dev/stdout java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -jar /usr/lib/selenium-server-standalone.jar &
# one liner test - this is one veery long one
perl -e 'use strict ; use warnings ; use Data::Printer ; my $host="127.0.0.1"; use Selenium::Remote::Driver;my $driver = Selenium::Remote::Driver->new( "browser_name" =>"chrome", "error_handler" => sub { print $_[1]; croak 'goodbye'; }, "remote_server_addr" => "$host","port"=> "4444");$driver->debug_on();$driver->get("http://www.google.com"); print $driver->get_title();$driver->quit();' &
```
Here is the code in the one-liner as a perl script
#!/usr/bin/env perl
use strict ;
use warnings ;
use Carp ;
use Data::Printer ;
use Selenium::Remote::Driver;
my $host="127.0.0.1";
my $driver = Selenium::Remote::Driver->new(
"browser_name" =>"chrome"
, "error_handler" => sub { print $_[1]; croak 'goodbye' ; }
, "remote_server_addr" => "$host"
, "port"=> "4444") ;
$driver->debug_on() ;
$driver->get("http://www.google.com");
print $driver->get_title();
$driver->quit();
The output should look something like:
```
Prepping get
Executing get
REQ: POST, http://127.0.0.1:4444/wd/hub/session/ddb9c2575ab026cdb8c640bdc554181b/url, {"url":"http://www.google.com"}
RES: {"sessionId":"ddb9c2575ab026cdb8c640bdc554181b","status":0,"value":null}
Prepping getTitle
Executing getTitle
REQ: GET, http://127.0.0.1:4444/wd/hub/session/ddb9c2575ab026cdb8c640bdc554181b/title, {}
RES: {"sessionId":"ddb9c2575ab026cdb8c640bdc554181b","status":0,"value":"Google"}
GooglePrepping quit
Executing quit
REQ: DELETE, http://127.0.0.1:4444/wd/hub/session/ddb9c2575ab026cdb8c640bdc554181b, {}
RES: {"sessionId":"ddb9c2575ab026cdb8c640bdc554181b","status":0,"value":null}
```
Try running the below code:
#!/usr/bin/perl
use warnings;
use strict;
use Selenium::Remote::Driver;
my $host = "10.10.1.1"; //Enter your server IP in this place
my $driver = new Selenium::Remote::Driver('remote_server_addr' => $host,
'port' => '4444',
'auto_close' => 0);
$driver->get('http://www.google.com');

Undefined symbol in SSLeay.so: OpenSSL_add_all_algorithms

I'm trying to do a download of files from a remote host using LWP. Here is a simplified version of my code that reproduces the error:
#!/usr/local/bin/perl5.8 -w
use strict;
use LWP::UserAgent;
my $userAgent = LWP::UserAgent->new;
$userAgent->agent("p_o_c");
my $request = HTTP::Request->new(GET=>"https://ajax.googleapis.com/ajax/libs/prototype/1.7.2.0/prototype.js");
my $response = $userAgent->request($request);
When I run this code, I get the following:
[dev] /home/me > ./tmp/poc.pl
/usr/local/bin/perl5.8: symbol lookup error: /usr/local/ext/perl/5.8.0/lib/site_perl/i686-linux-thread-multi/auto/Crypt/SSLeay/SSLeay.so: undefined symbol: OpenSSL_add_all_algorithms
Also, as seen above, I'm running perl 5.8. I'm running RHEL 6.4 (Santiago).
I've tried downloading a file over http (not https) and that works.
Does anyone have any idea on how to get this https download to work? I'm open to other methods too besides LWP, but this seemed like the easiest way.
As Oesor mentioned, I am using (not by choice) a very old version of Perl. Using a newer version fixed the issue.

Perl LWP::UserAgent request does not return using http1.1 but working fine using http1.0!? Why?

I have a very simple script (on a SLES11 system) to send a http1.1 request to a server. This worked fine for a long time. Since a couple of days it stopped working. I have no idea why. After some investigation I found out forcing the script to use http1.0 it is working again. I have no idea why. And I want to know why it is not working in its default http1.1 mode.
I already enabled debugging as much as I know it (see code below). But I do not see anything - my script just hangs. There is no network activity. I neiter see that a socket will be openend (netstat -a), nor I see any incoming traffic form my client on the server. (A "telnet myserver myport" works fine.)
Can someone please help me how to track down this problem? How can I enable more debug to see where the real problem is?
#!/usr/bin/perl -w
use strict;
use warnings;
use Data::Dumper;
use HTTP::Request::Common;
use LWP::Debug qw(+);
use LWP::UserAgent;
# Workaround: forcing http1.0 instead of using http1.1, it works again!
use LWP::Protocol::https10 ();
LWP::Protocol::implementor('https', 'LWP::Protocol::https10');
# EO workaround
my $ua = LWP::UserAgent->new;
$ua->ssl_opts( verify_hostname => 0 );
my $response = $ua->request(
POST 'https://myuser:mypassword#myserver:8888/service/myservice',
Connection => 'close', # Edit: added, see comments below
Content_Type => 'text/xml',
Content => '... my content ...'
);
$ua->request() does not return! I need to kill/^C the script!
Edit: ok, no one seems to have an idea how to continue. So I startet debugging it using perl debugger.
LWP::UserAgent::post(/usr/lib/perl5/site_perl/5.10.0/LWP/UserAgent.pm:418):
418: return $self->request( HTTP::Request::Common::POST( #parameters ), #suff );
So I can see it is not returning from request().
Anyway, due to the fact that LWP::UserAgent calls HTTP::Request::Common I changed my example code above back to use HTTP::Request::Common to skip that step while debugging.
Ok... New result:
LWP::Protocol::implementor(/usr/lib/perl5/site_perl/5.10.0/Net/HTTPS.pm:26):
26: eval { require IO::Socket::SSL; };
Inside request() it hangs at IO::Socket::SSL. Means, this script it enough for further debug:
#!/usr/bin/perl
require IO::Socket::SSL;
This statement does not come back.
Further down, inside IO::Socket::SSL it hangs at:
IO::Socket::SSL::CODE(0x1274370)(/usr/lib/perl5/site_perl/5.10.0/IO/Socket/SSL.pm:92):
92: Net::SSLeay::SSLeay_add_ssl_algorithms();
Ahhhh! There is already a bugreport concerning this issue: Net-SSLeay hangs on Suse 11 P2 pointing to Bug #81575 that says:
I enountered this same issue on SLES 11 SP2, which has openssl-0.9.8j installed.
An upgrade to openssl-0.9.8r did resolve the problem. ...
The packages for 0.9.8r can be found in this repository:
http://download.opensuse.org/repositories/security:/fips/
Guess that's it!
There is already a bugreport concerning this issue: Net-SSLeay hangs on Suse 11 P2 pointing to Bug #81575 that says:
I enountered this same issue on SLES 11 SP2, which has openssl-0.9.8j installed. An upgrade to openssl-0.9.8r did resolve the problem. ... The packages for 0.9.8r can be found in this repository: http://download.opensuse.org/repositories/security:/fips/
Guess that's it!

Why might Perl allow for http websites using TOR but not https?

I am having difficulty using perl to visit a website via TOR if it is an https site but not if it is an http site.
#!/usr/bin/perl
use strict;
use WWW::Mechanize;
use LWP::Protocol::socks;
use LWP::Protocol::https;
use utf8;
my $mech = WWW::Mechanize->new(timeout => 60*5);
$mech->proxy(['http', 'https'], 'socks://localhost:9150');
$mech->get("https://www.google.com");
I am receiving the error message: Error GETing https://www.google.com: Status read failed: Bad file descriptor at line 10," where line i10 is the last line of the program.
In the TOR browser, I can successfully view: "https://www.google.com" with a port of 9150.
I am using ActivePerl 5.16.2; Vadalia 0.2.21 and Tor 0.2.3.25.
I have a Windows machine and my primary internet browser is Mozilla.
I have tried installing packages with the commands:
cpan LWP::UserAgent
ppm install LWP::Protocol::https
cpan LWP::Protocol::https
ppm install LWP::Protocol::socks
cpan LWP::Protocol::socks
ppm install Mozilla::CA
ppm install IO::Socket::SSL
ppm install Crypt::SSLeay
cpan Crypt::SSLeay
Thank you for any help! Please let me know whether there is any further information that I can provide.
Time ago, i'd found the way to go throught https sites with Tor using WWW::Curl::Easy to fetch those kind of sites, because using LWP i found the same problems.
After that i save all html in files and parsing them using WWW::Mechanzie or HTML::TreeBuilder.
If you want more interactivity with site like post forms , etc. This solutions may be more tedious because you'll need to interact with curl.
package Curl;
use warnings;
use WWW::Curl::Easy;
use WWW::UserAgent::Random;
my $curl = WWW::Curl::Easy->new;
my $useragent = rand_ua("browsers");
my $host = 'localhost';
my $port = '9070';
my $timeout = '20';
my $connectTimeOut= '20';
&init;
sub get
{
my $url = shift;
$curl->setopt(CURLOPT_URL, $url);
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
my $retcode = $curl->perform;
if ($retcode == 0) {
print("Transfer went ok Http::Code = ".$curl->strerror($retcode)."\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
# judge result and next action based on $response_code
return \$response_body;
} else {
# Error code, type of error, error message
print("An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n");
return 0;
}
}
sub init
{
#setejem el proxy
$curl->setopt(CURLOPT_PROXY,"$host:".$port);
$curl->setopt(CURLOPT_PROXYTYPE,CURLPROXY_SOCKS4);
#posem les altres dades
$curl->setopt(CURLOPT_USERAGENT, $useragent);
$curl->setopt(CURLOPT_CONNECTTIMEOUT, $connectTimeOut);
$curl->setopt(CURLOPT_TIMEOUT, $timeout);
$curl->setopt(CURLOPT_SSL_VERIFYPEER,0);
$curl->setopt(CURLOPT_HEADER,0);
}
Hope this will help you!
Maybe the proxy that you are using is already an HTTPS proxy (ie. CONNECT proxy). In that case this should work (untested):
#!/usr/bin/perl
use strict;
use WWW::Mechanize;
use LWP::Protocol::socks;
use LWP::Protocol::https;
use utf8;
my $mech = WWW::Mechanize->new(timeout => 60*5);
$mech->proxy(['http'], 'socks://localhost:9150');
$mech->proxy(['https'], 'https://localhost:9150'); ### <-- make https go over https-connect proxy
$mech->get("https://www.google.com");
I cannot find the origin but I fought with that a long time ago. Basically the problem I had was with the imlpementation that LWP::UserAgent used for the https requests.
Possibly this question can help you: How do I force LWP to use Crypt::SSLeay for HTTPS requests?

Undefined subroutine LWP::Protocol::https::Socket::can_read called

I'm trying to script some interaction between a Linux box and an ESXi host, using the VMware Perl library (which appears to call a SOAP service on the ESXi host).
I am getting an error for which I can't find a solution: Undefined subroutine &LWP::Protocol::https::Socket::can_read called at /usr/local/share/perl5/LWP/Protocol/http.pm line 22
I am currently running Net-HTTP-6.03. I did try with Net-HTTP-6.05 but it makes the web-based calls very slow (and subsequent ones also eventually fail, instead with about 1.2MB of XML as the error message) -- which various forums suggest is only resolvable by reverting to 6.03 (via cpan install GAAS/Net-HTTP-6.03.tar.gz).
The crux of my code is as follows:
use VMware::VIRuntime;
my $context = {
options => ParseOptions(), # result of using GetOpts::Long to parse command-line
session => undef
};
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL"; # Suppresses SSL_VERIFY_NONE
if ($context->{options}->{'ignore-ssl-errors'}) {
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; # Ignores certificate errors
}
## next line fails
$context->{session} = Vim::login(
service_url => $context->{options}->{url},
username => $context->{options}->{username},
password => $context->{options}->{password}
);
Is there something else I (can|need to) do to get this working?
I've got it. I needed to reinstall Bundle::LWP from an older version, not just Net-HTTP. Without it, I had LWP v6.05 trying to talk to Net-HTTP v6.03.
Once I realised that LWP == libwww-perl (yes, I'm that new), I found the overall package I needed:
cpan install GAAS/libwww-perl-6.03.tar.gz
Reproduced with LWP 6.05 and Net::HTTP 6.01, which is the cocktail that MacPorts installs at the time of this writing. Upgrading Net::HTTP to 6.06 (current) solved it:
sudo cpan Net::HTTP