500 SSL negotiation failed - perl

I have a new-onset problem on my Windows XP Pro system, demonstrated by the Perl code below (which is, of course, a very cut down example from a much larger program).
It used to work until a few days ago, and I'm pulling my hair out trying to figure out what might have changed on the system to stop it working, and I'm hoping someone here might be able to give me some clues. (It still works fine on my Windows 8.1 system.)
The issue is that the code below (now) fails with "500 SSL negotiation failed".
use strict;
use warnings;
use HTTP::Request;
use LWP::UserAgent;
$ENV{HTTPS_DEBUG} = 1;
my $url = "https://secure.quksdns4.net:2087/";
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new (GET => $url);
my $res = $ua->request($req);
my $sts = $res->code;
my $hdr = $res->headers_as_string;
my $txt = $res->content;
print "\n".$sts."\n\n".$hdr."\n";
print $txt if ($sts == 500);
exit;
The output is:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:error in SSLv2 read server hello A
500
Content-Type: text/plain
Client-Date: Sat, 25 Oct 2014 14:52:43 GMT
Client-Warning: Internal response
500 SSL negotiation failed:
Curiously however it works (albeit not very usefully!) if the port number (:2087) is removed.
Active Perl v5.8.8 (which I haven't changed in years), ssleay32 & libeay32 dlls are 0.9.8.1 (also unchanged in years), and while there's a few on the system those in C:\Perl\bin are the only ones in the path.
Any hints as to what might have changed to stop the above working gratefully received!

In short: I guess the peer just disabled SSL 3.0 (at least on port 2087) because of the POODLE attack and because you are still using really old software on an unsupported OS you still attempt to connect with SSL 3.0.
Edit: It looks like version 0.57 of Crypt::SSLeay (needed for LWP at this time) used already SSLv23 handshakes which should in theory be compatible with TLS 1.x. This can also be seen in the debug output (SSLv2/v3 write client hello). So I guess that the reasons might be at least one of the following:
You are using an openssl version without support for TLS1.0. You give as the version number 0.9.8.1, but this kind of version never existed. Either you mean 0.9.8l which looks similar (and supported TLS1.0) or you mean something completely different.
They not only removed SSL 3.0 from the peer, but also straightened the ciphers so that it now requires ciphers which your old OpenSSL does not support yet.
Or they not only require TLS 1.0+ but TLS 1.1+. But support for TLS1.1 is only included since OpenSSL version 1.0.1.

Related

Switch from SSL to TLS connection

I have a Perl script which was using SSL for some file upload to server. Now we need to change this to TLS. As I am not aware how to do this, could someone help. Below is the block handling the certificate function.
sub check_ssl_cert {
use Net::SSLeay;
Net::SSLeay::set_proxy("proxye.hypo.de",80);
# $Net::SSLeay::trace = 9 if ($opt_d);
my ($page, $response, $headers, $server_cert) = Net::SSLeay::get_https3("$buba_srv", 443, "/");
print DEBUG "\n","#"x80,"\n", 'CERT: ' . $server_cert . "\n","-"x80,"\n" . $headers . "\n","-"x80,"\n" if ($opt_d);
if (!defined($server_cert) || ($server_cert == 0)) {
my $subject="Der BuBa Server gibt kein Zertifikat raus. Mit denen red ich nicht.";
my $message="HTTP Response: <" . $response . ">\n" .
"HTTP Headers: ---------------------------------------------------------------\n" .
$headers .
"HTTP Body: -------------------------------------------------------------------\n" .
$page .
"------------------------------------------------------------------------------\n" ;
&send_errormail($buba_srv, $subject, $message);
exit 5;
}
my $ssl_subject_name = Net::SSLeay::X509_NAME_oneline(Net::SSLeay::X509_get_subject_name($server_cert));
my $ssl_issuer_name = Net::SSLeay::X509_NAME_oneline(Net::SSLeay::X509_get_issuer_name($server_cert));
}
}
You are already using TLS. “SSL” is the name for old versions of the protocol, “TLS” is the name for new versions. Most websites no longer accept connections with the versions of the protocol that are officially called “SSL”. Here's a summary of the status of TLS versions:
SSLv1: so broken it wasn't even released publicly.
SSLv2: old and buried.
SSLv3: broken, don't use unless you have an ancient device that you can't upgrade and that is on a well-protected network, not Internet-facing.
TLSv1.0: deprecated but can still be used safely.
TLSv1.1: deprecated but can still be used safely.
TLSv1.2: recommended.
TLSv1.3: coming soon.
You can call Net::SSLeay::get_options to check whether SSLv3 is disabled, i.e. to check that your client will refuse to connect to an SSLv3-only server. You should ensure that SSLv3 is disabled on your client: an attacker who wanted to break your security might pretend to be the legitimate server, get your client to connect, and then exploit a vulnerability of the old protocol to be able to successfully impersonate the legitimate server. Since you're connecting to a specific server, if that server supports TLS 1.2, also disable TLS 1.0 and 1.1, by calling Net::SSLeay::set_options.
This is an sample how to switch to LWP and enforce TLS1.2:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(
ssl_opts => {
SSL_version => 'TLSv12:!SSLv2:!SSLv3:!TLSv1:!TLSv11',
}
);
$ua->timeout(30);
my $response = $ua->get('https://www.example.com/');
... = Net::SSLeay::get_https3("$buba_srv", 443, "/");
There is nothing in your code which limits the access to SSL (you probably mean SSL 3.0) only. In fact, unless $Net::SSLeay::ssl_version was explicitly set differently get_https3 will create a default SSL context which will use by default the best SSL/TLS version the underlying OpenSSL library supports, which is TLS 1.2 since OpenSSL 1.0.1 (released 2012).
Thus, if your code does not already use TLS 1.2 (have you even checked?) then check if $Net::SSLeay::ssl_version is set somewhere in your code or if your OpenSSL version is too old.
Apart from that, I really recommend against using Net::SSLeay::get_https3 and similar. These function do not properly validate the certificate (this is documented) so you will not realize if their is a man in the middle attack unless you explicitly add your own validation - which is not trivial and your code does not even attempt to do it.
Instead use higher level modules like LWP::UserAgent which makes doing HTTPS simple, uses sane defaults (at least in the current versions) and also properly validates the certificate:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->proxy(https => "http://proxye.hypo.de:80");
my $response = $ua->get("https://$buba_srv/");
my $page = $response->decoded_content;
my $headers = $response->headers->as_string;
You can also use HTTP::Tiny or Mojo::UserAgent. But contrary to LWP you need to explicitly enable proper certificate validation when using these modules by using verify_SSL option with HTTP::Tiny and the ca argument with Mojo::UserAgent.

Perl https proxy problems

I can't seem to get https through a proxy.
Example:
require LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->proxy('https', 'https://proxy:8080');
# $ua->proxy(['https'], 'https://proxy:8080'); # Fails
# $ua->env_proxy; # This also fails.
my $response = $ua->get('https://aws.amazon.com/cloudwatch/');
if ($response->is_success) {
print $response->decoded_content; # or whatever
}
else {
die $response->status_line;
}
Result:
500 Can't connect to aws.amazon.com:443 (timeout) at test.pl line 17.
But if I try the same proxy with curl (also wget) it works just fine.
$ curl --head --proxy https://proxy:8080 https://aws.amazon.com/cloudwatch/
HTTP/1.1 200 Connection established
HTTP/1.1 200 OK
Server: Server
Date: Thu, 08 Dec 2016 16:42:01 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 214187
Perl versions
$ perl -MLWP -le "print(LWP->VERSION)"
6.15
$ perl --version
This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi
I also tried with and without these:
export HTTPS_VERSION=3
export PERL_NET_HTTPS_SSL_SOCKET_CLASS="Net::SSL"
export PERL_LWP_ENV_PROXY=1
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
My actual goal here is to get aws-scripts-mon working on a machine behind a proxy but it also uses LWP::UserAgent so if I get this working then that will probably also.
Added info
It turns out that if I change to http by
$ua->proxy('http', 'http://proxy:8080'); and accesses a http url then it works just fine. The problem is that I need this to work with https.
The error from mon-put-instance-data.pl is:
./mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/
ERROR: Failed to call CloudWatch: HTTP 500. Message: Can't connect to monitoring.eu-west-1.amazonaws.com:443 (timeout)
LWP::Protocol::https::Socket: connect: timeout at /usr/local/share/perl5/LWP/Protocol/http.pm line 47.
Try LWP::Protocol::connect found in https://stackoverflow.com/a/17787133/44620
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');
$ua->get('https://www.somesslsite.com');
$ua->proxy('https', 'https://proxy:8080');
LWP does not support the use of HTTP proxies which are accessed using HTTPS. But my guess is that your proxy does not gets access with HTTPS at all, i.e. it gets accessed with HTTP even though it proxies HTTPS requests(*). Thus the code should instead use a http:// URL to access the proxy and not a https:// URL:
$ua->proxy('https', 'http://proxy:8080/');
Note that this only works in the usual setup, i.e. with IO::Socket::SSL installed on the system and used by LWP. Especially with setting PERL_NET_HTTPS_SSL_SOCKET_CLASS to Net::SSL or explicitly importing Net::SSL into the program the obsolete Crypt::SSLeay will be used where proxy handling is completely different.
(*) Even though the proxy will be accessed by HTTP and not HTTPS the connection is still encrypted. This is done by the client requesting the proxy to create a tunnel to the original target using the CONNECT method and then doing end-to-end SSL inside this tunnel. While there are a some proxies and some clients which support to be accessed by HTTPS too this would essentially mean to build an SSL connection between client and proxy and inside this SSL connection another SSL connection between client and final target, i.e. double encryption.

Perl PayPal IPN Listener Returning "500 SSL Handshake Failure" message

A perl script that has been running ok for two years just began returning "500 SSL Handshake Failure" errors upon postback of IPN message. My host service supports SHA 256, TLS 1.2 and HTTP/1.1. I'm using my host service's shared cert.
The code (from PayPal's sample script):
# read post from PayPal system and add 'cmd'
read (STDIN, $query, $ENV{'CONTENT_LENGTH'});
$query .= '&cmd=_notify-validate';
# post back to PayPal system to validate
$ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
$req = HTTP::Request->new('POST', 'https://www.sandbox.paypal.com/cgi-bin/webscr');
$req->content_type('application/x-www-form-urlencoded');
$req->header(Host => 'www.sandbox.paypal.com');
$req->content($query);
$res = $ua->request($req);
if ($res->is_error) {
# HTTP error
$retmsg = "HTTP Error:<br />";
$retmsg .= $res->status_line();
}
...
How do I best troubleshoot this?
According to SSLLabs the site is TLS 1.2 only, that it it does not accept TLS 1.0 or TLS 1.1. And according to your comment you are using OpenSSL version 1.0.0e (1000005f). But this version does not support TLS 1.2 yet because such support was only added with version 1.0.1.
This means that your system is too old to support TLS 1.2. You need to upgrade at least the OpenSSL version but preferable the whole system. Note that it does not help to just install a new version of OpenSSL. Instead you would need to recompile all the modules in Perl which make use of the library, i.e. especially Net::SSLeay in this case.

LWP refuses to connect via HTTPS

I am running the following Perl snippet on Debian using Perl v5.14.2 and libwww-perl v6.04-1
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new("GET", "https://google.com/");
my $rep = $ua->request($req);
print $rep->status_line;
This instantly returns "500 Can't connect to google.com:443". I have tried using LWP::Simple, Net::SSLeay, Crypt::SSLeay, etc. without any success.
Oddly enough, executing the same code on another Debian system running exactly the same Perl and LWP versions works.
So I thought, there is some error with the underlying system, but other applications - like cURL for any browser - are working fine.
Also, openssl s_client -connect google.com:443 returns Verify return code: 20 (unable to get local issuer certificate) on both systems.
Has anyone ever encountered this phenomenon and has a solution?
Instead of this:
$ua = LWP::UserAgent->new;
Try to use this:
$ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 });
FYI for others battling LWP 500 erors:
Some LWP 500 errors are apparently caused by another type of SSL problem (which isn't solved with the verify_hostname setting). Instead, LWP may be communicating with the HTTPS server, but the response is not what LWP expects. In my case, the solution was to force SSLv3, by the following means:
my %ssl_options = (SSL_version => 'SSLv3');
$ua = LWP::UserAgent->new(ssl_opts => \%ssl_options),
Also, for anyone trying to figure out what kind of 500 error you are getting, output this along with your error:
print $response->as_string;
For my problem (solved by setting SSLv3), my $response->as_string output included:
"SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message LWP"
I'll also say that my code ran fine for years on an earlier version of Ubuntu. This problem only appeared once I upgraded Ubuntu. I've concluded that there are probably multiple ways that newer versions of LWP differ from older ones, so developers beware!
Good luck solving your LWP problem!
I had this issue on Windows Server 2003 with Strawberry Perl. The error 500 was a problem with IO::Socket::IP.
Performing a re-install of the module (in cpan, force install IO::Socket::IP) resolved the issue.

How can I get LWP to validate SSL server certificates?

How can I get LWP to verify that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it doesn't even check that the certificate claims to be for the hostname I'm connecting to. That seems like a major security hole (especially with the recent DNS vulnerabilities).
Update: It turns out what I really wanted was HTTPS_CA_DIR, because I don't have a ca-bundle.crt. But HTTPS_CA_DIR=/usr/share/ca-certificates/ did the trick. I'm marking the answer as accepted anyway, because it was close enough.
Update 2: It turns out that HTTPS_CA_DIR and HTTPS_CA_FILE only apply if you're using Net::SSL as the underlying SSL library. But LWP also works with IO::Socket::SSL, which will ignore those environment variables and happily talk to any server, no matter what certificate it presents. Is there a more general solution?
Update 3: Unfortunately, the solution still isn't complete. Neither Net::SSL nor IO::Socket::SSL is checking the host name against the certificate. This means that someone can get a legitimate certificate for some domain, and then impersonate any other domain without LWP complaining.
Update 4: LWP 6.00 finally solves the problem. See my answer for details.
This long-standing security hole has finally been fixed in version 6.00 of libwww-perl. Starting with that version, by default LWP::UserAgent verifies that HTTPS servers present a valid certificate matching the expected hostname (unless $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} is set to a false value or, for backwards compatibility if that variable is not set at all, either $ENV{HTTPS_CA_FILE} or $ENV{HTTPS_CA_DIR} is set).
This can be controlled by the new ssl_opts option of LWP::UserAgent. See that link for details on how the Certificate Authority certificates are located. But be careful, the way LWP::UserAgent used to work, if you provide a ssl_opts hash to the constructor, then verify_hostname defaulted to 0 instead of 1. (This bug was fixed in LWP 6.03.) To be safe, always specify verify_hostname => 1 in your ssl_opts.
So use LWP::UserAgent 6; should be sufficient to have server certificates validated.
There are two means of doing this depending on which SSL module you have installed. The LWP docs recommend installing Crypt::SSLeay. If that's what you've done, setting the HTTPS_CA_FILE environment variable to point to your ca-bundle.crt should do the trick. (the Crypt::SSLeay docs mentions this but is a bit light on details). Also, depending on your setup, you may need to set the HTTPS_CA_DIR environment variable instead.
Example for Crypt::SSLeay:
use LWP::Simple qw(get);
$ENV{HTTPS_CA_FILE} = "/path/to/your/ca/file/ca-bundle";
$ENV{HTTPS_DEBUG} = 1;
print get("https://some-server-with-bad-certificate.com");
__END__
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:unknown CA
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:bad certificate
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:error in SSLv2 read server hello B
Note that get doesn't die, but it does return an undef.
Alternatively, you can use the IO::Socket::SSL module (also available from the CPAN). To make this verify the server certificate you need to modify the SSL context defaults:
use IO::Socket::SSL qw(debug3);
use Net::SSLeay;
BEGIN {
IO::Socket::SSL::set_ctx_defaults(
verify_mode => Net::SSLeay->VERIFY_PEER(),
ca_file => "/path/to/ca-bundle.crt",
# ca_path => "/alternate/path/to/cert/authority/directory"
);
}
use LWP::Simple qw(get);
warn get("https:://some-server-with-bad-certificate.com");
This version also causes get() to return undef but prints a warning to STDERR when you execute it (as well as a bunch of debugging if you import the debug* symbols from IO::Socket::SSL):
% perl ssl_test.pl
DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:269: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:271: socket connected
DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 139403496 open=139403496
DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:1135: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0)
500 Can't connect to some-server-with-bad-certificate.com:443 (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)
I landed on this page looking for a way to bypass SSL validation but all answers were still very helpful. Here are my findings. For those looking to bypass SSL validation (not recommended but there may be cases where you will absolutely have to), I'm on lwp 6.05 and this worked for me:
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use Net::SSL;
my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 }, );
my $req = GET 'https://github.com';
my $res = $ua->request($req);
if ($res->is_success) {
print $res->content;
} else {
print $res->status_line . "\n";
}
I also tested on a page with POST and it also worked. The key is to use Net::SSL along with verify_hostname = 0.
If you use LWP::UserAgent directly (not via LWP::Simple) you can validate the hostname in the certificate by adding the "If-SSL-Cert-Subject" header to your HTTP::Request object. The value of the header is treated as a regular expression to be applied on the certificate subject, and if it does not match, the request fails. For example:
#!/usr/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'https://yourdomain.tld/whatever');
$req->header('If-SSL-Cert-Subject' => '/CN=make-it-fail.tld');
my $res = $ua->request( $req );
print "Status: " . $res->status_line . "\n"
will print
Status: 500 Bad SSL certificate subject: '/C=CA/ST=Ontario/L=Ottawa/O=Your Org/CN=yourdomain.tld' !~ //CN=make-it-fail.tld/
All the solutions presented here contain a major security flaw in that they only verify the validity of the certificate's trust chain, but don't compare the certificate's Common Name to the hostname you're connecting to. Thus, a man in the middle may present an arbitrary certificate to you and LWP will happily accept it as long as it's signed by a CA you trust. The bogus certificate's Common Name is irrelevant because it's never checked by LWP.
If you're using IO::Socket::SSL as LWP's backend, you can enable verification of the Common Name by setting the verifycn_scheme parameter like this:
use IO::Socket::SSL;
use Net::SSLeay;
BEGIN {
IO::Socket::SSL::set_ctx_defaults(
verify_mode => Net::SSLeay->VERIFY_PEER(),
verifycn_scheme => 'http',
ca_path => "/etc/ssl/certs"
);
}
You may also consider Net::SSLGlue ( http://search.cpan.org/dist/Net-SSLGlue/lib/Net/SSLGlue.pm ) But, take care, it depends on recent IO::Socket::SSL and Net::SSLeay versions.
You are right to be concerned about this. Unfortunately, I don't think it's possible to do it 100% securely under any of the low-level SSL/TLS bindings I looked at for Perl.
Essentially you need to pass in the hostname of the server you want to connect to the SSL library before the handshaking gets underway. Alternatively, you could arrange for a callback to occur at the right moment and abort the handshake from inside the callback if it doesn't check out. People writing Perl bindings to OpenSSL seemed to have troubles making the callback interface consistently.
The method to check the hostname against the server's cert is dependent on the protocol, too. So that would have to be a parameter to any perfect function.
You might want to see if there are any bindings to the Netscape/Mozilla NSS library. It seemed pretty good at doing this when I looked at it.
Just perform execute the following command in Terminal:
sudo cpan install Mozilla::CA
It should solve it.