perl - Net::SSLeay and Server Name Indications - perl

Hello my friendly stackoverflow users,
I have the following code that "should" print out the cert for 'cyclingnews.com'
#!/usr/bin/env perl
use strict;
use Net::SSLeay qw(get_https get_https3 make_headers);
my $site = "cyclingnews.com";
my ($p, $resp, $headers, $server_cert) = get_https3($site, 443, '/');
print "$headers\n";
my $PEM = Net::SSLeay::PEM_get_string_X509( $server_cert);
print $PEM, "\n";
Upon inspection of the .pem, I see that the cert belongs to:
Subject: CN = *.ssl.hwcdn.net
X509v3 Subject Alternative Name:
DNS:*.ssl.hwcdn.net, DNS:ssl.hwcdn.net
So what I understand, this looks like a problem with SNI where Net::SSLeay is not passing a SSL_hostname to 'cyclingnews.com'. With IO::SOCKETS::SSL this can be done with SSL_hostname ( https://metacpan.org/pod/IO::Socket::SSL#Basic-SSL-Client )
The Net::SSLeay doc ( https://metacpan.org/pod/Net::SSLeay#Basic-set-of-functions ) says "By default get_https() supplies Host (to make virtual hosting easy) and Accept (reportedly needed by IIS) headers."
I am not sure if this relates to get_https3() so I have also tried:
#!/usr/bin/env perl
use strict;
use Net::SSLeay qw(get_https get_https3 make_headers);
my $site = "cyclingnews.com";
my ($p, $resp, $headers, $server_cert) = get_https3($site, 443, '/',
make_headers( 'Host:' => $site));
#);
print "$headers\n";
my $PEM = Net::SSLeay::PEM_get_string_X509( $server_cert);
print $PEM, "\n";
and this looks to pass the Host header but still same unwanted result.
so I am a bit lost, I'm a noob, so I know the folk on stackoverflow have a reputation for being freindly, maybe y'all could give me some pointers

get_https3 like many similar functions ultimately ends up in https_cat where the SSL context setup and the SSL handshake are done. Unfortunately, setting the server_name extension (SNI) is not done in this really old part of the code, which comes from a time where SNI wasn't that essentially for using HTTPS as it is today.

Related

Perl LWP::Simple::get($url) does not work for some urls

I am using LWP::Simple::get($url) library to access data from web pages. The problem is that the get function is not working for the below url.
Below is the code snippet:
#!/usr/bin/perl
use LWP::Simple;
use JSON;
use Data::Dumper;
my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";
my $json = get( $url);
die "Could not get $url!" unless defined $json;
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
After running this code it gives the below error:
Could not get https://www.cryptopia.co.nz/api/GetCurrencies!
When I replace the url with : $url = "https://api.coinmarketcap.com/v1/ticker/" it works fine.
Please can you tell me what is the root cause and how I can fix it.
Also the url mentioned in the code snippet worked once and now suddenly it does not work.
This is essentially the same question as this one, only for Perl LWP instead of NodeJS.
The problem is that the target site has a broken setup and is missing an intermediate certificate. This can be seen from the SSLabs report:
This server's certificate chain is incomplete. Grade capped to B.
To work around this you have two secure and one insecure option. The insecure but, unfortunately, often proposed option is to disable all certificate validation. Don't do this because it makes your application vulnerable against man-in-the-middle attacks.
The next option is to have your own trust store which includes all necessary certificates, i.e. both the root CA and also the missing intermediate CA:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_ca_file => 'myca.pem',
# disable OCSP stapling since it results in problems with this site
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
myca.pem is in this case the concatenation of the PEM representation for the certificates of the intermediate "COMODO RSA Extended Validation Secure Server CA" and the root "COMODO RSA Certification Authority". I've provided it at Pastebin, here.
The third option is to trust this specific leaf certificate only. When used this will trust a server using this certificate, no matter if the certificate is expired, revoked, the hostname does not match the URL or (as in this case) the trust chain cannot be built because of a missing intermediate certificate:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_fingerprint => 'sha256$70bca153ac950b8fa92d20f04dceca929852c42dc1d51bdc3c290df256ae05d3',
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE,
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
The fingerprint you see here is the one you can also see in the browser when looking at the certificate.

WWW::Mechanize ignore SSL

I'm using the following code, with the following snippet:
my $mech = WWW::Mechanize->new( 'ssl_opts' => { 'verify_hostname' => 0 } );
the following error is still being thrown:
Error GETing https://www.1031exchangeinc.com/: Can't connect to www.1031exchangeinc.com:443 (SSL connect attempt failed error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure) at crawl.pl line 29.
I want to ignore the SSL handshake.
#!/usr/bin/perl
use Modern::Perl;
use WWW::Mechanize;
use IO::Socket::SSL;
my $root = 'https://www.1031exchangeinc.com/';
my $domain = 'https://www.1031exchangeinc.com';
#my $mech = WWW::Mechanize->new;
my $mech = WWW::Mechanize->new( 'ssl_opts' => { 'verify_hostname' => 0 } );
sub visit {
my $url = shift;
my $indent = shift || 0;
my $visited = shift || {};
my $tab = ' ' x $indent;
# Already seen that.
return if $visited->{$url}++;
# Leaves domain.
if ($url !~ /^$domain/) {
say $tab, "-> $url";
return;
}
# Not seen yet.
say $tab, "- $url ";
$mech->get($url);
visit($_, $indent+2, $visited) for
map {$_->url_abs} $mech->links;
}
visit($root);
I want to ignore the SSL handshake.
The SSL handshake can not be ignored with https since it is an integral part of a TLS connection (and thus https). At most you could try to skip validation of the certificate (bad idea) which is what you are trying. But, this does not make handshake failures vanish.
Such handshake errors are instead caused for example by non overlapping cipher suites between client and server, unsupported protocol versions or missing but required TLS extensions.
It is unclear what exactly is the problem in your case. But given that the server requires SNI according to the SSLLabs report and that it requires modern ciphers (ECDHE and/or GCM or ChaCha20) my guess is that you are using a too old version of OpenSSL. This is typically the case on MacOS X which ships with a very old version of OpenSSL, i.e. version 0.9.8.
You can check the version you use with
perl -MNet::SSLeay -e 'warn Net::SSLeay::SSLeay_version(0)'
If this reports anything like 0.9.8 then you've found the reason on why it is failing and you need to upgrade to a newer version of OpenSSL and recompile Net::SSLeay against this.
If you instead have at least OpenSSL 1.0.1 then it is a different problem and you should add the versions of Net::SSLeay and IO::Socket::SSL to your question and also the output when running with perl -MIO::Socket::SSL=debug4 your-program.pl.

HTTPS Proxy and LWP::UserAgent

I have read a number of threads on a number of sites and am still unable to make this work.
I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an HTTPS site (https://github.com in the example) via a WinGate proxy that I have running on a Windows VM. Note that my actual application is attaching to an SSL webservice that I have no control over.
From Firefox, pointed to the proxy everything is copacetic. The page loads successfully and I see the connections in the Proxy software Activity monitor. I'll be darned if I can make it work in Perl though. I've started with the code from this Stack Overflow Question : How do I force LWP to use Crypt::SSLeay for HTTPS requests? And added some debugging and additional output. Here's were I stand now:
#!/usr/bin/perl
use strict;
use warnings;
use Net::SSL (); # From Crypt-SSLeay
BEGIN {
$Net::HTTPS::HTTPS_SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
$ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
# $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print "--\n";
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
print "--\n";
if ($response->is_success) {
print $response->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $response->status_line ."\n";
exit(1);
}
Here's the output from this code:
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
--
Fail:
500 Can't connect to github.com:443 (Crypt-SSLeay can't verify hostnames)
If I then uncomment $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;, I do see a single connect to github.com:443 on the proxy and then nothing. (Note it works great from a web browser through the proxy). After much hanging I get the following output from the script:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
Crypt/SSLeay/CTX.pm
Crypt/SSLeay/MainContext.pm
--
Fail:
500 SSL negotiation failed:
If anyone can provide some direction here I would greatly appreciate it!
I just uploaded the LWP::Protocol::connect module to CPAN.
This module adds the missing HTTP/CONNECT method support to LWP.
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');
$ua->get('https://www.somesslsite.com');
With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.
Why would you want "Force use of Net::SSL".
Try
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
# $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print $response->code ."\n";
An out put of 200 should mean that there were no errors.
A below sample code of mine works perfectly
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080'; #Valid HTTPS proxy taken from http://hidemyass.com/proxy-list/
$ENV{HTTPS_DEBUG} = 1;
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://www.nodeworks.com');
my $res = $ua->request($req);
print $res->code, "\n";
Output-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
With https://github.com/ the output is-
200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
Tool completed successfully
So having said all this. Your code version (below) should work fine-
use warnings;
use LWP::UserAgent;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080';
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; #works even with this
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://github.com/');
my $res = $ua->request($req);
print $res->code, "\n";
if ($res->is_success) {
print $res->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $res->status_line ."\n";
exit(1);
}
I pretty much ran into the same problem. Here are the things that fixed it for me:
make sure you use Crypt::SSLeay 0.58 or lower. At least 0.64 blocked endlessly for me (as described at http://cpanforum.com/posts/13808 ), while 0.57/0.58 worked.
make sure Net::HTTP has been updated to at least 6.03 (to avoid https://rt.cpan.org/Public/Bug/Display.html?id=72790 )
Instead of using Net::SSL which does not provide much host verification (and no SNI) you can use Net::SSLGlue::LWP. This monkey-patches LWP so that https_proxy can be used with the default SSL backend IO::Socket::SSL:
use Net::SSLGlue::LWP; # do this first
use LWP::Simple;
... continue with normal LWP stuff..
I know this may be a dead question, but if anyone else hits it I've another angle... I can't promise any answers, but we've faced a long-standing problem at $work in this area, but with the Squid proxy, and maybe specific to use of X509 client certs.
The use of the Net::SSL override is part of the solution, but I would fear that WinGate could be the problem (and not something I can help with) although in our case we contact the proxy over http (not sure how LWP deals with proxy+https).
For the record, here's an example of the precise form of code we use:
use Net::SSL;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS}="Net::SSL";
use LWP::UserAgent;
use LWP::Protocol::https;
my $ua = LWP::UserAgent->new;
$ENV{HTTPS_PROXY} = 'http://cache.local.employer.co.uk:80';
$ua->get("https://example.com/");
This is Perl 5.8.8 with recent CPAN installs (hence separation of L:P:https), so we have a fresh Net::HTTP.
I was going to mention some versions of Net::HTTP are crocked, but I just realised that was my CPAN bug in Martin's reply :)
Sorry if this doesn't add anything.
I've sent a pull-request on libwww-perl repository to fix (or may be workaround ...) the issue.
The comment of this PR shows a simple program that connects with https to github.com through a proxy. With this patch there's no need to mess around with %ENV in your program.
Another advantage is that you can re-use the usual https_proxy setup.
There has been a error in earl 5.8 and some further module where environment variable HTTP_PROXY does not appropriately sets the proxy connection.
Your case has a issue where the bug reported is as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1094440
Better way to use this is without environment variable and use LWP UserAgent
`use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');`
#!/usr/bin/env perl
#
# mimvp.com
# 2017-03-28
use CGI;
use strict;
use LWP::UserAgent;
our %proxy_https = ("https", "connect://173.233.55.118:443");
our $mimvp_url = "https://proxy.mimvp.com/exist.php";
## https
## 1. download LWP-Protocol-connect (wget http://search.cpan.org/CPAN/authors/id/B/BE/BENNING/LWP-Protocol-connect-6.09.tar.gz)
## 2. tar zxvf LWP-Protocol-connect-6.09.tar.gz
## cd LWP-Protocol-connect-6.09
## perl Makefile.PL
## make
## sudo make install
sub test_connect {
my ($url, %proxy) = #_;
print "proxy : $proxy{'http'}\n";
print "https : $proxy{'https'}\n";
print "socks4 : $proxy{'socks4'}\n";
print "socks5 : $proxy{'socks5'}\n";
print "url : $url\n";
my $browser = LWP::UserAgent->new();
$browser->env_proxy();
# # 设置的代理格式
$browser->proxy(%proxy);
$browser->timeout(30);
$browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36');
# my $req = new HTTP::Request('GET', $url);
# my $response = $browser->request($req);
my $response = $browser->get($url); # 爬取的网址
my $is_success = $response->is_success(); # 1
my $content_type = $response->content_type(); # text/html
my $content = $response->content(); # 网页正文
my $content_length = length($content); # 网页正文长度
print "$is_success\n";
print "$content_type\n";
print "$content_length\n";
print "$content\n";
}
test_connect($mimvp_url, %proxy_https); # https
## perl mimvp-proxy-perl.pl

How do I use Perl to send mail via a relay that only supports certificate based authentication

I have been using Email::Sender with Email::Sender::Transport::SMTP::TLS send email using a STARTTLS enabled relay. The relay has recently changed to using X.509 certificates for authentication. However I note that Email::Sender::Transport::SMTP::TLS has no option to point to my certificate. Neither does Net::SMTP::TLS on which Email::Sender::Transport::SMTP::TLS is based.
Can someone suggest a solution? Another module perhaps that will allow me to authenticate using a certificate.
Thank you
Checking the dependency tree: Email::Sender::Transport::SMTP::TLS ==> Net::SMTP::TLS::ButMaintained ==> IO::Socket::SSL ==> Net::SSLeay
Both IO::Socket::SSL and Net::SSLeay supporting the X.509 client certificates. So Net::SMTP::TLS::ButMaintained and Email::Sender::Transport::SMTP::TLS should be enhanced to support client certificates. A proper way should update both modules to allow the SSL_key|SSL_key_file and SSL_cert|SSL_cert_file parameters.
Here is a quick dirty one -- you need to create modified Net::SMTP::TLS::ButMaintained and keep it locally.
# modify the Net::SMTP::TLS::ButMaintained;
# built a private version which have client certificates
sub starttls {
my $me = shift;
$me->_command("STARTTLS");
my ( $num, $txt ) = $me->_response();
if ( not $num == 220 ) {
croak "Invalid response for STARTTLS: $num $txt\n";
}
if (
not IO::Socket::SSL::socket_to_SSL(
$me->{sock},
SSL_version => "SSLv3 TLSv1",
### private changes begin: append following two lines.
SSL_use_cert => 1,
SSL_cert_file => "path_to/your/certificate_file.pem",
SSL_key_file => "path_to/your/private_key_file.pem"
### private changes end:
)
)
{
croak "Couldn't start TLS: " . IO::Socket::SSL::errstr . "\n";
}
$me->hello();
}
Please leave the User and Password parameter as blank.
Good luck!
for completeness sake, Net::SMTP can do this if IO::Socket::SSL is available (don't ask me since when, but I know it works with recent versions of both modules).
Code:
use strict;
use warnings;
use Net::SMTP;
my $smtp = Net::SMTP->new(
Host => 'host.domain.tld',
Hello => 'hi there',
Port => 587,
Timeout => 5,
Debug => 4,
);
$smtp->starttls(
SSL_cert_file => "/path/to/crt",
SSL_key_file => "/path/to/key",
);
$smtp->mail("user\+perl\#domain\.nl\n");
$smtp->to("user\#otherdomain\.tld\n");
$smtp->data;
$smtp->datasend("From: Your Name <user\#domain.tld>\n");
$smtp->datasend("To: Recipient <user\#otherdomain.tld>\n");
$smtp->datasend("Subject: certificate based relay testing\n");
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-Type: text/plain; charset=us-ascii\n");
$smtp->datasend("X-Mailer: Net::SMTP IO::Socket::SSL\n");
$smtp->datasend( "X-mydate: " . localtime() . "\n" );
$smtp->datasend("\n");
$smtp->datasend("testing again\n");
$smtp->dataend;
$smtp->quit;
The host must exactly match the subject in the certificate, or it will not validate.
I assume your system trusts the certificate authority of the relay server, otherwise you need to fix that too.

Why can't I connect to my CAS server with Perl's AuthCAS?

I'm attempting to use an existing CAS server to authenticate login for a Perl CGI web script and am using the AuthCAS Perl module (v 1.3.1). I can connect to the CAS server to get the service ticket but when I try to connect to validate the ticket my script returns with the following error from the IO::Socket::SSL module:
500 Can't connect to [CAS Server]:443 (Bad hostname '[CAS Server]')
([CAS Server] substituted for real server name)
Symptoms/Tests:
If I type the generated URL for the authentication into the web browser's location bar it returns just fine with the expected XML snippet. So it is not a bad host name.
If I generate a script without using the AuthCAS module but using the IO::Socket::SSL module directly to query the CAS server for validation on the generated service ticket the Perl script will run fine from the command line but not in the browser.
If I add the AuthCAS module into the script in item 2, the script no longer works on the command line and still doesn't work in the browser.
Here is the bare-bones script that produces the error:
#!/usr/bin/perl
use strict;
use warnings;
use CGI;
use AuthCAS;
use CGI::Carp qw( fatalsToBrowser );
my $id = $ENV{QUERY_STRING};
my $q = new CGI;
my $target = "http://localhost/cgi-bin/testCAS.cgi";
my $cas = new AuthCAS(casUrl => 'https://cas_server/cas');
if ($id eq ""){
my $login_url = $cas->getServerLoginURL($target);
printf "Location: $login_url\n\n";
exit 0;
} else {
print $q->header();
print "CAS TEST<br>\n";
## When coming back from the CAS server a ticket is provided in the QUERY_STRING
print "QUERY_STRING = " . $id . "</br>\n";
## $ST should contain the received Service Ticket
my $ST = $q->param('ticket');
my $user = $cas->validateST($target, $ST); #### This is what fails
printf "Error: %s\n", &AuthCAS::get_errors() unless (defined $user);
}
Any ideas on where the conflict might be?
The error is coming from the line directly above the snippet Cebjyre quoted namely
$ssl_socket = new IO::Socket::SSL(%ssl_options);
namely the socket creation. All of the input parameters are correct. I had edited the module to put in debug statements and print out all the parameters just before that call and they are all fine. Looks like I'm going to have to dive deeper into the IO::Socket::SSL module.
As usually happens when I post questions like this, I found the problem. It turns out the Crypt::SSLeay module was not installed or at least not up to date. Of course the error messages didn't give me any clues. Updating it and all the problems go away and things are working fine now.
Well, from the module source it looks like that IO::Socket error is coming from get_https2
[...]
unless ($ssl_socket) {
$errors = sprintf "error %s unable to connect https://%s:%s/\n",&IO::Socket::SSL::errstr,$host,$port;
return undef;
}
[...]
which is called by callCAS, which is called by validateST.
One option is to temporarily edit the module file to put some debug statements in if you can, but if I had to guess, I'd say the casUrl you are supplying isn't matching up to the _parse_url regex properly - maybe you have three slashes after the https?