Perl Search::Elasticsearch doesn't work with SSL enabled node with Self-signed certificate - perl

I am using Perl wrapper for Elasticsearch: "Search::Elasticsearch"
I create an object as follows:
use Search::Elasticsearch;
my $e = Search::Elasticsearch->new( nodes => "192.168.0.66:9200", debug => 1);
It works fine and I can query Elasticsearch node using this object. Then I try to connect towards an Elasticsearch node that is enabled for SSL using self-signed certificate. I use the following command:
use Search::Elasticsearch;
my $e = Search::Elasticsearch->new( nodes => "192.168.0.66:9200", use_https => 1, userinfo => "testuser:testpwd", debug => 1);
But it doesn't work.
If I do curl to query Elasticsearch, I use the following command:
curl https://testuser:testpwd#192.168.0.67:9200
This also doesn't work. When I use -k switch with curl:
curl https://testuser:testpwd#192.168.0.67:9200 -k
Then it works fine. I get response from Elasticsearch node.
Now I want to use the same procedure to query same SSL enabled Elasticsearch node using Perl module: Search::Elasticsearch. What is the right syntax and how and where to define '-k' switch?

Using -k is synonym to --insecure so curl doesn't verify the certificate. The question is then, how useful is it to query over an encrypted SSL channel while allowing insecure connections?
That kind of defeats the purpose of using SSL in the first place, right?
If you have the CA that signed the certificate (which you should), then you should use it with the --cacert curl switch
curl --cacert /path/to/cacert.pem https://testuser:testpwd#192.168.0.67:9200
or with Perl you can also specify the CA cert in the ssl_options
use Search::Elasticsearch;
use IO::Socket::SSL;
my $es = Search::Elasticsearch->new(
nodes => [
"192.168.0.66:9200"
],
userinfo => "testuser:testpwd",
debug => 1,
ssl_options => {
SSL_verify_mode => SSL_VERIFY_PEER,
SSL_ca_file => '/path/to/cacert.pem',
SSL_verifycn_scheme => 'http',
}
);

Related

2-Way-TLS support

Server application is using following code to get the request from client to process. Now I got a requirement to make this call secure using HTTPS (2-way-tls). I need to convert this call to secure call in Perl and implement client authentication for trusted client. I am not sure how can I do that and how to provide certificates.
my $daemonname = "SOAP::Transport::HTTP::Daemon";
$daemon = $daemonname
->new(Scheme => 'http', LocalPort => 8080, Listen => 128)
->dispatch_to('MyCode')
->options({compress_threshold => 10000});
$daemon->handle;

Certificate error in Perl

I am connecting to a CAS server. But My CAS server certificate is expired and due to this getting below error:
error SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed unable to connect https://<domain Name>:443/
To avoid this error few suggestion is like verify_hostname & verify_ssl to "0". But it's not solving the issue. Can anyone help?
Perl version: 5.22
LWP:6.0.16
To avoid this error few suggestion is like verify_hostname & verify_ssl to "0"
If you would follow these suggestions then you should ask yourself why do you use https at all. Because ignoring certificate errors means that man in the middle attacks are possible and thus the protection TLS should offer simply vanishes.
To connect to a server where the certificate cannot be properly validated by normal means you have to use a different kind of verification instead of no verification at all. Support for https in current versions of LWP is realized using IO::Socket::SSL. This module offers a simple mechanism to deal with such problems by comparing the fingerprint of the certificate against the expected fingerprint.
First you need to get the current fingerprint of the certificate. This can be done with some openssl commands or if you are sure that there is currently no man in the middle attack you could simply access the server:
use strict;
use warnings;
use IO::Socket::SSL 1.980;
my $dst = 'bad-cert.example.com';
my $cl = IO::Socket::SSL->new(
PeerAddr => $dst,
PeerPort => 443,
# certificate cannot be validated the normal way, so we need to
# disable validation this one time in the hope that there is
# currently no man in the middle attack
SSL_verify_mode => 0,
) or die "connect failed";
my $fp = $cl->get_fingerprint;
print "fingerprint: $fp\n";
This will give you a fingerprint with hash algorithm, i.e. something like sha256$55a5dfaaf.... This fingerprint then can be used to validate the certificate in future calls:
use strict;
use warnings;
use IO::Socket::SSL 1.980;
use LWP::UserAgent;
my $dst = ....; # from above example
my $fp = ....; # from above example
my $ua = LWP::UserAgent->new(ssl_opts => { SSL_fingerprint => $fp });
my $resp = $ua->get("https://$dst");
print $resp->content;
Apart from that please not that there is a reason certificates expire. After the expiration time no more revocations will be tracked. This means you have to really know that this certificate is definitely not revoked, because no CA will tell you.

Trust self-signed certificate for Email::Sender::Transport::SMTPS in perl

Trying to use Email::Sender::Transport::SMTPS for sending email. My transport is:
my $transport = Email::Sender::Transport::SMTPS->new({
host => $smtpserver,
ssl => 'starttls',
sasl_username => $smtpuser,
sasl_password => $smtppassword,
debug => 1,
});
When trying send the email, the debug says:
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 250-SIZE 52428800
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 250-8BITMIME
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 250-PIPELINING
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 250-STARTTLS
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 250 HELP
Net::SMTPS=GLOB(0x7f893b2b00f0)>>> STARTTLS
Net::SMTPS=GLOB(0x7f893b2b00f0)<<< 220 TLS go ahead
DEBUG: .../IO/Socket/SSL.pm:735: local error: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:738: fatal SSL error: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
unable to establish SMTP connection
Probably because the server using self-signed certificate. (When using the same setting in the Thunderbird it is needed to add the "trust this certificate" setting.)
The question is: How to add the "trust this certificate" for the Email::Sender::Transport::SMTPS - thus allow sending email.
There is no direct way to give Email::Sender::Transport::SMTPS SSL specific arguments. But, since ultimately IO:Socket::SSL is used you can hack your way around this limitation. From the man page:
This is a list of typical errors seen with the use of IO::Socket::SSL:. ....
Make SSL settings inacessible by the user, together with bad builtin settings.
Some modules use IO::Socket::SSL, but don't make the SSL settings available to the user. This is often combined with bad builtin settings or defaults (like switching verification off). Thus the user needs to hack around these restrictions by using set_args_filter_hack or similar.
Thus what you could do is
IO::Socket::SSL::set_args_filter_hack( sub {
my ($is_server,$args) = #_;
$args->{SSL_fingerprint} = 'sha1$437104....'
});
With this option you could make certificate pinning trust a certificate as long as it's fingerprint matches the given one. You can get the fingerprint for instance with:
openssl s_client -connect mail.example.org:25 -starttls smtp |\
openssl x509 -fingerprint -noout
Alternative ways would be to use the certificate as trusted with the SSL_ca_file option. You could also set SSL_verify_mode to 0 (SSL_VERFY_NONE) but since this disables any kind of validation you should only use this for testing.
Please note that set_args_filter_hack is global, that is it affects all IO::Socket::SSL objects in your program. Thus depending on the program you should only set it directly before you establish the connection with Email::Sender::Transport::SMTPS and reset it immediately afterwards.
For more information about these options please read the documentation of IO::Socket::SSL.
You can control the SSL usage with the options of new() constructor method
"SSL_verify_mode"=>'SSL_VERIFY_NONE'
IN Email/Sender/Transport/SMTPS.pm
# compatible
my $ssl = $self->ssl;
$ssl = 'ssl' if $self->ssl and $self->ssl ne 'starttls';
return (
$self->host,
Port => $self->port,
Timeout => $self->timeout,
defined $ssl ? (doSSL => $ssl) : (),
defined $self->helo ? (Hello => $self->helo) : (),
defined $self->localaddr ? (LocalAddr => $self->localaddr) : (),
defined $self->localport ? (LocalPort => $self->localport) : (),
defined $self->debug ? (Debug => $self->debug) : (),
"SSL_verify_mode"=>'SSL_VERIFY_NONE',#ADDED LINE
);
EDIT: This is really bad, because as per comments you trust everything. Instead if on linux/*nix
cat tobetrusted.crt >> /etc/ssl/cert.pem

how do you validate server cert & each of the certificate in the chain up to the ROOT in perl - 4 levels deep

how do you validate server cert & each of the certificate in the chain up to the ROOT in perl - 4 levels deep.
I want to be able to check if they are valid & the signing is done right, also want to check if any of them are revoked in perl.
check: server cert-> chain1->chain2->root.
I am not sure how to accomplish this step by step & error out accordingly.
Any references to a working example? This is what I have:
my $certclient = IO::Socket::SSL->new(
PeerHost => "$host:$port",
SSL_ca_file => Mozilla::CA::SSL_ca_file(),
SSL_verify_mode => 0,
SSL_version => 'TLSv1',
SSL_cipher_list => 'RC4-SHA',
Proto => 'tcp',
Timeout => '5',
);
$certclient->verify_hostname($host, "http");
my $cn = $certclient->peer_certificate("cn");
$certclient->verify_hostname($host, "http");
thanks!
http://search.cpan.org/~sullr/IO-Socket-SSL-2.003/lib/IO/Socket/SSL.pod
IO::Socket::SSL will do all the verification for you if you don't disable it by setting SSL_verify_mode to 0. It will also check if the certificate is revoked if you provide the necessary CRL (SSL_crl_file) or use OCSP by setting SSL_ocsp_mode and call $ssl->ocsp_resolver->resolve_blocking().
It does not provide an interface to do this steps by hand and neither does the underlying Net::SSLeay module. The OpenSSL functions necessary to do the verification are not exposed to Perl.

How can I get LWP to validate SSL server certificates?

How can I get LWP to verify that the certificate of the server I'm connecting to is signed by a trusted authority and issued to the correct host? As far as I can tell, it doesn't even check that the certificate claims to be for the hostname I'm connecting to. That seems like a major security hole (especially with the recent DNS vulnerabilities).
Update: It turns out what I really wanted was HTTPS_CA_DIR, because I don't have a ca-bundle.crt. But HTTPS_CA_DIR=/usr/share/ca-certificates/ did the trick. I'm marking the answer as accepted anyway, because it was close enough.
Update 2: It turns out that HTTPS_CA_DIR and HTTPS_CA_FILE only apply if you're using Net::SSL as the underlying SSL library. But LWP also works with IO::Socket::SSL, which will ignore those environment variables and happily talk to any server, no matter what certificate it presents. Is there a more general solution?
Update 3: Unfortunately, the solution still isn't complete. Neither Net::SSL nor IO::Socket::SSL is checking the host name against the certificate. This means that someone can get a legitimate certificate for some domain, and then impersonate any other domain without LWP complaining.
Update 4: LWP 6.00 finally solves the problem. See my answer for details.
This long-standing security hole has finally been fixed in version 6.00 of libwww-perl. Starting with that version, by default LWP::UserAgent verifies that HTTPS servers present a valid certificate matching the expected hostname (unless $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} is set to a false value or, for backwards compatibility if that variable is not set at all, either $ENV{HTTPS_CA_FILE} or $ENV{HTTPS_CA_DIR} is set).
This can be controlled by the new ssl_opts option of LWP::UserAgent. See that link for details on how the Certificate Authority certificates are located. But be careful, the way LWP::UserAgent used to work, if you provide a ssl_opts hash to the constructor, then verify_hostname defaulted to 0 instead of 1. (This bug was fixed in LWP 6.03.) To be safe, always specify verify_hostname => 1 in your ssl_opts.
So use LWP::UserAgent 6; should be sufficient to have server certificates validated.
There are two means of doing this depending on which SSL module you have installed. The LWP docs recommend installing Crypt::SSLeay. If that's what you've done, setting the HTTPS_CA_FILE environment variable to point to your ca-bundle.crt should do the trick. (the Crypt::SSLeay docs mentions this but is a bit light on details). Also, depending on your setup, you may need to set the HTTPS_CA_DIR environment variable instead.
Example for Crypt::SSLeay:
use LWP::Simple qw(get);
$ENV{HTTPS_CA_FILE} = "/path/to/your/ca/file/ca-bundle";
$ENV{HTTPS_DEBUG} = 1;
print get("https://some-server-with-bad-certificate.com");
__END__
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:unknown CA
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL3 alert write:fatal:bad certificate
SSL_connect:error in SSLv3 read server certificate B
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:error in SSLv2 read server hello B
Note that get doesn't die, but it does return an undef.
Alternatively, you can use the IO::Socket::SSL module (also available from the CPAN). To make this verify the server certificate you need to modify the SSL context defaults:
use IO::Socket::SSL qw(debug3);
use Net::SSLeay;
BEGIN {
IO::Socket::SSL::set_ctx_defaults(
verify_mode => Net::SSLeay->VERIFY_PEER(),
ca_file => "/path/to/ca-bundle.crt",
# ca_path => "/alternate/path/to/cert/authority/directory"
);
}
use LWP::Simple qw(get);
warn get("https:://some-server-with-bad-certificate.com");
This version also causes get() to return undef but prints a warning to STDERR when you execute it (as well as a bunch of debugging if you import the debug* symbols from IO::Socket::SSL):
% perl ssl_test.pl
DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:269: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:271: socket connected
DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 139403496 open=139403496
DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 139403496
DEBUG: .../IO/Socket/SSL.pm:1135: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0)
500 Can't connect to some-server-with-bad-certificate.com:443 (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)
I landed on this page looking for a way to bypass SSL validation but all answers were still very helpful. Here are my findings. For those looking to bypass SSL validation (not recommended but there may be cases where you will absolutely have to), I'm on lwp 6.05 and this worked for me:
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use Net::SSL;
my $ua = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0 }, );
my $req = GET 'https://github.com';
my $res = $ua->request($req);
if ($res->is_success) {
print $res->content;
} else {
print $res->status_line . "\n";
}
I also tested on a page with POST and it also worked. The key is to use Net::SSL along with verify_hostname = 0.
If you use LWP::UserAgent directly (not via LWP::Simple) you can validate the hostname in the certificate by adding the "If-SSL-Cert-Subject" header to your HTTP::Request object. The value of the header is treated as a regular expression to be applied on the certificate subject, and if it does not match, the request fails. For example:
#!/usr/bin/perl
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'https://yourdomain.tld/whatever');
$req->header('If-SSL-Cert-Subject' => '/CN=make-it-fail.tld');
my $res = $ua->request( $req );
print "Status: " . $res->status_line . "\n"
will print
Status: 500 Bad SSL certificate subject: '/C=CA/ST=Ontario/L=Ottawa/O=Your Org/CN=yourdomain.tld' !~ //CN=make-it-fail.tld/
All the solutions presented here contain a major security flaw in that they only verify the validity of the certificate's trust chain, but don't compare the certificate's Common Name to the hostname you're connecting to. Thus, a man in the middle may present an arbitrary certificate to you and LWP will happily accept it as long as it's signed by a CA you trust. The bogus certificate's Common Name is irrelevant because it's never checked by LWP.
If you're using IO::Socket::SSL as LWP's backend, you can enable verification of the Common Name by setting the verifycn_scheme parameter like this:
use IO::Socket::SSL;
use Net::SSLeay;
BEGIN {
IO::Socket::SSL::set_ctx_defaults(
verify_mode => Net::SSLeay->VERIFY_PEER(),
verifycn_scheme => 'http',
ca_path => "/etc/ssl/certs"
);
}
You may also consider Net::SSLGlue ( http://search.cpan.org/dist/Net-SSLGlue/lib/Net/SSLGlue.pm ) But, take care, it depends on recent IO::Socket::SSL and Net::SSLeay versions.
You are right to be concerned about this. Unfortunately, I don't think it's possible to do it 100% securely under any of the low-level SSL/TLS bindings I looked at for Perl.
Essentially you need to pass in the hostname of the server you want to connect to the SSL library before the handshaking gets underway. Alternatively, you could arrange for a callback to occur at the right moment and abort the handshake from inside the callback if it doesn't check out. People writing Perl bindings to OpenSSL seemed to have troubles making the callback interface consistently.
The method to check the hostname against the server's cert is dependent on the protocol, too. So that would have to be a parameter to any perfect function.
You might want to see if there are any bindings to the Netscape/Mozilla NSS library. It seemed pretty good at doing this when I looked at it.
Just perform execute the following command in Terminal:
sudo cpan install Mozilla::CA
It should solve it.