I want to access a website where the certificate cannot be verified (hostname not correct and I cannot change/update the certificate on the server which my application point). I'm using Mojo::UserAgent to get request. So how would go about ignoring this and continues to connect to the website?
I've seen that there is not an option.
I don't want to use LWP::UserAgent.
I've done it using WWW::Curl and WWW::Curl::Easy but I want to clean the code using Mojo::UserAgent (as used in my entire application).
hostname not correct ... So how would go about ignoring this and continues to connect to the website?
It is a very bad idea just to abandon any kind of validation just because the hostname does not match the certificate. Why do you use TLS at all?
A much better way is to know up front which certificate you expect and verify that you exactly get this one. This can easily be done with the option SSL_fingerprint. Unfortunately Mojo::UserAgent does not offer a way to set connection specific arguments, so you need to set it immediately before the connection and back before you do other connections:
use IO::Socket::SSL 1.980;
IO::Socket::SSL::set_client_defaults(
SSL_fingerprint => "sha256$55a5dfaaf..."
);
... use Mojo::UserAgent to connect ..
IO::Socket::SSL::set_client_defaults(); # set back
For more information about to use this option and how to get the fingerprint see Certificate error in Perl.
Another way in case only the hostname is bad would be to use the SSL_verifycn_name option to specify the hostname you expect inside the certificate.
IO::Socket::SSL::set_client_defaults(
SSL_verifycn_name => 'foo.example.com',
);
Another way could be done with the set_args_filter_hack function which is intended to deal with modules which set strange defaults or which don't let the user set its own values:
my $hostname = undef;
IO::Socket::SSL::set_args_filter_hack(
sub {
my ($is_server,$args) = #_;
$args->{SSL_verifycn_name} = $hostname if $hostname;
}
);
...
$hostname = 'foo.example.com';
... do something with Mojo::UserAgent ...
$hostname = undef;
This way you can adapt the settings for each SSL handshake.
For more information see the documentation of IO::Socket::SSL, especially the part about the common usage errors. This part also documents what you should do instead of disabling any kind of validation if some part of the certificate is wrong.
'SSL connect attempt failed error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol'
curl ... SSL connection using TLS_RSA_WITH_RC4_128_MD5
My guess is what you are facing here is unrelated to the certificate validation. Given that this server is using a very old cipher RC4-MD5 I will assume that the server can only handle SSL 3.0. This version is disabled since a while for security reasons in IO::Socket::SSL. To explicitly use this insecure version temporarily:
IO::Socket::SSL::set_client_defaults(
SSL_version => 'SSLv3'
);
Mojo::UserAgent uses IO::Socket::SSL for SSL/TLS support, so you can disable server certificate verification using
IO::Socket::SSL::set_defaults(
SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE,
);
This is an old question, but Mojolicious is alive and kicking. As such I've battled with this recently. Directly from the documents:
my $bool = $ua->insecure;
$ua = $ua->insecure($bool);
Do not require a valid TLS certificate to access HTTPS/WSS sites, defaults to the value of the MOJO_INSECURE environment variable.
# Disable TLS certificate verification for testing
say $ua->insecure(1)->get('https://127.0.0.1:3000')->result->code;
In my application $bool is set from a configuration file, so I can switch it back on, where we need it.
Related
I have a Perl script that uses WWW::Mechanize to connect to a site over
https, and that script just stopped working the other day. The status
and error message I get back are 500 and "Can't connect to
jobs.illinois.edu:443". The URL I'm trying to connect to is
https://jobs.illinois.edu/. I can connect from my browser (Firefox).
My platform is Linux -- up-to-date Arch Linux. I can also connect
(using WWW::Mechanize) to other https sites.
I tried using LWP::UserAgent, and the behavior is the same.
I'm using ssl_opts => { SSL_version => 'TLSv1' }; I don't remember why
I added that -- it may have been necessary to get it working at some
point.
Any ideas on how to fix this, or how I might get more information as
to what the problem is? Are there other ssl options I can try?
I have a feeling there was some slight configuration change on the
site that led to this problem -- maybe some SSL-protocol version
change or something like that. (I don't think I updated anything
on my machine inbetween the times it worked and stopped working.)
Thanks.
Here's sample code that fails:
#!/usr/bin/perl
use strict;
use warnings;
use constant AJB_URL => 'https://jobs.illinois.edu/academic-job-board';
use WWW::Mechanize;
my $mech = WWW::Mechanize->new( ssl_opts => { SSL_version => 'TLSv1' } );
$mech->get( AJB_URL );
It returns:
Error GETing https://jobs.illinois.edu/academic-job-board: Can't connect to jobs.illinois.edu:443 at ./test2.pl line 12.
... that script just stopped working the other day.
Which in most cases is caused by server-side or client-side changes. But I assume that you did not make any changes on the client side.
Calling your code with perl -MIO::Socket::SSL=debug4... gives:
DEBUG: ...SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Looking at the SSLLabs report you see two trust paths, where one requires an extra download. The root-CA "USERTrust RSA Certification Authority" for the first trust path is not installed on my system (Ubuntu 14.04), and I guess it is not installed on yours (no information about your OS is known, so just guessing). This means the second trust chain will be used and the relevant Root-CA "AddTrust External CA Root" is also installed on my system. Unfortunately this trust chain is missing an intermediate certificate ("Extra download"), so the verification fails.
To fix the problem, find the missing root-CA which should match the fingerprint 2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e and use it:
$ENV{PERL_LWP_SSL_CA_FILE} = '2b8f1b57330dbba2d07a6c51f70ee90ddab9ad8e.pem';
Looking at the certificate you see that it was issued on 22 May 2015, i.e. three days ago. This explains why the problem happened just now.
[Running perl 5.16.2 on OS X 10.9.5]
I have a little Secret Santa perl script I dust off once a year, and now this time it has decided to give me a hassle. (I do have a new computer as well, so there does exist a different environment from last year.) I am calling:
$smtp = Net::SMTP::SSL->new(Host => "mail.mydomain.org", Port => 465);
and when this returns, $smtp contains no value ('p $smtp' in the perl debugger just displays a blank line) and subsequent accesses like $stmp->domain (and $smtp->auth()) fail with the error
Can't call method "domain" on an undefined value at ./secretsanta.pl line 67.
What am I missing here? Thanks for any pointers.
EDIT: when I turn on SSL debugging (perl -MIO::Socket::SSL=debug4 secretsanta.pl) I get:
DEBUG: .../IO/Socket/SSL.pm:1769: Invalid default certificate authority locations
SSL error: 8606: 1 - error:2006D002:BIO routines:BIO_new_file:system lib
SSL error: 8606: 2 - error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
DEBUG: .../IO/Socket/SSL.pm:1774: Invalid default certificate authority locations error:0200100D:system library:fopen:Permission denied
DEBUG: .../IO/Socket/SSL.pm:529: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:531: socket connected
DEBUG: .../IO/Socket/SSL.pm:553: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:1769: SSL structure creation failed
DEBUG: .../IO/Socket/SSL.pm:1774: SSL structure creation failed error:140BA0C3:SSL routines:SSL_new:null ssl ctx
DEBUG: .../IO/Socket/SSL.pm:1758: IO::Socket::INET configuration failed
(I am using an SSL cert provided by my hosting company that doesn't match the DNS name of my mail server, but obviously something changed from last year to cause this not to work.)
Have you checked whether, by upgrading, you are affected by one of those bugs (first perhaps):
Cpan Bug reports
Seems to me that fits: new computer -> perhaps updated versions ->
bug with SMTP 2.35+ ( first in list ).
Perhaps you should simply use Net::SMTP itself?
I tried a few things. I installed Perl 5.18.2 (x86) for Windows and got past the empty SMTP-SSL connector, but was having problems resolving a '554 5.7.1 client host rejected' error. I ended up going back to the Mac, changing my password temporarily and using the insecure Net::SMTP to send the email, then changing password again. Obviously some bug, but I just needed this to be done with.
IO::Socket::SSL: 2.002
Which is the newest version. If you run your script only once a year there changed a lot in the mean time.
DEBUG: .../IO/Socket/SSL.pm:1769: Invalid default certificate authority locations
SSL error: 8606: 1 - error:2006D002:BIO routines:BIO_new_file:system lib
SSL error: 8606: 2 - error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
Looks like it found a CA path, but has some problems to use it. Could you please check what it found, that is
perl -MIO::Socket::SSL -MData::Dumper -e 'warn Dumper({ IO::Socket::SSL::default_ca() })'
If this returns a SSL_ca_path setting with a directory as value make sure that all files in this directory are actually readable by the program.
(I am using an SSL cert provided by my hosting company that doesn't match the DNS name of my mail server, but obviously something changed from last year to cause this not to work.)
This will definitely give problems too, but only once the first problem got fixed. One of the changes in the last year was to enforce some kind of hostname checking by default, because modules using IO::Socket::SSL often forgot to set the proper verification schema for hostnames (Net::SMTP::SSL included).
If this hostname in the certificate differs from what you specify as the destination you need to explicitly use the SSL_verifycn_name setting to define the expected hostname.
BTW, since about a month libnet (which provides the CORE modules Net::SMTP, Net::FTP, ...) has support for SSL/TLS if IO::Socket::SSL is installed. This includes support for direct SSL and STARTTLS, so you don't need any special case modules like Net::SMTP::SSL (only direct SSL), Net::SMTP::TLS (only STARTTLS) or Net::SSLGlue::SMTP (monkey patches Net::SMTP to support both) any longer.
I am working with Web::ID and have some questions.
From the FAQ for Web::ID:
How can I use WebID in Perl?
[...]
Otherwise, you need to use Web::ID directly. Assuming you've configured your web server to request a client certificate from the browser, and you've managed to get that client certificate into Perl in PEM format, then it's just:
my $webid = Web::ID->new(certificate => $pem);
my $uri = $webid->uri;
And you have the URI.
Anyway I'm stuck at the .. get that client certificate into Perl .. part.
I can see the client certificate is being passed along to the script by examining the %ENVenvironment variable. But I am still unsure how to actually process it in the way that Web::ID does... like examine the SAN.
According to the documentation of mod_ssl you will find the PEM encoded client certificate in the environment variable SSL_CLIENT_CERT, so all you need is to call
my $webid = Web::ID->new(certificate => $ENV{SSL_CLIENT_CERT});
However, Apache does not set the SSL_CLIENT_CERT environment variable by default. This is for performance reasons - setting a whole bunch of environment variables before spawning your Perl script (via mod_perl, or CGI, or whatever) is wasteful if your Perl script doesn't use them, so it only sets a small set of environment variables by default. You need to configure Apache correctly to tell it you want ALL DA STUFFZ. In particular you want something like this in .htaccess, or your virtual host config, or server config file:
SSLOptions +StdEnvVars +ExportCertData
While you're at it, you also want to make sure Apache is configured to ask clients to present a certificate. For that you want something like:
SSLVerifyClient optional_no_ca
All this is kind of covered in the documentation for Web::ID but not especially thoroughly.
I'm trying to read out (later maybe even write) into a Google Spreadsheet with Net::Google::Spreadsheets.
The most boilerplate script dies with "Login failed" and no error:
use Net::Google::Spreadsheets;
my $service = Net::Google::Spreadsheets->new(
username => 'myusername#googlemail.com',
password => 'mypassword'
);
All I'm getting is
Net::Google::AuthSub login failed
Sadly, I don't know how one would diagnose or fix this issue. Anyone?
Thanks so much!
May be because of SSL certificate checking. You can skip the test with:
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
Though really you should set the certificate authorities correctly, as per the message returned by the Net::Google::AuthSub module:
Can't verify SSL peers without knowing which Certificate Authorities
to trust
This problem can be fixed by either setting the PERL_LWP_SSL_CA_FILE
envirionment variable or by installing the Mozilla::CA module.
To disable verification of SSL peers set the
PERL_LWP_SSL_VERIFY_HOSTNAME envirionment variable to 0. If you do
this you can't be sure that you communicate with the expected peer.
When trying to hit an environment with improperly configured SSL certificates, I get the following error:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:562)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)
at dispatch.BlockingHttp$class.dispatch$BlockingHttp$$execute(Http.scala:45)
at dispatch.BlockingHttp$$anonfun$execute$1$$anonfun$apply$3.apply(Http.scala:58)
at dispatch.BlockingHttp$$anonfun$execute$1$$anonfun$apply$3.apply(Http.scala:58)
at scala.Option.getOrElse(Option.scala:108)
at dispatch.BlockingHttp$$anonfun$execute$1.apply(Http.scala:58)
at dispatch.Http.pack(Http.scala:25)
at dispatch.BlockingHttp$class.execute(Http.scala:53)
at dispatch.Http.execute(Http.scala:21)
at dispatch.HttpExecutor$class.x(executor.scala:36)
at dispatch.Http.x(Http.scala:21)
at dispatch.HttpExecutor$class.when(executor.scala:50)
at dispatch.Http.when(Http.scala:21)
at dispatch.HttpExecutor$class.apply(executor.scala:60)
at dispatch.Http.apply(Http.scala:21)
at com.secondmarket.cobra.lib.delegate.UsersBDTest.tdsGet(UsersBDTest.scala:130)
at com.secondmarket.cobra.lib.delegate.UsersBDTest.setup(UsersBDTest.scala:40)
I would like to ignore the certificates entirely.
Update: I understand the technical concerns regarding improperly configured SSL certs and the issue isn't with our boxes but a service we're using. It happens mostly on test boxes rather than prod/stg so we're investigating but needed something to test the APIs.
You can't 'ignore the certificates entirely' for the following reasons:
The problem in this case is that the client didn't even provide one.
If you don't want security why use SSL at all?
I have no doubt whatsoever that many, perhaps most, of these alleged workarounds 'for development' have 'leaked' into production. There is a significant risk of deploying an insecure system if you build an insecure system. If you don't build the insecurity in, you can't deploy it, so the risk vanishes.
The following was able to allow unsafe SSL certs.
Http.postData(url, payload).options(HttpOptions.allowUnsafeSSL,
HttpOptions.readTimeout(5000))
For the newest version of Dispatch (0.13.2), you can use the following to create an http client that accepts any certificate:
val myHttp = Http.withConfiguration(config => config.setAcceptAnyCertificate(true))
Then you can use it for GET requests like this:
myHttp(url("https://www.host.com/path").GET OK as.String)
(Modify accordingly for POST requests...)
I found this out here: Why does dispatch throw "java.net.ConnectException: General SSLEngine ..." and "unexpected status" exceptions for a particular URL?
And to create an Http client that does verify the certificates, I found some sample code here: https://kevinlocke.name/bits/2012/10/03/ssl-certificate-verification-in-dispatch-and-asynchttpclient/.