WWW::Mechanize ignore SSL - perl

I'm using the following code, with the following snippet:
my $mech = WWW::Mechanize->new( 'ssl_opts' => { 'verify_hostname' => 0 } );
the following error is still being thrown:
Error GETing https://www.1031exchangeinc.com/: Can't connect to www.1031exchangeinc.com:443 (SSL connect attempt failed error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure) at crawl.pl line 29.
I want to ignore the SSL handshake.
#!/usr/bin/perl
use Modern::Perl;
use WWW::Mechanize;
use IO::Socket::SSL;
my $root = 'https://www.1031exchangeinc.com/';
my $domain = 'https://www.1031exchangeinc.com';
#my $mech = WWW::Mechanize->new;
my $mech = WWW::Mechanize->new( 'ssl_opts' => { 'verify_hostname' => 0 } );
sub visit {
my $url = shift;
my $indent = shift || 0;
my $visited = shift || {};
my $tab = ' ' x $indent;
# Already seen that.
return if $visited->{$url}++;
# Leaves domain.
if ($url !~ /^$domain/) {
say $tab, "-> $url";
return;
}
# Not seen yet.
say $tab, "- $url ";
$mech->get($url);
visit($_, $indent+2, $visited) for
map {$_->url_abs} $mech->links;
}
visit($root);

I want to ignore the SSL handshake.
The SSL handshake can not be ignored with https since it is an integral part of a TLS connection (and thus https). At most you could try to skip validation of the certificate (bad idea) which is what you are trying. But, this does not make handshake failures vanish.
Such handshake errors are instead caused for example by non overlapping cipher suites between client and server, unsupported protocol versions or missing but required TLS extensions.
It is unclear what exactly is the problem in your case. But given that the server requires SNI according to the SSLLabs report and that it requires modern ciphers (ECDHE and/or GCM or ChaCha20) my guess is that you are using a too old version of OpenSSL. This is typically the case on MacOS X which ships with a very old version of OpenSSL, i.e. version 0.9.8.
You can check the version you use with
perl -MNet::SSLeay -e 'warn Net::SSLeay::SSLeay_version(0)'
If this reports anything like 0.9.8 then you've found the reason on why it is failing and you need to upgrade to a newer version of OpenSSL and recompile Net::SSLeay against this.
If you instead have at least OpenSSL 1.0.1 then it is a different problem and you should add the versions of Net::SSLeay and IO::Socket::SSL to your question and also the output when running with perl -MIO::Socket::SSL=debug4 your-program.pl.

Related

Get reason for failed LWP::Simple head request

I have migrated my scrilpts from CentOS 7 to 8 and there's a new Perl version. I have the folowing snippet that uses head to check if a URL exists:
#!/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $sitemapurl = "https://www.prosinger.net";
if (head($sitemapurl)) {
...
}
else {
print "The $sitemapurl doesn't exist\n";
exit(1);
}
It now always returns that the URL doesn't exist. I'm quite sure that this has to do something with https (I have perl-LWP-Protocol-https installed), but I'm not sure how to get any feedback information from head method to check what the error code is.
Any ideas?
You can use LWP::UserAgent instead of LWP::Simple, which allows you to get an error message:
my $ua = LWP::UserAgent->new;
my $sitemapurl = "https://www.prosinger.net";
my $req = $ua->head($sitemapurl);
if ($req->is_success) {
...
} else {
die "Could not head($sitemapurl): " . $req->status_line;
}
Running this code prints:
Could not head(https://www.prosinger.net): 403 Forbidden at head.pl line 15.
You can fix this (for this specific website; this will not work for all website) by setting a User-Agent in your LWP::UserAgent object:
my $ua = LWP::UserAgent->new( agent => 'anything seems to work' );
Of interest is the decoded_content method of HTTP::Response that allows you to get the content of the request (you don't need it in that case, but you might later):
my $req = $ua->get(...);
if ($req->is_success) {
my $content = $req->decoded_content;
...
}
Your code that uses LWP::Simple and Dada's version that switches to LWP::UserAgent are basically doing the same thing, except that you can get details of the error when using LWP::UserAgent.
Running the LWP::UserAgent version gives this error:
Could not head(https://www.prosinger.net): 500 Can't connect to www.prosinger.net:443 (SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid)
And Googling that error message gives this SO answer as the first result. Is it possible that the clocks on your your client machine and the server are out of sync?

Perl LWP::Simple::get($url) does not work for some urls

I am using LWP::Simple::get($url) library to access data from web pages. The problem is that the get function is not working for the below url.
Below is the code snippet:
#!/usr/bin/perl
use LWP::Simple;
use JSON;
use Data::Dumper;
my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";
my $json = get( $url);
die "Could not get $url!" unless defined $json;
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
After running this code it gives the below error:
Could not get https://www.cryptopia.co.nz/api/GetCurrencies!
When I replace the url with : $url = "https://api.coinmarketcap.com/v1/ticker/" it works fine.
Please can you tell me what is the root cause and how I can fix it.
Also the url mentioned in the code snippet worked once and now suddenly it does not work.
This is essentially the same question as this one, only for Perl LWP instead of NodeJS.
The problem is that the target site has a broken setup and is missing an intermediate certificate. This can be seen from the SSLabs report:
This server's certificate chain is incomplete. Grade capped to B.
To work around this you have two secure and one insecure option. The insecure but, unfortunately, often proposed option is to disable all certificate validation. Don't do this because it makes your application vulnerable against man-in-the-middle attacks.
The next option is to have your own trust store which includes all necessary certificates, i.e. both the root CA and also the missing intermediate CA:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_ca_file => 'myca.pem',
# disable OCSP stapling since it results in problems with this site
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
myca.pem is in this case the concatenation of the PEM representation for the certificates of the intermediate "COMODO RSA Extended Validation Secure Server CA" and the root "COMODO RSA Certification Authority". I've provided it at Pastebin, here.
The third option is to trust this specific leaf certificate only. When used this will trust a server using this certificate, no matter if the certificate is expired, revoked, the hostname does not match the URL or (as in this case) the trust chain cannot be built because of a missing intermediate certificate:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_fingerprint => 'sha256$70bca153ac950b8fa92d20f04dceca929852c42dc1d51bdc3c290df256ae05d3',
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE,
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
The fingerprint you see here is the one you can also see in the browser when looking at the certificate.

Perl HTTPS over proxy using LWP::UserAgent

I wish to request a URL via a HTTPS proxy using perl's LWP::UserAgent module. There is quite a few reference around this, but nothing could help me get it work.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use Data::Dumper;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://<IP>:<PORT>';
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY_USERNAME} = '<API_KEY>';
$ENV{HTTPS_PROXY_PASSWORD} = '';
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = LWP::UserAgent->new(ssl_opts => {verify_hostname => 0}, SSL_version => 'SSLv3', allowed_protocols => ['https', 'http']);
$ua->proxy(['https', 'http'], 'https://<IP>:<PORT>');
my $req = HTTP::Request->new('GET','https://<DOMAIN_URL>');
print STDERR Dumper($ua);
my $response = $ua->request($req);
print $response->code ."\n";
print STDERR Dumper($response);
I get this error:
SSL connect attempt failed error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol at
/home/user/project/local/lib/perl5/LWP/Protocol/http.pm line 51.
Despite me specifying allowed_protocols in LWP, https scheme in proxy and the url scheme being https, it still goes to LWP::Protocol::HTTP above and not to LWP::Protocol::HTTPS.
I also verified that the version of LWP::Protocol::HTTPS is 6.06 which is the same as LWP::UserAgent (which was mentioned in one of the forums)
This worked for me:
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://<USER>:<PSWD>#<IP>:<PORT>/');
$ua->get('https://www.somesslsite.com');
HTTPS Proxy and LWP::UserAgent
Note: The environment credentials (HTTPS_PROXY_USERNAME) didnt work for me. I had to enter it in the URL like above.
The correct setup with recent versions of LWP (starting with version 6.06 which you have) is to just use the same syntax as found in other applications together with the proxy function:
my $ua = LWP::UserAgent->new;
$ua->proxy(https => 'http://user:pass#proxy');
$ua->get('https://server');
Alternatively you could set the environment variable https_proxy to the same value, i.e. http://user:pass#proxy.
Before 6.06 proxy support was broken at least when used together with IO::Socket::SSL (default since version 6.0). The syntax you have in your question is for the old backend Crypt::SSLeay which is no longer recommended because it does not check the certificates properly.

LWP HTTPS proxy timing out

I am trying to connect via HTTPS to a site using LWP::Simple. I have set the environment variables both inside of the script and in the shell. I continue to get a 500 Connection Timed Out. I can connect to a HTTP site just fine.
The proxy is letting communication through. I can connect to the HTTPS site through the proxy using curl without a problem.
Any suggestions?
#!/usr/bin/perl
use warnings;
use Net::SSL;
use LWP::UserAgent;
use LWP::Debug qw(+);
use Data::Dumper;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{HTTPS_DEBUG} = 1;
$ENV{HTTPS_VERSION} = 2;
my $ua = LWP::UserAgent->new (verify_hostname => 0);
$ua->ssl_opts(verify_hostname => 0,
SSL_verify_mode => 0x00);
$ua->proxy('https' => 'http://x.x.x.x:3128');
print $ua->proxy('https');
print Dumper($ua);
my $response = $ua->get('https://qualys.com/');
print Dumper ($response);
if ($response->is_success) {
print $response->decoded_content; # or whatever
exit(0);
}
else {
print "\nFail:\n";
print $response->status_line ."\n";
exit(1);
}
HTTPS proxy support for LWP is (or at least was until some days ago) broken, at least if you use IO::Socket::SSL, which is the default for LWP versions >=6.0 because it provides better security. It should be much better now with LWP 6.0.6 and LWP::Protocol::https 6.0.6 which were released few days ago. If you cannot upgrade you might try using Net::SSLGlue::LWP which patches LWP for better https proxy support.

How do I use Perl to send mail via a relay that only supports certificate based authentication

I have been using Email::Sender with Email::Sender::Transport::SMTP::TLS send email using a STARTTLS enabled relay. The relay has recently changed to using X.509 certificates for authentication. However I note that Email::Sender::Transport::SMTP::TLS has no option to point to my certificate. Neither does Net::SMTP::TLS on which Email::Sender::Transport::SMTP::TLS is based.
Can someone suggest a solution? Another module perhaps that will allow me to authenticate using a certificate.
Thank you
Checking the dependency tree: Email::Sender::Transport::SMTP::TLS ==> Net::SMTP::TLS::ButMaintained ==> IO::Socket::SSL ==> Net::SSLeay
Both IO::Socket::SSL and Net::SSLeay supporting the X.509 client certificates. So Net::SMTP::TLS::ButMaintained and Email::Sender::Transport::SMTP::TLS should be enhanced to support client certificates. A proper way should update both modules to allow the SSL_key|SSL_key_file and SSL_cert|SSL_cert_file parameters.
Here is a quick dirty one -- you need to create modified Net::SMTP::TLS::ButMaintained and keep it locally.
# modify the Net::SMTP::TLS::ButMaintained;
# built a private version which have client certificates
sub starttls {
my $me = shift;
$me->_command("STARTTLS");
my ( $num, $txt ) = $me->_response();
if ( not $num == 220 ) {
croak "Invalid response for STARTTLS: $num $txt\n";
}
if (
not IO::Socket::SSL::socket_to_SSL(
$me->{sock},
SSL_version => "SSLv3 TLSv1",
### private changes begin: append following two lines.
SSL_use_cert => 1,
SSL_cert_file => "path_to/your/certificate_file.pem",
SSL_key_file => "path_to/your/private_key_file.pem"
### private changes end:
)
)
{
croak "Couldn't start TLS: " . IO::Socket::SSL::errstr . "\n";
}
$me->hello();
}
Please leave the User and Password parameter as blank.
Good luck!
for completeness sake, Net::SMTP can do this if IO::Socket::SSL is available (don't ask me since when, but I know it works with recent versions of both modules).
Code:
use strict;
use warnings;
use Net::SMTP;
my $smtp = Net::SMTP->new(
Host => 'host.domain.tld',
Hello => 'hi there',
Port => 587,
Timeout => 5,
Debug => 4,
);
$smtp->starttls(
SSL_cert_file => "/path/to/crt",
SSL_key_file => "/path/to/key",
);
$smtp->mail("user\+perl\#domain\.nl\n");
$smtp->to("user\#otherdomain\.tld\n");
$smtp->data;
$smtp->datasend("From: Your Name <user\#domain.tld>\n");
$smtp->datasend("To: Recipient <user\#otherdomain.tld>\n");
$smtp->datasend("Subject: certificate based relay testing\n");
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-Type: text/plain; charset=us-ascii\n");
$smtp->datasend("X-Mailer: Net::SMTP IO::Socket::SSL\n");
$smtp->datasend( "X-mydate: " . localtime() . "\n" );
$smtp->datasend("\n");
$smtp->datasend("testing again\n");
$smtp->dataend;
$smtp->quit;
The host must exactly match the subject in the certificate, or it will not validate.
I assume your system trusts the certificate authority of the relay server, otherwise you need to fix that too.