I am trying to access a remote server via the WebDav protocol, and more specifically Perl's HTTP::DAV module.
According to its documentation the coupling to a remote directory takes place in the following manner:
use HTTP::DAV;
$d = HTTP::DAV->new();
$url = "http://host.org:8080/dav/";
$d->credentials(
-user => "pcollins",
-pass => "mypass",
-url => $url,
-realm => "DAV Realm"
);
$d->open( -url => $url )
or die("Couldn't open $url: " .$d->message . "\n");
I created a local webdav directory and can access it flawlessly over the http protocol.
According to HTTP::DAV's documentation, there should be and https support as well using the Crypt::SSLeay module.
The Crypt::SSLeay's documention offers us the following synopsys using inside the LWP::UserAgent module, thus providing for us web resource access over the https protocol:
use Net::SSL;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(
ssl_opts => { verify_hostname => 0 },
);
my $response = $ua->get('https://www.example.com/');
print $response->content, "\n";
My question is:
How can I combine the HTTP::DAV and Crypt::SSLeay modules in order to have web resource access over the WebDav and https protocols?
Something like the following:
use HTTP::DAV;
$d = HTTP::DAV->new();
$url = "https://host.org:8080/dav/";
#...
This is untested, but from skimming the documentation, this should work:
$d->get_user_agent->ssl_opts( verify_hostname => 0 );
The documentation for HTTP::DAV says:
get_user_agent
Returns the clients' working HTTP::DAV::UserAgent object.
You may want to interact with the HTTP::DAV::UserAgent object to modify request headers or provide advanced authentication procedures.
HTTP::DAV::UserAgent isn't documented, but its source code shows it's a subclass of LWP::UserAgent. The documentation for LWP::UserAgent mentions the ssl_opts method for setting SSL options for the user agent object.
Related
My question: why does my perl script--successful via home laptop--not work when run in the context of my hosting website? (Perhaps they have a firewall, for example. Perhaps my website needs to provide credentials. Perhaps this is in the realm of cross-site scripting. I DON'T KNOW and appeal for your help in my understanding what could be the cause and then the solution. Thanks!)
Note that all works fine IF I run the perl script from my laptop at home.
But if I upload the perl script to my web host, where I have a web page whose javascript successfully calls that perl script, there is an error back from the site whose URL is in the perl script (finance.yahoo in this example).
To bypass the javascript, I'm just typing the URL of my perl script, e.g. http://example.com/blah/script.pl
Here is the full error message from finance.yahoo when $url starts with http:
Can't connect to finance.yahoo.com:80 nodename nor servname provided, or not known at C:/Perl/lib/LWP/Protocol/http.pm line 47.
Here is the full error message from finance.yahoo when $url starts with https:
Can't connect to finance.yahoo.com:443 nodename nor servname provided, or not known at C:/Perl/lib/LWP/Protocol/http.pm line 47.
Code:
#!/usr/bin/perl
use strict; use warnings;
use LWP 6; # one site suggested loading this "for all important LWP classes"
use HTTP::Request;
### sample of interest: to scrape historical data and feed massaged facts to my private web page via js ajax
my $url = 'http://finance.yahoo.com/quote/sbux/profile?ltr=1';
my $browser = LWP::UserAgent->new;
# one site suggested having this empty cookie jar could help
$browser->cookie_jar({});
# another site suggested I should provide WAGuess
my #ns_headers = (
'User-Agent' =>
# 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0',
'Accept' => 'text/html, */*',
'Accept-Charset' => 'iso-8859-1,*,utf-8',
'Accept-Language' => 'en-US',
);
my $response = $browser->get($url, #ns_headers);
# for now, I just want to confirm, in my web page itself, that
# the target web page's contents was returned
my $content = $response->content;
# show such content in my web page
print "Content-type: text/html\n\n" . $content;
Well it is not obvious what is your final goal and it is possible that you over complicate the task.
You can retrieve above mentioned page with simpler perl code
#!/usr/bin/env perl
#
# vim: ai:ts=4:sw=4
#
use strict;
use warnings;
use feature 'say';
use HTTP::Tiny;
my $debug = 1;
my $url = 'https://finance.yahoo.com/quote/sbux/profile?ltr=1';
my $responce = HTTP::Tiny->new->get($url);
if ($responce->{success}) {
my $html = $responce->{content};
say $html if $debug;
}
In your post you indicated that javascript is somehow involved -- it is not clear how and what it's purpose in retrieving of the page.
Error message has a reference to at C:/Perl/lib/LWP/Protocol/http.pm line 47 which indicates that web hosting is taking place on Windows machine -- it would be nice to indicate it in your message.
Could you shed some light on purpose of following block in your code?
# WAGuess
$browser->env_proxy;
# WAGuess
$browser->cookie_jar({});
I do not see cookie_jar be utilized in your code anywhere.
Do you plan to use some authentication approach to extract some data under your personal account which is not accessible otherwise?
Please state in a few first sentences what you try to achieve on grand scale.
Perhaps it's about cookies or about using yahoo's "query" url instead.
Yahoo Finance URL not working
I am using LWP::Simple::get($url) library to access data from web pages. The problem is that the get function is not working for the below url.
Below is the code snippet:
#!/usr/bin/perl
use LWP::Simple;
use JSON;
use Data::Dumper;
my $url = "https://www.cryptopia.co.nz/api/GetCurrencies";
my $json = get( $url);
die "Could not get $url!" unless defined $json;
my $decoded_json = decode_json($json);
print Dumper($decoded_json);
After running this code it gives the below error:
Could not get https://www.cryptopia.co.nz/api/GetCurrencies!
When I replace the url with : $url = "https://api.coinmarketcap.com/v1/ticker/" it works fine.
Please can you tell me what is the root cause and how I can fix it.
Also the url mentioned in the code snippet worked once and now suddenly it does not work.
This is essentially the same question as this one, only for Perl LWP instead of NodeJS.
The problem is that the target site has a broken setup and is missing an intermediate certificate. This can be seen from the SSLabs report:
This server's certificate chain is incomplete. Grade capped to B.
To work around this you have two secure and one insecure option. The insecure but, unfortunately, often proposed option is to disable all certificate validation. Don't do this because it makes your application vulnerable against man-in-the-middle attacks.
The next option is to have your own trust store which includes all necessary certificates, i.e. both the root CA and also the missing intermediate CA:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_ca_file => 'myca.pem',
# disable OCSP stapling since it results in problems with this site
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
myca.pem is in this case the concatenation of the PEM representation for the certificates of the intermediate "COMODO RSA Extended Validation Secure Server CA" and the root "COMODO RSA Certification Authority". I've provided it at Pastebin, here.
The third option is to trust this specific leaf certificate only. When used this will trust a server using this certificate, no matter if the certificate is expired, revoked, the hostname does not match the URL or (as in this case) the trust chain cannot be built because of a missing intermediate certificate:
use strict;
use warnings;
use LWP::UserAgent;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_fingerprint => 'sha256$70bca153ac950b8fa92d20f04dceca929852c42dc1d51bdc3c290df256ae05d3',
SSL_ocsp_mode => SSL_OCSP_NO_STAPLE,
);
my $resp = $ua->get('https://www.cryptopia.co.nz/api/GetCurrencies');
print $resp->decoded_content;
The fingerprint you see here is the one you can also see in the browser when looking at the certificate.
I wish to request a URL via a HTTPS proxy using perl's LWP::UserAgent module. There is quite a few reference around this, but nothing could help me get it work.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use Data::Dumper;
BEGIN {
$ENV{HTTPS_PROXY} = 'https://<IP>:<PORT>';
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY_USERNAME} = '<API_KEY>';
$ENV{HTTPS_PROXY_PASSWORD} = '';
$ENV{HTTPS_DEBUG} = 1; #Add debug output
}
my $ua = LWP::UserAgent->new(ssl_opts => {verify_hostname => 0}, SSL_version => 'SSLv3', allowed_protocols => ['https', 'http']);
$ua->proxy(['https', 'http'], 'https://<IP>:<PORT>');
my $req = HTTP::Request->new('GET','https://<DOMAIN_URL>');
print STDERR Dumper($ua);
my $response = $ua->request($req);
print $response->code ."\n";
print STDERR Dumper($response);
I get this error:
SSL connect attempt failed error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol at
/home/user/project/local/lib/perl5/LWP/Protocol/http.pm line 51.
Despite me specifying allowed_protocols in LWP, https scheme in proxy and the url scheme being https, it still goes to LWP::Protocol::HTTP above and not to LWP::Protocol::HTTPS.
I also verified that the version of LWP::Protocol::HTTPS is 6.06 which is the same as LWP::UserAgent (which was mentioned in one of the forums)
This worked for me:
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://<USER>:<PSWD>#<IP>:<PORT>/');
$ua->get('https://www.somesslsite.com');
HTTPS Proxy and LWP::UserAgent
Note: The environment credentials (HTTPS_PROXY_USERNAME) didnt work for me. I had to enter it in the URL like above.
The correct setup with recent versions of LWP (starting with version 6.06 which you have) is to just use the same syntax as found in other applications together with the proxy function:
my $ua = LWP::UserAgent->new;
$ua->proxy(https => 'http://user:pass#proxy');
$ua->get('https://server');
Alternatively you could set the environment variable https_proxy to the same value, i.e. http://user:pass#proxy.
Before 6.06 proxy support was broken at least when used together with IO::Socket::SSL (default since version 6.0). The syntax you have in your question is for the old backend Crypt::SSLeay which is no longer recommended because it does not check the certificates properly.
I'm trying to create a pre signed URL for Amazon S3.
The below code generates the same signature as the signature in the example in the S3 docs.
(https://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html)
However, when I plug in my own credentials etc, I get an error from AWS:
"The request signature we calculated does not match the signature you provided. Check your key and signing method."
Given the code below seems to work, I presume my signing method is ok. Also, the credentials used are the ones
used to upload the file, so they are good too. They are root credentials, not an IAM user.
I have tried:
1) A file name with and without a period.
2) A file in US standard as well as Ireland.
3) HTTP and HTTPS
4) SHA256 (although this breaks the example in the docs.)
5) Different web clients: wget, curl and Firefox.
I seem to be doing exactly the same as the php given here:
Creating a signed S3 URL with Javascript
This is the URL generated for the US standard. Bucket is "buck1bar", file is "foo.txt"
https://s3.amazonaws.com/buck1bar/foo.txt?AWSAccessKeyId=AKIAJG5VR56AB5USIG5Q&Expires=1467893299&Signature=emdimX%2FWKYrcdhtPtE7eQ9ErWv30HNlAHaa9vlDODCY%3D
So, what am I doing wrong?
#!/usr/bin/perl
use strict;
use warnings;
use diagnostics;
use MIME::Base64;
use Digest::HMAC;
use Digest::SHA;
use URI::Escape;
my $hmac3 = Digest::HMAC->new( 'OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV', 'Digest::SHA');
$hmac3->add("GET\n\n\n1141889120\n/quotes/nelson");
my $signature = uri_escape( encode_base64 $hmac3->digest, '');
print "vjbyPxybdZaNmGa%2ByT272YEAiv4%3D\n$signature\n";
# vjbyPxybdZaNmGa%2ByT272YEAiv4%3D
# vjbyPxybdZaNmGa%2ByT272YEAiv4%3D
After further hackery, it turns out that the module I was using already does it.
my $client = Net::Amazon::S3::Client->new( s3 => $s3 );
my $bucket = $client->bucket( name => $bucket_name );
my $o = $bucket->object( key => 'key1', expires => '2016-07-08' );
my $uri = $o->query_string_authentication_uri();
print "$uri\n";
Here's my code
#!/path/to/perl
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
use Crypt::SSLeay;
$ENV{HTTPS_PROXY} = 'http://proxy:8080/';
$ENV{HTTPS_DEBUG} = 1;
my $myurl = "https://www.redhat.com";
my $ua = new LWP::UserAgent;
$ua->cookie_jar( {} );
$ua->protocols_allowed( [ 'http','https'] );
$ua->proxy(['http', 'https'], 'http://proxy:8080/');
my $page = $ua->get($myurl);
die "Error $myurl\n ", $page->status_line, "\n Aborting"
unless $page->is_success;
print "Success", $page1->content_type, " document!\n";
It returns
Error at https://www.redhat.com
400 Bad Request
Aborting at test.pl line 30.
what's wrong?
Edit:
Apparently, Its a bug. But the workaround doesn't work for me.
Ha! I got the answer!
1) remove the '/' after the port of ENV{HTTPS_PROXY}
2) Apparently, LWP's proxy system send 'GET' requests instead of CONNECT requests so use Crypt::SSLeay's proxy system by just setting the environment variable and remove the proxy command.
On some systems, e.g. Debian, you need to install the appropriate SSL library for this to work. The error messages on theses systems can sometimes be at bit missleading. I think the Debian package would be libnet-ssleay-perl.
I just uploaded the LWP::Protocol::connect module to CPAN. This module adds the missing HTTP/CONNECT method support to LWP.
use LWP::UserAgent;
$ua = LWP::UserAgent->new();
$ua->proxy('https', 'connect://proxyhost.domain:3128/');
$ua->get('https://www.somesslsite.com');
With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.
It looks like your proxy server does not accept HTTPS connections. Have you tried setting it up in your favorite browser and viewing the URL?