I am using the LWP::UserAgent to connect to fetch some connection
use LWP::UserAgent;
use LWP::ConnCache;
use LWP::Debug qw(+);
my $ua = LWP::UserAgent->new( conn_cache => 1);
my $cache = $ua->conn_cache(LWP::ConnCache->new( ));
$ua->conn_cache->total_capacity(undef);
$ua->cookie_jar({});
$ua->agent('Mozilla/5.0');
$ua->add_handler("request_send", sub { shift->dump; return });
push #{$ua->requests_redirectable}, 'GET';
$page = $ua->get('https://www.foo.com');
I tested the script its unable and checked the Requested Header. It does not have any the below key- value pairs.
Keep-Alive 115
Connection keep-alive
valuable input required.
I believe you just need to specify LWP::UserAgent->new(keep_alive => $maxrequests) to enable keepalive. It will automatically set up the connection cache for you.
I don't see a way to make the number unlimited, though.
Related
Problem: Scrape from tor .onion site using Web::Scraper
I would like to modify my code to connect to .onion site. I believe I need to connect to the SOCKS5 proxy, but unsure of how to do it with Web::Scraper
Existing code:
use Web::Scraper;
my $piratelink=$PIRATEBAYSERVER.'/search/' . $srstring . '%20'. 's'.$sval[1].'e'.$epinum.'/0/7/0';
my $purlToScrape = $piratelink;
my $ns = scraper {
process "td>a", 'mag[]' => '#href';
process "td>div>a", 'tor[]' => '#href';
process "td font.detDesc", 'sizerow[]' => 'TEXT';
};
my $mres = $ns->scrape(URI->new($purlToScrape));
Web::Scraper uses LWP if you pass a URI to scrape.
You can either fetch the HTML using some other HTTP library that uses SOCKS, or using the shared UserAgent variable from Web::Scraper, you can set up LWP to use SOCKS and pass that as the agent.
use strict;
use LWP::UserAgent;
use Web::Scraper;
# set up a LWP object with Tor socks address
my $ua = LWP::UserAgent->new(
agent => q{Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.2.0; .NET CLR 1.1.4322)},
);
$ua->proxy([qw/ http https /] => 'socks://localhost:9050'); # Tor proxy
$ua->cookie_jar({});
my $PIRATEBAYSERVER = 'http://uj3wazyk5u4hnvtk.onion';
my $srstring = 'photoshop';
my $piratelink=$PIRATEBAYSERVER.'/search/' . $srstring; # . '%20'. 's'.$sval[1].'e'.$epinum.'/0/7/0';
my $purlToScrape = $piratelink;
my $ns = scraper {
process "td>a", 'mag[]' => '#href';
process "td>div>a", 'tor[]' => '#href';
process "td font.detDesc", 'sizerow[]' => 'TEXT';
};
# override Scraper's UserAgent with our SOCKS LWP object
$Web::Scraper::UserAgent = $ua;
my $mres = $ns->scrape(URI->new($purlToScrape));
print $mres;
Note, you will also need to install the CPAN module LWP::Protocol::socks
I'm trying to use Asana API with HTTP Basic Auth. The following program prints
{"errors":[{"message":"Not Authorized"}]}
It seems that LWP doesn't send the auth credentials to the server.
#!/usr/bin/perl
use v5.14.0;
use LWP;
my $ua = new LWP::UserAgent;
$ua->credentials('app.asana.com:443', 'realm', 'api_key_goes_here' => '');
my $res = $ua->get("https://app.asana.com/api/1.0/users/me");
say $res->decoded_content;
I've run into something similar (on a completely different service), and couldn't get it working. I think it's to do with a realm/hostname mismatch.
As you note - if you hit that URL directly, from a web browser, you get the same answer (without an auth prompt).
But what I ended up doing instead:
my $request = HTTP::Request -> new ( 'GET' => 'https://path/to/surl' );
$request -> authorization_basic ( 'username', 'password' );
my $results = $user_agent -> request ( $request );
I am working against the salesforce rest api with lwp::useragent.
I have to use the http patch request.
For get and post requests we get use the following code:
require LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
my $get_response = $ua->get('http://search.cpan.org/',x=>'y');
my $post_response = $ua->post('http://search.cpan.org/',x=>'y');
Unfortunately this does not work
my $patch_response = $ua->patch('http://search.cpan.org/',x=>'y');
I don't find how to do it with this module.
There is a workaround to this problem like explained here How do I send a request using the PATCH method for a Salesforce update?
This works but this is not a nice solution.
I saw that with python it is possible to make explicitly patch requests How do I make a PATCH request in Python? so i assume that there is also an option with perl.
my $request = HTTP::Request->new(PATCH => $url);
... Add any necessary headers and body ...
my $response = $ua->request($request);
This has recently got a whole lot easier. PATCH is now implemented (like POST) in HTTP::Message.
First, update the HTTP::Message module (to 6.13 or later).
Then
my %fields = ( title => 'something', body => something else');
my $ua = LWP::UserAgent->new();
my $request = HTTP::Request::Common::PATCH( $url, [ %fields ] );
my $response = $ua->request($request);
I get that error. Here is my code.
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);
use HTTP::Cookies;
$URL="https://example.com/my.plicy";
$UA = LWP::UserAgent->new();
$UA->ssl_opts( verify_hostnames => 0 );
#UA->ssl_opts( SSL_ca_file => Mozilla::CA::SSL_ca_file() );
$req =HTTP::Request::Common::POST("$URL",
Content_type=>'form-data',
Content =>[
'username'=>'111',
'password'=>'2222',
'vhost'=>'standard'
]
);
$req->header('Cookie' =>q(TIN=287000; LastMRH_Session=439960f5; MRHSession=78c9c47291c1fcedae166121439960f5));
$resp=$UA->request($req);
if( ($resp->code() >= 200) && ($resp->code() <400) ) {
print $resp->decoded_content;
}else{
print "Error: ". $resp->status_line. "\n";
}
The problem is that I have no real certificate to provide, because the site is in development stages, and a certificate of the localhost is used... the browsers don't recognize it.
Is there a way to bypass the verification?? and avoid the error?
UPDATE:
I changed my code a bit. Added another library and added this function:
use CACertOrg::CA;
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$UA->ssl_opts(
verify_hostnames => 0,
SSL_ca_file => CACertOrg::CA::SSL_ca_file()
);
Now I get this:
Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
together with SSL_ca_file|SSL_ca_path for verification.
If you really don't want to verify the certificate and keep the
connection open to Man-In-The-Middle attacks please set
SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
at C:/Perl/lib/LWP/Protocol/http.pm line 31.
So I changed the options to this:
$UA->ssl_opts(
SSL_verify_mode => 'SSL_VERIFY_NONE',
verify_hostnames => 0,
SSL_ca_file => CACertOrg::CA::SSL_ca_file()
);
I get nothing printed... I don't get an error, though. Is it a good sign?
You need to add this line after use .. part:
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
Try this
my $UA = LWP::UserAgent->new( ssl_opts => { verify_hostname => 0, } );
Digging up corpses here but I was in the same position and here is what you need to include in order to fix it ;)
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{PERL_LWP_SSL_CA_FILE} = "/path/to/your/cert.pem";
Only setting verify_hostnames to 0 is not enough when you play around with self signed certificates.
None of those solutions worked for me (two and a half years after the last update.. but still it is the top search result so I thought I would add the solution that worked for me) - I needed an additional package to the installation on my Server. It is the LWP::Protocol::https package that I installed via cpan. To install it I typed into the commandline on my server shell:
cpan install LWP::Protocol::https
After the package is installed the code works as it was before without any alterations to the code.
I'm having trouble getting the following code to work and at a point where I am stuck. I am trying to perform client side authentication using a certificate during a POST request. I'm only interested in sending the client cert to the server and don't really need to check the server certificate.
Here is the cUrl command that trying to replicate:
curl --cacert caCertificate.pem --cert clientCerticate.pem -d "string" https://xx.xx.xx.xx:8443/postRf
I keep getting the following error in my Perl script:
ssl handshake failure
I guess I have two questions: what should I be pointing to for CRT7 AND KEY8 variables? and is this the best way to send a POST request using client cert authentication?
!/usr/bin/perl
use warnings;
use strict;
use Net::SSLeay qw(post_https);
my $$hostIp = "xx.xx.xx.xx"
my $hostPort = "8443"
my $postCommand = "/postRf/string";
my $http_method = 'plain/text';
my $path_to_crt7 = 'pathToCert.pem';
my $path_to_key8 = 'pathToKey.pem';
my ($page, $response, %reply_headers) =
post_https($hostIp, $hostPort, $postCommand, '',
$http_method, $path_to_crt7, $path_to_key8
);
print $page . "\n";
print $response . "\n";
See LWP::UserAgent and IO::Socket::SSL.
use strictures;
use LWP::UserAgent qw();
require LWP::Protocol::https;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
SSL_ca_file => 'caCertificate.pem',
SSL_cert_file => 'clientCerticate.pem',
);
$ua->post(
'https://xx.xx.xx.xx:8443/postRf',
Content => 'string',
);
I haven't tested this code.