I am using below code to post JSON data using LWP::useragent. I want to keep my session open and post two requests but it seems that its not working on linux machine (two POST requests are sent in two sessions instead of one).
Any suggestions? thanks in advance
#!/usr/bin/perl
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common;
open (JSON, "json3.txt") or die "$!";
$raw_string1 = do{ local $/ = undef; <JSON>;
};
my $req = HTTP::Request->new(POST => 'http://www.example.com');
$hdr1 = 'User-Agent';
$val1 = 'Java/1.7.0_45';
$hdr2 = 'Connection';
$val2 = 'keep-alive';
$hdr3 = 'Accept';
$val3 = 'application/json, application/*+json';
$hdr4 = 'Host';
$val4 = 'example.com';
$hdr5 = 'Content-Type';
$val5 = 'application/json;charset=UTF-8';
$req -> header($hdr3 => $val3);
$req -> header($hdr5 => $val5);
$req -> header($hdr1 => $val1);
$req -> header($hdr4 => $val4);
$req -> header($hdr2 => $val2);
$req->content_type("application/json");
$req->content("$raw_string1");
my $ua = LWP::UserAgent->new(keep_alive => 1);
$res = $ua->request($req);
print $res->content;
$res = $ua->request($req);
print $res->content;
Keep-Alive is just recommending the server to not close the TCP connection after the request because their will be more requests. The server does not need to follow the recommendation and in fact lots of servers don't to keep the number of open TCP connections low which all use up resources on the system.
Apart from that you should not need to set the Connection and Host header explicitly.
I've tried the following simplified example and a packet capture shows that keep alive is working if the server supports it (LWP 6.05). Supporting means that the server keeps the connection open and does not set a "Connection: close" header and either uses HTTP/1.1 or uses HTTP/1.0 together with "Connection: keep-alive" header.
my $req = HTTP::Request->new(POST => 'http://www.example.com/');
$req->content_type("application/json");
$req->content("foo");
my $ua = LWP::UserAgent->new(keep_alive => 1);
$res = $ua->request($req);
print $res->content;
$res = $ua->request($req);
print $res->content;
its resolved...it wasn't due to backend server closing the connection. i think i was using old perl (5.10) with old fedora version. I spun a new instance of CentOs and its working on it. thanks
Related
I am trying to send multiple HTTP get requests using Perl. I need to send those request using sock proxy.
If I use following code, I am able to send a request with sock proxy
#!/usr/bin/perl
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new(
agent => q{Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.2.0; .NET CLR 1.1.4322)},
);
$ua->proxy([qw/ http https /] => 'socks://localhost:9050'); # Tor proxy
#$ua->cookie_jar({});
$a = 10;
while ( $a < 20 ) {
my $rsp = $ua->get('http://example.com/type?parameter=1¶meter=2');
print $rsp->content;
$a = $a + 1;
}
It works successfully but I need to use AnyEvent to send multiple GET requests in parallel
#!/usr/bin/perl
use strict;
use AnyEvent;
use AnyEvent::HTTP;
use Time::HiRes qw(time);
use LWP::Protocol::socks;
use AnyEvent::Socket;
my $cv = AnyEvent->condvar( cb => sub {
warn "done";
});
my $urls = [
"http://url-1-withallparameters",
"http://url-2-withallparameters",
"http://url-3-withallparameters",
];
my $start = time;
my $result;
$cv->begin(sub { shift->send($result) });
for my $url ( #$urls ) {
$cv->begin;
my $now = time;
my $request;
$request = http_request(
GET => $url,
timeout => 2, # seconds
sub {
my ($body, $hdr) = #_;
if ($hdr->{Status} =~ /^2/) {
push (#$result,
join("\t",
($url,
" has length ",
$hdr->{'content-length'},
" and loaded in ",
time - $now,
"ms"))
);
}
else {
push (#$result,
join("",
"Error for ",
$url,
": (",
$hdr->{Status},
") ",
$hdr->{Reason})
);
}
undef $request;
$cv->end;
}
);
}
$cv->end;
warn "End of loop\n";
my $foo = $cv->recv;
print join("\n", #$foo), "\n" if defined $foo;
print "Total elapsed time: ", time-$start, "ms\n";
This is working fine but I am not able to send these requests using sock proxy.
Even in the terminal, if I export proxy commands like curl and wget they work fine with sock proxy, but when I use a Perl command to execute this script as a sock proxy it does not work.
I could integrate it with LWP::UserAgent but it is not working with AnyEvent.
I have used
proxy => [ $host, $port ],
below
GET => $url,
but it works for HTTP and HTTPS proxy only, not for sock proxy.
This url is working for HTTP/HTTPS proxy but not for sock proxy.
The
documentation for AnyEvent::HTTP has this
Socks proxies are not directly supported by AnyEvent::HTTP
If you read that section it may describe a workaround that suits you
Alternatively, take a look at
AnyEvent::HTTP::Socks which says
This module adds new ‘socks’ option to all http_* functions exported by AnyEvent::HTTP. So you can specify socks proxy for HTTP requests.
Or AnyEvent::HTTP::LWP::UserAgent which should allow you to use ideas from the working LWP::UserAgent code that you already have
I got the the following code that I thought it will not verify server cert, but it still does:
500 Can't connect to 10.0.0.9:443 (certificate verify failed)
Not sure why. Here is the code snippet:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
my $req = HTTP::Request->new(POST => 'https://10.0.0.9/test1234');
$ua->ssl_opts( verify_hostnames => 0 );
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
}
else {
print $res->status_line, "\n";
}
Any ideas? Thanks!
I was following the suggestion from link.
Turned out the "verify_hostnames" should be "verify_hostname".
Getting 500 handshaker error:443 over https. The host service I am sending XML to does not support TLS 1.2, they do support 1.0 and 1.1. Currently using LWP 6.03 on CentOS 6. Using the code below they claim I am still sending using TLS1.2
use LWP::UserAgent;
$ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0,SSL_version => 'SSLv23:!TLSv12' });
$req = HTTP::Request->new(GET => 'https://secure-host-server');
$res = $ua->request($req);
if ($res->is_success) {
print $res->content;
} else {
print "Error: " . $res->status_line . "\n";
}
Is it possible to print the TLS version as it is sent to the host? Anything I can do to verify I am using TLS1.1?
Setting SSL_version via LWP::UserAgent would not work. I tried countless methods to try to get my code to send XML via TLSv1 without luck, The following code did the trick.
use Net::SSLGlue::LWP;
use IO::Socket::SSL;
my $context = new IO::Socket::SSL::SSL_Context(
SSL_version => 'tlsv1',
SSL_verify_mode => Net::SSLeay::VERIFY_NONE(),
);
IO::Socket::SSL::set_default_context($context);
use LWP::UserAgent;
use HTTP::Request::Common;
I am using LWP::UserAgent to check the response from a server. I get a response from port 443 but I am not able to get any response from port 8443.
When I use cURL for Windows I get a response code from both ports.
Please help me.
This example program (adapted from perldoc lwpcook) shows how to connect with a different port
It also allows turning off of the SSL verify, in case you have a home brew cert that is causing a problem
#!/usr/bin/perl
$port = $ARGV[1] || 443;
$host = $ARGV[0] || 'pause.perl.org';
$verify =$ARGV[2] || 0;
use LWP::UserAgent;
$ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => $verify});;
#$ua->agent("$0/0.1 " . $ua->agent);
$ua->agent("Mozilla/8.0"); # pretend we are very capable browser
$req = HTTP::Request->new( GET => "https://$host:$port" );
$req->header( 'Accept' => 'text/html' );
# send request
$res = $ua->request($req);
# check the outcome
if ( $res->is_success ) {
print $res->decoded_content;
}
else {
print "Error: " . $res->status_line . "\n";
}
The following code ...
my $user_agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(GET => $url);
my $response = $user_agent->request($request);
if ($response->is_success) {
print "OK\n";
} else {
die($response->status_line);
}
.. will fail with ..
500 Can't connect to <hostname> (Bad hostname '<hostname>')
.. if the hostname in $url is an IPv6 only address (that is: presence of an AAAA record, but no A record).
My questions are:
How do I enable IPv6 support in LWP?
How do I configure LWP's settings for "prefer-IPv4-over-IPv6" (A vs. AAAA) / "prefer-IPv6-over-IPv4" (AAAA vs. A)?
It looks like you just need to use Net::INET6Glue::INET_is_INET6. To quote its example:
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
print get( 'http://[::1]:80' );
print get( 'http://ipv6.google.com' );
I believe you'll have to change the module to use the IPV6 net module. By default it does not have this enabled: http://eintr.blogspot.com/2009/03/bad-state-of-ipv6-in-perl.html. I don't believe there is something as simple as "prefer-ipv6"
Debian Wheezy (perl 5.14)
Work nice:
use LWP::Simple;
print get( 'http://ip6-localhost:80' );
Not working (1)
use LWP::Simple;
print get( 'http://[::1]:80' );
Not working (2) [Return: Bad hostname]
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Not working (3) [Return: Connection refused]
use Net::INET6Glue::INET_is_INET6;
use LWP::Simple;
$ua = new LWP::UserAgent();
my $req = new HTTP::Request("GET", "http://[::1]/");
my $res = $ua->request($req);
Soo, if you don't need IPv6 address in http request, it's fine. :(