I have code like this
my $ua = new LWP::UserAgent;
$ua->timeout($timeout);
$ua->agent($useragent);
$response = $ua->post($domain,['login_name'=>$login,'login_password'=> $password])->as_string;
Content of page so large, thatI can't receive it. How to get only headers with sending post data?
I think this should do it for you.
my $ua = LWP::UserAgent->new();
$ua->timeout($timeout);
$ua->agent($useragent);
my $response = $ua->post(
$domain,
[ 'login_name' => $login, 'login_password' => $password ]
);
use Data::Dumper;
print Dumper( $response->headers() );
print $response->request()->content(), "\n";
To first, check if you can pass this login_name and login_password via HEAD (in url string: domain/?login_name=...&login_password=...). If this will not work, you are in bad case.
You cannot use POST with behavior of HEAD. LWP will wait full response.
Using POST the server will send you the content anyway, but you can avoid receiving all content using sockets tcp by yourself: gethostbyname, connect, sysread until you get /\r?\n\r?\n/ and close socket after this. Some traffic will be utilized anyway, but you can save memory and receive time.
Its not normal thing to do this with sockets, but sometimes when you have highload/big data - there is no better way than such mess.
Related
I want to send xml request using HTTP streaming protocol . where transfer-encoding is "chunked". Currently i am using LWP::UserAgent to send the xml transaction.
my $userAgent = LWP::UserAgent->new;
my $starttime = time();
my $response = $userAgent->request(POST $url,
Content_Type => 'application/xml',
Transfer_Encoding => 'Chunked',
Content => $xml);
print "Response".Dumper($response);
But i am getting http status code 411 Length Required. Which means "client error response code indicates that the server refuses to accept the request without a defined "
How we can handle this while sending a request in chunked ?
LWP::UserAgent's API isn't designed to send a stream, but it is able to do so with minimal hacking.
use strict;
use warnings qw( all );
use HTTP::Request::Common qw( POST );
use LWP::UserAgent qw( );
my $ua = LWP::UserAgent->new();
# Don't provide any content.
my $request = POST('http://stackoverflow.org/junk',
Content_Type => 'application/xml',
);
# POST() insists on adding a Content-Length header.
# We need to remove it to get a chunked request.
$request->headers->remove_header('Content-Length');
# Here's where we provide the stream generator.
my $buffer = 'abc\n';
$request->content(sub {
return undef if !length($buffer); # Return undef when done.
return substr($buffer, 0, length($buffer), ''); # Return a chunk of data otherwise.
});
my $response = $ua->request($request);
print($response->status_line);
Using a proxy (Fiddler), we can see this does indeed send a chunked request:
There's no point in using a chunked request if you already have the entire document handy like in the example you gave. Instead, let's say wanted to upload the output of some external tool as it produced its output. To do that, you could use the following:
open(my $pipe, '-|:raw', 'some_tool');
$request->content(sub {
my $rv = sysread($pipe, my $buf, 64*1024);
die $! if !defined($rv);
return undef if !$rv;
return $buf;
});
But i am getting http status code 411 Length Required.
Not all servers understand a request with a chunked payload even though this is standardized in HTTP/1.1 (but not in HTTP/1.0). For example nginx only supports chunking within a request since version 1.3.9 (2012), see Is there a way to avoid nginx 411 Content-Length required errors?. If the server does not understand a request with chunked encoding there is nothing you can do from the client side, i.e. you simply cannot use chunked transfer encoding then. If you have control over the server make sure that the server actually supports it.
I've also never experienced browsers send such requests, probably since they cannot guarantee that the server will support such request. I've only seen mobile apps used where the server and app is managed by the same party and thus support for chunked requests can be guaranteed.
I'm trying to retrieve interface statistics data via HTTPS GET from a remote network device. The script is working, however when a response is larger than 4098 characters the connection hangs for 60 seconds. Please help ! I'm slowly going crazy.
my $interface_getAll = HTTP::Request->new( GET => $URL );
$interface_getAll->content( 'session_id=' . $sessionID . '&format=json&method=network.interface.fetchAllStatistics' );
my $res = $ua->request($interface_getAll);
$interface_getAll_rData = $res->content;
my $InterfaceValues = decode_json($interface_getAll_rData);
print Dumper($InterfaceValues);
I can't replicate this issue when using curl or wget, I get a response instantly. The API on the remote device responds to GET requests only. If you think that my code is invalid I'm open to any suggestions.
request URL example/format:
https://192.168.99.51:443/services/rest/V2.1/session_id=b121c16aa0c0361e9bebe5bd67e60a&format=json&method=network.interface.fetchStatistics
You say you need to fetch url
https://192.168.99.51:443/services/rest/V2.1/session_id=b121c16aa0c0361e9bebe5bd67e60a&format=json&method=network.interface.fetchStatistics
but your code is fetching url
https://192.168.99.51:443/services/rest/V2.1/
The following is how you fetch the correct url:
use URI;
my $url = URI->new("https://192.168.99.51:443/services/rest/V2.1/");
$url->query_form(
session_id => $sessionID,
format => 'json',
method => 'network.interface.fetchAllStatistics',
);
my $response = $ua->get($url);
Using $url->query_form ensures your values are properly encoded.
The API on the remote device responds to GET requests only.
Using a message body with a GET request has no defined behavior. In fact RFC 7231 clearly says
A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.
Thus if your API really requires a GET with a message body you should consider the API broken.
But given the URL you show I would rather suggest that the API does not require a message body but that you simply use LWP in the wrong way.
According to your example URL the session_id=... is not inside the message body but actually part of the URL and should be added to this, i.e.
$URL = "https://192.168.99.51:443/services/rest/V2.1/session_id=...."
And while you could just append it directly to the URL a more clean and readable way would be to construct your URL with URI as suggested by ikegami#:
use URI;
my $url = URI->new("https://192.168.99.51:443/services/rest/V2.1/");
$url->query_form(
session_id => $sessionID,
format => 'json',
method => 'network.interface.fetchAllStatistics',
);
my $interface_getAll = HTTP::Request->new( GET => $url );
# no setting content() here
In my application, I fetch webpages periodically using LWP. Is there anyway to check whether between two consecutive fetches the webpage has got changed in some respect (other than explicitly doing a comparison) ? Is there any signature(say CRC) that is being generated at lower protocol layers which can be extracted and compared against older signatures to see possible changes ?
There are two possible approaches. One is to use a digest of the page, e.g.
use strict;
use warnings;
use Digest::MD5 'md5_hex';
use LWP::UserAgent;
# fetch the page, etc.
my $digest = md5_hex $response->decoded_content;
if ( $digest ne $saved_digest ) {
# the page has changed.
}
Another option is to use an HTTP ETag, if the server provides one for the resource requested. You can simply store it and then set your request headers to include an If-None-Match field on subsequent requests. If the server ETag has remained the same, you'll get a 304 Not Modified status and an empty response body. Otherwise you'll get the new page. (And new ETag.) See Entity Tags in RFC2616.
Of course, the server could be lying, and sending the same ETag even though the content has changed. There's no way to know unless you look.
You should use the If-Modified-Since request header, noting the gotchas in the RFC. You send this header with the request. If the server supports it and thinks the content is newer, it sends it to you. If it thinks you have the most recent version, it returns a 304 with no message body.
However, as other answers have noted, the server doesn't have to tell you the truth, so you're sometimes stuck downloading the content and checking yourself. Many dynamic things will always claim to have new content because many developers have never thought about supporting basic HTTP things in their web apps.
For the LWP bits, you can create a single request with an extra header:
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => $url );
$r->header( 'If-Modified-Since' => $time );
$ua->request( $request );
For all requests, you can set a request handler:
$ua->add_handler(
request_send => sub {
my($request, $ua, $h) = #_;
# ... look up time from local store
$r->header( 'If-Modified-Since' => $time );
}
);
However, LWP can do most of this for you with mirror if you want to save the files:
$ua->mirror( $url, $filename )
I'm querying a webserver for a document and I want to capture the both the document and the related server response headers (esp. Content-Type: ...). I have trouble finding out how to read the headers. Here are some sniplets from my Perl script, I left error checking out for clarity:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent( 'requiredCustomUserAgent' ); # I'm required to set a custom user agent
$imageData = $response->content; # This is the received document
So at this point I can retrieve the web document, but I want to know what Content-Type the server sent with it. Unfortunately this is not always the same as the mime type found by the bash 'file' command. This file-method fails in case of .js or .css documents.
http://search.cpan.org/perldoc?HTTP::Response
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $response = $ua->get("http://google.co.uk");
print $response->headers()->as_string;
print $response->header('content-type');
the thing that request returns contains a HTTP::Headers object, so look at the docs for HTTP::Headers to see how to use it. For instance
my $response = $ua->request($req);
my $headers = $response->headers();
my #header_field_names = $headers->header_field_names();
$logger->info("$_: ".$headers->header($_)) for grep {/Hogwarts/} #header_field_names;
Suppose you're given an URL, http://site.com . How do you find out what it's content type is without downloading it? Can Perl's WWW::Mechanize or LWP issue a HEAD request?
You can use head() method of LWP in following manner
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->head('<url>');
Here's a full example:
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
my $response = $ua->head( 'http://www.perl.com' );
my $type = $response->content_type;
print "The type is $type\n";
Some servers choke on HEAD requests, so when I do this and get an error of any sort, I retry it with a GET request and only request the first couple hundred of bytes of the resources.