I'm trying to check the header of a redirection page, and get the 302 status,
but with my code I get the 200 OK status of the forwarded page. What should I do to get the redirection page 302 staus. My code:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','http://host.com');
my $res = $ua->request($req);
print $res->status_line;
After initializing $ua, set its requests_redirectable property to undef:
$ua->requests_redirectable(undef);
That way LWP::UserAgent will not follow redirects and instead stop after the first request.
Then yoy can get the code( "302", "301", etc) using:
$res->code()
Here's the official docs for LWP::UserAgent.
$response->previous() will be get you the previous response in the chain.
Or if you want to disable redirection, pass requests_redirectable => [] to LWP::UserAgent's constructor.
Related
I'm trying to retrieve interface statistics data via HTTPS GET from a remote network device. The script is working, however when a response is larger than 4098 characters the connection hangs for 60 seconds. Please help ! I'm slowly going crazy.
my $interface_getAll = HTTP::Request->new( GET => $URL );
$interface_getAll->content( 'session_id=' . $sessionID . '&format=json&method=network.interface.fetchAllStatistics' );
my $res = $ua->request($interface_getAll);
$interface_getAll_rData = $res->content;
my $InterfaceValues = decode_json($interface_getAll_rData);
print Dumper($InterfaceValues);
I can't replicate this issue when using curl or wget, I get a response instantly. The API on the remote device responds to GET requests only. If you think that my code is invalid I'm open to any suggestions.
request URL example/format:
https://192.168.99.51:443/services/rest/V2.1/session_id=b121c16aa0c0361e9bebe5bd67e60a&format=json&method=network.interface.fetchStatistics
You say you need to fetch url
https://192.168.99.51:443/services/rest/V2.1/session_id=b121c16aa0c0361e9bebe5bd67e60a&format=json&method=network.interface.fetchStatistics
but your code is fetching url
https://192.168.99.51:443/services/rest/V2.1/
The following is how you fetch the correct url:
use URI;
my $url = URI->new("https://192.168.99.51:443/services/rest/V2.1/");
$url->query_form(
session_id => $sessionID,
format => 'json',
method => 'network.interface.fetchAllStatistics',
);
my $response = $ua->get($url);
Using $url->query_form ensures your values are properly encoded.
The API on the remote device responds to GET requests only.
Using a message body with a GET request has no defined behavior. In fact RFC 7231 clearly says
A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.
Thus if your API really requires a GET with a message body you should consider the API broken.
But given the URL you show I would rather suggest that the API does not require a message body but that you simply use LWP in the wrong way.
According to your example URL the session_id=... is not inside the message body but actually part of the URL and should be added to this, i.e.
$URL = "https://192.168.99.51:443/services/rest/V2.1/session_id=...."
And while you could just append it directly to the URL a more clean and readable way would be to construct your URL with URI as suggested by ikegami#:
use URI;
my $url = URI->new("https://192.168.99.51:443/services/rest/V2.1/");
$url->query_form(
session_id => $sessionID,
format => 'json',
method => 'network.interface.fetchAllStatistics',
);
my $interface_getAll = HTTP::Request->new( GET => $url );
# no setting content() here
I'm playing with WWW::Mechanize, i.e.
my $mech = WWW::Mechanize->new(\%opts);
$mech->get($url);
my $reponse = $mech->follow_link(regex_url => qr/some link/);
$response is returned as an HTTP::Response object. My question is, can I use my $mech to continue to follow links in the response, submit forms, etc? What can I do with the $response object?
The HTTP::Response has everything you want back from the other site:
$response->is_success() will tell you if the request was successful,
$response->code() will return you the HTTP Response Code,
$response->header('Content-Type') will return the Content-Type HTTP Header,
$response->content() will give you the response content,
etc. Check out the perldoc on HTTP::Response for more details.
As for $mech, you can continue to use it for links, etc.
Check out WWW::Mechanize::Examples for some good examples.
I have code like this
my $ua = new LWP::UserAgent;
$ua->timeout($timeout);
$ua->agent($useragent);
$response = $ua->post($domain,['login_name'=>$login,'login_password'=> $password])->as_string;
Content of page so large, thatI can't receive it. How to get only headers with sending post data?
I think this should do it for you.
my $ua = LWP::UserAgent->new();
$ua->timeout($timeout);
$ua->agent($useragent);
my $response = $ua->post(
$domain,
[ 'login_name' => $login, 'login_password' => $password ]
);
use Data::Dumper;
print Dumper( $response->headers() );
print $response->request()->content(), "\n";
To first, check if you can pass this login_name and login_password via HEAD (in url string: domain/?login_name=...&login_password=...). If this will not work, you are in bad case.
You cannot use POST with behavior of HEAD. LWP will wait full response.
Using POST the server will send you the content anyway, but you can avoid receiving all content using sockets tcp by yourself: gethostbyname, connect, sysread until you get /\r?\n\r?\n/ and close socket after this. Some traffic will be utilized anyway, but you can save memory and receive time.
Its not normal thing to do this with sockets, but sometimes when you have highload/big data - there is no better way than such mess.
I'm querying a webserver for a document and I want to capture the both the document and the related server response headers (esp. Content-Type: ...). I have trouble finding out how to read the headers. Here are some sniplets from my Perl script, I left error checking out for clarity:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent( 'requiredCustomUserAgent' ); # I'm required to set a custom user agent
$imageData = $response->content; # This is the received document
So at this point I can retrieve the web document, but I want to know what Content-Type the server sent with it. Unfortunately this is not always the same as the mime type found by the bash 'file' command. This file-method fails in case of .js or .css documents.
http://search.cpan.org/perldoc?HTTP::Response
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $response = $ua->get("http://google.co.uk");
print $response->headers()->as_string;
print $response->header('content-type');
the thing that request returns contains a HTTP::Headers object, so look at the docs for HTTP::Headers to see how to use it. For instance
my $response = $ua->request($req);
my $headers = $response->headers();
my #header_field_names = $headers->header_field_names();
$logger->info("$_: ".$headers->header($_)) for grep {/Hogwarts/} #header_field_names;
Lets say I have "http://www.ritzcarlton.com" and that redirects me to "http://www.ritzcarlton.com/en/Default.htm". Is there a way in Perl to find the end url after all the redirects?
Using LWP will follow the redirections for you. You can then interrogate the HTTP::Request object to find out the URI it requested.
use LWP::UserAgent qw();
my $ua = LWP::UserAgent->new;
my $response = $ua->get('http://www.ritzcarlton.com');
print $response->request->uri . "\n";
Output is:
http://www.ritzcarlton.com/en/Default.htm
If you're issuing HTTP requests yourself, then the redirect URL will be in the returned Location: header. If you're using a proper HTTP client like LWP::UserAgent or WWW::Mechanize, which is what you should be doing, redirection is handled automatically.