i'm using Perl to access a Rest-Api:
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request::Common::PUT("http://xxx:yyy/...");
$req->header('content-type' => 'application/json');
$req->authorization_basic('abc','xyz');
my $put_data = '{
"description" : "TestPut"
}';
$req->content($put_data);
my $resp = $ua->request($req);
if ($resp->is_success){
print $resp->content() . "\n";
}
else{
print "PUT failed:\n";
print $resp->message . "\n";
}
But i am getting a "Method not allowed" Message.
The GET works fine.
Could this be a Problem by the Http-Server (Tomcat) or a firewall?
$req->as_string:
PUT #URL
Authorization: Basic xxx=
Content-Type: application/json
{
"description" : "TestPut"
}
The GET works fine. Could this be a Problem by the Http-Server (Tomcat) or a firewall?
Yes, you have to look there. GET and POST are the usual methods to access a web site while PUT is usually used for REST or WebDAV and not used by the web browser (unless you do your own XHR requests). Thus it might be that a firewall or HTTP server restricts access to this method.
Related
I'm trying to call the KeyForge API with a simple Perl program but it doesn't work. I'm using what's in the LWP::UserAgent documentation:
use strict;
use warnings;
use LWP::UserAgent ();
my $ua = LWP::UserAgent->new;
my $response = $ua->get('https://www.keyforgegame.com/api/decks/');
if ($response->is_success) {
print $response->decoded_content;
}
else {
die $response->status_line;
}
The program prints:
500 write failed: at test.pl line 16.
If I use the URL https://www.google.com or http://www.example.com, it works. The HTML is correctly displayed.
If I use this simple PowerShell program, it works too:
$Url = "https://www.keyforgegame.com/api/decks/"
$decks = Invoke-RestMethod ($url)
$decks
It displays:
count data
743719 {#{name=Dr. "The Old" Jeffries; expansion=341; power_level=0; chains=0; wins=0; losses=0; id=ec86db52-e41e-4e...
What am I missing?
PS: I'm using Perl 5.16.3 on Windows 10.
EDIT:
Thank you all for your help. I finally found out what was happening. It turns out I had a very old version of Net::HTTP (from 2013). I upgraded it and now it works out of the box, without configuring agent, cookies or e-mail. The error message I had was actually from the client and not from the server.
$ perl -MLWP::UserAgent -e'
my $ua = LWP::UserAgent->new();
my $response = $ua->get("https://www.keyforgegame.com/api/decks/");
print $response->as_string;
'
HTTP/1.1 403 Forbidden
...
Content-Type: text/html; charset=UTF-8
...
<!DOCTYPE html>
...
<title>Access denied | www.keyforgegame.com used Cloudflare to restrict access</title>
...
<h2 data-translate="what_happened">What happened?</h2>
<p>The owner of this website (www.keyforgegame.com) has banned your access based on your browser's signature (4bfe0c0e2e86ab84-ua22).</p>
...
But,
$ perl -MLWP::UserAgent -e'
use version; our $VERSION = qv("v1.0.0");
my $ua = LWP::UserAgent->new(
agent => "NameOfTool/$VERSION",
from => q{me#example.com},
);
my $response = $ua->get("https://www.keyforgegame.com/api/decks/");
print $response->as_string;
'
HTTP/1.1 200 OK
...
Content-Type: application/json
...
{"count":...
If they want to block you, they can. So it's your best interest to provide a unique application name, a proper version and a valid email address (even if providing junk for the agent and leaving out from field works). This gives them more options to resolve any issues they have with your program.
I have a simple code like this:
use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Request;
my $cookies = HTTP::Cookies->new();
my $browser = LWP::UserAgent->new();
$browser->agent(' ... ');
$browser->cookie_jar($cookies);
my $request = HTTP::Request->new();
my $response;
my $url;
my $referer;
$referer = '';
$url = 'https:// ...'; # url #1
$request->url($url);
$request->method('GET');
$request->header('Referer' => $referer);
$response = $browser->request($request);
print $response->request()->uri() . "\n\n" .
$response->headers()->as_string . "\n\n" .
$response->content . "\n\n";
$referer = $response->request()->uri();
$url = 'https:// ... '; # url #2
$request->url($url);
$request->method('GET');
$request->header('Referer' => $referer);
$response = $browser->request($request);
print $response->request()->uri() . "\n\n" .
$response->headers()->as_string . "\n\n" .
$response->content . "\n\n";
Now, I want to see full HTTP request headers as well, not just response headers.
How can I do it? What has to be added to this code?
I think you almost have it in your existing code. You are accessing the request URI with $response->request()->uri(). The ->request() is your HTTP::Request object. I believe that you can use $response->request->headers->as_string to get what you want.
print $response->request->as_string
This will show you requests as well as responses.
use LWP::UserAgent;
use LWP::ConsoleLogger::Easy qw( debug_ua );
my $browser = LWP::UserAgent->new();
debug_ua( $browser );
$request->headers->as_string and $response->request->headers->as_string will you get you the headers of the first and last request passed to Net::HTTP by LWP[1], but these aren't quite what Net::HTTP sends. For example, Net::HTTP can add a Content-Length header, a TE header, and/or a number of others.
Net::HTTP doesn't keep a record of the headers it actually sends. You will need a wire sniffer (e.g. tcpdump) or a debugging proxy (e.g. Fiddler) for that. You could also use a debugger or trace statements to view the request prepared in Net::HTTP::Methods's format_request. The most convenient, however, might be to wrap Net::HTTP::Methods's format_request.
These are the same unless the initial request was redirected. To get all the requests (and responses), you can use:
while ($response) {
my $request = $response->request;
...
$response = $response->previous;
}
See: https://developer.spotify.com/web-api/authorization-guide/
I'm using the "client credentials flow" method.
sub get_token {
my $req = HTTP::Request->new(POST => $SPOTIFY_TOKEN);
$req->header('Authorization' => 'Basic MYBASE64HERE');
my $post_data = 'grant_type=client_credentials';
$req->content($post_data);
my $resp = $ua->request($req); #this is LWP
if ($resp->is_success) {
my $token = $resp->decoded_content;
print "$token\n";
return \$token;
}
else {
print "HTTP POST error code: ", $resp->code, "\n";
print "HTTP POST error message: ", $resp->message, "\n";
}
}
I get back HTTP POST error code: 400 / bad request
I know that it doesn't have to do with the header information or the URL. I tested via Curl and used Data::Dumper to verify it was formatted properly.
I'm not sure on the format I need to send the POST body data. I've tried the example above my $post_data = 'grant_type=client_credentials'; as well as every variation I could think of. Is there a proper way to do this in Perl using HTPP::Request to build the POST request?
I think following should work, Please try:
$req->content(grant_type => 'client_credentials');
my $post_data = "grant_type=client_credentials";
Turns out this is the answer. I'm not sure how I missed this previously.
When I try to submit a POST request with Perl, it often ends in a 301 redirect to the homepage. Here is the code :
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
# This does not work
my $url = 'http://www.opensubtitles.org/en/search2';
my $req = HTTP::Request->new(POST => $url);
$req->content('MovieName=the+terminator+(1996)');
# Pass request to the user agent and get a response back
print $req->as_string."\n";;
my $res = $ua->request($req);
if (!$res->is_success) {
print $res->status_line, "\n";
}
else {
print "Success in posting search\n";
}
In order to make it work, I have to manually use Firefox, go to the url (!). Then the script works. However, using a GET request works flawlessly :
# This works
my $url = 'http://www.opensubtitles.org/en/search2?MovieName=the+terminator+(1996)';
my $req = HTTP::Request->new(GET => $url);
Why is that ?
The site doesn't expect a POST to that URL, so it redirects you to back to the search page.
Firefox will use GET, not POST, if you just put the URL into the address line, that's why it works.
If I have a URL (eg. http://www.foo.com/alink.pl?page=2), I want to determine if I am being redirected to another link. I'd also like to know the final URL (eg. http://www.foo.com/other_link.pl). Finally, I want to be able to do this in Perl and Groovy.
In Perl:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => 'http://google.com/' );
my $response = $ua->request($request);
if ( $response->is_success and $response->previous ) {
print $request->url, ' redirected to ', $response->request->uri, "\n";
}
Well, I know nothing about either Perl or groovy, so I'll give you an another from an HTTP point of view, and you'll have to adapt.
Normally, you make an HTTP request, and you get back some HTML text along with a response code. The response code for Success is 200. Any response code in the 300 range is some form of a redirect.
Referring to James's answer - sample HTTP session:
$ telnet www.google.com 80
HEAD / HTTP/1.1
HOST: www.google.com
HTTP/1.1 302 Found
Location: http://www.google.it/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: ##############################
Date: Thu, 30 Oct 2008 20:03:36 GMT
Server: ####
Content-Length: 218
Using HEAD instead of GET you get only the header. "302" means a temporary redirection, "Location:" is where you are redirected to.
A quick & dirty groovy script to show the concepts -- Note, this is using java.net.HttpURLConnection
In order to detect the redirect, you have to use setFollowRedirects(false). Otherwise, you end up on the redirected page anyway with a responseCode of 200. The downside is you then have to navigate the redirect yourself.
URL url = new URL ('http://google.com')
HttpURLConnection conn = url.openConnection()
conn.followRedirects = false
conn.requestMethod = 'HEAD'
println conn.responseCode
// Not ideal - should check response code too
if (conn.headerFields.'Location') {
println conn.headerFields.'Location'
}
301
["http://www.google.com/"]
In Perl you can use LWP::Useragent for that. I guess the easiest way is to add a response_redirect handler using add_handler.
I think this will work for 301 redirects.
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => 'http://google.com/' );
my $response = $ua->request($request);
if ( $response->is_redirect ) {
print $request->url . " redirected to location " . $response->header('Location') . "\n";
}