I'm trying to call the KeyForge API with a simple Perl program but it doesn't work. I'm using what's in the LWP::UserAgent documentation:
use strict;
use warnings;
use LWP::UserAgent ();
my $ua = LWP::UserAgent->new;
my $response = $ua->get('https://www.keyforgegame.com/api/decks/');
if ($response->is_success) {
print $response->decoded_content;
}
else {
die $response->status_line;
}
The program prints:
500 write failed: at test.pl line 16.
If I use the URL https://www.google.com or http://www.example.com, it works. The HTML is correctly displayed.
If I use this simple PowerShell program, it works too:
$Url = "https://www.keyforgegame.com/api/decks/"
$decks = Invoke-RestMethod ($url)
$decks
It displays:
count data
743719 {#{name=Dr. "The Old" Jeffries; expansion=341; power_level=0; chains=0; wins=0; losses=0; id=ec86db52-e41e-4e...
What am I missing?
PS: I'm using Perl 5.16.3 on Windows 10.
EDIT:
Thank you all for your help. I finally found out what was happening. It turns out I had a very old version of Net::HTTP (from 2013). I upgraded it and now it works out of the box, without configuring agent, cookies or e-mail. The error message I had was actually from the client and not from the server.
$ perl -MLWP::UserAgent -e'
my $ua = LWP::UserAgent->new();
my $response = $ua->get("https://www.keyforgegame.com/api/decks/");
print $response->as_string;
'
HTTP/1.1 403 Forbidden
...
Content-Type: text/html; charset=UTF-8
...
<!DOCTYPE html>
...
<title>Access denied | www.keyforgegame.com used Cloudflare to restrict access</title>
...
<h2 data-translate="what_happened">What happened?</h2>
<p>The owner of this website (www.keyforgegame.com) has banned your access based on your browser's signature (4bfe0c0e2e86ab84-ua22).</p>
...
But,
$ perl -MLWP::UserAgent -e'
use version; our $VERSION = qv("v1.0.0");
my $ua = LWP::UserAgent->new(
agent => "NameOfTool/$VERSION",
from => q{me#example.com},
);
my $response = $ua->get("https://www.keyforgegame.com/api/decks/");
print $response->as_string;
'
HTTP/1.1 200 OK
...
Content-Type: application/json
...
{"count":...
If they want to block you, they can. So it's your best interest to provide a unique application name, a proper version and a valid email address (even if providing junk for the agent and leaving out from field works). This gives them more options to resolve any issues they have with your program.
Related
So I am going over the Attachment API for ServiceNow, documented here:
https://docs.servicenow.com/integrate/inbound_rest/reference/r_AttachmentAPI-POST.html
For an application I'm working on, I need to write up some Perl code to handle attachments. Unfortunately, the ServiceNow Perl API libraries do not handle attachments larger than 5mb, so I need to use the stock Attachment API that comes with the instance.
From the above link, I saw this example on how to post files with this python code.
#Need to install requests package for python
#easy_install requests
import requests
# Set the request parameters
url = 'https://instance.service-now.com/api/now/attachment/file?table_name=incident&table_sys_id=d71f7935c0a8016700802b64c67c11c6&file_name=Issue_screenshot.jpg'
# Specify the file To send. When specifying fles to send make sure you specify the path to the file, in
# this example the file was located in the same directory as the python script being executed.
data = open('Issue_screenshot.jpg', 'rb').read()
# Eg. User name="admin", Password="admin" for this code sample.
user = 'admin'
pwd = 'admin'
# Set proper headers
headers = {"Content-Type":"image/jpeg","Accept":"application/json"}
# Do the HTTP request
response = requests.post(url, auth=(user, pwd), headers=headers, data=data)
# Check for HTTP codes other than 201
if response.status_code != 201:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)
I've used REST::Client a lot for posting, but unfortunately, I can't find a good example on how to handle the above ^^ but in Perl. How does one use REST::Client to post a file like above?
I've done a temp workaround with this by invoking curl in my scripts, but using REST::Client would be more elegant.
You can use LWP::UserAgent Perl module to achieve the same:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request;
use Fcntl;
use JSON qw[decode_json];
use Data::Dumper;
my $ua = LWP::UserAgent->new;
my $url = 'https://instance.service-now.com/api/now/attachment/file?table_name=incident&table_sys_id=d71f7935c0a8016700802b64c67c11c6&file_name=Issue_screenshot.jpg';
my $user = 'admin';
my $pwd = 'admin';
$ua->credentials( 'instance.service-now.com', '<REALM>', $user, $pwd);
my $file = 'Issue_screenshot.jpg';
my $request = HTTP::Request->new( POST => $url );
$request->header( 'Content-Type' => 'image/jpeg');
$request->header( 'Accept' => 'application/json');
$request->header( 'Content-Length' => -s $file);
sysopen(my $fh,$file,O_RDONLY);
$request->content( sub {
sysread($fh,my $buf,1024);
return $buf;
});
my $res = $ua->request($request);
unless($res->code == 201) {
print 'Status: '.$res->code, 'Headers:',$res->headers_as_string,'Error Response:',$res->content;
exit;
}
my $data = decode_json($res->content);
print Dumper($data);
i'm using Perl to access a Rest-Api:
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request::Common::PUT("http://xxx:yyy/...");
$req->header('content-type' => 'application/json');
$req->authorization_basic('abc','xyz');
my $put_data = '{
"description" : "TestPut"
}';
$req->content($put_data);
my $resp = $ua->request($req);
if ($resp->is_success){
print $resp->content() . "\n";
}
else{
print "PUT failed:\n";
print $resp->message . "\n";
}
But i am getting a "Method not allowed" Message.
The GET works fine.
Could this be a Problem by the Http-Server (Tomcat) or a firewall?
$req->as_string:
PUT #URL
Authorization: Basic xxx=
Content-Type: application/json
{
"description" : "TestPut"
}
The GET works fine. Could this be a Problem by the Http-Server (Tomcat) or a firewall?
Yes, you have to look there. GET and POST are the usual methods to access a web site while PUT is usually used for REST or WebDAV and not used by the web browser (unless you do your own XHR requests). Thus it might be that a firewall or HTTP server restricts access to this method.
I am new to perl and any help will be appreciated!!
I have to invoke some URLs through perl (On unix machine).URLs are both http and https
if URL gets invoked successfully,then its fine else create a log file stating that unable to invoke a URL.
For invoking the URL,I am thinking to use for e.g.
exec 'firefox http://www.yahoo.com';
But how to get http and https request status code? Something like if status is 200,then ok else error..
Kindly help!!
Rather than using a browser such a Firefox you should use an HTTP client library such as HTTP::Tiny or LWP::UserAgent.
For exmaple:
#!/usr/bin/env perl
use strict;
use warnings;
use feature 'say';
use HTTP::Tiny;
my $Client = HTTP::Tiny->new();
my #urls = (
'http://www.yahoo.com',
'https://www.google.com',
'http://nosuchsiteexists.com',
);
for my $url (#urls) {
my $response = $Client->get($url);
say $url, ": ", $response->{status};
}
Which outputs:
alex#yuzu:~$ ./return_status.pl
http://www.yahoo.com: 200
https://www.google.com: 200
http://nosuchsiteexists.com: 599
If you want to correctly recognise redirect status codes (3XX) you would have to set the max_redirect parameter to 0.
alex#yuzu:~$ perl -MHTTP::Tiny -E 'say HTTP::Tiny->new(max_redirect => 0)->get("http://www.nestoria.co.uk/soho")->{status};'
301
If all you care about is success then the response hashref contains a 'success' field which will be true on success and false on failure.
alex#yuzu:~$ perl -MHTTP::Tiny -E 'say HTTP::Tiny->new()->get("http://www.google.com")->{success};'
1
I'm talking to what seems to be a broken HTTP daemon and I need to make a GET request that includes a pipe | character in the URL.
LWP::UserAgent escapes the pipe character before the request is sent.
For example, a URL passed in as:
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1|01
is passed to the HTTP daemon as
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1%7C01
This is correct, but doesn't work with this broken server.
How can I override or bypass the encoding that LWP and its friends are doing?
Note
I've seen and tried other answers here on StackOverflow addressing similar problems. The difference here seems to be that those answers are dealing with POST requests where the formfield parts of the URL can be passed as an array of key/value pairs or as a 'Content' => $content parameter. Those approaches aren't working for me with an LWP request.
I've also tried constructing an HTTP::Request object and passing that to LWP, and passing the full URL direct to LWP->get(). No dice with either approach.
In response to Borodin's request, this is a sanitised version of the code I'm using
#!/usr/local/bin/perl -w
use HTTP::Cookies;
use LWP;
my $debug = 1;
# make a 'browser' object
my $browser = LWP::UserAgent->new();
# cookie handling...
$browser->cookie_jar(HTTP::Cookies->new(
'file' => '.cookie_jar.txt',
'autosave' => 1,
'ignore_discard' => 1,
));
# proxy, so we can watch...
if ($debug == 1) {
$browser->proxy(['http', 'ftp', 'https'], 'http://localhost:8080/');
}
# user agent string (pretend to be Firefox)
$agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.7.12) Gecko/20050919 Firefox/1.0.7';
# set the user agent
$browser->agent($agent);
# do some things here to log in to the web site, accept session cookies, etc.
# These are basic POSTs of filled forms. Works fine.
# [...]
my $baseURL = 'https://hostname/url/doSomethingScript?ss=1234&activities=VALUEA|VALUEB';
#values = ['Lec1', '01', 'Lec1', '02'];
while (1) {
if (scalar(#values) < 2) { last; }
my $vala = shift(#values);
my $valb = shift(#values);
my $url = $basEURL;
$url =~ s/VALUEA/$vala/g;
$url =~ s/VALUEB/$valb/g;
# simplified. Would usually check request for '200' response, etc...
$content = $browser->get($url)->content();
# do something here with the content
# [...]
# fails because the '|' character in the url is escaped after it's handed
# to LWP
}
# end
As #bchgys mentions in his comment, this is (almost) answered in the linked thread. Here are two solutions:
The first and arguably cleanest one is to locally override the escape map in URI::Escape to not modify the pipe character:
use URI;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $res;
{
# Violate RFC 2396 by forcing broken query string
# local makes the override take effect only in the current code block
local $URI::Escape::escapes{'|'} = '|';
$res = $ua->get('http://server/script?q=a|b');
}
print $res->request->as_string, "\n";
Alternatively, you can simply undo the escaping by modifying the URI directly in the request after the request has been created:
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'http://server/script?q=a|b');
# Violate RFC 2396 by forcing broken query string
${$req->uri} =~ s/%7C/|/;
my $res = $ua->request($req);
print $res->request->as_string, "\n";
The first solution is almost certainly preferable because it at least relies on the %URI::Escape::escapes package variable which is exported and documented, so that's probably as close as you're gonna get to doing this with a supported API.
Note that in either case you are in violation of RFC 2396 but as mentioned you may have no choice when talking to a broken server that you have no control over.
I've used Perl a bit for small applications and test code, but I'm new to networking and CGI.
I get how to make the header of a request (using CGI.pm and printing the results of the header() function), but haven't been able to find any info on how to access the headers being sent to my CGI script. Could someone point me in the right direction?
This could be from a request like this:
curl http://127.0.0.1:80/cgi-bin/headers.cgi -H "HeaderAttribute: value"
The CGI module has a http() function you can use to that purpose:
#!/usr/bin/perl --
use strict;
use warnings;
use CGI;
my $q = CGI->new;
my %headers = map { $_ => $q->http($_) } $q->http();
print $q->header('text/plain');
print "Got the following headers:\n";
for my $header ( keys %headers ) {
print "$header: $headers{$header}\n";
}
Try it out; the above gives me:
$ curl http://localhost/test.cgi -H "HeaderAttribute: value"
Got the following headers:
HTTP_HEADERATTRIBUTE: value
HTTP_ACCEPT: */*
HTTP_HOST: localhost
HTTP_USER_AGENT: curl/7.21.0 (i686-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
In addition to the CGI.pm http() method you can get HTTP headers information from the environment variables.
So in case you are using something like CGI::Minimal, which doesn't have the http method. you can do something like:
my $header = 'HTTP_X_REQUESTED_WITH';
if (exists $ENV{$header} && lc $ENV{$header} eq 'xmlhttprequest') {
_do_some_ajaxian_stuff();
}
They're supplied as environment variables, such as
HTTP_HEADERATTRIBUTE=value
You may have to do something to configure your web server to supply such a variable, though.