I am trying to run a simple API request from a perl script. But it seems not working. Same request if copied to web browser works without any problem.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $query = 'http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9944384761';
my $result = get($query);
print $result."\n";
when I use getprint($query) - it gives 403 error.
If you take a look at the body of the response (i.e. not only at the status code 403) you will find:
The owner of this website (checkdnd.com) has banned your access based on your browser's signature (2f988642c0f02798-ua22).
This means that it is blocking the client because it probably looks too much like a non-browser. For this site a simple fix is to include some User-Agent header. The following works for me:
my $ua = LWP::UserAgent->new;
$ua->default_header('User-Agent' => 'Mozilla/5.0');
my $resp = $ua->get('http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9944384761');
my $result = $resp->decoded_content;
The site in question seems to be served by Cloudflare which has some thing they call "Browser Integrity Check". From the support page for this feature:
... looks for common HTTP headers abused most commonly by spammers and denies access to your page. It will also challenge visitors that do not have a user agent or a non standard user agent (also commonly used by abuse bots, crawlers or visitors).
Related
The HTTP Request I used to send to a specific website is now getting redirected which eventually broke my code. I realized that the cookies are not working anymore for the redirected domain (of course). I read the docs of LWP but I did not find any related option to preserve/maintain cookies automatically. Is there an easy way to do it?
Just for a side note: this behavior works out of the box using Python's Requests class.
The following adds support for cookies to LWP::UserAgent.
my $ua = LWP::UserAgent->new( cookie_jar => {} );
It causes cookies returned in a response to be sent with subsequent matching requests, just like a browser does.
I have a situation where i need to check for certain conditions of an Internal web application.
First i need to check if the application is loading or not. -- For this i have used Perl Mechanize module and using get method to load
the URl. The problem which i am facing was it was showing 401
unauthorized and if i send the username and password as parameters to
function "credentials" it works fine.
I Just want to check if the webpage is loading or not without entering the credentials? Printing a message if it loads looks fine.
You can do a direct request with LWP and check the return code. If it is 401 you know that the server was responding. If this also means that your application is working depends on who is responsible for checking the authorization.
use LWP::UserAgent;
my $resp = LWP::UserAgent->new->get('http://example.com');
if ($resp->code == 401) ...
My work requires an authorization for internet use. I log in, and after that it recognizes me and lets me access whatever I need.
I have been using POSTMAN to test send to and receive from a company RESTful service. It automatically uses my same internet use auth at the other end to give my user account POST and GET permissions.
Now, I am trying to automate with a perl script and it won't authorize. The owner of the RESTful service says if I make a windows/.net application it will authorize automatically, but that isn't an option.
Any suggestions? I would think I could just do special headers or something and duplicate whatever windows is doing....
I have been asked to provide what I have done so far
#!/usr/local/bin/perl
use strict;
use LWP::UserAgent;
my $ua=LWP::UserAgent->new;
my $server_endpoint = "The post destination";
my $req= HTTP::Request->new(POST => $server_endpoint);
$req->header('content-type' => 'application/json');
my $post_data="[ SOME JSON HERE ]";
$req->content($post_data);
my $resp = $ua->request($req);
if($resp->is_success){
my $message = $resp->decoded_content;
print "received reply : $message\n";
}
else{
print "post error code : ",$resp->code,"\n";
print "post error message : ",$resp->message,"\n";
}
In the past when I had to authenticate against an IIS server I had to use LWP::Authen::Ntlm to get it to authenticate.
For more information about LWP::Authen::Ntlm, see https://metacpan.org/pod/LWP::Authen::Ntlm
The main "pitfalls" I had is that keepalive is required, and that newer versions of IIS now use Digest, and not NTLM
In those cases, I simply switched to the built-in LWP::Authen::Digest (it comes inside LWP)
Have a look at a similar question (scroll up to the top see the question) and see if the included bit of Perl code doesn't help...
LWP::UserAgent HTTP Basic Authentication
The short version is that it doesn't appear that your Perl code above includes any login information and this POSTMAN plugin may be sending over cached login info that your Perl code is not yet aware of.
On a certain project in Perl, I've written several "parsers", which allow me to visit websites with LWP::UserAgent. However, I'm having a problem with one website: it's behaving exactly as if I had visited the site with my browser, having turned off Cookies, so instead of giving me the page I want, it gives me a page with the message that I must turn on cookies. The entire code of my script is below. Any ideas? Thanks in advance.
(Note that I looked at the following url, which seems to be addressing my question, but unfortunately, I was unable to get a working script based on its suggestion: Cookies in perl lwp.)
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Cookies;
my $useragent = LWP::UserAgent->new;
$useragent->cookie_jar(HTTP::Cookies->new);
my $request = HTTP::Request->new(GET => "http://www.the-site-im-trying-to-parse.com");
my $response = $useragent->request($request);
print "Content-type: text/html\n\n";
print $response->as_string;
Have you considered using WWW::Mechanize module? It does collect cookies automatically by default. And it's a bit easier to use since there are plenty of included methods which are very useful.
All you are doing is downloading the html data over HTTP, so there is no browser interaction until you decide to view the result in one. That being said, the HTTP server has no way of knowing if your request is from a client that has cookies enabled. So doing so won't actually do anything to change the result.
The WWW:Mechanize module is useful for easily traversing web sites, but it won't fix the problem you are facing. So it won't actually help you resolve the issue you are having.
More realistically what it is going on is there is some sort of client-side javascript code that isn't working correctly once you download the file and display it in your browser. This could be any number of things, such as breaking the cross-domain policy implemented in the javascript code. Without providing the URL you are accessing, it is impossible to say.
Try setting up cookie_jar to temporary storage (give it empty hashref):
$useragent->cookie_jar( {} );
I am writing a perl script that should login to Gowalla, fetch some information and check in. Looking at the http://api.gowalla.com/api/docs I don't find a way to "login". It seems they want every request to include the username and the password.
I thought it should be possible to first "login" and then use the supplied cookie to keep the conversation.
Am I missing something there or is that the case?
It says they use Basic Authentication. If you are using the LWP type of thing:
my $req = HTTP::Request->new( POST => 'http://somesite.com/');
$req->authorization_basic('username', 'password');
# using data supplied by the other answer.
$req->header( 'X-Gowalla-API-Key' => 'YOURKEY' );
my $resp = $ua->request($req);
Their API simply does not allow what you ask for. Quoting,
All authentication is handled with HTTP Basic Authentication. All
calls must also include your
Gowalla API Key in a X-Gowalla-API-Key
request header.
Nowhere in the document is mentioned "cookie", either.
So every time you want to make a request to them, you must supply both the HTTP basic auth info, and the X-Gowalla_API-Key HTTP header.