I have a problem. I asked this question yesterday but not the way I really wanted it to be done, so my apologies. What I want to do is post to a page email and password but that page also requires specific headers to be sent. (I believe they are called request headers). I tried LWP::UserAgent but I could only post data, not the headers with it. How would I do this? I am new to Perl. Thanks for your help.
You previously mentioned the user Agent and cookies. While you could add those to the request, that's usually something you want over multiple requests and thus can be set in the UA itself.
use HTTP::Cookies qw( );
use HTTP::Request::Common qw( POST );
use LWP::UserAgent qw( );
my $cookies = HTTP::Cookies->new();
$cookies->set_cookie(...);
my $ua = LWP::UserAgent->new(
agent => ...,
cookie_jar => $cookies,
);
my $request = POST(
'http://www.example.com/',
Referer => '...',
Content => [
username => ...,
...
],
);
my $response = $ua->request($request);
use strict;
use warnings;
use LWP::UserAgent qw( );
my $ua = LWP::UserAgent->new();
$ua->default_header('Cookie' => '...');
$ua->default_header('Referer' => '...');
my $response = $ua->post('url_goes_here', {username=>'test', password=>'pass'});
if($response->is_success)
{
print $response->as_string;
}
this worked for me. Hopefully someone can learn a thing or two from this answer. The trick is to use default_header
Related
I can connect to a remote url using the proxy from tsocks in the following way:
tsocks telnet host port
How can I do the same using the Perl's LWP::UserAgent module?
I've been so far trying it like this, but it doesn't work:
use strict;
use warnings;
use v5.16;
use LWP::UserAgent;
use HTTP::Request::Common;
use Data::Dumper;
#my $ua = LWP::UserAgent->new();
my $ua = LWP::UserAgent->new(timeout => 10,
ssl_opts => {
#verify_hostname => 0,
verify_hostname => 0,
SSL_verify_mode => '0x01',
SSL_version => 'SSLv23:!SSLv3:!SSLv2',
}
);
$ua->proxy(['http', 'https' ], 'https://proxy_host:proxy_port' );
my $request = GET ( 'https://remote_url', Accept => 'application/json' );
$request->authorization_basic( 'username', 'password' );
say $request->as_string();
my $response = $ua->request( $request );
say $response->as_string();
By the way, I don't have socks installed on this server. So I'll need to do it without them.
Thanks!
LWP::UserAgent and SOCKS
You may use LWP::Protocol::socks perl package make LWP::UserAgent SOCKS capable.
IMHO it is much better way for your own perl scripts.
tsocks may be better for legacy perl scripts you do not want to modify.
I found a solution which might be very specific for my case:
Before running the script, I just call tsocks like this:
tsocks script_name.pl
I am writing a Perl script to POST an attachment to JIRA using
REST::Client to access the API
but I am getting an error.
use REST::Client;
use warnings;
use strict;
use File::Slurp;
use MIME::Base64;
my $user = 'user';
my $pass = 'pass';
my $url = "http://******/rest/api/2/issue/BugID/attachments";
my $client = REST::Client->new();
$client->addHeader( 'Authorization', 'Basic' . encode_base64( $user . ':' . $pass ) );
$client->addHeader( 'X-Atlassian-Token', 'no-check' );
$client->setHost( $url );
# my %header = ('Authorization' => 'Basic'. encode_base64($user . ':' . $pass),'X-Atlassian-Token' => 'no-check');
my $attachment = "C:\\Folder\\Test.txt";
$client->POST(
$url,
'Content_Type' => 'form-data',
'Content' => [ 'file' => [$attachment] ]
);
if ( $client->responseCode() eq '200' ) {
print "Updated\n";
}
# print the result
print $client->responseContent() . "\n";
The error I get is
REST::Client exception: headers must be presented as a hashref at C:\Users\a\filename.pl line 24.
As shown in the code, I have tried setting headers in different ways but I still get same error.
Please suggest if there is any other method.
I have tried using JIRA module but it gives error too.
According to the documentation, the POST method:
Takes an optional body content and hashref of custom request headers.
You need to put your headers in a hashref, e.g.:
$client->POST($url, $content, {
foo => 'bar',
baz => 'qux'
});
But...it looks like you're expecting REST::Client to use HTTP::Request::Common to construct a multipart/form-data request. Unfortunately, that's not the case, so you'll have to build the content by hand.
You could use HTTP::Request::Common directly like this:
use strict;
use warnings 'all';
use 5.010;
use HTTP::Request::Common;
use REST::Client;
my $client = REST::Client->new;
my $url = 'http://www.example.com';
my $req = POST($url,
Content_Type => 'form-data',
Content => [ file => [ 'foo.txt' ] ]
);
$client->POST($url, $req->content(), {
$req->headers->flatten()
});
But this is a bit convoluted; I would recommend dropping REST::Client and using LWP::UserAgent instead. REST::Client is just a thin wrapper for LWP::UserAgent with a few convenience features, like prepending a default host to all requests. In this case, it's just getting in the way and I don't think the conveniences are worth the trouble.
From the documentation:
POST ( $url, [$body_content, %$headers] )
And you're doing:
$client->POST(
$url,
'Content_Type' => 'form-data',
'Content' => [ 'file' => [$attachment] ]
);
So - passing a list of scalars, with an arrayref at the end.
Perhaps you want something like:
$client->POST(
$url,
$attachment,
{ 'Content-Type' => 'form-data' }
);
Note the {} to construct an anonymous hash for the headers.
Although you probably want to open and include the 'attachment', because there's nothing in REST::Client about opening files and sending them automagically.
The if statement is showing me that there is a response, but when I try to print the response I get nothing
use LWP::UserAgent;
use strict;
use warnings;
use HTTP::Request::Common;
# use this {"process": "mobileGps","phone": "9565551236"}
my $url = "the url goes here";
my $json = '{data :{"process" : "mobileGps", "phone" : "9565551236"}}';
my $req = HTTP::Request->new( POST => $url );
$req->header( 'Content-Type' => 'application/json' );
$req->content( $json );
my $ua = LWP::UserAgent->new;
my $res = $ua->request( $req );
if ( $res->is_success ) {
print "It worked";
print $res->decoded_content;
}
else {
print $res->code;
}
I do have the URL: I just took it out for the purpose of this example.
What am I missing?
Try debugging your script like this:
use strict;
use warnings;
use HTTP::Request::Common;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use LWP::UserAgent;
# use this {"process": "mobileGps","phone": "9565551236"}
my $url = "the url goes here";
my $json = '{data :{"process" : "mobileGps", "phone" : "9565551236"}}';
my $req = HTTP::Request->new(POST => $url);
$req->header('Content-Type' =>'application/json');
$req->content($json);
my $ua = LWP::UserAgent->new;
debug_ua( $ua );
my $res = $ua->request($req);
if ($res->is_success) {
print "It worked";
print $res->decoded_content;
} else {
print $res->code;
}
That will (hopefully) give you a better idea of what's going on.
Can you not use the debugger, or add some print statements to see how your program is progressing?
If not then this is going to be another case of on-line turn-by-turn debugging, which benefits no one except the OP, and the ultimate diagnosis is that they should have learned the language first
The internet can be wise, but it will make many more artisans Pretender than craftsmen
Please don't ever expect to make a half-hearted attempt at a sketch, and then rope in the rest of the world to finish your job. It takes a huge amount of experience, aptitude, and understanding to get even a "What's your name" .. "Hello" program working, and things only get harder thereafter
If you don't like being careful and thorough, and would rather ask for people to do your stuff for you than discover a solution by experimentation, then you are a manager, not a programmer. I hope you will never try to advance a software career by getting great at delegating, because that doesn't work with software
Here. Use this as you will. The world is full of managers; it is good programmers that we need
use strict;
use warnings 'all';
use feature 'say';
use constant URL => 'http://example.com/';
use LWP;
my $ua = LWP::UserAgent->new;
my $json = '{}';
my $req = HTTP::Request->new( POST => URL );
$req->header( content_type => 'application/json' );
$req->content( $json );
my $res = $ua->request( $req );
say $res->as_string;
The code is fine. The problem must be with the server that is serving the request upon status code 200. You should check at server's end.
Trying to write the exact equivalence in perl of the following:
curl -H "Content-Type: application/json" -X POST -d '{"user": { "uid":"13453"},"access_token":"3428D3194353750548196BA3FD83E58474E26B8B9"}' https://platform.gethealth.io/v1/health/account/user/
Unexperienced with perl, this is what I have tried:
use HTTP::Request::Common;
use LWP::UserAgent;
get '/gethealthadduser/:id' => sub {
my $ua = LWP::UserAgent->new;
$ua->request(POST 'https://platform.gethealth.io/v1/health/account/user', [{"user": { "uid":param("id")},"access_token":config->{gethealthtoken}}]);
};
I take it you are working with Dancer already, or you are adding something to an existing application, and the goal is to hide the POST request to another service behind your API.
In your curl example, you have the Content-Type application/json, but in your Perl code you are sending a form. That's likely going to be the Content-Type application/x-www-form-urlencoded. It might not be what the server wants.
In addition to that, you were passing the form data as an array reference, which makes POST believe they are headers. That not what you want.
In order to do the same thing you are doing with curl, you need a few more steps.
You need to convert the data to JSON. Luckily Dancer brings a nice DSL keyword to_json that does that easily.
You need to tell LWP::UserAgent to use the right Content-Type header. That's application/json and you can set it either at the request level, or as a default for the user agent object. I'll do the former.
In addition to that, I recommend not using HTTP::Request::Common to import keywords into a Dancer app. GET and POST and so on are upper-case and the Dancer DSL has get and post which is lower-case, but it's still confusing. Use HTTP::Request explicitly instead.
Here's the final thing.
use LWP::UserAgent;
use HTTP::Request;
get '/gethealthadduser/:id' => sub {
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
POST => 'https://platform.gethealth.io/v1/health/account/user',
[ 'Content-Type' => 'application/json' ], # headers
to_json { user => param("id"), access_token => config->{gethealthtoken} }, # content
);
my $res = $ua->request($req);
# log the call with log level debug
debug sprintf( 'Posted to gethealth.io with user %s and got a %s back.',
param('id'),
$res->status_line
);
# do things with $res
};
Try using HTTP::Tiny (it's on CPAN). IMHO, it's a much cleaner module than LWP::UserAgent, although the latter is much more popular.
Here's some code that should work out of the box:
use HTTP::Tiny 0.064; # use a recent version or better
my $url = 'https://api.somewhere.com/api/users';
my $data = {
first_name => "joe",
last_name => "blow"
};
my $method = 'POST';
my $default_headers = {
'Authorization' => "Bearer ".$token, # if needed
'Accept' => 'application/json'
};
my $tiny = HTTP::Tiny->new(
agent => 'mywebsite.com',
default_headers => $default_headers,
timeout => 30
);
my $response;
if ( ($method eq 'POST') || ($method eq 'PUT') ) {
$response = $tiny->request($method, $url, {
headers => {
'Content-Type' => 'application/json'
},
content => &toJSON($data)
});
}
else {
if ($data) {
die "data cannot be included with method $method";
}
$response = $tiny->request($method, $url);
}
die unless $response->{'success'};
Good luck on your project!
Here is the solution with the correct format and structure of posted parameters:
get '/api/gethealthadduser/:id' => sub {
my %user = (
uid => param("id")
);
# my $user = {
# uid => param("id")
# };
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(
POST => 'https://platform.gethealth.io/v1/health/account/user/',
[ 'Content-Type' => 'application/json' ], # headers
JSON::to_json({ user => \%user, access_token => config->{gethealthtoken} }) # content
);
my $res = $ua->request($req);
print STDERR Dumper($res);
$res;
};
So right now I'm using REST::Client which does a perfectly good job when it comes to making GET requests and fetching JSON data. However, the API in question, when issuing a POST request, should pass a CSRF token and session id and then if entering the right credentials through JSON should then be used further on for all POST requests.
Thing is, I see no way to fetch the cookie using REST::CLient, so I tried LWP, I was able to execute a JSON request, set up the cookie settings and still no cookies.
I tried storing it in a file, tried in a variable, still nothing
$mech->cookie_jar($cookies);
So how do I get these cookies?
P.S I'm sure the request is executed, since I'm seeing the right output and I'm seeing the cookies with a third party rest client.
EDIT:
#!/usr/bin/perl
use REST::Client;
use JSON;
use Data::Dumper;
use MIME::Base64;
use 5.010;
use LWP::UserAgent;
use HTTP::Cookies;
my $first = $ARGV[0];
my $username = 'user#user.com';
my $password = 'password';
my $cookies = HTTP::Cookies->new();
my $ua = LWP::UserAgent->new( cookie_jar => $cookies );
my $headers = {Content-type => 'application/json'};
my $client = REST::Client->new( { useragent => $ua });
my $res = $client->POST('https://URL/action/?do=login',
'{"username": "user#user.com", "password":"password"}', {"Content-type" => 'application/json'});
chkerr($client->responseCode());
print $client->responseContent();
#print $client->responseHeaders();
#$cookies->extract_cookies($res);
print "\n" . $cookies->as_string;
sub chkerr {
my $res = shift;
if($res eq '200') {
print "Success\n";
} else { print "API Call failed: $res\n";
#exit(1);
}
}
The code is really dirty since I've tried about 50 different things now.
The output is as follows:
Success
true -> this indicated that login is successful
Set-Cookie3: __cfduid=d3507306fc7b69798730649577c267a2b1369379851; path="/"; domain=.domain.com; path_spec; expires="2019-12-23 23:50:00Z"; version=0
From the documentation, it seems that REST::Client is using LWP::UserAgent internally. By default, LWP::UserAgent ignores cookies unless you set its cookie_jar attribute.
So you could do something like this (untested code):
my $ua = LWP::UserAgent->new( cookie_jar => {} );
my $rc = REST::Client->new( { useragent => $ua } );
While innaM's answer will work this seems more straight forward to me:
$client->getUseragent()->cookie_jar({}); # empty internal cookie jar
This assumes you already have a REST::Client object in $client.