My situation: Want to create a perl script to automate web login page. I need also to be able to pass in (POST) components such as HTTP Headers, i.e X-forwarded-for, username, password, csrf tokens and the likes. I know of Mechanize, but which other modules can I use to do the mentioned? Can LWP do it?
Library recommendations are off-topic, so I'll focus the question about LWP.
Yes, LWP::UserAgent (and its subclass WWW::Mechanize) can be used to send arbitrary headers.
$ua->request takes a request object with custom headers. You can create this object and perform the request as follows:
use HTTP::Request::Common qw( GET );
my $request = GET($url,
HeaderName => 'HeaderValue',
HeaderName => 'HeaderValue',
);
my $response = $ua->request($request);
LWP:UserAgent provides a shorthand for this.
my $response = $ua->get($url,
HeaderName => 'HeaderValue',
HeaderName => 'HeaderValue',
);
Related
I'm trying to use Mojo::UserAgent to access the eBay API.
One of the options is to use API requests with an XML payload, but I have had no success doing it with Mojo::UserAgent. I didn't find any options for the $ua->post method.
I also tried
my $tx = $ua->build_tx(POST => $ebay_api_url => $headers);
$tx->req->body($xml_body);
my $res = $ua->start($tx)->res->json;
with no success. The XML body is not set for the request.
What do I need to do to achieve the desired result?
I know about the possibility of using JSON requests, but that is a reserve plan.
Try to post your $xml_body like so:
my $tx = $ua->post($ebay_api_url => form => $xml_body);
You likely want (2nd example in post doc):
my $tx = $ua->post($ebay_api_url => {Accept => '*/*'} => $xml_body);
I faced the similar issue like yours but later I realized that the problem was with the xml data. Please ensure that you do not have any trailing or leading white spaces in your $xml_body. This works:
my $tx = $ua->post($ebay_api_url => $headers => $xml_body);
Mojolicious and Mojo::UserAgent is awesome and light.
I am trying to automate sms sending through Way2sms in Perl LWP. I am able to login successfully. Then I click the Send SMS link in the browser to go the sms-sending page. From there, based upon the page url and the url of the iframe within which the sms fields are located, I try to construct the absolute URL of the page to which the form should be posted, and post it with the correct parameters (you can see it all in the image). However, the sms isn't being sent. Can anybody tell me what am I doing wrong here? (There is a similar module in CPAN which accomplishes this through Mechanize, I am trying a different approach)
use LWP::UserAgent;
use HTTP::Cookies;
my $cookie_jar = HTTP::Cookies->new(
file => "cookies.txt",
autosave => 1,
);
my $ua = LWP::UserAgent->new(
agent =>
'Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1',
cookie_jar => $cookie_jar,
);
my $response = $ua->post(
'http://site2.way2sms.com/contentt/bar/Login1.action',
{
username => $mob,
password => $pass,
}
);
if ( $response->is_redirect ) {
$response = $ua->get( $response->header('Location') );
print 5 if $response->decoded_content =~ /Kaustav Mukherjee/i; #prints it, showing that the login is successful
}
my $smsresp = $ua->post("http://site5.way2sms.com/jsp/quicksms.action",[MobNo=>$mob,textArea=>'Hello World']);
if ( $smsresp->is_redirect ) {
$smsresp = $ua->post($smsresp->header('Location'),[MobNo=>$mob,textArea=>'Hello World']);
}
Here is an indirect answer that may help you out:
If you have to use this website, try using something like CasperJS. There may be some JavaScript magic that needs to happen.
Maybe you could use something like Google Voice or some other smarter service to automate send text messages. Looks like someone on CPAN already has a sweet Google::Voice module for this.
I'm using Perl's SOAP::Lite to access a remote web service defined by a WSDL. That means I have:
use strict;
use Data::Dumper;
use SOAP::Lite +trace => 'debug';
my $service = SOAP::Lite->service('http://path/wsdl');
Ok so far. Problem is that I need access to the HTTP::Request object to send along custom HTTP request headers (and I'm not talking about authentication headers). It looks like I can access the request object after doing a successful call:
my $result = $service->getClient('parameters');
print Dumper($service->transport->http_request);
That'll give me the correct HTTP::Request object:
$VAR1 = bless( {
'_content' => '',
'_uri' => undef,
'_headers' => bless( {}, 'HTTP::Headers' ),
'_method' => undef
}, 'HTTP::Request' );
If I try to access the request object before doing an autoDispatch (the $service->getClient part), the transport object is empty and I have no way of modifying the request. It seems like everything would work fine if I were going the SOAP::Lite->proxy way -- but that defeats the helpfulness of having a pre-defined service definition.
Any ideas how I'm suppose to access the request object from a service definition without first having to make a call? Chicken and egg problem really...
Thanks!
What I'm trying to accomplish is populating the transport before doing a service call.
And you do exactly that by adding the appropriate handler, because the transport is not empty
Add a handler to the transport, see LWP::Debug for example, see LWP::UserAgent for documentation, or perlmonks.org/?node_id=904166 for example
I've built a REST Server and now I want to rapidly test it from a Perl Client, using REST::Client module.
It works fine if I perform GET Request (explicitly setting parameters in the URL) but I can't figure out how to set those params in POST Requests.
This is how my code looks like:
#!/usr/bin/perl
use strict;
use warnings;
use REST::Client;
my $client = REST::Client->new();
my $request_url = 'http://myHost:6633/my_operation';
$client->POST($request_url);
print $client->responseContent();
I've tried with something similar to:
$client->addHeader ('my_param' , 'my value');
But it's clearly wrong since I don't want to set an HTTP predefined Header but a request parameter.
Thank you!
It quite straight forward. However, you need to know what kind of content the server expects. That will typically either be XML or JSON.
F.ex. this works with a server that can understand the JSON in the second parameter, if you tell it what it is in the header in the third parameter.
$client->POST('http://localhost:3000/user/0/', '{ "name": "phluks" }', { "Content-type" => 'application/json'});
The REST module accepts a body content parameter, but I found to make it work with a string of parameters, you need to set a proper content type.
So the following code works for me:
$params = $client->buildQuery([username => $args{username},
password => $args{password}]);
$ret = $client->POST('api/rest/0.001/login', substr($params, 1),
{'Content-type' => 'application/x-www-form-urlencoded'});
I've not used the REST module, but looking at the POST function, it accepts a body content parameter, try creating a string of the parameters and send that within the function
$client->POST($request_url, "my_param=my+value");
print $client->responseContent();
I would like to write a script to login to a web application and then move to other parts
of the application:
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use Data::Dumper;
$ua = LWP::UserAgent->new(keep_alive=>1);
my $req = POST "http://example.com:5002/index.php",
[ user_name => 'username',
user_password => "password",
module => 'Users',
action => 'Authenticate',
return_module => 'Users',
return_action => 'Login',
];
my $res = $ua->request($req);
print Dumper(\$res);
if ( $res->is_success ) {
print $res->as_string;
}
When I try this code I am not able to login to the application. The HTTP status code returned is 302 that is found, but with no data.
If I post username/password with all required things then it should return the home page of the application and keep the connection live to move other parts of the application.
You may be able to use WWW::Mechanize for this purpose:
Mech supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you've visited, which can be queried and revisited.
I'm guessing that LWP isn't following the redirect:
push #{ $ua->requests_redirectable }, 'POST';
Any reason why you're not using WWW::Mechanize?
I've used LWP to log in to plenty of web sites and do stuff with the content, so there should be no problem doing what you want. Your code looks good so far but two things I'd suggest:
As mentioned, you may need to make the requests redirectable
You may also need to enable cookies:
$ua->cookie_jar( {} );
Hope this helps