my %parameters = (
key => 'value'
);
my $response = $ua->get('http://example.com/i', %parameters);
I'm trying to get content of http://example.com/i?key=value,but after debugging I found the %parameters are stored in http headers instead of url parameters.
What's wrong in my code?
Though perldoc tells me that :
$ua->get( $url , $field_name => $value, ... )
But it should also work if I put those parameters in a %parameters,right?
The additional parameters to get are HTTP headers. For GET requests, arguments are included in the URL itself, URLencoded. You can use the URI module to create the appropriate URLs including GET variables, or construct them yourself (probably using URI::Escape to urlencode the values).
e.g.:
my %parameters = (
key => 'value'
);
my $url = URI->new("http://example.com/i");
$url->query_form(%parameters);
my $response = $ua->get($url);
From the fine manual:
$ua->get( $url )
$ua->get( $url , $field_name => $value, ... )
This method will dispatch a GET request on the given $url. Further arguments can be given to initialize the headers of the request.
Emphasis mine. You're misreading the documentation, the extra parameters for get() are HTTP header fields, not CGI parameters. If you want to include some CGI parameters then you'll have to add them to the URI yourself (preferably with URI).
Related
I am trying to write a function to create HTTP requests (POST and GET mostly) in Perl. I am keeping everything generic by using variables so that I don't have to worry about the type of request, the payload, headers, etc, however HTTP::Request->header() doesn't seem to like my variable:
my($req_type, $headers, $endpoint, $args, $request, $jsonfile) = #_;
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new($req_type => $endpoint);
$req->content_type('application/json');
foreach (#$headers) {
$req->push_header($_);
}
$req->content($args);
$req->content($request);
print "request : ".$req->as_string;
I tried a few different approches, and using push_header got me the closest, but I realize it may not be the best solution. I think it might have something to do with single quotes getting passed in:
#headers = "'x-auth-token' => '$_token'";
I can post more of the code if it is helpful. I'm hoping some Perl guru will know exactly what I'm doing wrong. I'm sure it's something to do with the format of the string I'm passing in.
#headers = "'x-auth-token' => '$_token'";
The header function expects to be passed two arguments. The header name and the header value.
You are passing it one argument: a string containing a fragment of Perl code.
You need to format your data more sensibly.
my %headers = (
"x-auth-token" => $_token;
);
and
foreach my $header_name (keys %headers) {
$req->push_header($header_name => $headers{$header_name});
}
I need to send an HTTP POST request with the following HTTP headers:
Content-type: 'application/atom+xml'
Authorization: MyLogin auth=$token
The token is coming from an authorization subroutine. Here is the Perl making the actual request after the subroutine is successful:
my $ua = LWP::UserAgent->new;
my $req = $ua->post ( $url );
$req = header('Content-type' => 'application/atom+xml');
$req = header('Authorization' => "MyLogin auth=$token");
However, I receive the following error when run:
Undefined subroutine &main::header called ...
How can I accomplish this?
According to the LWP::UserAgent documentation, you can set additional headers by passing them as arguments to post:
my $ua = LWP::UserAgent->new;
my $response = $ua->post($url,
'Content-type' => 'application/atom+xml',
'Authorization' => "MyLogin auth=$token"
);
Note that $ua->post actually sends the request, so trying to set the headers after calling it, as you do in your example code, is useless. (Not to mention the fact that there is no header function in the main namespace unless you import it from somewhere or write it yourself.)
I'm talking to what seems to be a broken HTTP daemon and I need to make a GET request that includes a pipe | character in the URL.
LWP::UserAgent escapes the pipe character before the request is sent.
For example, a URL passed in as:
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1|01
is passed to the HTTP daemon as
https://hostname/url/doSomethingScript?ss=1234&activities=Lec1%7C01
This is correct, but doesn't work with this broken server.
How can I override or bypass the encoding that LWP and its friends are doing?
Note
I've seen and tried other answers here on StackOverflow addressing similar problems. The difference here seems to be that those answers are dealing with POST requests where the formfield parts of the URL can be passed as an array of key/value pairs or as a 'Content' => $content parameter. Those approaches aren't working for me with an LWP request.
I've also tried constructing an HTTP::Request object and passing that to LWP, and passing the full URL direct to LWP->get(). No dice with either approach.
In response to Borodin's request, this is a sanitised version of the code I'm using
#!/usr/local/bin/perl -w
use HTTP::Cookies;
use LWP;
my $debug = 1;
# make a 'browser' object
my $browser = LWP::UserAgent->new();
# cookie handling...
$browser->cookie_jar(HTTP::Cookies->new(
'file' => '.cookie_jar.txt',
'autosave' => 1,
'ignore_discard' => 1,
));
# proxy, so we can watch...
if ($debug == 1) {
$browser->proxy(['http', 'ftp', 'https'], 'http://localhost:8080/');
}
# user agent string (pretend to be Firefox)
$agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.7.12) Gecko/20050919 Firefox/1.0.7';
# set the user agent
$browser->agent($agent);
# do some things here to log in to the web site, accept session cookies, etc.
# These are basic POSTs of filled forms. Works fine.
# [...]
my $baseURL = 'https://hostname/url/doSomethingScript?ss=1234&activities=VALUEA|VALUEB';
#values = ['Lec1', '01', 'Lec1', '02'];
while (1) {
if (scalar(#values) < 2) { last; }
my $vala = shift(#values);
my $valb = shift(#values);
my $url = $basEURL;
$url =~ s/VALUEA/$vala/g;
$url =~ s/VALUEB/$valb/g;
# simplified. Would usually check request for '200' response, etc...
$content = $browser->get($url)->content();
# do something here with the content
# [...]
# fails because the '|' character in the url is escaped after it's handed
# to LWP
}
# end
As #bchgys mentions in his comment, this is (almost) answered in the linked thread. Here are two solutions:
The first and arguably cleanest one is to locally override the escape map in URI::Escape to not modify the pipe character:
use URI;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $res;
{
# Violate RFC 2396 by forcing broken query string
# local makes the override take effect only in the current code block
local $URI::Escape::escapes{'|'} = '|';
$res = $ua->get('http://server/script?q=a|b');
}
print $res->request->as_string, "\n";
Alternatively, you can simply undo the escaping by modifying the URI directly in the request after the request has been created:
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => 'http://server/script?q=a|b');
# Violate RFC 2396 by forcing broken query string
${$req->uri} =~ s/%7C/|/;
my $res = $ua->request($req);
print $res->request->as_string, "\n";
The first solution is almost certainly preferable because it at least relies on the %URI::Escape::escapes package variable which is exported and documented, so that's probably as close as you're gonna get to doing this with a supported API.
Note that in either case you are in violation of RFC 2396 but as mentioned you may have no choice when talking to a broken server that you have no control over.
I'm trying to submit a form by post method using WWW::Mechanize perl module.
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
...
$mech->get($url);
...
my $response = $mech->submit_form(
form_name => $name,
fields => {
$field_name => $field_value
},
button => 'Button'
);
$field_name is generally speaking a text field (though the type is not specified explicitly in the form), which has a preset value.
$field_name => $field_value in $mech->submit_form on whatever reason does not replace the value, instead $field_value is added into the form after the original value:
{submitted_field_value} = {original_value},{provided_value}
How to replace {original_value} with {provided_value} in the form to be submitted ?
What happens if you add this single line to your code before calling $mech->submit_form():
$mech->field( $name, [$field_value], 1 );
This makes sure that the first value is added, or overwritten if it already exists.
1 is the number parameter (or position index)
See the documentation of WWW::Mechanize:
$mech->field( $name, \#values, $number )
Given the name of a field, set its value to the value specified. [...]
The optional $number parameter is used to distinguish between two
fields with the same name. The fields are numbered from 1.
It's important to remember WWW::Mechanize is better thought of as a 'headless browser' as opposed to say LWP or curl, which only handle all the fiddly bits of http requests for you. Mech keeps its state as you do things.
You'll need to get the form by using $mech->forms or something similar (its best to decide from the documentation. I mean there so many ways to do it.), and then set the input field you want to change, using the field methods.
I guess the basic way to do this comes out as so:
$mech->form_name($name);
$mech->field($field_name, $field_value);
my $response = $mech->click('Button');
Should work. I believe it will also work if you get the field and directly use that (ie my $field = $mech->form_name($name); then use $field methods instead of $mech.
I managed to make it working at my will. Thanks Timbus and knb for your suggestions. Though my case may not be completely general (I know the preset value) but I'd share what I've found (by trails & errors).
my $mech = WWW::Mechanize->new();
$mech->get($url);
$mech->form_name( $name );
my $fields = $mech->form_name($name);
foreach my $k ( #{$fields->{inputs}}){
if ($k->{value} eq $default_value){
$k->{value}=$field_value;
}
}
my $response = $mech->click('Button_name');
I am having some trouble creating a request to this WSDL that works; it requires authHeaders and I am not having much luck adding them. This is what I am trying:
# make proxy for the service
my $soap = SOAP::Lite->service($wsdl);
# add fault hanlder
$soap->on_fault(
sub { # SOAP fault handler
my $soap = shift;
my $res = shift;
# Map faults to exceptions
if(ref($res) eq '') {
die($res);
}
else {
die($res->faultstring);
}
return new SOAP::SOM;
}
);
# authentication request headers
my #headers = (
SOAP::Header->name('user')->value('myemail#whatever.com')->uri($apins),
SOAP::Header->name('password')->value('mypassword')->uri($apins),
SOAP::Header->name('appName')->value('TestApp')->uri($apins),
SOAP::Header->name('appVersion')->value('0.02')->uri($apins)
);
# request method
print $soap->getCompanyInfo('NB', #headers);
The response I get when doing this is:
String value expected instead of SOAP::Header reference
The method I am requesting has two string parameters, both optional. And suggestions?
I was able to get help form the SOAP::Lite mailing list. If I want to pass my own headers, I have to use the call method instead of the actually method name.
# create header for requests
my $authHeader = SOAP::Header->name("xsd:authHeader" =>
\SOAP::Header->value(
SOAP::Header->name('xsd:user')->value($s7user)->type(''),
SOAP::Header->name('xsd:password')->value($s7pass)->type(''),
SOAP::Header->name('xsd:appName')->value('TestApp')->type(''),
SOAP::Header->name('xsd:appVersion')->value('0.03')->type('')
));
# create data to pass as method paramaters
my $params = SOAP::Data->name('ns:email')->value($s7user)->type('');
# request method
$soap->call('checkLogin', $params, $authHeader);
In order to use the call method, you will need to define a proxy (endpoint) on your soap object. Hope this is helpful for someone else down the road.