I am trying to fill out a form on a separate page and return the data, but my hosting does not support the www::mechanize module. I saw that HTML::FORM would accomplish the same thing but I am getting the error
Can't call method "value" on an undefined value at G:\Programming\test.pl line 12.
Here is the code I have been testing with
use strict;
use LWP::Simple;
use LWP::UserAgent;
use HTML::Form;
use HTML::Strip;
my $base_uri = "UTF-8";
my $url = 'xxxxxxx';
my $form = HTML::Form->parse($url, $base_uri);
$form->value(Zip => '74014');
my $ua = LWP::UserAgent->new;
my $response = $ua->request($form->click);
The first argument for parse is HTML document itself but not the URL.
The required arguments is the HTML document to parse ($html_document)
and the URI used to retrieve the document ($base_uri). The base URI is
needed to resolve relative action URIs. The provided HTML document
should be a Unicode string (or US-ASCII).
So you need to first get this document (with LWP::UserAgent) and parse response:
my $response = $ua->get($url);
if ($response->is_success) {
my $form = HTML::Form->parse($response->decoded_content, $base_uri);
...
}
else {
die $response->status_line;
}
Related
I want to get last of redirect URL.
like
url_1 : http://on.fb.me/4VGeu
url_2 : https://www.facebook.com/
I want to get url_2 by url_1 in perl.
Previous source is below.
sub get_redirect_location
{
my ($url) = #_;
my $ua = LWP::UserAgent->new;
$ua->proxy('http', 'SAMPLE_PROXY');
my $req = new HTTP::Request(GET => $url);
my $res = $ua->request($req);
return $res->headers_as_string;
}
Thanks in advance.
You can find the request that lead to a response using
$response->request()
You can get the previous response in the chain using
$response->previous()
All together:
while ($response) {
say $response->request()->uri();
$response = $response->previous();
}
You could look at WWW::Mechanize. I have used it before to do something like this.
http://search.cpan.org/~jesse/WWW-Mechanize-1.72/lib/WWW/Mechanize.pm#$mech->redirect_ok()
You may also find this post helpful:
Perl WWW::Mechanize (or LWP) get redirect url
This request returns a file of type ZIP how can I retrieve that file from that request?
# put timeouts, proxy etc into the useragent if needed
my $ua = LWP::UserAgent->new();
my $req = POST $in_u, Content_Type => 'form-data', Content => $in_r;
my $response = $ua->request($req);
if ($response->is_success())
{
print $response->content;
}
I think you can use the content method on your $req object to get the raw content returned as a result of the POST. If the content is huge, then content_ref method is more suitable and offers to directly manipulate the content.
my $zfile = $req->content;
and crack on $zfile with Archive::Zip as DVK suggested.
You can use Archive::Zip CPAN module
I call in this mode:
my $ua = new LWP::UserAgent;
my $response= $ua->post('www.example.com', {param1=>'val1',param2=>'val2'...} );
Can I call the above in the same way passing the values in GET form?:
my $response= $ua->post('www.example.com?param=val1¶m2=val2' );
It is because I'm using Firebug and when I go to Net tab under the "POST" tab it shows individual parameters as well as a GET string for POST submitted requests.
So I was wondering if I use GET string in this function call.
Parametersapplication/x-www-form-urlencoded
Itemid 4 option com_search
searchword dsd task search Source
Content-Type:
application/x-www-form-urlencoded
Content-Length: 53
searchword=dsd&task=search&option=com_search&Itemid=4
In short you can pass GET strings yes, but if your end code does not accept GET METHOD it will fail.
Also you might still need to specify some parameters since the post method asks for post(url,array_with_parameters).
sub post {
require HTTP::Request::Common;
my($self, #parameters) = #_;
my #suff = $self->_process_colonic_headers(\#parameters, (ref($parameters[1]) ? 2 : 1));
return $self->request( HTTP::Request::Common::POST( #parameters ), #suff );
}
Using along with HTTP::Request you can specify it at the content in the way you prefer:
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
# Create a request
my $req = HTTP::Request->new(POST => 'http://www.example.com');
$req->content_type('application/x-www-form-urlencoded');
$req->content('searchword=dsd&task=search&option=com_search&Itemid=4');
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
} else {
print $res->status_line, "\n";
}
Read more...
I am having some difficulties getting results from a form via Perl. I believe I have successfully found the form and submitted the value I want to the appropriate field, but am unsure of how to turn the response object into something useful (If I print it out it shows up as the following).
HTTP::Request=HASH(0x895b8ac)
Here is the relevant code (assume $url is correct)
my $ua = LWP::UserAgent->new;
my $responce = $ua->get($url);
my #form = HTML::Form->parse($responce);
my $chosen = $form[0];
$chosen->value('netid', $user);
my $ro = $chosen->click('Search');
What can I do to make $ro useful?
Thanks!
To quote the HTML::Form docs on click:
The result of clicking is an HTTP::Request object that can then be passed to LWP::UserAgent if you want to obtain the server response.
So you can do:
my $ua = LWP::UserAgent->new;
my $response = $ua->get($url);
my #form = HTML::Form->parse($response);
my $chosen = $form[0];
$chosen->value('netid', $user);
my $ro = $chosen->click('Search');
# If you want to see what you're sending to the server:
print $ro->as_string;
# Fetch the server's response:
$response = $ua->request($ro);
What you do with $response next depends on what you're trying to do.
P.S. "responce" is usually spelled without a C. But HTTP does have a history of misspellings. (I'm looking at you, "Referer".)
I have some code I've written in PHP for consuming our simple webservice, which I'd also like to provide in Perl for users who may prefer that language. What's the simplest method of making a HTTP request to do that? In PHP I can do it in one line with file_get_contents().
Here's the entire code I want to port to Perl:
/**
* Makes a remote call to the our API, and returns the response
* #param cmd {string} - command string ID
* #param argsArray {array} - associative array of argument names and argument values
* #return {array} - array of responses
*/
function callAPI( $cmd, $argsArray=array() )
{
$apikey="MY_API_KEY";
$secret="MY_SECRET";
$apiurl="https://foobar.com/api";
// timestamp this API was submitted (for security reasons)
$epoch_time=time();
//--- assemble argument array into string
$query = "cmd=" .$cmd;
foreach ($argsArray as $argName => $argValue) {
$query .= "&" . $argName . "=" . urlencode($argValue);
}
$query .= "&key=". $apikey . "&time=" . $epoch_time;
//--- make md5 hash of the query + secret string
$md5 = md5($query . $secret);
$url = $apiurl . "?" . $query . "&md5=" . $md5;
//--- make simple HTTP GET request, put the server response into $response
$response = file_get_contents($url);
//--- convert "|" (pipe) delimited string to array
$responseArray = explode("|", $response);
return $responseArray;
}
LWP::Simple:
use LWP::Simple;
$contents = get("http://YOUR_URL_HERE");
LWP::Simple has the function you're looking for.
use LWP::Simple;
$content = get($url);
die "Can't GET $url" if (! defined $content);
Take a look at LWP::Simple.
For more involved queries, there's even a book about it.
I would use the LWP::Simple module.
Mojo::UserAgent is a great option too!
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
# Say hello to the Unicode snowman with "Do Not Track" header
say $ua->get('www.☃.net?hello=there' => {DNT => 1})->res->body;
# Form POST with exception handling
my $tx = $ua->post('https://metacpan.org/search' => form => {q => 'mojo'});
if (my $res = $tx->success) { say $res->body }
else {
my ($err, $code) = $tx->error;
say $code ? "$code response: $err" : "Connection error: $err";
}
# Quick JSON API request with Basic authentication
say $ua->get('https://sri:s3cret#example.com/search.json?q=perl')
->res->json('/results/0/title');
# Extract data from HTML and XML resources
say $ua->get('www.perl.org')->res->dom->html->head->title->text;`
Samples direct from CPAN page. I used this when I couldn 't get LWP::Simple to work on my machine.
Try the HTTP::Request module.
Instances of this class are usually passed to the request() method of an LWP::UserAgent object.
If it's in Unix and if LWP::Simple isn't installed, you can try:
my $content = `GET "http://trackMyPhones.com/"`;
I think what Srihari might be referencing is Wget, but I would actually recommend (again, on *nix without LWP::Simple) to use cURL:
$ my $content = `curl -s "http://google.com"`;
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
The -s flag tells curl to be silent. Otherwise, you get curl's progress bar output on standard error every time.