I am trying to post data to a page with Perl, but the page also requires headers. How would I post the headers and send headers (cookies, user agents, etc)?
I tried using LWP::UserAgent, but I couldn't figure out how to send the headers even though I could post to the page.
One more thing about this topic. When I posted on that page and printed the response content I could see the html just fine except the numbers that were supposed to show.
Try doing this :
use LWP::UserAgent;
use HTTP::Request;
my $userAgent = LWP::UserAgent->new();
my $request = HTTP::Request->new(
POST => "http://domain.tld/path"
);
$request->content("stuff=foobar");
$request->content_type("application/x-www-form-urlencoded");
my $response = $userAgent->request($request);
if ($response->code == 200) {
print $response->as_string;
}
else {
die $response->status_line;
}
Related
See: https://developer.spotify.com/web-api/authorization-guide/
I'm using the "client credentials flow" method.
sub get_token {
my $req = HTTP::Request->new(POST => $SPOTIFY_TOKEN);
$req->header('Authorization' => 'Basic MYBASE64HERE');
my $post_data = 'grant_type=client_credentials';
$req->content($post_data);
my $resp = $ua->request($req); #this is LWP
if ($resp->is_success) {
my $token = $resp->decoded_content;
print "$token\n";
return \$token;
}
else {
print "HTTP POST error code: ", $resp->code, "\n";
print "HTTP POST error message: ", $resp->message, "\n";
}
}
I get back HTTP POST error code: 400 / bad request
I know that it doesn't have to do with the header information or the URL. I tested via Curl and used Data::Dumper to verify it was formatted properly.
I'm not sure on the format I need to send the POST body data. I've tried the example above my $post_data = 'grant_type=client_credentials'; as well as every variation I could think of. Is there a proper way to do this in Perl using HTPP::Request to build the POST request?
I think following should work, Please try:
$req->content(grant_type => 'client_credentials');
my $post_data = "grant_type=client_credentials";
Turns out this is the answer. I'm not sure how I missed this previously.
When I try to submit a POST request with Perl, it often ends in a 301 redirect to the homepage. Here is the code :
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
# This does not work
my $url = 'http://www.opensubtitles.org/en/search2';
my $req = HTTP::Request->new(POST => $url);
$req->content('MovieName=the+terminator+(1996)');
# Pass request to the user agent and get a response back
print $req->as_string."\n";;
my $res = $ua->request($req);
if (!$res->is_success) {
print $res->status_line, "\n";
}
else {
print "Success in posting search\n";
}
In order to make it work, I have to manually use Firefox, go to the url (!). Then the script works. However, using a GET request works flawlessly :
# This works
my $url = 'http://www.opensubtitles.org/en/search2?MovieName=the+terminator+(1996)';
my $req = HTTP::Request->new(GET => $url);
Why is that ?
The site doesn't expect a POST to that URL, so it redirects you to back to the search page.
Firefox will use GET, not POST, if you just put the URL into the address line, that's why it works.
I want to get last of redirect URL.
like
url_1 : http://on.fb.me/4VGeu
url_2 : https://www.facebook.com/
I want to get url_2 by url_1 in perl.
Previous source is below.
sub get_redirect_location
{
my ($url) = #_;
my $ua = LWP::UserAgent->new;
$ua->proxy('http', 'SAMPLE_PROXY');
my $req = new HTTP::Request(GET => $url);
my $res = $ua->request($req);
return $res->headers_as_string;
}
Thanks in advance.
You can find the request that lead to a response using
$response->request()
You can get the previous response in the chain using
$response->previous()
All together:
while ($response) {
say $response->request()->uri();
$response = $response->previous();
}
You could look at WWW::Mechanize. I have used it before to do something like this.
http://search.cpan.org/~jesse/WWW-Mechanize-1.72/lib/WWW/Mechanize.pm#$mech->redirect_ok()
You may also find this post helpful:
Perl WWW::Mechanize (or LWP) get redirect url
This request returns a file of type ZIP how can I retrieve that file from that request?
# put timeouts, proxy etc into the useragent if needed
my $ua = LWP::UserAgent->new();
my $req = POST $in_u, Content_Type => 'form-data', Content => $in_r;
my $response = $ua->request($req);
if ($response->is_success())
{
print $response->content;
}
I think you can use the content method on your $req object to get the raw content returned as a result of the POST. If the content is huge, then content_ref method is more suitable and offers to directly manipulate the content.
my $zfile = $req->content;
and crack on $zfile with Archive::Zip as DVK suggested.
You can use Archive::Zip CPAN module
I call in this mode:
my $ua = new LWP::UserAgent;
my $response= $ua->post('www.example.com', {param1=>'val1',param2=>'val2'...} );
Can I call the above in the same way passing the values in GET form?:
my $response= $ua->post('www.example.com?param=val1¶m2=val2' );
It is because I'm using Firebug and when I go to Net tab under the "POST" tab it shows individual parameters as well as a GET string for POST submitted requests.
So I was wondering if I use GET string in this function call.
Parametersapplication/x-www-form-urlencoded
Itemid 4 option com_search
searchword dsd task search Source
Content-Type:
application/x-www-form-urlencoded
Content-Length: 53
searchword=dsd&task=search&option=com_search&Itemid=4
In short you can pass GET strings yes, but if your end code does not accept GET METHOD it will fail.
Also you might still need to specify some parameters since the post method asks for post(url,array_with_parameters).
sub post {
require HTTP::Request::Common;
my($self, #parameters) = #_;
my #suff = $self->_process_colonic_headers(\#parameters, (ref($parameters[1]) ? 2 : 1));
return $self->request( HTTP::Request::Common::POST( #parameters ), #suff );
}
Using along with HTTP::Request you can specify it at the content in the way you prefer:
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
# Create a request
my $req = HTTP::Request->new(POST => 'http://www.example.com');
$req->content_type('application/x-www-form-urlencoded');
$req->content('searchword=dsd&task=search&option=com_search&Itemid=4');
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
} else {
print $res->status_line, "\n";
}
Read more...