I am using mason to call an API(web based, I can use GET to call it) so that a json file can be returned.
I know m->comp() can be used inside. But what function can be used externally?
Sounds like you want to make an http request to an external url. Since you're just embedding perl in html you could just have a perl block that uses HTTP::Request and LWP::UserAgent to make that request. Something like this:
my $ua = LWP::UserAgent->new;
my $response = $ua->request( HTTP::Request->new( "GET", "http://https://api.twitter.com/1/users/show.json?screen_name=aplusk" ) );
my $data = $response->content();
Then have HTML::Mason do whatever you want it to do with the json $data
Related
I have code like this
my $ua = new LWP::UserAgent;
$ua->timeout($timeout);
$ua->agent($useragent);
$response = $ua->post($domain,['login_name'=>$login,'login_password'=> $password])->as_string;
Content of page so large, thatI can't receive it. How to get only headers with sending post data?
I think this should do it for you.
my $ua = LWP::UserAgent->new();
$ua->timeout($timeout);
$ua->agent($useragent);
my $response = $ua->post(
$domain,
[ 'login_name' => $login, 'login_password' => $password ]
);
use Data::Dumper;
print Dumper( $response->headers() );
print $response->request()->content(), "\n";
To first, check if you can pass this login_name and login_password via HEAD (in url string: domain/?login_name=...&login_password=...). If this will not work, you are in bad case.
You cannot use POST with behavior of HEAD. LWP will wait full response.
Using POST the server will send you the content anyway, but you can avoid receiving all content using sockets tcp by yourself: gethostbyname, connect, sysread until you get /\r?\n\r?\n/ and close socket after this. Some traffic will be utilized anyway, but you can save memory and receive time.
Its not normal thing to do this with sockets, but sometimes when you have highload/big data - there is no better way than such mess.
Need to send HTTP OPTIONS Request in Perl. Looked through several CPAN modules; read the docs, no mention of OPTIONS request method, just GET, POST, PUT, DELETE.
Do I need to format this manually? Or is there possibly another library/module that my google-fu is missing out on?
The documentation for the HTTP::Request module says:
The method should be a short string like "GET", "HEAD", "PUT" or "POST".
So:
use v5.16;
use warnings;
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new(OPTIONS => 'http://www.example.com/');
my $response = $ua->request($request);
I don't have a server that gives a useful response to an OPTIONS request to test the response with, but the request looks OK when I examine it after setting a proxy.
In my application, I fetch webpages periodically using LWP. Is there anyway to check whether between two consecutive fetches the webpage has got changed in some respect (other than explicitly doing a comparison) ? Is there any signature(say CRC) that is being generated at lower protocol layers which can be extracted and compared against older signatures to see possible changes ?
There are two possible approaches. One is to use a digest of the page, e.g.
use strict;
use warnings;
use Digest::MD5 'md5_hex';
use LWP::UserAgent;
# fetch the page, etc.
my $digest = md5_hex $response->decoded_content;
if ( $digest ne $saved_digest ) {
# the page has changed.
}
Another option is to use an HTTP ETag, if the server provides one for the resource requested. You can simply store it and then set your request headers to include an If-None-Match field on subsequent requests. If the server ETag has remained the same, you'll get a 304 Not Modified status and an empty response body. Otherwise you'll get the new page. (And new ETag.) See Entity Tags in RFC2616.
Of course, the server could be lying, and sending the same ETag even though the content has changed. There's no way to know unless you look.
You should use the If-Modified-Since request header, noting the gotchas in the RFC. You send this header with the request. If the server supports it and thinks the content is newer, it sends it to you. If it thinks you have the most recent version, it returns a 304 with no message body.
However, as other answers have noted, the server doesn't have to tell you the truth, so you're sometimes stuck downloading the content and checking yourself. Many dynamic things will always claim to have new content because many developers have never thought about supporting basic HTTP things in their web apps.
For the LWP bits, you can create a single request with an extra header:
use HTTP::Request;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $request = HTTP::Request->new( GET => $url );
$r->header( 'If-Modified-Since' => $time );
$ua->request( $request );
For all requests, you can set a request handler:
$ua->add_handler(
request_send => sub {
my($request, $ua, $h) = #_;
# ... look up time from local store
$r->header( 'If-Modified-Since' => $time );
}
);
However, LWP can do most of this for you with mirror if you want to save the files:
$ua->mirror( $url, $filename )
I'm querying a webserver for a document and I want to capture the both the document and the related server response headers (esp. Content-Type: ...). I have trouble finding out how to read the headers. Here are some sniplets from my Perl script, I left error checking out for clarity:
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent( 'requiredCustomUserAgent' ); # I'm required to set a custom user agent
$imageData = $response->content; # This is the received document
So at this point I can retrieve the web document, but I want to know what Content-Type the server sent with it. Unfortunately this is not always the same as the mime type found by the bash 'file' command. This file-method fails in case of .js or .css documents.
http://search.cpan.org/perldoc?HTTP::Response
use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $response = $ua->get("http://google.co.uk");
print $response->headers()->as_string;
print $response->header('content-type');
the thing that request returns contains a HTTP::Headers object, so look at the docs for HTTP::Headers to see how to use it. For instance
my $response = $ua->request($req);
my $headers = $response->headers();
my #header_field_names = $headers->header_field_names();
$logger->info("$_: ".$headers->header($_)) for grep {/Hogwarts/} #header_field_names;
Lets say I have "http://www.ritzcarlton.com" and that redirects me to "http://www.ritzcarlton.com/en/Default.htm". Is there a way in Perl to find the end url after all the redirects?
Using LWP will follow the redirections for you. You can then interrogate the HTTP::Request object to find out the URI it requested.
use LWP::UserAgent qw();
my $ua = LWP::UserAgent->new;
my $response = $ua->get('http://www.ritzcarlton.com');
print $response->request->uri . "\n";
Output is:
http://www.ritzcarlton.com/en/Default.htm
If you're issuing HTTP requests yourself, then the redirect URL will be in the returned Location: header. If you're using a proper HTTP client like LWP::UserAgent or WWW::Mechanize, which is what you should be doing, redirection is handled automatically.