I have this perl script to extract the source code of a webpage:
#!/usr/bin/perl
use LWP::UserAgent;
my $ou = new LWP::UserAgent;
my $url = "http://google.com";
my $source = $ou->get("$url")->decoded_content;
print "$source\n";
Now, I want to check the internet status if it is connected or not before extracting the source code .
The simplest way to detect whether a remote server is off line is to attempt to connect to it. Using LWP to send a head request (instead of get) retrieves just the HTTP header information without any content, and you should get a swift response from any server that is on line
The default timeout of LWP::UserAgent object is three minutes, so you will need to set it to something much shorter for a rapid test
This program temporarily sets the timeout to 0.5 seconds, sends a head request, and reports that the server is not responding if the result is an error of any sort. The original timeout value is restored before carrying on
Depending on the real server that you want to test, you will need to adjust the timeout carefully to avoid getting false negatives
use strict;
use warnings 'all';
use constant URL => 'http://www.google.com/';
use LWP;
my $ua = LWP::UserAgent->new;
{
my $to = $ua->timeout(0.5);
my $res = $ua->head(URL);
unless ( $res->is_success ) {
die sprintf "%s is not responding (%s)\n", URL, $res->status_line;
}
$ua->timeout($to);
}
Related
$url = "http://203.155.220.231/radar/pics/radarh.jpg";
use LWP::UserAgent;
$ua = new LWP::UserAgent;
$request = new HTTP::Request;
$request->method('GET');
$request->url($url);
$response = $ua->request($request);
if ( $response->is_error or
$response->header('Content-Type') ne 'image/jpeg' or
$response->header('Content-Length') ne length($response->content)
)
{
print $response->status_line . "\n";
print $response->header('Content-Length') . "\n";
print length($response->content) . "\n";
die "$!";
}
By checking $response->header('Content-Length') compare with length($response->content) is easiest way to verify data is really complete 100% downloaded ?
To check download complete, You should get total size of file before download. Then You can check download complete easily.
It depends on the response from the server.
If the server sends a Content-Length header then you could use this to verify the length. But if the server uses chunked Transfer-Encoding or simply closes the connection at the end of the response you cannot use this information. Such kind of responses are typical for dynamically generated content. And as far as I can see there are no information in these cases which let you determine if the download was complete or not.
When using the LWP::UserAgent module, one makes a request to a URL and
receives an HTTP::Response object which contains the response code
(hopefully 200!) and a status line.
My problem is that I can't figure out how to determine whether the
response code was returned from the webserver or from LWP::UserAgent.
For example, I believe that if the domain name doesn't resolve or you
simply cannot connect to the host, LWP::UserAgent reports this in the
form a 500 code, which is indistinguisable from a 500 "Internal Server
Error" code reported from the actual web server that's up but
experiencing some issues.
The problem is further amplified when going through a proxy server, as
there are now three possible "sources" of an error message:
the target webserver
the proxy server
the LWP::UserAgent library
How is one supposed to know if the 500 code means a) the server is up
but unhappy, b) the proxy could not connect to the server, or c)
LWP::UserAgent could not connect to the proxy?
I posted the same question here also:
http://www.justskins.com/forums/lwp-useragent-determining-source-43810.html
Error responses that LWP generates internally will have the
"Client-Warning" header set to the value "Internal response". If you
need to differentiate these internal responses from responses that a
remote server actually generates, you need to test this header value.
(from LWP::UserAgent -> REQUEST-METHODS)
#!/usr/bin/perl
use strict;
use LWP::UserAgent;
use HTTP::Request;
use IO::Socket::SSL;
my $ua = LWP::UserAgent->new(
ssl_opts => {
SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE
}
);
my $request = HTTP::Request->new(GET => "www.example.com");
my $response = $ua->request($request);
my $clientWarning = $response->header("Client-Warning");
if(defined $clientWarning and length($clientWarning) != 0) {
if($clientWarning =~ /Internal response/) {
print "$server UNAVAILABLE";
}
} else {
print "server AVAILABLE";
}
I need a Perl CGI script that fetches a URL and then returns the result of the fetch - the status, headers and content - unaltered to the CGI environment so that the "proxied" URL is returned by the web server to the user's browser as if they'd accessed the URL directly.
I'm running my script from cgi-bin in an Apache web server on an Ubuntu 14.04 host, but this question should be independent of server platform - anything that can run Perl CGI scripts should be able to do it.
I've tried using LWP::UserAgent::request() and I've got very close. It returns an HTTP::Response object that contains the status code, headers and content, and even has an "as_string" method that turns it into a human-readable form. The problem from a CGI perspective is that "as string" converts the status code to "HTTP/1.1 200 OK" rather than "Status: 200 OK", so the Apache server doesn't recognise the output as a valid CGI response.
I can fix this by using other methods in HTTP::Response to split out the various parts, but there seems to be no public way of getting at the encapsulated HTTP::Headers object in order to call its as_string method; instead I have to hack into the Perl blessed object hash and yank out the private "_headers" member directly. To me this seems slightly evil, so is there a better way?
Here's some code to illustrate the above. If you put it in your cgi-bin directory then you can call it as
http://localhost/cgi-bin/lwp-test?url=http://localhost/&http-response=1&show=1
You can use a different URL for testing if you want. If you set http-response=0 (or drop the param altogether) then you get the working piece-by-piece solution. If you set show=0 (or drop it) then the proxied request is returned by the script. Apache will return the proxied page if you have http-response=0 and will choke with a 500 Internal Server Error if it's 1.
#!/usr/bin/perl
use strict;
use warnings;
use CGI::Simple;
use HTTP::Request;
use HTTP::Response;
use LWP::UserAgent;
my $q = CGI::Simple->new();
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => $q->param('url'));
my $res = $ua->request($req);
# print a text/plain header if called with "show=1" in the query string
# so proxied URL response is shown in browser, otherwise just output
# the proxied response as if it was ours.
if ($q->param('show')) {
print $q->header("text/plain");
print "\n";
}
if ($q->param('http-response')) {
# This prints the status as "HTTP/1.1 200 OK", not "Status: 200 OK".
print $res->as_string;
} else {
# This works correctly as a proxy, but using {_headers} to get at
# the private encapsulated HTTP:Response object seems a bit evil.
# There must be a better way!
print "Status: ", $res->status_line, "\n";
print $res->{_headers}->as_string;
print "\n";
print $res->content;
}
Please bear in mind that this script was written purely to demonstrate how to forward an HTTP::Response object to the CGI environment and bears no resemblance to my actual application.
You can go around the internals of the response object at $res->{_headers} by using the $res->headers method, that returns the actual HTTP::Headers instance that is used. HTTP::Response inherits that from HTTP::Message.
It would then look like this:
print "Status: ", $res->status_line, "\n";
print $res->headers->as_string;
That looks less evil, though it's still not pretty.
As simbabque pointed out, HTTP::Response has a headers method through inheritance from HTTP::Message. We can tidy up the handling of the status code by using HTTP::Response->header to push it into the embedded HTTP::Headers object, then use headers_as_string to print out the headers more cleanly. Here's the final script:-
#!/usr/bin/perl
use strict;
use warnings;
use CGI::Simple;
use HTTP::Request;
use HTTP::Response;
use LWP::UserAgent;
my $q = CGI::Simple->new();
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new(GET => $q->param('url'));
my $res = $ua->request($req);
# print a text/plain header if called with "show=1" in the query string
# so proxied URL response is shown in browser, otherwise just output
# the proxied response as if it was ours.
if ($q->param('show')) {
print $q->header("text/plain");
}
# $res->as_string returns the status in a "HTTP/1.1 200 OK" line rather than
# a "Status: 200 OK" header field so it can't be used for a CGI response.
# We therefore have a little more work to do...
# convert status from line to header field
$res->header("Status", $res->status_line);
# now print headers and content - don't forget a blank line between the two
print $res->headers_as_string, "\n", $res->content;
I have written a Perl script which would check a list of URLs and connect to them by sending a GET request.
Now, let's say that one of these URLs has a file which is very big in size, for instance, has a size > 100 MB.
When a request is sent to download this file using this:
$mech=WWW::Mechanize->new();
$url="http://somewebsitename.com/very_big_file.txt"
$mech->get($url)
Once the GET request is sent, it will start downloading the file. I want this to be cancelled using WWW::Mechanize. How can I do that?
I checked the documentation of this Perl Module here:
http://metacpan.org/pod/WWW::Mechanize
However, I could not find a method which would help me do this.
Thanks.
Aborting a GET request
Using the :content_cb option, you can provide a callback function to get() that will be executed for each chunk of response content received from the server. You can set* the chunk size (in bytes) using the :read_size_hint option. These options are documented in LWP::UserAgent (get() in WWW::Mechanize is just an overloaded version of the same method in LWP::UserAgent).
The following request will be aborted after reading 1024 bytes of response content:
use WWW::Mechanize;
sub callback {
my ($data, $response, $protocol) = #_;
die "Too much data";
}
my $mech = WWW::Mechanize->new;
my $url = 'http://www.example.com';
$mech->get($url, ':content_cb' => \&callback, ':read_size_hint' => 1024);
print $mech->response()->header('X-Died');
Output:
Too much data at ./mechanize line 12.
Note that the die in the callback does not cause the program itself to die; it simply sets the X-Died header in the response object. You can add the appropriate logic to your callback to determine under what conditions a request should be aborted.
Don't even fetch URL if content is too large
Based on your comments, it sounds like what you really want is to never send a request in the first place if the content is too large. This is quite different from aborting a GET request midway through, since you can fetch the Content-Length header with a HEAD request and perform different actions based on the value:
my #urls = qw(http://www.example.com http://www.google.com);
foreach my $url (#urls) {
$mech->head($url);
if ($mech->success) {
my $length = $mech->response()->header('Content-Length') // 0;
next if $length > 1024;
$mech->get($url);
}
}
Note that according to the HTTP spec, applications should set the Content-Length header. This does not mean that they will (hence the default value of 0 in my code example).
* According to the documentation, the "protocol module which will try to read data from the server in chunks of this size," but I don't think it's guaranteed.
I have code like this
my $ua = new LWP::UserAgent;
$ua->timeout($timeout);
$ua->agent($useragent);
$response = $ua->post($domain,['login_name'=>$login,'login_password'=> $password])->as_string;
Content of page so large, thatI can't receive it. How to get only headers with sending post data?
I think this should do it for you.
my $ua = LWP::UserAgent->new();
$ua->timeout($timeout);
$ua->agent($useragent);
my $response = $ua->post(
$domain,
[ 'login_name' => $login, 'login_password' => $password ]
);
use Data::Dumper;
print Dumper( $response->headers() );
print $response->request()->content(), "\n";
To first, check if you can pass this login_name and login_password via HEAD (in url string: domain/?login_name=...&login_password=...). If this will not work, you are in bad case.
You cannot use POST with behavior of HEAD. LWP will wait full response.
Using POST the server will send you the content anyway, but you can avoid receiving all content using sockets tcp by yourself: gethostbyname, connect, sysread until you get /\r?\n\r?\n/ and close socket after this. Some traffic will be utilized anyway, but you can save memory and receive time.
Its not normal thing to do this with sockets, but sometimes when you have highload/big data - there is no better way than such mess.