first off I've searched the forums and didn't find exactly my issue.
I'm running Ubuntu with perl 5.10 installed.
I'm receiving the following error after executing my script:
"Can't use an undefined value as filehandle reference at scraper.pl line 17"
Here is my script....
#!/usr/bin/perl -W
use strict;
use warnings;
use WWW::Curl::Easy;
my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER, 1);
$curl->setopt(CURLOPT_URL, 'http://something.com');
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
my $return_code = $curl->perform;
if ($return_code == 0)
{
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
print ("Success ".$response_code);
}
else
{
# Error Code
print ("An error occured: ".$return_code." ".$curl->strerror($return_code)." ".$curl->errbuf."\n");
}
# EOF
Any help here would be much appreciated.
Thanks,
Ben
In place of:
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
do:
my $response_body = '';
open(my $fileb, ">", \$response_body);
$curl->setopt(CURLOPT_WRITEDATA,$fileb);
If you check the documentation for the version of WWW-Curl you actually have installed, I think you'll see it passes a filehandle, not a scalar reference.
Alternatively, upgrade WWW-Curl.
Also note that -W is not generally advisable; often modules will disable warnings for a particular scope and the capital W switch prevents that. Use -w instead (or just use warnings; for your own code, which you are already doing).
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Curl::Easy;
use File::Temp qw/tempfile/;
my $response_body = tempfile();
my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER, 1);
$curl->setopt(CURLOPT_URL, 'http://yiddele.com/categoryindex.aspx');
#$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
$curl->setopt(CURLOPT_WRITEDATA, \$response_body);
my $return_code = $curl->perform;
if ($return_code == 0)
{
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
print ("Success ".$response_code);
}
else
{
# Error Code
print ("An error occured: ".$return_code." ".$curl->strerror($return_code)." ".$curl->errbuf."\n");
}
# EOF
Output is:
Success 200
There is bad code at:
print ("Success ".$response_code);
Look at the documentation for print: due to the way arguments are parsed when you use parentheses, the first argument will be interpreted to be a filehandle, which is not what you intended. In your code, the parentheses are unnecessary; just pass a concatenated string, or better, avoid the concatenation and pass a list of strings:
print 'Success ', $response_code;
Also, please please always include use strict; use warnings; at the top of every script you write. You will discover that many errors are highlighted that may otherwise remain hidden for quite some time, and it also saves everyone's time when you catch an error before ever having to ask on Stack Overflow. :)
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
You've declared $response_body, but haven't assigned a value to it. I assume that this would work if you made it a string.
my $response_body = "";
That said, I can't be sure as I can't reproduce the error. Perhaps installing a newer version of the module would help too.
use Data::Printer ;
use URI::Encode qw(uri_encode uri_decode);
use JSON ();
use JSON::Parse ':all' ;
use WWW::Curl;
use HTTP::Response ;
use utf8 ;
use Carp ;
use Cwd qw ( abs_path ) ;
use Getopt::Long;
use WWW::Curl::Easy;
my $curl = WWW::Curl::Easy->new;
$curl->setopt(WWW::Curl::Easy::CURLOPT_HEADER(),1);
$curl->setopt(WWW::Curl::Easy::CURLOPT_URL(), 'https://www.pivotaltracker.com/services/v5/me?fields=%3Adefault');
$curl->setopt(WWW::Curl::Easy::CURLOPT_HTTPHEADER() , ['X-TrackerToken: ' . $TOKEN] );
#$curl->setopt(WWW::Curl::Easy::CURLOPT_POST(), 1);
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(WWW::Curl::Easy::CURLOPT_WRITEDATA(),\$response_body);
# Starts the actual request
my $ret = $curl->perform;
if ($ret == 0) {
print("Transfer went ok\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
# judge result and next action based on $response_code
$response_body = HTTP::Response->parse($response_body);
print("Received response: $response_body\n");
p($response_body);
my $json_data = $response_body->content ;
$json_data = JSON->new->utf8->decode($json_data);
p($json_data);
} else {
# Error code, type of error, error message
print("An error happened: $ret ".$curl->strerror($ret)." ".$curl->errbuf."\n");
}
# my $cmd='curl -X GET -H "X-TrackerToken: ' . "$TOKEN" . '" "https://www.pivotaltracker.com/services/v5/me?fields=%3Adefault"' ;
# my $json_str = `$cmd`;
# p($json_str);
# my $json_data = JSON->new->utf8->decode($json_str);
# p($json_data);
Related
The task is quite simple: access a url and parse it based on the result. In case there is an error (404, 500 etc etc), take appropriate action. The last piece is the one that I am having issue with.
I have listed both the pieces of code that I currently use. The longer one (LWP+TreeBuilder) works for both conditions ; the shorter one (TreeBuilder) works for the first condition but does not work for the error condition. If I use TreeBuilder and the site returns a 404 or some other error, the script simply exits ! Any ideas ?
Longer code that works
use LWP::Simple;
use LWP::UserAgent;
use HTML::TreeBuilder;
$url="http://some_url.com/blahblah" ;
$response = LWP::UserAgent->new->request( HTTP::Request->new( GET => $url ));
if ($response->is_success) {
$p = HTML::TreeBuilder->new();
$p->parse($response->content);
} else {
warn "Couldn't get $url: ", $response->status_line, "\n";
}
Shorter one that does not
use HTML::TreeBuilder;
$url="http://some_url.com/blahblah" ;
$tree = HTML::TreeBuilder->new_from_url($url) ;
To quote the docs:
If LWP is unable to fetch the URL, or the response is not HTML (as determined by content_is_html in HTTP::Headers), then new_from_url dies, and the HTTP::Response object is found in $HTML::TreeBuilder::lwp_response.
Try this:
use strict;
use warnings;
use HTML::TreeBuilder 5; # need new_from_url
use Try::Tiny;
my $url="http://some_url.com/blahblah" ;
my $p = try { HTML::TreeBuilder->new_from_url($url) };
unless ($p) {
my $response = $HTML::TreeBuilder::lwp_response;
if ($response->is_success) {
warn "Content of $url is not HTML, it's " . $response->content_type . "\n";
} else {
warn "Couldn't get $url: ", $response->status_line, "\n";
}
}
the script simply exits
No, it throws an exception. You could always catch the exception with eval BLOCK if you so desired.
my $tree = eval { HTML::TreeBuilder->new_from_url($url) }
or warn($#);
I'm trying to use Geo::Coder::Google to get a list of coordinates from an array of locations. My problem is that the array of locations is generated by an another script which sometimes puts in it some strange locations which can not be found in google maps, i.e. CorseMétéo.
This generates the following error message :
"Google Maps API returned error: 500 Can't connect to maps.google.com:80 (Bad hostname) at geoTest.pl line 24.".
My code looks like this :
#!/usr/bin/perl -w
use strict;
use locale;
use warnings;
#use diagnostics;
use utf8;
binmode(STDIN, "encoding(utf8)");
binmode(STDOUT, "encoding(utf8)");
binmode(STDERR, "encoding(utf8)");
use Geo::Coder::Google;
my #place = ('Daluis', 'Corse', 'CorseMétéo');
my ($long, $lat);
foreach my $place(#place){
my $geocoder = Geo::Coder::Google->new(apikey => '{MyAPIkey}');
my $response;
until (defined $response){
$response = $geocoder->geocode(location => $place);
}
($long, $lat) = #{ $response->{Point}{coordinates} };
print "$long\n";
print "$lat\n";
}
Usually this perl module is used to geolocate street address, however it seems to run pretty good on bigger geographic locations.
Anybody had a similar problem ?
Thank you.
If you want your code to keep going even if there is an error, use an eval block:
until (defined $# or defined $response){
eval {
$response = $geocoder->geocode(location => $place);
}
}
if ($#)
{
#some error handling.
}
Note that eval { BLOCK } is not the same as eval "string of code". It doesn't compile the code at run-time, and it isn't a security problem. It is just a simple way of exception handling.
I managed to find a way to keep it working, it looks like this :
#!/usr/bin/perl -w
use strict;
use locale;
use warnings;
#use diagnostics;
use utf8;
binmode(STDIN, "encoding(utf8)");
binmode(STDOUT, "encoding(utf8)");
binmode(STDERR, "encoding(utf8)");
use Geo::Coder::Google;
my #place = ('Daluis', 'Corse', 'CorseMétéo', 'New Delhi');
my ($long, $lat);
foreach my $place(#place){
my $geocoder = Geo::Coder::Google->new(apikey => '{MyAPIkeyHere}');
my $response;
until (defined $response){
eval{
$response = $geocoder->geocode(location => $place);
if ($#){
"Couldn't get location : $place\n";
}else{
($long, $lat) = #{ $response->{Point}{coordinates} };
}
}
}
print "$place\n";
print "$long\n";
print "$lat\n";
}
The thing is now, each time a location isn't found, the coordinates from the previous location are pushed in it. So afterwards it's just a matter of getting rid of the duplicates. I use the coordinates to fill a JavaScript in order to generate a location map with google maps API.
However, the code will still generate an error output :
Useless use of a constant (Couldn't get location) in void context at get_LatLng.pl line 48.
But the code works for now. If anybody has any idea how to manage this error it would be nice.
Thank you anyway !!!
Try to debug this script. I think it maybe an issue of variable interpolation? I'm not sure.
It works using options if I pass the values like so:
perl test-file-exists.pl --file /proj/Output/20111126/_GOOD
I am trying to remove the option of passing in --file since I need to generate the date
dynamically.
perl test-file-exists.pl
Given the code changes below (I commented out the options piece). I am trying to create the string (see $chkfil). I am getting errors passing in $dt4. Somehow, its not passing in the file string that I am creating into this other module.
use strict;
use warnings;
use lib '/home/test/lib';
use ProxyCmd;
use Getopt::Long;
#
### Set up for Getopt
#
#my $chkfil;
#my $help;
#usage() if ( #ARGV < 1 or
# ! GetOptions('help|?' => \$help,
# 'file=s' => \$chkfil)
# or defined $help );
my $cmd = ProxyCmd->new( User=>"test_acct",
AuthToken=>"YToken",
loginServer=>"host.com");
# Get previous day
my $dt4 = qx {date --date='-1day' +'%Y%m%d'};
# Check file
my $chkfil = qq{/proj/Output/$dt4/_GOOD};
# Now test the fileExists function
print "Checking 'fileExists':\n";
my $feResults = $cmd->fileExists("$chkfil");
if ($feResults == 0) {
print "File Exists!\n";
} else {
print "File Does Not Exist\n";
}
sub usage
{
print "Unknown option: #_\n" if ( #_ );
print "usage: program [--file /proj/Output/20111126/_GOOD] [--help|-?]\n";
exit;
}
When you use backticks or qx, you get the trailing newline included so chomp it off:
my $dt4 = qx {date --date='-1day' +'%Y%m%d'};
chomp $dt4;
and you'll get a sensible filename.
You could also use DateTime and friends to avoid shelling out entirely.
I need to know how to customize my own errors in Perl. For instance, here's some code:
my $filename = 'filaname1.exe';
print "Copying $filename";
copy("$dir_root\\$filename", "$spcl_dir\\$filename");
if ($? == "0") {
print " - Success!\n";
}
else { print " - Failure!\n"; }
I tried to write this and "catch" the error and print "Failure" when I don't get an exit code of 0, and print "Success" when I do. I need to know how I can customize this; I don't really want to use die or anything like that where it will give a somewhat cryptic error (to the end user).
Thanks!
You need to read the documentation on $? in perlvar. This value is:
The status returned by the last pipe
close, backtick ("``") command,
successful call to wait() or
waitpid(), or from the system()
operator.
Your call to copy (presumably from File::Copy) doesn't far into any of those categories, so $? isn't set.
However, if you read the documentation for File::Copy, you'll see that its function all "return 1 on success, 0 on failure". So you can simplify your code a lot.
#!/usr/bin/perl
use strict; use warnings;
use File::Copy;
if (copy('notthere', 'somewhere else')) {
warn "success\n";
} else {
warn "failure: $!\n";
}
Note that I've used "warn" rather than "print" so that the errors go to STDERR. Note, also, the use of $! to display the operating system error. This can, of course, be omitted if it's not user-friendly enough.
Are you using File::Copy? You must be using something, because copy() isn't a perl keyword or built-in function.
The documentation of File::Copy doesn't refer to $? at all, so that's probably your mistake. You want to check the return value, and only if it's zero, refer to $!.
use strict;
use File::Copy qw(copy);
my ($from, $to) = #ARGV;
my $res = copy ($from, $to);
if( $res ){
print "Okay\n";
}
else{
print "Not Okay: $!\n";
}
I have CGI proxy that works on my localhost, but when I try to get it work on another server I get Premature end of script headers. I have included the source below. I also tried print header instead of the text/xml and it worked localhost but it failed on the server.
use strict;
#use warnings;
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
use HTTP::Request::Common qw(POST);
use HTTP::Request::Common;
use LWP::UserAgent;
use URI::Escape;
use Data::Dumper;
my $url = param('url');
sub writeXML($) {
my $response = shift #_;
if ($response->is_success) {
print CGI->header('text/xml');
print $response->content;
print STDERR "content response:#" . $response->content . "#\n";
}
else {
print STDERR "Status Code: " . $response->status_line . "\n";
print STDERR Dumper ($response);
}
}
sub makeRequest(){
if ($url){
my $ua = LWP::UserAgent->new;
my $response = $ua->request(GET $url);
if ($response){
writeXML($response);
}
else{
print STDERR "No response exists";
}
}
else{
print STDERR "URL must be specified";
}
}
makeRequest();
0;
__END__
The script "works" when I try it from the command line:
$ t.pl url=http://www.unur.com/
gives me the home page of my web site.
That means, the host on which you are trying this is missing some libraries. To figure out which ones, you should examine the server's error log, or try running your script from the shell as shown above.
See DEBUGGING.
PS: There is absolutely no good reason for those prototypes on makeRequest and writeXML. Plus, try warn sprintf "Status: %s\n", $response->status_line; instead of those unsightly print STDERR lines.
See my Troubleshooting Perl CGI scripts guide for all the steps you can go through to find the problem.
You only output a header if the program succeeds, all your error conditions are going to cause the premature end of script headers.
Put a 'print CGI->header();' and a suitable error message to STDOUT at all the points where you're output an error message to STDERR, and you'll be to see what's going wrong.