I'm trying to get the response of a curl call into a variable in perl.
my $foo = `curl yadd yadda`;
print $foo;
does not work. When I run this at the command line the curl call prints all its output correctly in the terminal, but the variable is not filled with that data.
Is there a way to do this without installing and calling the Perl curl lib?
It probably sends its stuff to stderr. Try
my $foo = `curl yadd yadda 2>&1`;
You also might consider looking at LWP::UserAgent or even LWP::Simple.
What do you really want to do? Use curl at all costs, or grab the contents of a web page?
A more perlish way of doing this (which relies on no external programs that may or may not be installed on the next machine where you need to do this) would be:
use LWP::Simple;
my $content = get("http://stackoverflow.com/questions/1015438/")
or die "no such luck\n";
If you want to see why the GET failed, or grab multiple pages from the same site, you'll need to use a bit more machinery. perldoc lwpcook will get you started.
In the shell 2> means redirect fileno 2. Fileno 2 is always what a program sees as stderr. Similarly, fileno 0 is stdin and fileno 1 is stdout. So, when you say 2>&1 you are telling the shell to redirect stderr (fileno 2) into stdout (fileno 1). Since the backticks operator uses the the shell to run the command you specify, you can use shell redirection, so
my $foo = `curl yadda yadda 2>&1`;
is telling curl to redirect its output into stdout, and since the backtick operator catches stdout, you get what you were looking for.
Very old post, but the real way of using curl in backticks is using the appropriate switch of curl.
This switch is -o which defines where to send the output.
More from the curl man page:
Specifying the output as '-' (a single dash) will force the output to
be done to stdout.
This also prevents having possible errors in $foo, which would happen if you redirect the complete STDERR to STDOUT on errors:
my $foo = `curl -o - yadd yadda`;
Try this:
$var = `curl "http://localhost" 2>/dev/null`;
print length($var)
curl displays progress information on stderr, redirecting that to /dev/null makes it easier to see what's going on.
This works on my system:
#!/usr/bin/perl
use strict;
use warnings;
my $output = `curl www.unur.com`;
print $output;
__END__
C:\> z1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
etc.
You can open a pipe as if it were a file.
$url = "\"http://download.finance.yahoo.com/d/quotes.csv?s=" .
"$symbol&f=sl1d1t1c1ohgvper&e=.csv\"";
open CURL, "curl -s $url |" or die "single_stock_quote: Can't open curl $!\n";
$line = <CURL>;
close CURL;
It might be that some of the output you want to capture is in standard err, not standard out. Try this:
my $foo = system "curl http://www.stackoverflow.com";
print $foo;
Related
I have an issue with downloading a page,
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
I can browse following with a browser but when I run bash command in perl or linux shell,
GET $url >OUTPUT1; # Even it does not write anything to file "OUPUT1"
When I try wget, It downloads but not correct ,I mean with --> <title>Error - Nucleotide - NCBI</title>. I want the page with items , but it returns me a page without items.
my $html = qx{wget --quiet --output-document=OUTPUT1 $url};
**Note: I noticed a few minutes ago, url is ok with Mozilla firefox, but it can not be browsed via google chrome. it is weird, probably my issue related with this too. Any idea?
Code from link:
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
my $html = qx{wget --quiet --output-document=OUTPUT11 $url};
# wget get something, but it does not get items, it gets what I get via google chrome
`GET $url2 >OUTPUT11`; # it does not write anything to file,
OK, given your code - the problem is almost certainly one of interpolation. Because the & in your URL is going to be interpreted by the shell you're spawning as 'background this process'.
That's almost certainly not what you want. Why not just use LWP natively?
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
my $content = get $url;
print $content;
open ( my $output_fh, '>', 'output.html' ) or die $!;
print {$output_fh} $content;
close ( $output_fh );
I am trying to convert this shell command into perl I am using use WWW::Curl::Easy;
I want to write the verbose in a different file and i want to get the URL content into a different file . Till now i have been able to curl the URL and add header to it as well .
Now I want to just write the as shell command mentions output from 1> into a file and
2> into antohter file as it is there in shell
my $curl = WWW::Curl::Easy->new();
$curl->setopt(CURLOPT_HEADER,1);
$curl->pushopt(WWW::Curl::Easy::CURLOPT_HTTPHEADER,['AUTH_USER:John']);
$curl->setopt(WWW::Curl::Easy::CURLOPT_URL,"www.abc.com");
$curl->setopt(CURLOPT_VERBOSE,1);
how do i put the content of www.abc.com that is
<html>
<body>
<marquee>abc is up </marquee>
</body>
</html>
into one file
And this into another file
> GET / HTTP/1.1
Host: abc-01
Accept: */*
AUTH_USER:hojn
< HTTP/1.1 200 OK
......
A cursory reading of the documentation and libcurl docs reveals that you probably need:
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
my $response_header;
$curl->setopt(CURLOPT_WRITEHEADER,\$response_header);
When using libcurl, it's useful to have its documentation handy.
Browsing through the options, we find CURLOPT_HEADERFUNCTION & WRITEHEADER and CURLOPT_WRITEFUNCTION & WRITEDATA.
use strict;
use warnings;
use Net::Curl::Easy qw( /^CURLOPT_/ );
my $url = 'http://stackoverflow.com/';
my $easy = Net::Curl::Easy->new();
open(my $fh_header, '>:raw', 'header.out') or die $!;
open(my $fh_data, '>:raw', 'data.out' ) or die $!;
$easy->setopt(CURLOPT_URL, $url);
$easy->setopt(CURLOPT_WRITEHEADER, $fh_header);
$easy->setopt(CURLOPT_WRITEDATA, $fh_data);
$easy->perform();
Note: I used Net::Curl over WWW::Curl because I know and trust it. The above should also work with WWW::Curl as long as it provides a Perl-aware default for CURLOPT_WRITEFUNCTION and CURLOPT_HEADERFUNCTION. It it doesn't, you'll need to provide values for those options too.
i wrote a little script which executes the ls command. Now I want to store the output of that command in a variable to work with it in the further perl script. Unfortunately I cannot grab the output from the ls command.
my ($core) = "servername";
my $tel_con = "ssh ".$core. " -l $user";
$ssh = Expect->spawn("$tel_con");
$ssh->log_stdout(1);
unless ( $ssh->expect( 5, -re => '$' ) ) {
return "Never connected " . $ssh->exp_error() . "\n";
}
sleep 1;
$ssh->send_slow(0, "ls -l\r");
$ssh->clear_accum();
my $str = $ssh->exp_after();
print "STR = '$str'\n";
Maybe you can give me some help please?
use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($core, user => $user);
$ssh->error and die $ssh->error;
my $output = $ssh->capture('ls -l');
print "command output:\n$output\n\n";
In case you can not or don't want to use Net::OpenSSH you may do:
my #output = $exp->expect(5);
print 'OUT: '.$output[3].'END';
To get the whole output (including the used command, return string, console information)
you could call expect in a seperate process and grab the output via qx or open a pipe
What is send_slow? Depending on how this command sends the ls command, the output can be received in different ways.
Most probably, the error output of the command is stored in the $? variable, possibly byte-shifted.
It seems that Expect will redirect everything to STDOUT and log it internally. Since you enabled output with $ssh->log_stdout(1), you should be able to get the results of ls -l directly from STDOUT (maybe by redirecting standard out to a variable). You can also see try grab the data from the internal logging. Just make sure to grab the output before doing clear_accum().
From CPAN: $object->send_slow($delay, #strings);
... After each character $object will be checked to determine whether or not it has any new data ready and if so update the accumulator for future expect() calls and print the output to STDOUT and #listen_group if log_stdout and log_group are appropriately set.
Layman's terms?
I must obligatorily post this link - I still go there from time-to-time, it is the most layman-y explanation I've ever found of all things regex:
http://www.anaesthetist.com/mnm/perl/Findex.htm
novice Perl programmer here!.
I'm using the system() function to get the return code for an external program (in this case - php), however, the command's output is still printed to the screen.
How do I prevent it from doing so?
My code is:
use strict; use warnings;
print 'Return code:', system('php -l wrong.php'), "\n";
This code does print the return code, but it also print the the output of the command executed.
Any help will be appreciated!
EDIT: further testing shown that this happens while only using the php lint command.. using it with other commands doesnt print anything...
What you want is IPC::Open3:
use IPC::Open3;
use Symbol qw(gensym);
my $err = gensym; #work around IPC::Open3's interface
my $pid = open3 my $wtr, my $rdr, $err,
'some cmd', 'arg1', 'arg2', ...;
The separates out stdin into $wtr, stdout into $rdr, and stderr into $err
This also gives you maximum control over communicating with the process
If you are on a UNIX-like OS, you should be able to redirect the output in the command:
Try:
system('php -l wrong.php >> /dev/null') to get rid of what's being sent to stdout.
You could also open the command with a pipe to handle the output directly, which should be more portable.
Perl Question. I'm trying to get this script running in a debugger.
I've got Aptana + Epic + ActivePerl 5.12.4 working on Windows 7x64. The script is starting fine but I'm getting an error:
curl -sS http://intranet.mycompany.org/directory/directory.xml
The above command works fine... but if I start the debugger I get this error:
curl: (1) Protocol 'http not supported or disabled in libcurl
First part of the script below:
#!/usr/bin/perl
use strict;
use XML::Parser;
use Data::Dumper;
my $url = 'http://intranet.atlanticgeneral.org/directory/directory.xml';
my $output = 'C:\global.gabook';
my $file = "curl -sS '$url' |";
my $parser = new XML::Parser(Style => 'Tree');
my $tree = $parser->parsefile($file)->[1];
Windows doesn't like single quotes in commands. Try using double quotes in the command, using qq{} escaping. Just change one line:
my $file = qq{curl -sS "$url" |};
Wooble~
"I'm guessing it's the extra single quotes around $url that's
causing it"
When I removed the quotes around the '$url' it worked. Quotes worked in redhat perl, but didn't work in my windows perl debugger:
#!/usr/bin/perl
use strict;
use XML::Parser;
use Data::Dumper;
my $url = 'http://intranet.atlanticgeneral.org/directory/directory.xml';
my $output = 'C:\global.gabook';
my $file = "curl -sS $url |";
my $parser = new XML::Parser(Style => 'Tree');
my $tree = $parser->parsefile($file)->[1];
Posting as answer since Wooble didn't.
I was getting the same error when I was using the curl command in my java program as follows
String command = "curl 'http://google.com'";
try
{
Process proc = Runtime.getRuntime().exec(command);
.......
}catch(Exception e){}
Changing command to the following fixed this error
String command = "curl http://google.com";
Actually, It may be an issue because of shell interpretor.
I used curl command like below example
String command = "curl -duser.name=hdfs -dexecute=select+*+from+logdata.test; -dstatusdir=test.output http://hostname:50111/templeton/v1/hive";
As an alternative (and not needing an external program), you could use LWP::UserAgent to fetch the document.
You pass a wrongly spelled protocol part as in "htpt://example.com" or as in the less evident case if you prefix the protocol part with a space as in " http://example.com/".
Source : https://curl.haxx.se/docs/faq.html