I am trying to convert this shell command into perl I am using use WWW::Curl::Easy;
I want to write the verbose in a different file and i want to get the URL content into a different file . Till now i have been able to curl the URL and add header to it as well .
Now I want to just write the as shell command mentions output from 1> into a file and
2> into antohter file as it is there in shell
my $curl = WWW::Curl::Easy->new();
$curl->setopt(CURLOPT_HEADER,1);
$curl->pushopt(WWW::Curl::Easy::CURLOPT_HTTPHEADER,['AUTH_USER:John']);
$curl->setopt(WWW::Curl::Easy::CURLOPT_URL,"www.abc.com");
$curl->setopt(CURLOPT_VERBOSE,1);
how do i put the content of www.abc.com that is
<html>
<body>
<marquee>abc is up </marquee>
</body>
</html>
into one file
And this into another file
> GET / HTTP/1.1
Host: abc-01
Accept: */*
AUTH_USER:hojn
< HTTP/1.1 200 OK
......
A cursory reading of the documentation and libcurl docs reveals that you probably need:
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
my $response_header;
$curl->setopt(CURLOPT_WRITEHEADER,\$response_header);
When using libcurl, it's useful to have its documentation handy.
Browsing through the options, we find CURLOPT_HEADERFUNCTION & WRITEHEADER and CURLOPT_WRITEFUNCTION & WRITEDATA.
use strict;
use warnings;
use Net::Curl::Easy qw( /^CURLOPT_/ );
my $url = 'http://stackoverflow.com/';
my $easy = Net::Curl::Easy->new();
open(my $fh_header, '>:raw', 'header.out') or die $!;
open(my $fh_data, '>:raw', 'data.out' ) or die $!;
$easy->setopt(CURLOPT_URL, $url);
$easy->setopt(CURLOPT_WRITEHEADER, $fh_header);
$easy->setopt(CURLOPT_WRITEDATA, $fh_data);
$easy->perform();
Note: I used Net::Curl over WWW::Curl because I know and trust it. The above should also work with WWW::Curl as long as it provides a Perl-aware default for CURLOPT_WRITEFUNCTION and CURLOPT_HEADERFUNCTION. It it doesn't, you'll need to provide values for those options too.
Related
I have an issue with downloading a page,
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
I can browse following with a browser but when I run bash command in perl or linux shell,
GET $url >OUTPUT1; # Even it does not write anything to file "OUPUT1"
When I try wget, It downloads but not correct ,I mean with --> <title>Error - Nucleotide - NCBI</title>. I want the page with items , but it returns me a page without items.
my $html = qx{wget --quiet --output-document=OUTPUT1 $url};
**Note: I noticed a few minutes ago, url is ok with Mozilla firefox, but it can not be browsed via google chrome. it is weird, probably my issue related with this too. Any idea?
Code from link:
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
my $html = qx{wget --quiet --output-document=OUTPUT11 $url};
# wget get something, but it does not get items, it gets what I get via google chrome
`GET $url2 >OUTPUT11`; # it does not write anything to file,
OK, given your code - the problem is almost certainly one of interpolation. Because the & in your URL is going to be interpreted by the shell you're spawning as 'background this process'.
That's almost certainly not what you want. Why not just use LWP natively?
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $url='http://www.ncbi.nlm.nih.gov/nuccore?linkname=pcassay_nucleotide&from_aid=504934,1806,1805,1674';
my $content = get $url;
print $content;
open ( my $output_fh, '>', 'output.html' ) or die $!;
print {$output_fh} $content;
close ( $output_fh );
I have written a Java Applet as a school project and I need a CGI file to create a file in the cgi-bin directory. The problem is when I run the code from the browser, the code executes but my file is not created with the variable name. Nothing is created. Here is the code
#!/usr/bin/perl -wT
use CGI;
print "content-type: text/plain\n\n";
my $q = CGI->new();
my $name = $q->param('username');
my $pw = $q->param('param');
my $bool = $q->param('bool');
my $rel = $q->param('rel');
my $ext = ".txt";
my $strt = "../cgi-bin/";
my $app = $strt . $name . $ext;
print $app;
open (FILE,'>',$app) or print "Error";
print FILE $pw . "\n";
print FILE $bool . "\n";
print FILE $rel;
close(FILE);
When I run the cgi it prints the $app variable and it is the correct address I want but the file is not created. If I change the line
open (FILE,'>',$app) or print "Error";
to
open (FILE,'>','../cgi-bin/test.txt') or print "Error";
it creates the file where I want it. Any ideas why this would happen when using the variable $app? Either way I never get Error printed to the browser.
SOLUTION:
Thanks guys for the help. When using:
use CGI::Carp qw(fatalsToBrowser);
I got this error:
Content-type: text/html
<H1>Software error:</H1>
<PRE>Insecure dependency in open while running with -T switch
</PRE>
<P>
For help, please send mail to the webmaster (or webmaster), giving this error message
and the time and date of the error.
It seems it was not liking the -T. Once I removed that it worked. Thanks again
Why you use ../cgi-bin to write into cgi-bin ?
Just use:
open (FILE, ">$name$ext") or die $!;
and use CGI::Carp qw(fatalsToBrowser); to carp fatals on the browser (suitable for this debug) with file creation
-T is Perl's "tainted data" flag. It stops you from doing unsafe operations with untrusted data. Yes, your script works without the -T flag, but now you have a very insecure script.
If someone passes in a username value of ../../../../../../../../home/badguy/secret, then you will write the username and password into secret.txt in badguy's home directory. -T prevents you from doing that. That's why -T exists.
I'm unable to execute the HTML::Template function in the CGI.
I'm following a simple tutorial that I found here: http://metacpan.org/pod/HTML::Template
I created a new file on my server in the home path as test.tmpl.
I created a new file named frt.cgi ... (is that the issue here? should it be a different file extention??)
#!/usr/local/bin/perl -w
use HTML::Template;
# open the html template
my $template = HTML::Template->new(filename => '/test.html');
# fill in some parameters
$template->param(HOME => $ENV{HOME});
$template->param(PATH => $ENV{PATH});
# send the obligatory Content-Type and print the template output
print "Content-Type: text/html\n\n", $template->output;
I've modified the 1st line to reflect my host provided program path for perl. I don't know what the -w does I just know I've tried this with and without it. Also I've tried changing the code a bit like this:
use warnings;
use strict;
use CGI qw(:standard);
use HTML::Template;
I've searched...
https://stackoverflow.com/search?q=HTML%3A%3ATEMPLATE+&submit=search
https://stackoverflow.com/search?q=HTML%3A%3ATEMPLATE
https://stackoverflow.com/search?q=HTML%3A%3ATEMPLATE+PERL&submit=search
Yet I still do not see the answer.
I even searched google for .TMPL Encoding because I thought there may be some special type needed. Please help.
If you look in your server logs, you'll probably see an error message along the lines of:
HTML::Template->new() : Cannot open included file /test.html : file not found.
You need to provide the path on the file system, not a URI relative to the generated document.
First, you likely specified the wrong path - change /test.html to test.html.
Also, it is possible that there is no $ENV{HOME} variable in your system so set up flag die_on_bad_params to 0:
my $template = HTML::Template->new(
filename => 'test.html',
die_on_bad_params => 0,
);
Also, don't forget to mark your Perl file as executable by chmod 755.
Option -w makes Perl to enable warnings, so there is no point to write use warnings; afterwards.
You can check what Perl command line options do by using module B::Deparse, like this ($^W variable disables/enables warnings):
perl -w -MO=Deparse -e "print;"
This would print:
BEGIN { $^W = 1; }
print $_;
I'm executing remote commands using Net::OpenSSH using a web frontend. My commands return without failure on the command line, but I get nothing in a web browser. I've done a couple hour research to no avail--any ideas?
Here is some code to give you an example (some removed for obvious reasons).
#!/usr/bin/perl -w
use strict;
use CGI ':standard';
use Net::OpenSSH;
# Here in the code is just the header and standard tags
print "1";
print "2"; # both display
my $ssh = Net::OpenSSH->new($host, user => $uname, key_path => $key); # all works
$ssh- error and die "Can't ssh to host" . $ssh->error;
print "3";
$ssh->system("uname -a") or
die "remote command failed: " . $ssh->error;
my #lsa = $ssh->capture("ls -a");
$ssh->error and
die "remote ls command failed: ". $ssh->error;
print "4";
print "5";
print #lsa; # won't display in browser, just terminal/CLI
Cheers!
I maintain CGI.pm. I recommend these additions to your simple script:
Before you print anything else, print the standard HTTP header: print header();
Add this after the use CGI line: use CGI::Carp qw(fatalsToBrowser); ... that will display any run-time problems in the browser. If you don't get any output after these changes, check that the script compiles with perl -cw script.pl
Below is about the minimum Perl code that worked for me on Debian machine. I suggest you go through it and compare it to your actual code.
However, it did not work out-of-the box on my Debian, I had make some decisions most of which probably aren't very safe, but that's more about specific environment:
make home for user that server runs writable (/var/www)
add host to ~/.ssh/known_hosts beforehand
use the strict_mode => 0 to bypass Net::OpenSSH's security checks instead of finding proper
ctl_dir (Net::OpenSSH requires that the folder and all above folders are 0755 or more strict,
so /tmp I used is normally not good enough)
I believe there are much safer practices than that, but as I said, that's specific to environment.
So the code:
#!/usr/bin/perl
use strict;
use warnings;
use Net::OpenSSH;
use File::Temp qw/ tempdir /;
# necessary minimum for CGI
print "Content-type: text/plain\n\n";
# prepare temp dir
my $temp = tempdir("/tmp/sshme.pl-XXXXXXXX", CLEANUP => 1);
# open SSH session
my %opts = (
user => "user",
password => "password",
ctl_dir => $temp,
strict_mode => 0 ## NOT recommended - see my comments
);
my $ssh = Net::OpenSSH->new("host", %opts);
$ssh->error
and die "Couldn't establish SSH connection: ". $ssh->error;
# perform command and print output
my #lines = $ssh->capture("ls")
or die "remote command failed: " . $ssh->error;
print #lines;
Perhaps your errors get directed to standard error, not standard output. In that case, they'll usually end up in the server log, not the browser window. Perhaps you can use POSIX::dup2 to avoid this:
use POSIX;
# Make sure to send HTTP headers before redirecting streams!
POSIX::close(2); # close original stderr stream. No more output to apache logs!!!
POSIX::dup2(1,2); # redirect error stream to standard output, so errors will appear in browser.
Processes launched by perl, like e.g. some ssh binary, will inherit these streams.
I'm trying to get the response of a curl call into a variable in perl.
my $foo = `curl yadd yadda`;
print $foo;
does not work. When I run this at the command line the curl call prints all its output correctly in the terminal, but the variable is not filled with that data.
Is there a way to do this without installing and calling the Perl curl lib?
It probably sends its stuff to stderr. Try
my $foo = `curl yadd yadda 2>&1`;
You also might consider looking at LWP::UserAgent or even LWP::Simple.
What do you really want to do? Use curl at all costs, or grab the contents of a web page?
A more perlish way of doing this (which relies on no external programs that may or may not be installed on the next machine where you need to do this) would be:
use LWP::Simple;
my $content = get("http://stackoverflow.com/questions/1015438/")
or die "no such luck\n";
If you want to see why the GET failed, or grab multiple pages from the same site, you'll need to use a bit more machinery. perldoc lwpcook will get you started.
In the shell 2> means redirect fileno 2. Fileno 2 is always what a program sees as stderr. Similarly, fileno 0 is stdin and fileno 1 is stdout. So, when you say 2>&1 you are telling the shell to redirect stderr (fileno 2) into stdout (fileno 1). Since the backticks operator uses the the shell to run the command you specify, you can use shell redirection, so
my $foo = `curl yadda yadda 2>&1`;
is telling curl to redirect its output into stdout, and since the backtick operator catches stdout, you get what you were looking for.
Very old post, but the real way of using curl in backticks is using the appropriate switch of curl.
This switch is -o which defines where to send the output.
More from the curl man page:
Specifying the output as '-' (a single dash) will force the output to
be done to stdout.
This also prevents having possible errors in $foo, which would happen if you redirect the complete STDERR to STDOUT on errors:
my $foo = `curl -o - yadd yadda`;
Try this:
$var = `curl "http://localhost" 2>/dev/null`;
print length($var)
curl displays progress information on stderr, redirecting that to /dev/null makes it easier to see what's going on.
This works on my system:
#!/usr/bin/perl
use strict;
use warnings;
my $output = `curl www.unur.com`;
print $output;
__END__
C:\> z1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
etc.
You can open a pipe as if it were a file.
$url = "\"http://download.finance.yahoo.com/d/quotes.csv?s=" .
"$symbol&f=sl1d1t1c1ohgvper&e=.csv\"";
open CURL, "curl -s $url |" or die "single_stock_quote: Can't open curl $!\n";
$line = <CURL>;
close CURL;
It might be that some of the output you want to capture is in standard err, not standard out. Try this:
my $foo = system "curl http://www.stackoverflow.com";
print $foo;