I'm having a bit of difficulty with HTTP::Request in Perl,
My script is set up like so:
use CGI;
use MIME::Base64;
use HTTP::Cookies;
use HTTP::Request;
use LWP::UserAgent;
$request = HTTP::Request->new(GET => 'http://www.example.com/');
$ua = LWP::UserAgent->new;
$response = $ua->request($request);
print "$response";
When I run the script on my website, I get a message like so:
HTTP::Response=HASH(0x987f8d8)
(I'm trying to get it to "print" example.com)
Yes its a hash. You have to access like below.
print $res->decoded_content(); ## if gziped
print $res->content();
print $res->status_line;
Also you can use the Data Dumper to print the whole $res and observe what actually it holds.
use Data::Dumper;
print Dumper $res;
http::Request returns indeed a hash.
$response->content has the result.
From the link:
$r->content( $bytes )
This is used to get/set the content and it is inherited from the HTTP::Message base class. See HTTP::Message for details and other methods that can be used to access the content.
Note that the content should be a string of bytes. Strings in perl can contain characters outside the range of a byte. The Encode module can be used to turn such strings into a string of bytes.
You need to do this ...
print $response->content();
... take a look at the docs http://metacpan.org/pod/LWP#An-Example
Related
I need some insight on my Perl CGI script.
First of all all this is running under webmin so i'm doing a custom module.
I'm calling a CGI Perl script passing 2 parameter from another Perl CGI. The link I'm calling is in the following format:
http://IP:8080/foobar/alat.cgi?sysinfo=xxxxxxx&SR=yyyyyyyy
The alat.cgi script look like this
#!/usr/bin/perl
use CGI qw(:standard);
ReadParse();
$q = new CGI;
my $dir = $in->param('SR');
my $s = $in->param('sysinfo');
ui_print_header(undef, $text{'edit_title'}.$dir, "");
print $dir."<br>";
print $s"<br>";
The only output I get printed is the value of $dir and $s seems to be empty.
What am I doing wrong?
As #Сухой27 said, add use strict;, but also use warnings; to the top of your script, right below the shebang (#!/usr/bin/perl) line. Those will tell you about syntax errors and other stuff where Perl is doing something other than you might intend.
With CGI (which is btw not part of the Perl core in the latest 5.22 release any more) and the object oriented approach you are tyring to take, you don't need to use ReadParse(). That is an abomination left in from Perl 4's cgilib.pl times.
I don't know what your ui_print_header function does. I'm guessing it outputs a bunch of HTML. Are you sure you defined it?
With fixing all your syntax errors and using modern syntax, your program would look like this. I'll break down what is happening for you.
#!/usr/bin/perl
use strict;
use warnings;
use CGI;
my $q = CGI->new;
my $dir = $q->param('SR');
my $s = $q->param('sysinfo');
# you need to declare this to use it below
my %text = ( edit_title => 'foo' );
# we declare this sub further down
ui_print_header(undef, $text{'edit_title'} . $dir, q{});
print $dir . '<br />';
print $s . '<br />';
sub ui_print_header {
my ( $foo, $title, $dir, $bar ) = #_;
# do stuff here...
}
Let's look at some of the things I did here.
Saying new CGI as the CGI docs suggest is fine, but since we are using the OOP way you can use the more common CGI->new. It's the same thing really, but it's consistent with the rest of the OOP Perl world and it's more clear that you are calling the new method on the CGI package.
If you have $q, keep using it. There is no $in.
Declare all your variables with my.
Declare %text so you can use $text{'edit_title'} later. Probably you imported that, or ommitted it from the code you showed us.
Declare ui_print_header(). See above.
q{} is the same as '', but it's clearer that it's an empty string.
thank you everyone for the very quick answer, and as I was suspecting I just had some silly mistake.
Adding here the corrected code that now works
#!/usr/bin/perl
# Run alat on selected sysinfo and allow display of output
#use strict;
use diagnostics;
require 'recoverpoint-lib.pl';
use CGI qw(:standard);
ReadParse();
my $q = new CGI;
my $dir = $q->param('SR');
my $s = $q->param('sysinfo');
ui_print_header(undef, $text{'edit_title'}.$dir, "");
print $dir."<br>";
print $s."<br>";
Just to clarify for some of previous answer, this is a custom module of webmin so variable $text is imported and function ui_print_header is a webmin defined one, it basically print the page header in HTML
As you enable strict and warnings you can easily know the errors.Also you should check Apache error logs, I think the script should be like this:
#!/usr/bin/perl
use CGI qw(:standard);
use strict;
use warnings;
ReadParse();
my $q = new CGI;
my $dir = $q->param('SR');
my $s = $q->param('sysinfo');
ui_print_header(undef, $text{'edit_title'}.$dir, "");
print $dir."<br>";
print $s."<br>";
I am trying to write a basic webscraping program in Perl. For some reason it is not working correctly and I don't have the slightest clue as to why.
Just the first part of my code where I am getting the content (just saving all of the HTML code from the webpage to a variable) does not work with certain websites.
I am testing it by just printing it out, and it does not print anything out with this specific website. It works with some other sites, but not all.
Is there another way of doing this that will work?
#use strict;
use LWP::Simple qw/get/;
use LWP::Simple qw/getstore/;
## Grab a Web page, and throw the content in a Perl variable.
my $content = get("https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650");
print $content;
You have a badly-written web site there. The request times out with a 500 Internal Server Error.
I can't suggest how to get around it, but the site almost certainly uses JavaScript as well which LWP doesn't support, so I doubt if an answer would be much use to you.
Update
It looks like the site has been written so that it goes crazy if there is no Accept-Language header in the request.
The full LWP::UserAgent module is necessary to set it up, like this
use strict;
use warnings;
use LWP;
my $ua = LWP::UserAgent->new(timeout => 10);
my $url = 'https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650';
my $resp = $ua->get($url, accept_language => 'en-gb,en', );
print $resp->status_line, "\n\n";
print $resp->decoded_content;
This returns with a status of 200 OK and some HTML.
To interact with a website that uses Javascript, I would advise that you use the following module:WWW::Mechanize::Firefox
use strict;
use warnings;
use WWW::Mechanize::Firefox;
my $url = "https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650"
my $mech = WWW::Mechanize::Firefox->new();
$mech->get($url);
print $mech->status();
my $content = $mech->content();
I have the following script:
#!/usr/bin/perl
use strict;
use warnings;
# Downloading the XML
use LWP::Simple;
my $url = "http://meny.no/WebServices/AJAXService.asmx/GetMarkets";
my $content = get($url);
# Creating data from XML
use XML::Simple;
use Data::Dumper;
my $xml = new XML::Simple;
my $data = $xml -> XMLin($content);
my $newdata = Dumper($data);
So what i wonder about is what type of datastorage do i have with $newdata, and how to loop over the data? I want to insert the data into database.
Data::Dumper::Dumper returns a serialized string representation of a Perl data structure. You can't really loop over a string in a meaningful sense, which makes me wonder why you ran this data through Dumper in the first place. Did you copy the code from somewhere? Generally, Dumper is used when you want to inspect data for debugging purposes or if you want to store data in a Perl-readable format.
The structured data that I presume you want to work with is in the $data variable, returned by the XML parser.
I have Perl CGI pages, (.pl extension). How Can I get a persons raw user agent string? There are ways of doing it in Javascript, (which I have been), though I'd rather move over to completely Perl, rather then having some Javascript and some Perl.
In example, to get a person's IP: $ENV{REMOTE_ADDR}
User agent is stored in $ENV{HTTP_USER_AGENT}
Use the CGI module:
#!/usr/bin/perl -wT
use strict;
use CGI qw(:standard);
use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
print header;
print start_html("Environment");
foreach my $key (sort(keys(%ENV))) {
print "$key = $ENV{$key}<br>\n";
}
print end_html;
As people have already mentioned, it's always available in the $ENV{HTTP_USER_AGENT} variable. But if you're using CGI.pm, you can also get it by calling the user_agent() function.
I'm trying to inspect cookie values in a cgi script; my test script looks like
#!/usr/bin/perl -w
use DBI;
use CGI qw/:standard/;
use CGI::Cookie;
my $cgiH = CGI->new;
print header;
print start_html(-title=>'Cookie Terms'), h1("Cookie Terms"), "<hr>\n";
%cookies = CGI::Cookie->fetch;
foreach $k (keys %cookies) {
my $term = "$cookies{$k}";
my $term =~ s/SubjectTerm//;
print "at $k is $term \n";
}
print end_html;
the relevant input to the script (from an HTTP GET) is
Cookie: SubjectTerm1=ponies
Cookie: SubjectTerm2=horses
(this is verified by using either fiddler or debugging the code in my delphi app). the result of my script (omitting the HTML wrapper) is either
at SubjectTerm1 is
at SubjectTerm2 is
or if I change the print statement to
print "at $k is $cookies{$k}\n";
it is
at SubjectTerm1 is SubjectTerm1=ponies; path=/
at SubjectTerm2 is SubjectTerm2=horses; path=/
What I want to arrive it is something like this
at SubjectTerm1 is ponies
at SubjectTerm2 is horses
I know I'm missing something about the hash usage but can't quite figure out what it is.
am I not addressing the %cookies hash correctly?
If you had use warnings you would see you are clobbering the $term with the line
my $term =~ s/SubjectTerm//;
Remove the my.
I think you might be able to extract the value is simply changing the assignment to
my $term = $cookies{$k}->value();
and get rid of the s/SubjectTerm//. But not sure about this sorry.
As an aside, before I answer the question, I'd strongly recommend you look at using something other than CGI.pm -- it's the best of late-90s Perl, and other than being included by default in the core perl distribution has little to recommend it.
Some more modern, more featureful alternatives include:
CGI::Application - http://cgi-app.org/
Mojolicious - http://mojolicio.us/
Dancer - http://www.perldancer.org/
Beyond that, as #Sodved says if you'd included use strict; and use warnings; at the top of your file the first of your problems -- clobbering $term by using my twice -- would have been highlighted immediately.
Once you're using the strict and warnings pragmas (and I'd say, don't leave home without them) you'll also want to change your foreach to either:
foreach (keys %cookies) {
my $term = $cookies{$_};
...
OR
foreach my $k (keys %cookies) {
my $term = $cookies{$k};
...
That is, you'll either need to declare the $k variable, or you may prefer to use Perl's built-in $_ inside the loop -- it's purely a matter of style.
Good luck!