I wrote a simple perl code:
use HTTP::Request::Common qw(POST);
use strict;
use warnings;
use LWP;
my $ua = new LWP::UserAgent(agent => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5');
my $res = $ua->get('http://60606.com/members/search',Cookie => 'incomerr_refid=0; ff_language_id=3; _t_=1; fcode2=4139; ss_time=1527663671; PHPSESSID=otfmog35f0iva0uld1rgld2kj7; _webpush_key_=none; dinged_id=1412807; _language_id=3; _gnum_=950212; refid=1; first_key=TOCz19ls8HfkFW6LoBrIj35p4xSyYPVE; last_valid_url=https%3A//60606.com/login; ping_last=1527663675.939');
print $res->content;
but when I run it I get this error:
Too many header lines (limit is 128) at C:/Perl64/lib/Net/HTTP/Methods.pm line 377
Please tell me why I get this error, I think it's for cookie format
Please tell me why I get this error, I think it's for cookie format
The error speaks about the number of lines in the header. That's not related to the cookie header, as that is only a single line.
The place where Net::HTTP::Methods complains is when it reads the incoming response header lines. Those are set to 128 by default, but this can be changed with an argument.
However, I haven't figured out a way yet to inject a different value. I've reproduced the problem stand-alone though.
$ perl -Mojo -E \
'a(sub ($c) { $c->res->headers->header("X-$_" => $_) for 1..128; $c->render(text => "Hello") })->start' \
daemon
This will give you a server on localhost:3000 that responds with loads of header lines. If you then make a response, the same problem occurs.
my $res = $ua->get('http://localhost:3000/');
Unfortunately that does not help us very much.
Related
I have a question here:
For some reason when I call get request in perl script, I am getting only half page.
However this is happening only on certain machines, but on some machines it is returning whole page as expected.
The requested page size is only 64 kb, and perl is running on windows 7 (RAM 8GB, 64 bit OS).
I just wondering if anyone have seen this issue before? Is there any way to solve it?
Thank you!
Here is my code
use strict;
use warnings;
use File::Slurp;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->default_header( 'Authorization' => 'Basic [encoded credential]' );
$ua->max_size(2000000000) ;
my $resp = $ua->get( 'url', ':content_file'=>'test.txt' );
say $resp->status_line;
my $content = read_file('test.txt');
print $content;
I've been going around on this for some time now and can't quite get it. This is Perl 5 on Ubuntu. I have a drop down list on my web page:
$output .= start_form . "Student: " . popup_menu(-name=>'student', -values=>['', #students], -labels=>\%labels, -onChange=>'Javascript:submit()') . end_form;
It's just a set of names in the form "Last, First" that are coming from a SQL Server table. The labels are created from the SQL columns like so:
$labels{uc($record->{'id'})} = $record->{'lastname'} . ", " . $record->{'firstname'};
The issue is that the drop down isn't displaying some Unicode characters correctly. For instance, "Søren" shows up in the drop down as "Søren". I have in my header:
use utf8;
binmode(STDOUT, ":utf8");
...and I've also played around with various takes on the "decode( )" function, to no avail. To me, the funny thing is that if I pull $labels into a test script and print the list to the console, the names appear just fine! So what is it about the drop down that is causing this? Thank you in advance.
EDIT:
This is the relevant functionality, which I've stripped down to this script that runs in the console and yields the correct results for three entries that have Unicode characters:
#!/usr/bin/perl
use DBI;
use lib '/home/web/library';
use mssql_util;
use Encode;
binmode(STDOUT, ":utf8");
$query = "[SQL query here]";
$dbh = &connect;
$sth = $dbh->prepare($query);
$result = $sth->execute();
while ($record = $sth->fetchrow_hashref())
{
if ($record->{'id'})
{
$labels{uc($record->{'id'})} = Encode::decode('UTF-8', $record->{'lastname'} . ", " . $record->{'nickname'} . " (" . $record->{'entryid'} . ")");
}
}
$sth->finish();
print "$labels{'ST123'}\n";
print "$labels{'ST456'}\n";
print "$labels{'ST789'}\n";
The difference in what the production script is doing is that instead of printing to the console like above, it's printing to HTTP:
$my_output = "<p>$labels{'ST123'}</p><br>
<p>$labels{'ST456'}</p><br>
<p>$labels{'ST789'}</p>";
$template =~ s/\$body/$my_output/;
print header(-cookie=>$cookie) . $template;
This gives, i.e., strings like "Zoë" and "Søren" on the page. BUT, if I remove binmode(STDOUT, ":utf8"); from the top of the production script, then the strings appear just fine on the page (i.e. I get "Zoë" and "Søren").
I believe that the binmode( ) line is necessary when writing UTF-8 to output, and yet removing it here produces the correct results. What gives?
Problem #1: Decoding inputs
53.C3.B8.72.65.6E is the UTF-8 encoding for Søren. When you instruct Perl to encode it all over again (by printing it to handle with the :utf8 layer), you are producing garbage.
You need to decode your inputs ($record->{id}, $record->{lastname}, $record->{firstname}, etc)! This will transform The UTF-8 bytes 53.C3.B8.72.65.6E ("encoded text") into the Unicode Code Points 53.F8.72.65.6E ("decoded text").
In this form, you will be able to use uc, regex matches, etc. You will also be able to print them out to a handle with an encoding layer (e.g. :encoding(UTF-8), or the improper :utf8).
You let on that these inputs come from a database. Most DBD have a flag that causes strings to be decoded. For example, if it's a MySQL database, you should pass mysql_enable_utf8mb4 => 1 to connect.
Problem #2: Communicating encoding
If you're going to output UTF-8, don't tell the browser it's ISO-8859-1!
$ perl -e'use CGI qw( :standard ); print header()'
Content-Type: text/html; charset=ISO-8859-1
Fixed:
$ perl -e'use CGI qw( :standard ); print header( -type => "text/html; charset=UTF-8" )'
Content-Type: text/html; charset=UTF-8
Hard to give a definitive solution as you don't give us much useful information. But here are some pointers that might help.
use utf8 only tells Perl that your source code is encoded as UTF-8. It does nothing useful here.
Reading perldoc perlunitut would be a good start.
Do you know how your database tables are encoded?
Do you know whether your database connection is configured to automatically decode data coming from the database into Perl characters?
What encoding are you telling the browser that you have encoded your HTTP response in?
I am trying to write a basic webscraping program in Perl. For some reason it is not working correctly and I don't have the slightest clue as to why.
Just the first part of my code where I am getting the content (just saving all of the HTML code from the webpage to a variable) does not work with certain websites.
I am testing it by just printing it out, and it does not print anything out with this specific website. It works with some other sites, but not all.
Is there another way of doing this that will work?
#use strict;
use LWP::Simple qw/get/;
use LWP::Simple qw/getstore/;
## Grab a Web page, and throw the content in a Perl variable.
my $content = get("https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650");
print $content;
You have a badly-written web site there. The request times out with a 500 Internal Server Error.
I can't suggest how to get around it, but the site almost certainly uses JavaScript as well which LWP doesn't support, so I doubt if an answer would be much use to you.
Update
It looks like the site has been written so that it goes crazy if there is no Accept-Language header in the request.
The full LWP::UserAgent module is necessary to set it up, like this
use strict;
use warnings;
use LWP;
my $ua = LWP::UserAgent->new(timeout => 10);
my $url = 'https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650';
my $resp = $ua->get($url, accept_language => 'en-gb,en', );
print $resp->status_line, "\n\n";
print $resp->decoded_content;
This returns with a status of 200 OK and some HTML.
To interact with a website that uses Javascript, I would advise that you use the following module:WWW::Mechanize::Firefox
use strict;
use warnings;
use WWW::Mechanize::Firefox;
my $url = "https://jobscout.lhh.com/Portal/Page/ResumeProfile.aspx?Mode=View&ResumeId=53650"
my $mech = WWW::Mechanize::Firefox->new();
$mech->get($url);
print $mech->status();
my $content = $mech->content();
I have a perl cgi program which output to a simple html form for user data input.
The form is in chinese big5 charset
When opened the cgi script, I have to manual switch web browser charset encoding to big5.
I searched on google and I found a method to set charset. Then
original code
$q = new CGI;
print $q->header;
to new code
$q = new CGI;
print $q->header(-charset=>'big5');
However, it just output a blank html.
This works for me:
use CGI;
my $q = CGI->new();
print $q->header(-charset => 'big5');
print '簡體字';
When i try it, it will be showed correctly. (Make sure, that your script is also saved in big5).
If those are the only two lines, then it's probably working.
Run the cgi from command line and you should see:
Content-Type: text/html; charset=big5
You're printing headers, but no content, so the page will be blank. Use Firebug or similar to verify the response from the server.
I have a perl script that I am running from a browser. I'm passing query variables to it, but the query variables are showing up like johndoe%40test%2Ecom instead of johndoe#test.com
I would like to debug the variables to see how they are showing up in the beginning of the script.
In PHP it would be when accessing the url:
http://localhost/dump-variables.php?foo=1&bar=2&bis=johndoe%40test%2Ecom
The PHP script would include this code:
header('Content-type: text/plain');
print_r($_GET);
And the result would be:
Array
(
[foo] => 1
[bar] => 2
[bis] => johndoe#test.com
)
Perl version : 5.010001
Important Note
I don't have any special access (root / sudo) on the system I'm working on - so modules may have to be included via source...
You've tagged this CGI, so I'll first suggest using something more efficient.
But if you are going to use CGI, and assuming the use of CGI.pm, then (from the CGI.pm docs):
use CGI;
use Data::Dump qw/ddx/;
my $q = CGI->new();
print $q->header('text/plain');
my %params = $q->Vars;
print "Content-type: text/plain\n\n"; # had to add this
ddx \%params;
Try Data::Printer http://metacpan.org/pod/Data::Printer
Just add
use DDP;
...
p $_GET