Correct way of rewriting `curl ...` to perl - perl

Question

I wrote a program that requests the source and the response header of a webpage, now I need it to run cross platform. I used the external command curl (in linux) to achieve it. I get the source like this::
#!/usr/bin/perl -w
use strict;
#declaring variables here#
my $result = `curl 'https://$host$request' -H 'Host: $host' -H 'User-Agent: $useragent' -H 'Accept: $accept' -H 'Accept-Language: $acceptlanguage' --compressed -H 'Cookie: $cookie' -H 'DNT: $dnt' -H 'Connection: $connection' -H 'Upgrade-Insecure-Requests: $upgradeinsecure' -H 'Cache-Control: $cachecontrol'`;
print "$result\n";
And the response header like this:
#!/usr/bin/perl -w
use strict;
#declaring variables here#
my $result = `curl -I 'https://$host$request' -H 'Host: $host' -H 'User-Agent: $useragent' -H 'Accept: $accept' -H 'Accept-Language: $acceptlanguage' --compressed -H 'Cookie: $cookie' -H 'DNT: $dnt' -H 'Connection: $connection' -H 'Upgrade-Insecure-Requests: $upgradeinsecure' -H 'Cache-Control: $cachecontrol'`;
print "$result\n";
These work fine, but I need to call these in perl and not as external commands.
I wrote some code using LWP::UserAgent to get the source:
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
#declaring variables here#
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => "https://$host$request HTTP/1.1");
$req->header('Host' => "$host");
$req->header('User-Agent' => "$useragent");
$req->header('Accept' => "$accept");
$req->header('Accept-Language' => "$acceptlanguage");
$req->header('Accept-Encoding' => "$acceptencoding");
$req->header('Cookie' => "$cookie");
$req->header('DNT' => "$dnt");
$req->header('Connection' => "$connection");
$req->header('Upgrade-Insecure-Requests' => "$upgradeinsecure");
$req->header('Cache-Control' => "$cachecontrol");
my $resp = $ua->request($req);
if ($resp->is_success) {
my $message = $resp->decoded_content;
print "$message\n";
}
This sometimes runs fine, but sometimes decoded_content returns nothing, I do get a response and i can print it using content, but its still encoded.
And requesting response headers using LWP::UserAgent is not possible so I wrote the request using Net::HTTP:
#!/usr/bin/perl -w
use strict;
use Net::HTTP;
#declaring variables here#
my $s = Net::HTTP->new(Host => "$host") || die $#;
$s->write_request(GET => "$request", 'Host' => "$host", 'User-Agent' => "$useragent", 'Accept' => "$accept", 'Accept-Language' => "$acceptlanguage", 'Accept-Encoding' => "$acceptencoding", 'Cookie' => "$cookie", 'DNT' => "$dnt", 'Connection' => "$connection", 'Upgrade-Insecure-Requests' => "$upgradeinsecure", 'Cache-Control' => "$cachecontrol");
my #headers;
while(my $line = <$s>) {
last unless $line =~ /\S/;
push #headers, $line;
}
print #headers;
This returns
HTTP/1.1 302 Found
Content-Type: text/html; charset=UTF-8
Connection: close
Content-Length: 0
Is the problem with my syntax of am I using the wrong tools? I know that WWW::Curl::Easy can request the source and the header at the same time, but I don't know how to pass my variables to its request. Could someone tell me what the problem is or just rewrite these requests correctly using the same variables with WWW:Curl::Easy? I'd appreciate a solution using WWW::Curl::Easy. Thanks in advance.

Answer 1

You can get the response headers in a couple of ways with LWP. Demonstrated here:
use LWP::UserAgent;
my($host,$request) = ('example.com', '/my/request');
my #header=( [Host => $host],
['User-Agent' => 'James Bond 2.0'],
[Accept => 'text/plain'],
[Cookie => 'cookie=x'],
);
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => "https://$host$request"); #dont add HTTP/1.1
$req->header(#$_) for #header;
my $resp = $ua->request($req);
if ($resp->is_success) {
my %h; $resp->headers->scan( sub{ $h{shift()}=shift() } );
printf "Header name: %-30s Value: %-30s\n", $_, $h{$_} for sort keys %h;
print "\n<<<".$resp->headers()->as_string.">>>\n\n"; #all header lines in one big string
print $resp->header('Content-Type'),"\n\n"; #get one specific header line
my $content = $resp->decoded_content;
print "$content\n";
}
Note: "HTTP/1.1" should not be a part of the string after GET =>.
And with calling curl as a sub process you don't need to call it twice. You can get both headers and content at once by using -i like this:
my $response = ` curl -s -i "http://somewhere.com/path" -H 'User-Agent: Yes' `;
my($headers,$content) = split /\cM?\cJ\cM?\cJ/, $response, 2;
print "Headers: <<<$headers>>>\n\n";
print "Content: <<<$content>>>\n\n";