program using WWW-Mechanize Module to get the http header information
i have done using LWP::simple below is the code and i require the same using WWW-Mechanize Module.
use strict;
use LWP::simple;
use LWP::UserAgent;
use HTTP::Request;
my $URL = 'https://www.gmail.com/';
my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
my $header = HTTP::Request->new(GET => $URL);
my $request = HTTP::Request->new('GET', $URL, $header);
my $response = $ua->request($request);
print "URL:$URL\nHeaders:\n";
print $response->headers_as_string;
open (FILE, ">output.csv");
print FILE "URL:$URL\nHeaders:\n";
print FILE $response->headers_as_string;
close(FILE);
The synopsis says:
WWW::Mechanize is a proper subclass of LWP::UserAgent and you can also use any of LWP::UserAgent's methods.
use strict;
use warnings;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->get( 'http://www.example.com' );
print $mech->dump_headers();
print " \n ==OR== \n\n";
my $header = HTTP::Request->new(GET => 'http://www.example.com');
my $request = HTTP::Request->new('GET', 'http://www.example.com', $header);
my $response = $mech->request($request);
print $response->headers_as_string;
Prints:
Connection: close
Date: Wed, 26 Jun 2013 03:57:01 GMT
Accept-Ranges: bytes
ETag: "780602-4f6-4db31b2978ec0"
Server: ECS (mdw/13C6)
Content-Length: 1270
Content-Type: text/html; charset=UTF-8
Last-Modified: Thu, 25 Apr 2013 16:13:23 GMT
Client-Date: Wed, 26 Jun 2013 03:57:01 GMT
Client-Peer: 93.184.216.119:80
Client-Response-Num: 1
Title: Example Domain
X-Cache: HIT
X-Meta-Charset: utf-8
X-Meta-Viewport: width=device-width, initial-scale=1
==OR==
Connection: close
Date: Wed, 26 Jun 2013 03:57:01 GMT
Accept-Ranges: bytes
ETag: "780602-4f6-4db31b2978ec0"
Server: ECS (mdw/13C6)
Content-Length: 1270
Content-Type: text/html; charset=UTF-8
Last-Modified: Thu, 25 Apr 2013 16:13:23 GMT
Client-Date: Wed, 26 Jun 2013 03:57:01 GMT
Client-Peer: 93.184.216.119:80
Client-Response-Num: 1
Title: Example Domain
X-Cache: HIT
X-Meta-Charset: utf-8
X-Meta-Viewport: width=device-width, initial-scale=1
Related
Perl 5.36.0 with latest WWW::Mechanize 2.15. I want to get https://web.metro.taipei/img/ALL/timetables/079a.PDF this PDF file's Last-Modified, in which both curl & HTTPie work well:
$ curl -i https://web.metro.taipei/img/ALL/timetables/079a.PDF
HTTP/1.1 200 OK
Content-Type: application/pdf
Last-Modified: Fri, 11 Nov 2022 16:20:50 GMT
Accept-Ranges: bytes
ETag: "93931790e9f5d81:0"
Date: Wed, 23 Nov 2022 05:24:16 GMT
Content-Length: 205866
Strict-Transport-Security: max-age=177211688
Set-Cookie: ...
$ http -p h https://web.metro.taipei/img/ALL/timetables/079a.PDF
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 205866
Content-Type: application/pdf
Date: Wed, 23 Nov 2022 05:24:52 GMT
ETag: "93931790e9f5d81:0"
Last-Modified: Fri, 11 Nov 2022 16:20:50 GMT
Set-Cookie: ...
Strict-Transport-Security: max-age=177211688
But with Perl WWW::Mechanize (based on LWP::UserAgent), it doesn't return the PDF file and instead the server has redirected me to root page https://web.metro.taipei/:
(btw, I need to workaround its SSL certificate verification issue due to lack of intermediate certificate on https://web.metro.taipei/ web server settings. The sectigo-rsa.pem file can be obtained from https://crt.sh/?d=924467857)
#!/usr/bin/env perl
use 5.010;
use strict;
use warnings;
use Data::Dumper;
use JSON;
use WWW::Mechanize;
INIT {
# Workaround with Sectigo's intermediate CA.
my $ua = WWW::Mechanize->new(
agent => 'Monitoring/0.20221123',
ssl_opts => {
SSL_ca_file => 'sectigo-rsa.pem'
},
);
my $res = $ua->get(
'https://web.metro.taipei/img/ALL/timetables/079a.PDF',
);
say $res->base;
# You can see details of redirects with:
say Dumper $res->redirects;
}
__END__
Now $res->base is:
$ ./monitoring-taipei-metro.pl
https://www.metro.taipei/
Also for the result of $res->headers, you can see there is a redirect response about https://www.metro.taipei/:
$VAR1 = bless( {
'_content' => '',
'_rc' => '302',
'_headers' => bless( {
'content-length' => '0',
'client-peer' => '60.244.85.177:443',
'location' => 'https://www.metro.taipei/',
By capturing WWW:Mechanize's raw request (using mitmproxy), I found these headers in the request:
TE: deflate,gzip;q=0.3
Connection: close, TE
It seems that web.metro.taipei will redirect all requests to https://www.metro.taipei/ if Connection: TE exists.
push(#LWP::Protocol::http::EXTRA_SOCK_OPTS, SendTE => 0);
This turns of sending TE header.
I'm trying to find a way to do a Curl request to MailChimps new API v3.0, that will subscribe a user to a given list. Here is what I have thus far:
use warnings;
use WWW::Curl::Easy;
use JSON;
my $apikey = 'xxxx';
my $listid = 'xxxx';
my $email = 'andy#test.co.uk';
my $endpoint = "https://us6.api.mailchimp.com/3.0/lists";
my $json = JSON::encode_json([
'email_address' => $email,
'status' => 'pending',
'merge_fields' => [
'FNAME' => "andy",
'LNAME' => "newby"
]
]);
my $curl = WWW::Curl::Easy->new;
my $url = "$endpoint/$listid/members/" . Digest::MD5::md5(lc($email));
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, $endpoint);
$curl->setopt(CURLOPT_USERPWD, 'user:' . $apikey);
$curl->setopt(CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
$curl->setopt(CURLOPT_TIMEOUT, 10);
$curl->setopt(CURLOPT_CUSTOMREQUEST, 'PUT');
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_POSTFIELDS, $json);
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
# Starts the actual request
my $retcode = $curl->perform;
# Looking at the results...
if ($retcode == 0) {
print("Transfer went ok\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
print "Received response: $response_body\n";
} else {
# Error code, type of error, error message
print "An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n";
}
The documentation is pretty scarce, due to it being a new API. Has anyone had any success with the MailChimp v3 API, for subscribing someone in Perl? (I'm also open to suggestions for command line curl requests... but everything I tried with regards to that, failed with "internal server errors" coming back from MailChimp, which wasn't very helpful)
UPDATE: As suggested below, I enabled verbose, and it now spits out:
Hostname was NOT found in DNS cache
Trying 184.86.100.251...
Connected to us6.api.mailchimp.com (184.86.100.251) port 443 (#0)
successfully set certificate verify locations:
CAfile: none CApath: /etc/ssl/certs
SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
Server certificate:
subject: C=US; ST=GA; L=Atlanta; O=ROCKET SCIENCE GROUP; OU=Rocket Science Group; CN=*.api.mailchimp.com
start date: 2015-09-22 14:39:14 GMT
expire date: 2016-09-22 14:39:13 GMT
subjectAltName: us6.api.mailchimp.com matched
issuer: C=NL; L=Amsterdam; O=Verizon Enterprise Solutions; OU=Cybertrust; CN=Verizon Akamai SureServer CA G14-SHA2
SSL certificate verify ok.
Server auth using Basic with user 'user'
PUT /3.0/lists HTTP/1.1 Authorization: Basic xxxx Host: us6.api.mailchimp.com Accept: / Content-Type: application/json
Content-Length: 108
upload completely sent off: 108 out of 108 bytes < HTTP/1.1 405 Method Not Allowed
Server nginx is not blacklisted < Server: nginx < Content-Type: application/problem+json; charset=utf-8 < Content-Length: 253 <
X-Request-Id: 5f6ab08f-69e7-4c9b-b22a-91714331d5b7 < Link:
https://us6.api.mailchimp.com/schema/3.0/ProblemDetailDocument.json;
rel="describedBy" < Allow: GET, POST < Date: Tue, 13 Oct 2015 11:24:32
GMT < Connection: close < Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/
<
Closing connection 0 Transfer went ok Received response: HTTP/1.1 405 Method Not Allowed Server: nginx Content-Type:
application/problem+json; charset=utf-8 Content-Length: 253
X-Request-Id: 5f6ab08f-69e7-4c9b-b22a-91714331d5b7 Link:
https://us6.api.mailchimp.com/schema/3.0/ProblemDetailDocument.json;
rel="describedBy" Allow: GET, POST Date: Tue, 13 Oct 2015 11:24:32 GMT
Connection: close Set-Cookie: _AVESTA_ENVIRONMENT=prod; path=/
{"type":"http://kb.mailchimp.com/api/error-docs/405-method-not-allowed","title":"Method
Not Allowed","status":405,"detail":"The requested method and resource
are not compatible. See the Allow header for this resource's available
methods.","instance":""}
I'm not really sure what to make of that though :/
Working code: Thanks to TooMuchPete, I managed to get it going. For anyone who may come across this while trying to use the MailChimp API (3.0) in Perl, below is a working sample (you just need to replace the values of the email, name, api key, and list id);
use WWW::Curl::Easy;
use JSON;
use Digest::MD5;
my $apikey = 'xxxx-us6';
my $listid = 'xxxx';
my $email = 'andy#testr.co.uk';
my $endpoint = "https://us6.api.mailchimp.com/3.0/lists";
my $json = JSON::encode_json({
'email_address' => $email,
'status' => 'pending',
'merge_fields' => {
'FNAME' => "andy",
'LNAME' => "newby"
}
});
my $curl = WWW::Curl::Easy->new;
my $url = "$endpoint/$listid/members/" . Digest::MD5::md5(lc($email));
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, $url);
$curl->setopt(CURLOPT_VERBOSE, 1);
$curl->setopt(CURLOPT_USERPWD, 'user:' . $apikey);
$curl->setopt(CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
$curl->setopt(CURLOPT_TIMEOUT, 10);
$curl->setopt(CURLOPT_CUSTOMREQUEST, 'PUT');
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_POSTFIELDS, $json);
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
# Starts the actual request
my $retcode = $curl->perform;
# Looking at the results...
if ($retcode == 0) {
print("Transfer went ok\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
# judge result and next action based on $response_code
print "Received response: $response_body\n";
} else {
# Error code, type of error, error message
print "An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n";
}
I hope this saves someone the grief I had :)
You're attempting to connect to $endpoint instead of $url.
my $url = "$endpoint/$listid/members/" . Digest::MD5::md5(lc($email));
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, $endpoint);
should be:
my $url = "$endpoint/$listid/members/" . Digest::MD5::md5(lc($email));
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, $url);
I received an illegal characters response from MailChimp using the code above until I switched to the md5_base64() call.
Trying to learn Mojolicious here. For the following request, I get 404 when I try to get to
http://hostname:3000/xml
Here is the simple script:
use Mojolicious::Lite;
use Data::Dumper;
get '/xml' => sub {
my $self = shift;
$self->render(xml => "<employees>
<employee>
<id>1001</id>
<name>John Smith</name>
</employee>
<employee>
<id>1002</id>
<name>Jane Dole</name>
</employee>
</employees>"
);
};
app->start;
This script was adopted from an example for json, which works fine. Not sure why xml doesn't work.
Just need to specify a format
get '/xml' => sub {
my $self = shift;
my $xml = <<'XML';
<employees>
<employee><id>1001</id><name>John Smith</name></employee>
<employee><id>1002</id><name>Jane Dole</name></employee>
</employees>
XML
$self->render(data => $xml, format => 'xml');
};
Response header equals the following:
Connection: keep-alive
Server: Mojolicious (Perl)
Content-Length: 140
Content-Type: application/xml
Date: Wed, 09 Apr 2014 05:36:05 GMT
200 OK
Could also place the data in a template, of course:
get '/xml' => sub {
my $self = shift;
$self->render('employees', format => 'xml');
};
app->start;
__DATA__
## employees.xml.ep
<employees>
<employee><id>1001</id><name>John Smith</name></employee>
<employee><id>1002</id><name>Jane Dole</name></employee>
</employees>
i am trying to learn to work with cookies using Perl. following is my code. but i dont know why the cookie is not getting saved in chrome. everytime i run this script a new cookie is created.
#!"C:\wamp\perl\bin\perl.exe" -w
print "Content-type: text/html\n\n";
use CGI::Carp qw( fatalsToBrowser );
use CGI;
my $q=new CGI;
$value=$q->cookie('lol');
$cookie=$q->cookie
(
-name=>'lol',
-value=>'gh',
-expires=>'+7d'
);
print $q->header(-cookie=>$cookie);
$q->start_html
(
-title=>'CGI.pm Cookies'
);
unless($value) {print "cookie is goint to set";}
else {print "Hi $value";}
$q->end_html;
exit;
Here's the output of your script:
Content-type: text/html
Set-Cookie: lol=gh; path=/; expires=Sat, 04-May-2013 11:16:12 GMT
Date: Sat, 27 Apr 2013 11:16:12 GMT
Content-Type: text/html; charset=ISO-8859-1
cookie is goint to set
You send the Content-Type response header twice: first, on line 2, and again on line 16 when you print $q->header(-cookie => $cookie).
In fact, the double newline on line 2 ends your HTTP headers. So the output of $q->header(-cookie => $cookie) will be treated as document body content, not as HTTP headers.
Quickest solution? Comment out line 2.
Your forgot to send your cookie to the client:
print header(-cookie=>$cookie);
I am making a request to remote perl server. but got the problem as
XMLHttpRequest cannot load http://otherdomain.com/getPub.pl?content=hello. Origin http://localhost is not allowed by Access-Control-Allow-Origin.
I already enable access_control_allow_origin to be "*" in perl script, codes as following:
#!/usr/bin/perl
use strict;
use CGI qw(:standard);
use warnings;
my $cgi = new CGI;
print $cgi -> header(
-type => 'text/plain',
-access_control_allow_origin => '*',
);
my $content = $cgi -> param('content');
open(CON,">content.txt") || die "can't open $!";
print CON $content;
close(CON);
and the js request as followings:
function sendData(){
var url = "http://otherdomain.com/getPub.pl?content=hello";
var xhr = createCORSRequest("GET", url);
if(!xhr){
throw new Error ('CORS not supported');
}
xhr.send();
}
function createCORSRequest(method, url){
var xhr = new XMLHttpRequest();
if("withCredentials" in xhr){
xhr.open(method,url,true);
}else if(typeof XDomainRequest != "undefined"){
xhr = new XDomainRequest();
xhr.open(method, url);
}else{
xhr = null;
}
return xhr;
}
the response header as:
Allow:GET,HEAD,POST,OPTIONS,TRACE
Connection:Keep-Alive
Content-Length:0
Content-Type:text/plain; charset=UTF-8
Date:Mon, 07 Jan 2013 16:55:44 GMT
Keep-Alive:timeout=15, max=99
Server:Apache/2.2.3 (CentOS)
What is the matter?
It finally works in PHP, although I still didn't see the difference.
Summary as :
1.
When using perl as:
my $cgi = new CGI;
print $cgi -> header(
-type => 'text/plain',
-access_control_allow_origin => '*',
-access_control_allow_headers => 'content-type,X-Requested-With',
-access_control_allow_methods => 'GET,POST,OPTIONS',
-access_control_allow_credentials => 'true',
);
The HTTP headers as:
Request header:
Request Method:OPTIONS
Status Code:200 OK
Request Headersview source
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Access-Control-Request-Headers:origin, x-requested-with, content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:example.org
Origin:http://localhost
Referer:http://localhost/testCORS.html
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11
Response header:
Allow:GET,HEAD,POST,OPTIONS,TRACE
Connection:Keep-Alive
Content-Length:0
Content-Type:text/plain; charset=UTF-8
Date:Tue, 08 Jan 2013 05:52:26 GMT
Keep-Alive:timeout=15, max=100
Server:Apache/2.2.3 (CentOS)
2.
BUT in PHP, it works!!!: I didn't see the differences!
code as:
<?php
header("Access-Control-Allow-Origin: *");
header("Access-Control-Allow-Methods: GET,POST,OPTIONS");
header("Access-Control-Allow-Headers: X-Requested-With,");
#header("Access-Control-Allow-Credentials: true");
?>
Response header :
Access-Control-Allow-Headers:X-Requested-With
Access-Control-Allow-Methods:GET,POST,OPTIONS
Access-Control-Allow-Origin:*
Connection:Keep-Alive
Content-Length:0
Content-Type:text/html; charset=UTF-8
Date:Tue, 08 Jan 2013 05:52:10 GMT
Keep-Alive:timeout=15, max=100
Server:Apache/2.2.3 (CentOS)
X-Powered-By:PHP/5.3.3
You need to make sure that your server is responding to an OPTIONS request to that URL with the proper Access-Control-Allow-Origin header. The browser will "preflight" your request by first making an OPTIONS request. If that fails, it will not try your request at all.