How do I tell WWW::Mechanize to ignore a secure cookie? - perl

I need to work with a legacy CGI program and I am writing tests for it. I use Test::WWW::Mechanize::CGI for that. The application runs on https in production, and the home-made session handling simply throws out a cookie, which has the secure option set.
my $cookie = $q->cookie(
-name => 'session',
-value => 'foobar',
-expires => '+24h',
-secure => 1, # this is the culprit
-httponly => 1,
-samesite => 'Strict',
;
While this makes sense under the https URL in production, it breaks my tests because I don't have SSL support there.
The obvious solution would be to put in a switch that only enables this option on the cookie if there is SSL available, but I don't want to do that at this point. Instead, I want to find out how to disable this thing from the testing end.
Here's an example that illustrates what I'm talking about. It uses things in CGI.pm that I would usually discourage people from using. Please bear with me to understand the issue.
use strict;
use warnings;
use CGI;
use Test::WWW::Mechanize::CGI;
use Test::More;
my $mech = Test::WWW::Mechanize::CGI->new;
$mech->cgi(
sub {
my $q = CGI->new;
if ( $q->param('behind_login') ) {
# check if we've got the session cookie
if ( $q->cookie('session') ) {
print $q->header, $q->start_html('Logged in'), $q->h1('Welcome back'), $q->end_html;
}
else {
print $q->header( 'text/plain', '403 Unauthorized' );
}
}
else {
# this is where the user gets logged in
my $cookie = $q->cookie(
-name => 'session',
-value => 'foobar',
-expires => '+24h',
-secure => 1, # this is the culprit
-httponly => 1,
-samesite => 'Strict'
);
print $q->header( -cookie => $cookie ),
$q->start_html('Hello World'),
$q->h1('Hello World'),
$q->end_html;
}
}
);
$mech->get_ok('http://localhost/');
$mech->get_ok('http://localhost/?behind_login=1');
done_testing;
If this program is run, the first test will pass, and the second one will fail. If the marked line with the -secure option is commented out, the second test will pass, too.
I've rummaged around in LWP::UserAgent a bit, but haven't found where this could be disabled. I'm aware that this is the default behavior and that it's good that it behaves like this.
There might be an option to turn this off that I have failed to see when I was studying the docs, but it's likely there is not. I am fine with monkey-patching this thing away once I understand where to do that.

The solution is trivial. Just call get_ok with https URL. Mechanize will simply do the right thing. The request will know that it's secure, and stuff will work.
$mech->get_ok('https://localhost/');
$mech->get_ok('https://localhost/?behind_login=1');
There is no need to monkey-patch anything at all.

Related

Adding language variable into WWW::Mailchimp (subscription)

I'm trying to work out how I can use WWW::Mailchimp ( http://search.cpan.org/~arcanez/WWW-Mailchimp/ ) to sign someone up to our list, but also assign the language of the person (i.e english, french, german, spanish, etc).
Here is what I have thus far:
my $mailchimp = WWW::Mailchimp->new(apikey => 'xxxx' );
$mailchimp->listSubscribe( id => "xxx", email_address => $in->{Email}, merge_vars => [ FNAME => $name[0], LNAME => $name[1], mc_language => "fr", LANG => "fr", LANGUAGE => "fr" ] );
mc_language => "fr", LANG => "fr", LANGUAGE => "fr" doesn't seem to do anything (been trying all the params I see laying around, in the vain hope one of them works!)
While it works (and asks you to confirm your subscription), all the language variables are ignored. Looking at their documents, I'm a bit confused as to what to use:
https://apidocs.mailchimp.com/api/2.0/lists/subscribe.php
The code "fr" is ok, but I'm unsure what params to pass along to it.
Has anyone had any experience with this before? Apart from the language, it works fine (but I need to be able to send the confirmation emails in their own language, and then also filter down when doing mailings)
UPDATE: Ok, so it looks like its not going to be a simple case of updating to the newer API. I've been looking into the v3.0 API, and its a total overhaul of the older one (new function names, new ways of sending requests, etc). What I'm going to do is look into a "Curl" method, so we can at least get it going with that. Once I've got that going, I'll probably have a look at coding something to work with LWP::UserAgent, as that'd be cleaner than doing lots of curl requests. Shame there isn't anything out there already for Perl and MailChimp (with the new API, or even version 2.0!)
From looking at the source, it defaults to API 1.3:
has api_version => (
is => 'ro',
isa => Num,
lazy => 1,
default => sub { 1.3 },
);
The documentation for that shows you need to use MC_LANGUAGE:
string MC_LANGUAGE Set the member's language preference. Supported
codes are fully case-sensitive and can be found here.
It looks like the module just shoves whatever data structure you provide into JSON and POSTs it to Mailchimp, so the appropriate Mailchimp API doc version for the API you target should be referenced as a primary source.
Ok, so I got there in the end! I have been talking with MailChimp support, and they were very helpful. Turns out it was a double issue.
1) Auto-Translate needed to be enabled for the list in question. This was their answer around that:
After taking a look at the call, it appears to be set up properly now, so you are all good on that front. That being said, I am seeing
that the Auto-translate option doesn't seem to be enabled for any of
your lists. In order for the Confirmation and all other response
emails to automatically translate, this will need to be enabled for
all of the lists being used.
We have a bit of additional information on that, here, if you'd like to check that out:
http://kb.mailchimp.com/lists/signup-forms/translate-signup-forms-and-emails#Auto-Translate-Forms
2) When making the request via the API, you need to specifically set the Accept-Language: xx value. For example, en, fr, es, de, etc.
Here is a working function for anyone who needs it in the future. Just be sure to update the apikey,listId and endpoint URL.
do_register_email_list('foo#bar.com','Andrew Test',"en")
sub do_register_email_list {
# (email,name,lang)
use WWW::Curl::Easy;
use Digest::MD5;
use JSON;
my #name = split /\s+/, $_[1];
my $apikey = 'xxxx-us6';
my $listid = 'xxxx';
my $email = $_[0];
my $endpoint = "https://us6.api.mailchimp.com/3.0/lists";
my $lang = $_[2]||'en';
my $json = JSON::encode_json({
'email_address' => $email,
'status' => 'pending',
'language' => $lang,
'merge_fields' => {
'FNAME' => $name[0]||'',
'LNAME' => $name[1]||''
}
});
my $curl = WWW::Curl::Easy->new;
my $url = "$endpoint/$listid/members/" . Digest::MD5::md5(lc($email));
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, $url);
# $curl->setopt(CURLOPT_VERBOSE, 1);
$curl->setopt(CURLOPT_USERPWD, 'user:' . $apikey);
$curl->setopt(CURLOPT_HTTPHEADER, ['Content-Type: application/json',"Accept-Language: $lang"]);
$curl->setopt(CURLOPT_TIMEOUT, 10);
$curl->setopt(CURLOPT_CUSTOMREQUEST, 'PUT');
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_POSTFIELDS, $json);
# A filehandle, reference to a scalar or reference to a typeglob can be used here.
my $response_body;
$curl->setopt(CURLOPT_WRITEDATA,\$response_body);
# Starts the actual request
my $retcode = $curl->perform;
#print "FOO HERE";
# Looking at the results...
if ($retcode == 0) {
print "Transfer went ok\n";
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
print "Received response: $response_body\n";
} else {
# Error code, type of error, error message
print "An error happened: $retcode ".$curl->strerror($retcode)." ".$curl->errbuf."\n";
}
}
Hopefully this saves someone else from all the grief I had with it :) (the MailChimp support lady also said she will get their team to add something about this in the developer notes, so its made a bit clearer!)

Twitter LED Timeline

Hello I have put together a script for a twitter timeline it works apart from i dont know how to authorize my twitter api key my led sign is just saying "Bad Authentication data"
here is my code
#!/usr/bin/perl
require LWP::UserAgent;
use JSON;
my $lwpua = LWP::UserAgent->new;
my $uagent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6";
my #header = ( 'Referer' => 'http://api.twitter.com/', 'User-Agent' => $uagent );
my $twuser = '<twitter_name>';
my $twurl = "http://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=$twuser";
my $response = $lwpua->get( $twurl, #header );
my $return = $response->content;
my $json = JSON->new->allow_nonref;
my $json_text = $json->decode($return);
my #tweets = #{$json_text};
my $message;
foreach $tweet (#tweets) {
$message .= $tweet->{text} . "\n";
}
use Device::MiniLED;
my $sign = Device::MiniLED->new( devicetype => "sign" );
$sign->addMsg(
data => "$message",
effect => "scroll",
speed => 4
);
$sign->send( device => "/dev/ttyUSB0" );
1;
First: use strict; and use warnings;. Even if you're the awesomest programmer ever, this should be your first port of call if you're having problems. (And everyone makes typos).
Secondly: $json_text is a hash ref, not an array ref. You probably want to use values or similar.
Thirdly: Bad Authentication Data is a twitter api error, not a code error. You need to authorize it with oAuth, and you're doing no twitter auth at all. From: https://dev.twitter.com/overview/api/response-codes
215 - Typically sent with 1.1 responses with HTTP code 400. The method requires authentication but it was not presented or was wholly invalid.
E.g. you can't do what you're doing without authenticating. I think what you need is this:
https://dev.twitter.com/oauth/overview/application-owner-access-tokens
Specifically - the 'easy answer' is generate an account specific authentication token, and send that in your request.
Once you have done this this web page on Twitter:
https://dev.twitter.com/oauth/tools/signature-generator/
allows you to generate a (time limited) example command that you could use to fetch your timeline. But you'll most likely need to build authentication into your script in order to do what you're trying to do. (user_timeline.json is a restricted access API). (There's a 'test oAuth' button on the app webpage).
Reading through the docs on creating authentication tokens, makes me thing that installing Net::Twitter or Net::Twitter::Lite might be the way to go.
First follow the instructions here: https://dev.twitter.com/oauth/overview/application-owner-access-tokens
Specifically, on https://apps.twitter.com/ you need to:
create an application
generate a token (from the application page).
(Under 'keys and access tokens' on the app specific page).
This will give you the 4 things you need to speak to twitter:
a consumer key
a consumer secret
an access token
an access token secret
By default, your access token will be read only. This is fine for what you want to do.
The process for turning them into Twitter auth is a bit complicated and involves RSA-HMAC encryption. So just let Net::Twitter do it for you: (I've removed my keys, because I'm not quite daft enough to post the equivalent of my Twitter password)
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use Net::Twitter;
my $twitter = Net::Twitter->new(
traits => [qw/API::RESTv1_1/],
consumer_key => 'long_string_of_stuff',
consumer_secret =>
'long_string_of_stuff',
access_token => '12345-long_string_of_stuff',
access_token_secret =>
'long_string_of_stuff',
ssl => 1,
);
my $tweets = $twitter->user_timeline();
print Dumper \$result;
my $message;
foreach my $tweet ( #{$tweets} ) {
$message .= $tweet->{text} . "\n";
}
print $message;
Tested this with my account, and it prints a list of my recent tweets, which I think was what you wanted?

How to remove login form from this CGI::Application example?

In this tutorial he creates a custom login form, just to show how it is done. Please search for
How do I remove the custom login and fall back to the default?
To code looks like this
sub cgiapp_init {
my $self = shift;
my %CFG = $self->cfg;
# ...
$self->authen->config(
DRIVER => [ 'Authen::Simple::LDAP',
host => '',
basedn => '',
],
STORE => 'Session',
LOGOUT_RUNMODE => 'logout',
LOGIN_RUNMODE => 'login',
POST_LOGIN_RUNMODE => 'okay',
RENDER_LOGIN => \&my_login_form,
);
$self->authen->protected_runmodes(
'mustlogin',
);
}
sub login : Runmode {
my $self = shift;
my $url = $self->query->url;
my $user = $self->authen->username;
if ($user) {
my $message = "User $user is already logged in!";
my $template = $self->load_tmpl('default.html');
$template->param(MESSAGE => $message);
$template->param(MYURL => $url);
return $template->output;
} else {
my $url = $self->query->self_url;
unless ($url =~ /^https/) {
$url =~ s/^http/https/;
return $self->redirect($url);
}
return $self->my_login_form;
}
}
Update
Here is mentions that CGI::Application have a default login that looks better than his.
Line 159 specifies a subroutine to use
to generate a login form. Note that
the Authentication plugin comes with a
default form that you can use. I'm
including this one just to demonstrate
how to go about creating one of your
own, in case you really want to. The
default one actually looks much better
than mine, so you might wish to
comment out line 159!
I'm the author of that tutorial. Sorry for the confusion!
What I should have said is "comment out lines 157, 158, and 159 of Login.pm".
To use the default form that's built in to the CGI::Application::Plugin::Authentication module, you don't need to specify LOGIN_RUNMODE, POST_LOGIN_RUNMODE, or RENDER_LOGIN.
Those are all provided just to help you customize your login page. I included a customized
version in the tutorial thinking that most people would need to know how to do so.
The default one actually looks much better than mine, so you might wish to comment out line 159!
Comment out line 159.

OpenID authentication to Google Apps via Perl and Net::OpenID::Consumer fails

I asked this over on Google's support forums for Apps integration, but got zero response. Maybe somebody here can help steer me in the right direction.
I'm trying to integrate a Perl application with Google Apps, and I'm having some trouble with the OpenID authentication. I've been using this PHP tutorial from Google as a kind of reference, since there are no Perl examples I can find.
My initial file, index.cgi (referred by manifest.xml, and the starting point of the OpenID transaction) is as follows:
use Net::OpenID::Consumer;
use CGI;
# ...
my $q = CGI->new();
my $domain = $q->param('domain');
if (!$domain) {
print $q->header(), 'Provide domain please.';
exit 0;
}
# my website
my $root = 'http://www.example.com/';
my $csr = Net::OpenID::Consumer->new(
# The user agent which sends the openid off to the server
ua => LWP::UserAgent->new,
# Who we are
required_root => $root,
# Consumer Key Secret from Google Apps Marketplace
consumer_secret => 'Zzzzzz9zzAAAAA....'
);
my $claimed_id = $csr->claimed_identity(
'https://www.google.com/accounts/o8/site-xrds?hd=' . $domain);
if ($claimed_id) {
my $check_url = $claimed_id->check_url(
# Upon validation, the user will be returned here, and real
# work may begin
return_to => $root . '/return.cgi',
trust_root => $root
);
print $q->redirect($check_url);
}
else {
print $q->header(), "Error";
}
This part seems to be working. That is, I get redirected to return.cgi with a bunch of openid.* parameters. However, at this point I get the following error:
no_identity_server The provided URL doesn't declare its OpenID identity server
I'm using the latest version of the Net::OpenID::Consumer module.
Here are the significant bits of return.cgi:
my $q = CGI->new();
my $csr = Net::OpenID::Consumer->new(
ua => LWP::UserAgent->new,
# The root of our URL
required_root => 'http://www.example.com/',
# Our password.
consumer_secret => 'Zzzzzz9zzAAAAA....',
# Where to get the information from.
args => $q
);
print $q->header();
$csr->handle_server_response(
not_openid => sub {
print "That's not an OpenID message. Did you just type in the URL?";
},
setup_required => sub {
my $setup_url = shift;
print 'You need to do something here.';
},
cancelled => sub {
print 'You cancelled your login.';
},
verified => sub {
my $vident = shift;
my $url = $vident->url;
print "You are verified as '$url'. ** FIN **";
},
error => sub { die "Can't figure it out: ", #_; }
);
As you can imagine, I'm wanting the verified sub to fire, but instead I'm getting an error. Anything obvious I'm missing? Any help would be appreciated.
So the solution, it turns out, is to switch modules. I changed to the skimpily documented Net::Google::FederatedLogin, and things are now working. The code is as follows (substitute example.com below for your actual developer's domain).
In your Google Apps Marketplace vendor profile, add the URL to index.cgi in the Application Manifest:
...
<Url>http://www.example.com/index.cgi?from=google&domain=${DOMAIN_NAME}</Url>
...
Then add the following code to your servers.
index.cgi
use CGI;
use Net::Google::FederatedLogin;
my $q = CGI->new();
my $domain = $q->param('domain');
if (!$domain) {
print $q->header(), 'Provide domain please.';
exit 0;
}
my $fl = Net::Google::FederatedLogin->new(
claimed_id =>
'https://www.google.com/accounts/o8/site-xrds?hd=' . $domain,
return_to =>
'http://www.example.com/return.cgi',
extensions => [
{
ns => 'ax',
uri => 'http://openid.net/srv/ax/1.0',
attributes => {
mode => 'fetch_request',
required => 'email',
type => {
email => 'http://axschema.org/contact/email'
}
}
}
] );
print $q->redirect($fl->get_auth_url());
return.cgi
use CGI;
use Net::Google::FederatedLogin;
my $q = CGI->new();
print $q->header();
my $fl = Net::Google::FederatedLogin->new(
cgi => $q,
return_to =>
'http://www.example.com/return.cgi' );
eval { $fl->verify_auth(); };
if ($#) {
print 'Error: ' . $#;
}
else {
# we've authenticated and gotten attributes --
my $ext = $fl->get_extension('http://openid.net/srv/ax/1.0');
print $ext->get_parameter('value.email');
}
(For a full sample, plus OAuth access to user data, see this post on my blog.)
In some cases, reportedly, this is caused by Perl missing Net::SSL, making it fail on Google's SSL URLs.
The step where it's failing is where it performs discovery on the identifier asserted by Google's server. For some reason discovery on that identifier URL is failing. It would be helpful to know what OpenID identifier the Google server is asserting to try to debug why discovery is not working for it.
I'm not sure what $domain is there, but you may need to escape it:
use URI::Escape 'uri_escape';
....
my $claimed_id = $csr->claimed_identity(
'https://www.google.com/accounts/o8/site-xrds?hd=' . uri_escape($domain) );
Also, the consumer_secret used by Net::OpenID::Consumer has no relationship to any other secret.

How can I log in to YouTube using Perl?

I am trying to write a Perl script to connect to me YouTube account but it doesnt seem to work. Basically I just want to connect to my account but apparently it is not working. I don't even have an idea on how I could debug this! Maybe it is something related to https protocol?
Please enlighten me! Thanks in advance.
use HTTP::Request::Common;
use LWP::UserAgent;
use strict;
my $login="test";
my $pass = "test";
my $res = "";
my $ua = "";
# Create user agent, make it look like FireFox and store cookies
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051213 Firefox/1.0.7");
$ua->cookie_jar ( {} );
# Request login page
$res = $ua->request(GET "https://www.google.com/accounts/ServiceLogin?service=youtube&hl=en_US&passive=true&ltmpl=sso&uilel=3&continue=http%3A//www.youtube.com/signup%3Fhl%3Den_US%26warned%3D%26nomobiletemp%3D1%26next%3D/index");
die("ERROR1: GET http://www.youtube.com/login\n") unless ($res->is_success);
# Now we login with our user/pass
$res = $ua->request(
POST "https://www.google.com/accounts/ServiceLoginAuth?service=youtube",
Referer => "http://www.youtube.com/login",
Content_Type => "application/x-www-form-urlencoded",
Content => [
currentform => "login",
next => "/index",
username => $login,
password => $pass,
action_login => "Log+In"
]
);
# YouTube redirects (302) to a new page when login is success
# and returns OK (200) if the login failed.
#die("ERROR: Login Failed\n") unless ($res->is_redirect());
print $res->content;
what i am doing is learning the web features of perl, so i dont want to use any library except wwwlib or mechanize to get the job done.
how can i just connect to my account using a perl script? this is my objective for now
hope someone can post a script or correct mine.
thank you guys for you help.
i am testing Webscarab now..
What data are you trying to grab? Why not just using an existing implementation like WebService::YouTube
Some comments on your code: I always avoided the shortcut $ua->request(GET/POST) method since I always ended up needing more flexibility that only the use of HTTP::Request and HTTP::Response allowed. I always felt the code was cleaner that way too.
Why is your code not working? Who knows.
Make sure your cookiejar is adding your cookies to the outgoing HTTP::Request. I'd suggest dumping all your headers when you do it in a browser and compare with the headers and data that libwww is sending. There may be some additional fields that they are checking for that vary for every hit. They may be checking for your UserAgent string. If you are just looking to learn libwww I'd suggest using a different site as a target as I'm sure YouTube has all sort of anti-scripting hardening.
Are you using YouTube's stable documented API?
Use an HTTP proxy such as WebScarab to watch the data flow.
Trey's suggestion to use somebody else's CPAN package for the mechanics is a good idea too.
Right right by and large, what you want to do is define a cookiejar for most of these websites that have a redirection login. This is what the package has done. Also the package tunes a lot of the lookups and scrapes based on the youtube spec.
Ajax content for example will be rough since its not there when your scraping
You just picked a somewhat rough page to start out with.
Enjoy
I'm actually working on this issue myself. Before, I would suggest read over this the API guide from Google as a good starting reference. If I'm reading it correctly, one begins with passing user credentials through a REST interface to get a Authentication Token. To handle that, I'm using the following:
sub getToken {
my %parms = #_;
my $response = LWP::UserAgent->new->post(
'https://www.google.com/youtube/accounts/ClientLogin',
[
Email => $parms{'username'},
Passwd => $parms{'password'},
service => "youtube",
source => "<<Your Value Here>>",
]
);
my $content = $response->content;
my ($auth) = $content =~ /^Auth=(.*)YouTubeUser(.*)$/msg
or die "Unable to authenticate?\n";
my ($user) = $content =~ /YouTubeUser=(.*)$/msg
or die "Could not extract user name from response string. ";
return ($auth, $user);
}
And I call that from the main part of my program as such:
## Get $AuthToken
my ($AuthToken, $GoogleUserName) = getToken((
username => $email, password => $password
));
Once I have these two things -- $AuthToken and $GoogleUserName, I'm still testing the LWP Post. I'm still writing this unit:
sub test {
my %parms = #_;
## Copy file contents. Use, foy's three param open method.
my $fileSize = -s $parms{'File'};
open(VideoFile, '<', "$parms{'File'}") or die "Can't open $parms{'File'}.";
binmode VideoFile;
read(VideoFile, my $fileContents, $fileSize) or die "Can't read $parms{'File'}";
close VideoFile;
my $r = LWP::UserAgent->new->post(
"http://uploads.gdata.youtube.com/feeds/api/users/$parms{'user'}/uploads",
[
Host => "uploads.gdata.youtube.com",
'Authorization' => "AuthSub token=\"$parms{'auth'}\"",
'GData-Version' => "2",
'X-GData-Key' => "key=$YouTubeDeveloperKey",
'Slug' => "$parms{'File'}",
'Content-Type' => "multipart/related; boundary=\"<boundary_string>\"",
'Content-Length' => "<content_length>",
'video_content_type'=> "video/wmv",
'Connection' => "close",
'Content' => $fileContents
]
);
print Dumper(\$r->content)
}
And that is called as
&test((auth=>$Auth, user=>$user, File=>'test.wmv'));