Same code different URL not working? - perl

OK, this is driving me mad. I currently run a background script on our server that logs in to our SagePay account every 5 minutes and updates currently logged in users on our local database.
SagePay rolled out a new admin system some time ago but due to many complaints kept the old version running alongside so I never bothered to update my script. SagePay have now decided to decommission the old version so I have no choice.
The following code is just the first part of script that logs in to SagePay:
use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Request;
use HTTP::Headers;
use HTTP::Request::Common qw(POST);
my $request = "";
my $content = "";
my $req = "";
my $cookie_jar = HTTP::Cookies->new();
my $ua = LWP::UserAgent->new();
$ua->cookie_jar($cookie_jar);
$ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
# LOGIN TO SAGEPAGY
$req = (POST 'https://portal.sagepay.com/myoldskool/loginpage.asp',
["vendorname" => "MYVENDORNAME",
"username" => "MYUSERNAME",
"password" => "MYPASSWORD",
"clickedButton" => "login"]);
$request = $ua->request($req);
$content = $request->content;
print $content;
This code works perfectly every time and prints the HTML of the SagePay dashboard as it would if you logged in.
The new POST url is https://portal.sagepay.com/mysagepay/j_spring_security_check and the parameters that need to be passed are pretty much the same, so technically I should just need to make the following alterations to the code and it should work:
$req = (POST 'https://portal.sagepay.com/mysagepay/j_spring_security_check',
["inp_vspvendorname" => "MYVENDORNAME",
"inp_username" => "MYUSERNAME",
"inp_password" => "MYPASSWORD",
"CSRFToken" => ""]);
When I change this code in my script $content prints nothing and I cannot for the life of me figure out why. I've even tried using HttpRequester Firefox addon and sent the same params to the same URL and that returns the dashboard HTML as expected.
$request->status_line is '302 Found'.
Obviously I can't give you my login details for SagePay but even just getting it to return the HTML for an invalid login, at least I'm making progress. I've been stuck on this for a few days now, I'm hoping someone can help.

You're receiving an HTTP 302 response, which is a redirect. By default, LWP::UserAgent doesn't follow redirects with POST requests. To change this, alter the user agent's requests_redirectable array before sending the request:
push #{ $ua->requests_redirectable }, 'POST';
my $response = $ua->request($req);

Related

Facebook login via Perl WWW::Mechanize, login error "Cookies Required" [duplicate]

This question already has answers here:
Python Mechanize log into Facebook cookie error
(2 answers)
Closed 6 years ago.
I have a Perl script that I run in Strawberry Perl in which I login to Facebook using WWW::Mechanize. For a short background, I use my script to automate an occasional "leaderboard" for a fun little game I invented to be played among my group of friends. I assign points based upon certain posts, and the objective is to earn the most points. Very simple. This script is complete, and it's worked for me in the past (I last ran it 4 days ago). However, I am unable to login to Facebook today. I dumped the content after the form submission to an HTML file, and it appears that I've returned back to the login screen with a red error box:
Cookies Required
Cookies are not enabled on your browser. Please enable cookies in your browser preferences to continue.
From what I understand, WWW::Mechanize features automatic cookies -- in other words by the default constructor, the bot should be accepting cookies. Cookies should be enabled by the browser. I should not see this screen, right?
I've read and experimented with WWW:Mechanize and HTTP::Cookies, but no matter what I try (fooling around with the cookie jar every which way), I cannot get past this "cookies required" error.
This is my code without any experimental fluff, very simple.
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->get("https://www.facebook.com/login.php");
$mech->submit_form(
fields => {
email => '<my email here>',
pass => '<my password here>',
}
);
open($out, ">", "output_page.html") or die "Can't open output_page.html: $!";
print $out $mech->content;
Capture a session of you logging in with your browser (if you're using firefox you can use firebug, and check what HTTP headers are set - how-to. Set the same headers in your script, including the user agent. Check if the same cookies are set. You may end up with something like the below:
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
my $cookie_jar = HTTP::Cookies->new(
hide_cookie2 => 1,
);
my $mech = WWW::Mechanize->new(
cookie_jar => $cookie_jar,
);
$mech->agent_alias('Windows Mozilla');
$mech->add_header( 'Connection' => 'keep-alive' );
$mech->add_header( 'Accept-Language' => 'en-GB,en;q=0.5' );
$mech->add_header( 'Accept-Encoding' => 'gzip, deflate' );
$mech->add_header( 'Accept' =>
'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' );

Unable to obtain cookie-based access to Iframe within a HTML page

I am trying to automate sms sending through Way2sms in Perl LWP. I am able to login successfully. Then I click the Send SMS link in the browser to go the sms-sending page. From there, based upon the page url and the url of the iframe within which the sms fields are located, I try to construct the absolute URL of the page to which the form should be posted, and post it with the correct parameters (you can see it all in the image). However, the sms isn't being sent. Can anybody tell me what am I doing wrong here? (There is a similar module in CPAN which accomplishes this through Mechanize, I am trying a different approach)
use LWP::UserAgent;
use HTTP::Cookies;
my $cookie_jar = HTTP::Cookies->new(
file => "cookies.txt",
autosave => 1,
);
my $ua = LWP::UserAgent->new(
agent =>
'Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/20100101 Firefox/14.0.1',
cookie_jar => $cookie_jar,
);
my $response = $ua->post(
'http://site2.way2sms.com/contentt/bar/Login1.action',
{
username => $mob,
password => $pass,
}
);
if ( $response->is_redirect ) {
$response = $ua->get( $response->header('Location') );
print 5 if $response->decoded_content =~ /Kaustav Mukherjee/i; #prints it, showing that the login is successful
}
my $smsresp = $ua->post("http://site5.way2sms.com/jsp/quicksms.action",[MobNo=>$mob,textArea=>'Hello World']);
if ( $smsresp->is_redirect ) {
$smsresp = $ua->post($smsresp->header('Location'),[MobNo=>$mob,textArea=>'Hello World']);
}
Here is an indirect answer that may help you out:
If you have to use this website, try using something like CasperJS. There may be some JavaScript magic that needs to happen.
Maybe you could use something like Google Voice or some other smarter service to automate send text messages. Looks like someone on CPAN already has a sweet Google::Voice module for this.

Loggin into website with LWP and Perl

Somewhat inexperienced programmer here trying to write a program to log into my courses site and download all the content (lectures homeworks etc). Obviously it is a password protected site so I have to give it that. I understand LWP::UserAgent and the likes well enough, and that I need to use credentials. What I cannot figure out is how to get to the next page. I can go to the log-in, but how does perl get the result of my log-in?
code example (i pulled out the log-info obviously):
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $url = 'login URL';
$ua -> credentials(
$url,
'',
'user',
'pass'
);
my $response = $ua ->get($url);
print $response->content;
the content from response is the same content as what I would have got as if I had not passed any credentials. Obviously I'm missing something here....
Oh one other thing my own courses site does not have a unique url as far as I know.
You probably want to be using WWW::Mechanize, a subclass of LWP::UserAgent designed to act more like a browser, allowing you to navigate through pages of a website with cookie storage already taken care of for you.
You only use credentials if the site uses HTTP basic auth, in which case you don't "log in", you just pass the credentials with every request.
If the site has a form based login system, then you need to use cookie_jar and request the form's action URI with whatever data it expects.
#!/usr/bin/perl
use LWP::UserAgent;
use HTTP::Cookies;
my $ua=LWP::UserAgent->new(timeout => 20);
$ua->agent('Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.8) Gecko/20100202 MRA 5.5 (build 02842) Firefox/3.5.8');
$ua->requests_redirectable(0);
my $cook = HTTP::Cookies->new;
$ua->cookie_jar($cook);
print = requester('http://urlexample/login.php', 'login=yourlogin&password=pass' )->as_string;
sub requester
{
my $type = 'GET';
if($_[1]){$type = 'POST'}
my $req = HTTP::Request->new($type => $_[0]);
$req->content_type('application/x-www-form-urlencoded; charset=UTF-8');
if($_[1]){$req->content($_[1])}
my $res = $ua->request($req);
return $res;
}

Why am I getting a new session ID on every page fetch in my Perl WWW::Mechanize script?

So I'm scraping a site that I have access to via HTTPS, I can login and start the process but each time I hit a new page (URL) the cookie Session Id changes. How do I keep the logged in Cookie Session Id?
#!/usr/bin/perl -w
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
use LWP::Debug qw(+);
use HTTP::Request;
use LWP::UserAgent;
use HTTP::Request::Common;
my $un = 'username';
my $pw = 'password';
my $url = 'https://subdomain.url.com/index.do';
my $agent = WWW::Mechanize->new(cookie_jar => {}, autocheck => 0);
$agent->{onerror}=\&WWW::Mechanize::_warn;
$agent->agent('Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100407 Ubuntu/9.10 (karmic) Firefox/3.6.3');
$agent->get($url);
$agent->form_name('form');
$agent->field(username => $un);
$agent->field(password => $pw);
$agent->click("Log In");
print "After Login Cookie: ";
print $agent->cookie_jar->as_string();
print "\n\n";
my $searchURL='https://subdomain.url.com/search.do';
$agent->get($searchURL);
print "After Search Cookie: ";
print $agent->cookie_jar->as_string();
print "\n";
The output:
After Login Cookie: Set-Cookie3: JSESSIONID=367C6D; path="/thepath"; domain=subdomina.url.com; path_spec; secure; discard; version=0
After Search Cookie: Set-Cookie3: JSESSIONID=855402; path="/thepath"; domain=subdomain.com.com; path_spec; secure; discard; version=0
Also I think the site requires a CERT (Well in the browser it does), would this be the correct way to add it?
$ENV{HTTPS_CERT_FILE} = 'SUBDOMAIN.URL.COM'; ## Insert this after the use HTTP::Request...
Also for the CERT In using the first option in this list, is this correct?
X.509 Certificate (PEM)
X.509 Certificate with chain (PEM)
X.509 Certificate (DER)
X.509 Certificate (PKCS#7)
X.509 Certificate with chain (PKCS#7)
When your user-agent isn't doing something you think it should be doing, compare it's requests with that of an interactive browser. A Firefox plugin are handy for this sort of thing.
You're probably missing part of the process that the server expects. You probably aren't logging in or interacting correctly, and that could be for all sorts of reasons. For instance, there might be JavaScript on the page that WWW::Mechanize isn't handling.
When you can pinpoint what an interactive browser is doing that you are not, you'll know where you need to improve your script.
In your script, you can also watch what is happening by turning on debugging in LWP, which Mech is built on:
use LWP::Debug qw(+);
rjh already answered the certificate part of your question.
If your session cookie changes every page load, then likely you are not logging in correctly. But you could try forcing the JSESSIONID to be the same for each request. Construct your own cookie jar and tell WWW::Mechanize to use it:
my $cookie_jar = HTTP::Cookies->new(file => 'cookies', autosave => 1, ignore_discard => 1);
my $agent = WWW::Mechanize->new(cookie_jar => $cookie_jar, autocheck => 0);
The ignore_discard => 1 means that even session cookies are saved to disk (normally they are discarded for security reasons).
Then, after logging in, call:
$cookie_jar->save;
Then, after each request:
$cookie_jar->revert; # re-loads the save
Alternately, you could sub-class HTTP::Cookies and override the set_cookie method to reject re-setting the session cookie if it already exists.
Also I think the site requires a CERT (Well in the browser it does), would this be the correct way to add it?
Some browsers (Internet Explorer for example) prompt for a security certificate even if one is not needed. If you are not getting any errors and the response content looks good, you probably don't need to set one.
If you do have a certificate file, check the POD for Crypt::SSLeay. Your certificate is PEM0-encoded so yes, you want to set $ENV{HTTPS_CERT_FILE} to the path of your cert. You might want to set $ENV{HTTPS_DEBUG} = 1 to see what's happening.
Setup the cookie jar, something akin to this:
my $cookie = HTTP::Cookies->new(file => 'cookie',autosave => 1,);
my $mech = WWW::Mechanize->new(cookie_jar => $cookie, ....);

How do I use Perl's LWP to log in to a web application?

I would like to write a script to login to a web application and then move to other parts
of the application:
use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use Data::Dumper;
$ua = LWP::UserAgent->new(keep_alive=>1);
my $req = POST "http://example.com:5002/index.php",
[ user_name => 'username',
user_password => "password",
module => 'Users',
action => 'Authenticate',
return_module => 'Users',
return_action => 'Login',
];
my $res = $ua->request($req);
print Dumper(\$res);
if ( $res->is_success ) {
print $res->as_string;
}
When I try this code I am not able to login to the application. The HTTP status code returned is 302 that is found, but with no data.
If I post username/password with all required things then it should return the home page of the application and keep the connection live to move other parts of the application.
You may be able to use WWW::Mechanize for this purpose:
Mech supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you've visited, which can be queried and revisited.
I'm guessing that LWP isn't following the redirect:
push #{ $ua->requests_redirectable }, 'POST';
Any reason why you're not using WWW::Mechanize?
I've used LWP to log in to plenty of web sites and do stuff with the content, so there should be no problem doing what you want. Your code looks good so far but two things I'd suggest:
As mentioned, you may need to make the requests redirectable
You may also need to enable cookies:
$ua->cookie_jar( {} );
Hope this helps