I've used WWW::Mechanize to login to the site.
Now that we are logged in, I want to make WWW::Mechanize script go to payments.php and then find the active user subscription (for example VIP Access) (class: <p class="description">).
From this I want to then read what that is, then select the correct action. For example if users package states VIP Small then print PKG: VIP Small and if users package states VIP Full then print PKG: VIP Full.
Does anyone know of a way to do this? Code used so far (being coded in my Ubuntu virtual machine):
#!/usr/bin/perl
use WWW::Mechanize;
my $forum = "http://localhost/forums/forum.php";
print "Username\r\n";
my $username = <>;
chomp($username);
print "Password\r\n";
my $password = <>;
# do login
my $mech = WWW::Mechanize->new(agentcheck => 1, agent => 'Perl WWW::Mechanize');
$mech->get($forum);
$mech->submit_form(form_number => 1, fields => { vb_login_username => $username, vb_login_password = $password });
print "this far";
$mech->follow_link(text => "Click here if your browser does not automatically redirect you.");
I think you need
$mech->get('http://localhost/forums/payments.php');
but I cannot help you get information from there without seeing the HTML of the page.
You need to parse result HTML file. I recommend to use HTML::TreeBuilder::XPath for such tasks:
my $tree = HTML::TreeBuilder::XPath->new_from_content( $mech->content() );
my ($description) = $tree->findvalues('//p[ #class = "description" ]');
Related
I would like to view my account balance (online banking) with a Perl script using WWW::Mechanize. The site is Sparkasse Duerenen (online banking) However, the field IDs seem to have a special encryption mechanism. On each new page load the id is generated with a new unique name.
If you view the HTML source you will see following in the field "Legimtation ID" located on the left where you can input login data.
<input id="TgQZqInrKGXTjHOP" class="loginfeld" type="text" onkeyup="testEmptyInput(event,this);" onblur="testEmptyInput(event,this);" onfocus="testEmptyInput(event,this);" value="" maxlength="16" size="10" name="TgQZqInrKGXTjHOP"></input>
Same thing on the PIN/Password.
The input ID seems to have every time an unique generated name. I'am not able to fill this field with a static pre-defined field-name with WWW::Mechanize. What would you folks suggest now? How to fill this field in order to submit a POST request.
I would suggesting using Mojo::DOM to parse the returned HTML and look for an input with class="loginfeld" and type="text". Then just pull the attribute name.
For a short 8 minute video on using Mojo::DOM check out Mojocast Episode 5
The following prints the login field names to STDOUT. You'll just have to feed the return html from WWW::Mechanize to Mojo::DOM instead of this method of using Mojo::UserAgent
#!/usr/bin/perl
use strict;
use warnings;
use Mojo::UserAgent;
my $url = 'https://bankingportal.sparkasse-dueren.de/portal/portal/Starten';
my $ua = Mojo::UserAgent->new;
my $dom = $ua->get($url)->res->dom;
# Print Login field names
for my $input ($dom->find('input')->each) {
if ($input->attr('class') eq 'loginfeld') {
if ($input->attr('type') eq 'text') {
print "Login field name = " . $input->attr('name') . "\n";
}
if ($input->attr('type') eq 'password') {
print "Password field name = " . $input->attr('name') . "\n";
}
}
}
I'm trying to fetch the date of the next episode of a specific tv show in this site using Mechanize in perl.
# getting episode number & date
# create a new browser
use WWW::Mechanize;
my $browser = WWW::Mechanize->new(autocheck => 0);
# fill search form, getting to tv show page
my $url= "http://next-episode.net/";
$browser->get($url);
$browser->form_name("search");
$browser->field("search", "big bang");
$browser->click();
print $browser->content();
I can't get to the tv show web-page. I only get the 404 page: "Sorry, the page you're looking for cannot be found! You may have typed a wrong url, or it may've been linked badly or moved."
am I filling the form wrongly?
What about this ? :
my $url = "http://next-episode.net";
my $search = "big bang";
use WWW::Mechanize;
use URI::Escape;
my $browser = WWW::Mechanize->new(autocheck => 1);
my $string = uri_escape $search;
$browser->get("$url/site-search-$string.html");
print $browser->content();
And if you'd like to know the number of days remaining to wait, add the extra line :
print "$1 days to wait\n" if $browser->content() =~ /(\d+)\s+Day\(s\)\s+/;
(I use regex here because HTML here is odd)
I use WWW::Mechanize::Shell to test stuff.
my code is this:
#!/usr/bin/perl
use WWW::Mechanize;
use HTTP::Cookies;
my $url = "http://mysite/app/login.jsp";
my $username = "username";
my $password = "asdfasdf";
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get($url);
$mech->form_number(1);
$mech->field(j_username => $username);
$mech->field(j_password => $password);
$mech->click();
$mech->follow_link(text => "LINK A", n => 1);
$mech->follow_link(text => "LINK B", n => 1);
........................
........................
........................
etc, etc.
the problem is the next:
LINK B (web_page_b.html), make a redirect to web_page_x.html
if I print the contents of $mech->content(), display web_page_b.html
but i need to display web_page_x.html,to automatically submit a HTML form (web_page_x.html)
The question is:
How I can get web_page_x.html ?
thanks
Why don't you first test to see if the code containing the redirect (I'm guessing it's a <META> tag?) exists on web_page_b.html, then go directly to the next page once you're sure that that's what a browser would have done.
This would look something like:
$mech->follow_link(text => "LINK B", n => 1);
unless($mech->content() =~ /<meta http-equiv="refresh" content="5;url=(.*?)">/i) {
die("Test failed: web_page_b.html does not contain META refresh tag!");
}
my $expected_redirect = $1; # This should be 'web_page_x.html'
$mech->get($expected_redirect); # You might need to add the server name into this URL
Incidentally, if you're doing any kind of testing with WWW::Mechanize, you should really check out Test::WWW::Mechanize and the other Perl testing modules! They make life a lot easier.
In case it doesn't really redirect, then you better use regex with that follow_link method rather than just plain text.
such as:
$mech->follow_link(url_regex => qr/web_page_b/i , n => 1);
same for the other link.
I asked this over on Google's support forums for Apps integration, but got zero response. Maybe somebody here can help steer me in the right direction.
I'm trying to integrate a Perl application with Google Apps, and I'm having some trouble with the OpenID authentication. I've been using this PHP tutorial from Google as a kind of reference, since there are no Perl examples I can find.
My initial file, index.cgi (referred by manifest.xml, and the starting point of the OpenID transaction) is as follows:
use Net::OpenID::Consumer;
use CGI;
# ...
my $q = CGI->new();
my $domain = $q->param('domain');
if (!$domain) {
print $q->header(), 'Provide domain please.';
exit 0;
}
# my website
my $root = 'http://www.example.com/';
my $csr = Net::OpenID::Consumer->new(
# The user agent which sends the openid off to the server
ua => LWP::UserAgent->new,
# Who we are
required_root => $root,
# Consumer Key Secret from Google Apps Marketplace
consumer_secret => 'Zzzzzz9zzAAAAA....'
);
my $claimed_id = $csr->claimed_identity(
'https://www.google.com/accounts/o8/site-xrds?hd=' . $domain);
if ($claimed_id) {
my $check_url = $claimed_id->check_url(
# Upon validation, the user will be returned here, and real
# work may begin
return_to => $root . '/return.cgi',
trust_root => $root
);
print $q->redirect($check_url);
}
else {
print $q->header(), "Error";
}
This part seems to be working. That is, I get redirected to return.cgi with a bunch of openid.* parameters. However, at this point I get the following error:
no_identity_server The provided URL doesn't declare its OpenID identity server
I'm using the latest version of the Net::OpenID::Consumer module.
Here are the significant bits of return.cgi:
my $q = CGI->new();
my $csr = Net::OpenID::Consumer->new(
ua => LWP::UserAgent->new,
# The root of our URL
required_root => 'http://www.example.com/',
# Our password.
consumer_secret => 'Zzzzzz9zzAAAAA....',
# Where to get the information from.
args => $q
);
print $q->header();
$csr->handle_server_response(
not_openid => sub {
print "That's not an OpenID message. Did you just type in the URL?";
},
setup_required => sub {
my $setup_url = shift;
print 'You need to do something here.';
},
cancelled => sub {
print 'You cancelled your login.';
},
verified => sub {
my $vident = shift;
my $url = $vident->url;
print "You are verified as '$url'. ** FIN **";
},
error => sub { die "Can't figure it out: ", #_; }
);
As you can imagine, I'm wanting the verified sub to fire, but instead I'm getting an error. Anything obvious I'm missing? Any help would be appreciated.
So the solution, it turns out, is to switch modules. I changed to the skimpily documented Net::Google::FederatedLogin, and things are now working. The code is as follows (substitute example.com below for your actual developer's domain).
In your Google Apps Marketplace vendor profile, add the URL to index.cgi in the Application Manifest:
...
<Url>http://www.example.com/index.cgi?from=google&domain=${DOMAIN_NAME}</Url>
...
Then add the following code to your servers.
index.cgi
use CGI;
use Net::Google::FederatedLogin;
my $q = CGI->new();
my $domain = $q->param('domain');
if (!$domain) {
print $q->header(), 'Provide domain please.';
exit 0;
}
my $fl = Net::Google::FederatedLogin->new(
claimed_id =>
'https://www.google.com/accounts/o8/site-xrds?hd=' . $domain,
return_to =>
'http://www.example.com/return.cgi',
extensions => [
{
ns => 'ax',
uri => 'http://openid.net/srv/ax/1.0',
attributes => {
mode => 'fetch_request',
required => 'email',
type => {
email => 'http://axschema.org/contact/email'
}
}
}
] );
print $q->redirect($fl->get_auth_url());
return.cgi
use CGI;
use Net::Google::FederatedLogin;
my $q = CGI->new();
print $q->header();
my $fl = Net::Google::FederatedLogin->new(
cgi => $q,
return_to =>
'http://www.example.com/return.cgi' );
eval { $fl->verify_auth(); };
if ($#) {
print 'Error: ' . $#;
}
else {
# we've authenticated and gotten attributes --
my $ext = $fl->get_extension('http://openid.net/srv/ax/1.0');
print $ext->get_parameter('value.email');
}
(For a full sample, plus OAuth access to user data, see this post on my blog.)
In some cases, reportedly, this is caused by Perl missing Net::SSL, making it fail on Google's SSL URLs.
The step where it's failing is where it performs discovery on the identifier asserted by Google's server. For some reason discovery on that identifier URL is failing. It would be helpful to know what OpenID identifier the Google server is asserting to try to debug why discovery is not working for it.
I'm not sure what $domain is there, but you may need to escape it:
use URI::Escape 'uri_escape';
....
my $claimed_id = $csr->claimed_identity(
'https://www.google.com/accounts/o8/site-xrds?hd=' . uri_escape($domain) );
Also, the consumer_secret used by Net::OpenID::Consumer has no relationship to any other secret.
I am trying to write a Perl script to connect to me YouTube account but it doesnt seem to work. Basically I just want to connect to my account but apparently it is not working. I don't even have an idea on how I could debug this! Maybe it is something related to https protocol?
Please enlighten me! Thanks in advance.
use HTTP::Request::Common;
use LWP::UserAgent;
use strict;
my $login="test";
my $pass = "test";
my $res = "";
my $ua = "";
# Create user agent, make it look like FireFox and store cookies
$ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051213 Firefox/1.0.7");
$ua->cookie_jar ( {} );
# Request login page
$res = $ua->request(GET "https://www.google.com/accounts/ServiceLogin?service=youtube&hl=en_US&passive=true<mpl=sso&uilel=3&continue=http%3A//www.youtube.com/signup%3Fhl%3Den_US%26warned%3D%26nomobiletemp%3D1%26next%3D/index");
die("ERROR1: GET http://www.youtube.com/login\n") unless ($res->is_success);
# Now we login with our user/pass
$res = $ua->request(
POST "https://www.google.com/accounts/ServiceLoginAuth?service=youtube",
Referer => "http://www.youtube.com/login",
Content_Type => "application/x-www-form-urlencoded",
Content => [
currentform => "login",
next => "/index",
username => $login,
password => $pass,
action_login => "Log+In"
]
);
# YouTube redirects (302) to a new page when login is success
# and returns OK (200) if the login failed.
#die("ERROR: Login Failed\n") unless ($res->is_redirect());
print $res->content;
what i am doing is learning the web features of perl, so i dont want to use any library except wwwlib or mechanize to get the job done.
how can i just connect to my account using a perl script? this is my objective for now
hope someone can post a script or correct mine.
thank you guys for you help.
i am testing Webscarab now..
What data are you trying to grab? Why not just using an existing implementation like WebService::YouTube
Some comments on your code: I always avoided the shortcut $ua->request(GET/POST) method since I always ended up needing more flexibility that only the use of HTTP::Request and HTTP::Response allowed. I always felt the code was cleaner that way too.
Why is your code not working? Who knows.
Make sure your cookiejar is adding your cookies to the outgoing HTTP::Request. I'd suggest dumping all your headers when you do it in a browser and compare with the headers and data that libwww is sending. There may be some additional fields that they are checking for that vary for every hit. They may be checking for your UserAgent string. If you are just looking to learn libwww I'd suggest using a different site as a target as I'm sure YouTube has all sort of anti-scripting hardening.
Are you using YouTube's stable documented API?
Use an HTTP proxy such as WebScarab to watch the data flow.
Trey's suggestion to use somebody else's CPAN package for the mechanics is a good idea too.
Right right by and large, what you want to do is define a cookiejar for most of these websites that have a redirection login. This is what the package has done. Also the package tunes a lot of the lookups and scrapes based on the youtube spec.
Ajax content for example will be rough since its not there when your scraping
You just picked a somewhat rough page to start out with.
Enjoy
I'm actually working on this issue myself. Before, I would suggest read over this the API guide from Google as a good starting reference. If I'm reading it correctly, one begins with passing user credentials through a REST interface to get a Authentication Token. To handle that, I'm using the following:
sub getToken {
my %parms = #_;
my $response = LWP::UserAgent->new->post(
'https://www.google.com/youtube/accounts/ClientLogin',
[
Email => $parms{'username'},
Passwd => $parms{'password'},
service => "youtube",
source => "<<Your Value Here>>",
]
);
my $content = $response->content;
my ($auth) = $content =~ /^Auth=(.*)YouTubeUser(.*)$/msg
or die "Unable to authenticate?\n";
my ($user) = $content =~ /YouTubeUser=(.*)$/msg
or die "Could not extract user name from response string. ";
return ($auth, $user);
}
And I call that from the main part of my program as such:
## Get $AuthToken
my ($AuthToken, $GoogleUserName) = getToken((
username => $email, password => $password
));
Once I have these two things -- $AuthToken and $GoogleUserName, I'm still testing the LWP Post. I'm still writing this unit:
sub test {
my %parms = #_;
## Copy file contents. Use, foy's three param open method.
my $fileSize = -s $parms{'File'};
open(VideoFile, '<', "$parms{'File'}") or die "Can't open $parms{'File'}.";
binmode VideoFile;
read(VideoFile, my $fileContents, $fileSize) or die "Can't read $parms{'File'}";
close VideoFile;
my $r = LWP::UserAgent->new->post(
"http://uploads.gdata.youtube.com/feeds/api/users/$parms{'user'}/uploads",
[
Host => "uploads.gdata.youtube.com",
'Authorization' => "AuthSub token=\"$parms{'auth'}\"",
'GData-Version' => "2",
'X-GData-Key' => "key=$YouTubeDeveloperKey",
'Slug' => "$parms{'File'}",
'Content-Type' => "multipart/related; boundary=\"<boundary_string>\"",
'Content-Length' => "<content_length>",
'video_content_type'=> "video/wmv",
'Connection' => "close",
'Content' => $fileContents
]
);
print Dumper(\$r->content)
}
And that is called as
&test((auth=>$Auth, user=>$user, File=>'test.wmv'));