I would like to view my account balance (online banking) with a Perl script using WWW::Mechanize. The site is Sparkasse Duerenen (online banking) However, the field IDs seem to have a special encryption mechanism. On each new page load the id is generated with a new unique name.
If you view the HTML source you will see following in the field "Legimtation ID" located on the left where you can input login data.
<input id="TgQZqInrKGXTjHOP" class="loginfeld" type="text" onkeyup="testEmptyInput(event,this);" onblur="testEmptyInput(event,this);" onfocus="testEmptyInput(event,this);" value="" maxlength="16" size="10" name="TgQZqInrKGXTjHOP"></input>
Same thing on the PIN/Password.
The input ID seems to have every time an unique generated name. I'am not able to fill this field with a static pre-defined field-name with WWW::Mechanize. What would you folks suggest now? How to fill this field in order to submit a POST request.
I would suggesting using Mojo::DOM to parse the returned HTML and look for an input with class="loginfeld" and type="text". Then just pull the attribute name.
For a short 8 minute video on using Mojo::DOM check out Mojocast Episode 5
The following prints the login field names to STDOUT. You'll just have to feed the return html from WWW::Mechanize to Mojo::DOM instead of this method of using Mojo::UserAgent
#!/usr/bin/perl
use strict;
use warnings;
use Mojo::UserAgent;
my $url = 'https://bankingportal.sparkasse-dueren.de/portal/portal/Starten';
my $ua = Mojo::UserAgent->new;
my $dom = $ua->get($url)->res->dom;
# Print Login field names
for my $input ($dom->find('input')->each) {
if ($input->attr('class') eq 'loginfeld') {
if ($input->attr('type') eq 'text') {
print "Login field name = " . $input->attr('name') . "\n";
}
if ($input->attr('type') eq 'password') {
print "Password field name = " . $input->attr('name') . "\n";
}
}
}
Related
My code is to enter an actor name and the program, via the given actor's filmography in IMDB, lists on a hash table all the cinematic genres of the movies he has acted in as well as their frequency. However, I have a problem: When I type a name like "brad pitt" or "bruce willis" after running the program at the prompt, execution takes indefinitely. How do you know what the problem is?
Another problem: when I type "nicolas bedos" (an actor name that I entered from the beginning), it works but it seems that the index is only made for a single movie selected in the #url_links list. Should the look_down function of the TreeBuilder module within a foreach loop be adapted? I was telling myself that the #genres list was overwritten on each iteration so I added a push () but the result remains the same.
use LWP::Simple;
use PerlIO::locale;
use HTML::TreeBuilder;
use WWW::Mechanize;
binmode STDOUT, ':locale';
use strict;
use warnings;
print "Enter the actor's name:";
my $acteur1 = <STDIN>; # the user enters the name of the actor
print "We will analyze the filmography of the actor $actor1 by genre\n";
#we put the link with the given actor in Mechanize variable in order to browse the internet links
my $lien1 = "https://www.imdb.com/find?s=nm&q=$acteur1";
my $mech = WWW::Mechanize->new();
$mech->get($lien1); #we access the search page with the get function
$mech->follow_link( url_regex => qr/nm0/i ); #we access the first result using the follow_link function and the regular expression nm0 which is in the URL
my #url_links= $mech->find_all_links( url_regex => qr/title\/tt/i ); #owe insert in an array all the links having as regular expression "title" in their URL
my $nb_links = #url_links; #we record the number of links in the list in this variable
my $tree = HTML::TreeBuilder->new(); #we create the TreeBuilder module to access a specific text on the page via the tags
my %index; #we create a hashing table
my #genres = (); #we create the genre list to insert all the genres encountered
foreach (#url_links) { #we make a loop to browse all the saved links
my $mech2 = WWW::Mechanize->new();
my $html = $_->url(); #we take the url of the link
if ($html =~ m=^/title=) { #if the url starts with "/title"
$mech2 ->get("https://www.imdb.com$html"); #we complete the link
my $content = $mech2->content; #we take the content of the page
$tree->parse($content); #we access the url and we use the tree to find the strings that interest us
#genres = $tree->look_down ('class', 'see-more inline canwrap', #We have as criterion to access the class = "see-more .."
sub {
my $link = $_[0]->look_down('_tag','a'); #new conditions: <a> tags
$link->attr('href') =~ m{genres=}; #autres conditions: "genres" must be in the URL
}
);
}
}
my #genres1 = (); #we create a new list to insert the words found (the genres of films)
foreach my $e (#genres){ #we create a loop to browse the list
my $genre = $e->as_text; #the text of the list element is inserted into the variable
#genres1 = split(/[à| ]/,$genre); #we remove the unnecessary characters that are spaces, at and | which allow to keep that the terms of genre cine
}
foreach my $e (#genres1){ #another loop to filter listing errors (Genres: etc ..) and add the correct words to the hash table
if ($e ne ("Genres:" or "") ) {
$index{$e}++;
}
}
$tree->delete; #we delete the tree as we no longer need it
foreach my $cle (sort{$index{$b} <=> $index{$a}} keys %index){
print "$cle : $index{$cle}\n"; #we display the hash table with the genres and the number of times that appear in the filmography of the given actor
}
Thank you in advance for your help,
wobot
The IMDB Conditions of Use say this:
Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below.
So you might want to reconsider what you're doing. Perhaps you could look at the OMDB API instead.
I have a html table that has 2 radio buttons for every row and a save button. I want to store the value of the radio button when saved and preset the value when the page is revisited.This is the html code I have written
<form action='table_extract.cgi' method = 'get'>
<td><input type='radio' name='signoff' value = 'approve'>Approve<br>
<input type='radio' name='signoff' value='review'>Review</td>
<td><input type='submit' name='button' value='Save'/></td></form>
This is what is in table_extract.cgi
#!usr/local/bin/perl
use CGI qw(:standard);
use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
use strict;
use warnings;
print <<END;
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
END
my $regfile = 'signoff.out';
my $sign;
$sign = param('signoff');
open(REG,">>$regfile") or fail();
print REG "$sign\n";
close(REG);
print "param value :", param('signoff');
print <<END;
<title>Thank you!</title>
<h1>Thank you!</h1>
<p>signoff preference:$sign </p>
END
sub fail {
print "<title>Error</title>",
"<p>Error: cannot record your registration!</p>";
exit; }
This is just first part of the problem. I was not able to find any output in console or in poll.out. Once I read the values, I need to preset the values to the radio buttons that was saved by the user in the previous visit.
The problem is in the HTML, your submit button has two type attributes. When I repaired this, the form worked for me.
You are doing too much work to save the form. See chapter SAVING THE STATE OF THE SCRIPT TO A FILE in the documentation.
To implement recaptcha in my website.
One option is google API . But for that i need to signup with domain name to get API key.
Is there any other way we can do it ?
You don't necessarily need a domain name to sign up, per se.
They have a concept of a "global key" where one single domain key would be used on several domains. When signing up, select the "Enable this key on all domains (global key)" option, and use a unique identifier (domainkey.abhilasha.com) and this will be fine, you can use the key from any domain in the end.
One way: add this code to your perl file that is called by an html form:
Simplified of course
my #field_names=qw(name branch email g-recaptcha-response);
foreach $field_name (#field_names)
{
if (defined param("$field_name"))
{
$FIELD{$field_name} = param("$field_name");
}
}
$captcha=$FIELD{'g-recaptcha-response'};
use LWP::Simple;
$secretKey = "put your key here";
$ip = remote_host;
#Remove # rem to test submitted variables are present
#print "secret= $secretKey";
#print " and response= $captcha";
#print " and remoteip= $ip";
$URL = "https://www.google.com/recaptcha/api/siteverify?secret=".$secretKey."&response=".$captcha."&remoteip=".$ip;
$contents = get $URL or die;
# contents variable takes the form of: "success": true, "challenge_ts": "2016-11-21T16:02:41Z", "hostname": "www.mydomain.org.uk"
use Data::Dumper qw(Dumper);
# Split contents variable by comma:
my ($success, $challenge_time, $hostname) = split /,/, $contents;
# Split success variable by colon:
my ($success_title, $success_value) = split /:/, $success;
#strip whitespace:
$success_value =~ s/^\s+//;
if ($success_value eq "true")
{
print "it worked";
}else{
print "it did not";
}
If you are just trying to block spam, I prefer the honeypot captcha approach: http://haacked.com/archive/2007/09/10/honeypot-captcha.aspx
Put an input field on your form that should be left blank, then hide it with CSS (preferably in an external CSS file). A robot will find it and will put spam in it but humans wont see it.
In your form validation script, check the length of the field, if it contains any characters, do not process the form submission.
I'm trying to fetch the date of the next episode of a specific tv show in this site using Mechanize in perl.
# getting episode number & date
# create a new browser
use WWW::Mechanize;
my $browser = WWW::Mechanize->new(autocheck => 0);
# fill search form, getting to tv show page
my $url= "http://next-episode.net/";
$browser->get($url);
$browser->form_name("search");
$browser->field("search", "big bang");
$browser->click();
print $browser->content();
I can't get to the tv show web-page. I only get the 404 page: "Sorry, the page you're looking for cannot be found! You may have typed a wrong url, or it may've been linked badly or moved."
am I filling the form wrongly?
What about this ? :
my $url = "http://next-episode.net";
my $search = "big bang";
use WWW::Mechanize;
use URI::Escape;
my $browser = WWW::Mechanize->new(autocheck => 1);
my $string = uri_escape $search;
$browser->get("$url/site-search-$string.html");
print $browser->content();
And if you'd like to know the number of days remaining to wait, add the extra line :
print "$1 days to wait\n" if $browser->content() =~ /(\d+)\s+Day\(s\)\s+/;
(I use regex here because HTML here is odd)
I've used WWW::Mechanize to login to the site.
Now that we are logged in, I want to make WWW::Mechanize script go to payments.php and then find the active user subscription (for example VIP Access) (class: <p class="description">).
From this I want to then read what that is, then select the correct action. For example if users package states VIP Small then print PKG: VIP Small and if users package states VIP Full then print PKG: VIP Full.
Does anyone know of a way to do this? Code used so far (being coded in my Ubuntu virtual machine):
#!/usr/bin/perl
use WWW::Mechanize;
my $forum = "http://localhost/forums/forum.php";
print "Username\r\n";
my $username = <>;
chomp($username);
print "Password\r\n";
my $password = <>;
# do login
my $mech = WWW::Mechanize->new(agentcheck => 1, agent => 'Perl WWW::Mechanize');
$mech->get($forum);
$mech->submit_form(form_number => 1, fields => { vb_login_username => $username, vb_login_password = $password });
print "this far";
$mech->follow_link(text => "Click here if your browser does not automatically redirect you.");
I think you need
$mech->get('http://localhost/forums/payments.php');
but I cannot help you get information from there without seeing the HTML of the page.
You need to parse result HTML file. I recommend to use HTML::TreeBuilder::XPath for such tasks:
my $tree = HTML::TreeBuilder::XPath->new_from_content( $mech->content() );
my ($description) = $tree->findvalues('//p[ #class = "description" ]');