I have this which is supposed to fill the email form on the http://faceoook.com/recover.php
and as you know, you can search by email,name or phone number.
So I am trying to search by email, and get the content of that page after the search has been completed to see whether the profile is found or not, but the code doesn't seem to work.
use HTTP::Request::Common;
use LWP::UserAgent;
$email="blabla\#hotmail.com";
my %data=(email=>$email);
my $user_agent = 'Mozilla/6.0';
my $Browser = LWP::UserAgent->new;
$Browser->agent($user_agent);
$ua=$Browser->post('https://www.facebook.com/recover.php',\%data);
if($ua->content=~/couldn\'t/){ #"couldn't" is part of the message displayed when
print "Not Found"; # input doesn't match
}
elsif ($ua->content=~/name/) {
print "Found";
}
else {
print "Not found";
}
$result=$ua->content;
open FILE,">","me.txt" or die $!;
print FILE $result;
close FILE;
use strict
make it compile under strict
review the manpage for LWP::UserAgent, there's a problem with your code that you'll have to discover on your own so you'll remember
review your variable names in light of the conventions used in the manpage
review the approach (considering the Facebook has an API, IIRC)
no need to escape the single quote in the regex
You should post your request to the URL in the action field of the form (while you're using the URL of the page which shows the form).
Also add any hidden field to your %data.
Have a look at the HTML code of the page (or use some sort of form inspector) to get the correct URL and the hidden fields (javascript code, if present, can further complicate things).
Then use strict (and use warnings as well) as already said by Lumi.
Related
So I'm trying to create a perl script that logs in to SAP BusinessObjects Central Management Console (CMC) page but it doesn't even look like it's finding the right form or finding the right field or even clicking Submit.
Here's my code:
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get("http://myserver:8080/BOE/CMC");
$mech->form_name("_id2");
$mech->field("_id2:logon:CMS", "MYSERVER:6400");
$mech->field("_id2:logon:SAP_SYSTEM", "");
$mech->field("_id2:logon:SAP_CLIENT", "");
$mech->field("_id2:logon:USERNAME", "MYUSER");
$mech->field("_id2:logon:PASSWORD", "MYPWD");
$mech->field("_id2:logon:AUTH_TYPE", "secEnterprise");
$mech->click;
print $mech->content();
When I run it, I don't get any errors but the output I get is the login page again. Even more puzzling, it doesn't seem to be accepting the field values I send it (the output would display default values instead of the values I assign it). Putting in a wrong user or password doesn't change anything - no error but I just get the login page back with default values
I think the script itself is fine since I changed the necessary fields and I was able to log in to our Nagios page (the output page definitely shows Nagios details). I think the CMC page is not so simple, but I need help in figuring out how to make it work.
What I've tried:
1
use Data::Dumper;
print $mech->forms;
print Dumper($mech->forms());
What that gave me is:
Current form is: WWW::Mechanize=HASH(0x243d828)
Part of the Dumper output is:
'attr' => {
'target' => 'servletBridgeIframe',
'style' => 'display:none;',
'method' => 'post'
},
'inputs' => []
I'm showing just that part of the Dumper output because it seems that's the relevant part. When I tried the same thing with our Nagios page, the 'attr' section had a 'name' field which the above doesn't. The Nagios page also had entries for 'inputs' such as 'useralias' and 'password' but the above doesn't have any entries.
2
$mech->form_number(1);
Since I wasn't sure I was referencing the form correctly, I just had it try using the first form it finds (the page only has one form anyway). My result was the same - no error and the output is the login page with default values.
3
I messed around with escaping (with '\') the underscore (_) and colon (:) in the field names.
I've searched and didn't find anything that said I had to escape any characters but it was worth a shot. All I know is, the Nagios page field names only contained letters and it worked.
I got field names from Chrome's developer tool. For example, the User Name form field showed:
<input type="text" id="_id2:logon:USERNAME" name="_id2:logon:USERNAME" value="Administrator">
I don't know if Mechanize has a problem with names starting with underscore or names containing colons.
4
$mech->click("_id2:logon:logonButton");
Since I wasn't sure the "Log On" button was being clicked I tried to specify it but it gave me an error:
No clickable input with name _id2:logon:logonButton at /usr/share/perl5/WWW/Mechanize.pm line 1676
That's probably because there is no name defined on the button (I used the id instead) but I thought it was worth a shot. Here's the code of the button:
<input type="submit" id="_id2:logon:logonButton" value="Log On" class="logonButtonNoHover logon_button_no_hover" onmouseover="this.className = 'logonButtonHover logon_button_hover';" onmouseout="this.className = 'logonButtonNoHover logon_button_no_hover';">
There's only one button on the form anyway so I shouldn't have needed to specify it (I didn't need to for the Nagios page)
5
The interactive shell of Mechanize
Here's the output when I tried to retrieve all forms on the page:
$ perl -MWWW::Mechanize::Shell -eshell
(no url)>get http://myserver:8080/BOE/CMC
Retrieving http://myserver:8080/BOE/CMC(200)
http://myserver:8080/BOE/CMC>forms
Form [1]
POST http://myserver:8080/BOE/CMC/1412201223/admin/logon.faces
Help!
I don't really know perl so I don't know how to troubleshoot this further - especially since I'm not seeing errors. If someone can direct me to other things to try, it would be helpful.
In this age of DOM and Javascript, there's lots of things that can go wrong with Web automation. From your results, it looks like maybe the form is built in browser space, which can be really hard to deal with programmatically.
The way to be sure is to dump the original response and look at the form code it contains.
If that turns out to be your problem, your simplest recourse is something like Mozilla::Mechanize.
When dealing with forms, it can sometimes be easier to replicate the request the form generates than to try to work with the form through Mechanize.
Try using your browser's developer tools to monitor what happens when you log into the site manually (in Firefox or Chrome it'll be under the Network tab), and then generate the same request with Mechanize.
For example, the resulting code MIGHT look something like:
my $post_data => {
'_id2:logon:CMS' => "MYSERVER:6400",
'_id2:logon:SAP_SYSTEM' => "",
'_id2:logon:SAP_CLIENT' => "",
'_id2:logon:USERNAME' => "MYUSER",
'_id2:logon:PASSWORD' => "MYPWD",
'_id2:logon:AUTH_TYPE' => "secEnterprise",
};
$mech->post($url, $post_data);
unless ($mech->success()){
warn "Failed to post to $url: " . $mech->response()->status_line() . "\n";
}
print $mech->content();
Where %post_data should match exactly the data that's passed in the manual post to the site and not just what's in the HTML--the keys or data could be transformed by javascript before the actual post is made.
I had someone more knowledgeable than me give me help. The main hurdle was how the page was constructed in frames and how it operated. Here are the details:
The URL of the frame that contained the login page is "http://myserver:8080/BOE/CMC/0000000000/myuser/logon.faces". The main frame of the page had a form in it, but it wasn't the logon form, which explains why the form from my original code didn't have the logon fields I was expecting.
The other "gotcha" that I ran into was that after a successful logon, the site redirects you to a different URL: "http://myserver:8080/BOE/CMC/0000000000/myuser/App/home.faces?service=%2Fmyuser%2FApp%2F". So to check a successful login, I had to get this URL and check for whatever text I decided to look for.
I also had to refer to the logon form by id and not by name (since the form did not have a name).
Here's the working code:
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get("http://myserver:8080/BOE/CMC/0000000000/myuser/logon.faces");
$mech->form_id("_id2");
$mech->field("_id2:logon:CMS", "MYSERVER:6400");
$mech->field("_id2:logon:SAP_SYSTEM", "");
$mech->field("_id2:logon:SAP_CLIENT", "");
$mech->field("_id2:logon:USERNAME", "MyUser");
$mech->field("_id2:logon:PASSWORD", "MyPwd");
$mech->field("_id2:logon:AUTH_TYPE", "secEnterprise");
$mech->click;
$mech->get("http://myserver:8080/BOE/CMC/0000000000/myuser/App/home.faces?service=%2Fmyuser%2FApp%2FappService.jsp&appKind=CMC");
$output_page = $mech->content();
if (index($output_page, "Welcome:") != -1)
{
print "\n\n+++++ Successful login! ++++++\n\n";
}
else
{
print "\n\n----- Login failed!-----\n\n";
}
For validating that I had successfully logged in, I kept it very simple and just searched for the "Welcome:" text (as in "Welcome: MyUser").
I'm pretty new to Perl. While I just created a simple scripts to retrieve a file with
getstore($url, $file);
But how do I know whether the task is done correctly or the connection interrupted in the middle, or authentication failed, or whatever response. I searched all the web and I found some, like a response list, and some talking about useragent stuff, which I totally can't understand, especially the operator $ua->.
What I wish is to an explanation about that operator stuff (I don't even know what -> used for), and the RC code meaning, and finally, how to use it.
Its a lot of stuff so I appreciate any answer given, even just partially. And, thanks first for whoever will to help. =)
The LWP::Simple module is just that: quite simplistic. The documentation states that the getstore function returns the HTTP status code which we can save into a variable. There are also the is_success and is_error functions that tell us whether a certain return value is ok or not.
my $url = "http://www.example.com/";
my $filename = "some-file.html";
my $rc = getstore($url, $filename)
if (is_error($rc)) {
die "getstore of <$url> failed with $rc";
}
Of course, this doesn't catch errors with the file system.
The die throws a fatal exception that terminates the execution of your script and displays itself on the terminal. If you don't want to abort execution use warn.
The LWP::Simple functions provide high-level controls for common tasks. If you need more control over the requests, you have to manually create an LWP::UserAgent. An user agent (abbreviated ua) is a browser-like object that can make requests to servers. We have very detailed control over these requests, and can even modify the exact header fields.
The -> operator is a general dereference operator, which you'll use a lot when you need complex data structures. It is also used for method calls in object-oriented programming:
$object->method(#args);
would call the method on $object with the #args. We can also call methods on class names. To create a new object, usually the new method is used on the class name:
my $object = The::Class->new();
Methods are just like functions, except that you leave it to the class of the object to figure out which function exactly will be called.
The normal workflow with LWP::UserAgent looks like this:
use LWP::UserAgent; # load the class
my $ua = LWP::UserAgent->new();
We can also provide named arguments to the new method. Because these UA objects are robots, it is considered good manners to tell everybody who sent this Bot. We can do so with the from field:
my $ua = LWP::UserAgent->new(
from => 'ss-tangerine#example.com',
);
We could also change the timeout from the default three minutes. These options can also be set after we constructed a new $ua, so we can do
$ua->timeout(30); # half a minute
The $ua has methods for all the HTTP requests like get and post. To duplicate the behaviour of getstore, we first have to get the URL we are interested in:
my $url = "http://www.example.com/";
my $response = $ua->get($url);
The $response is an object too, and we can ask it whether it is_success:
$response->is_success or die $response->status_line;
So if execution flows past this statement, everything went fine. We can now access the content of the request. NB: use the decoded_content method, as this manages transfer encodings for us:
my $content = $response->decoded_content;
We can now print that to a file:
use autodie; # automatic error handling
open my $fh, ">", "some-file.html";
print {$fh} $content;
(when handling binary files on Windows: binmode $fh after opening the file, or use the ">:raw" open mode)
Done!
To learn about LWP::UserAgent, read the documentation. To learn about objects, read perlootut. You can also visit the perl tag on SO for some book suggestions.
I am getting error as "Internal Server Error.The server encountered an internal error or misconfiguration and was unable to complete your request."
I am submitting a form in html and get its values.
HTML Code (index.cgi)
#!c:/perl/bin/perl.exe
print "Content-type: text/html; charset=iso-8859-1\n\n";
print "<html>";
print "<body>";
print "<form name = 'login' method = 'get' action = '/cgi-bin/login.pl'> <input type = 'text' name = 'uid'><br /><input type = 'text' name = 'pass'><br /><input type = 'submit'>";
print "</body>";
print "</html>";
Perl Code to fetch data (login.pl)
#!c:/perl/bin/perl.exe
use CGI::Carp qw(fatalsToBrowser);
my(%frmfields);
getdata(\%frmfields);
sub getdata {
my ($buffer) = "";
if (($ENV{'REQUEST_METHOD'} eq 'GET')) {
my (%hashref) = shift;
$buffer = $ENV{'QUERY_STRING'};
foreach (split(/&/,$buffer)) {
my ($key, $value) = split(/=/, $_);
$key = decodeURL($key);
$value= decodeURL($value);
$hashref{$key} = $value;
}
}
else{
read(STDIN,$buffer,$ENV{'CONTENT_LENGTH'})
}
}
sub decodeURL{
$_=shift;
tr/+/ /;
s/%(..)/pack('c', hex($1))/eg;
return($_);
}
The HTML page opens correctly but when i submit the form, i get internal server error.
Please help.
What does the web server's error log say?
Independent of what it says, you must stop parsing the form data yourself. There are modules for that, specifically CGI.pm. Using that, you can do this instead:
use CGI;
my $CGI = CGI->new();
my $uid = $CGI->param( 'uid' );
my $pass = $CGI->param( 'pass' );
# rest of your script
Much cleaner and much safer.
I agree with Tore that you must not parse this yourself. Your code has multiple errors. You don't allow multiple parameter values, you don't allow the ; alternate separator, you don't handle POST with a query string in the URL, and so on.
I don't know how long it will be online for free, but chapter 15 of my new "Beginning Perl" book covers Web programming. That should get you started on some decent basics. Note that the online version is an early, rough draft. The actual book also includes Chapter 19 which has a complete Web app example.
could it be this line that's the problem?
my (%hashref) = shift;
You're initialising a proper hash, but shift will give you a hash reference, since you did getdata(\%frmfields);. You probably want this, instead:
my $hashref = shift;
"500 Internal Server Error" just means that something didn't work the way the web server expected. Maybe you don't have CGI enabled. Maybe the script isn't executable. Maybe it's in a directory the web server isn't allowed to access. It's even possible that maybe the web server ran the script successfully and it worked perfectly, but didn't start its output with a valid set of HTTP headers. You need to look in the web server's error log to find out what it didn't like, which may or may not be a Perl issue.
Like everyone else has said, though, don't try to parse query strings and grovel though %ENV yourself. Use one of the many fine modules or frameworks which are available and already known to work correctly. CGI.pm is the granddaddy of them all and works well for smaller projects, but I'd recommend looking into a proper web application framework such as Dancer, Mojolicious, or Catalyst (there are many others, but those are the big three) if you're planning to build anything with more than a handful of relatively simple pages and forms.
I am trying to setup a login page to be used with awstats, so that the content is only viewable by authenticated users.
Ideally, I would like to create my own login page, and if a user is not logged in when the visit the stats page, they are redirected to the login page. (Right now there is no authentication)
The problem is that I don't know how to implement this. I have tried googling this, but the only solutions I could find were to use .htaccess (which I would rather not use in this case if I don't have to)
Has anyone implemented something similar to this?
.htaccess is the right tool for this job, but if you insist, the ancient ancient ancient way
#!/usr/bin/perl --
use strict;
use warnings;
use CGI;
Main( #ARGV );
exit( 0 );
sub Main {
my ( $q ) = CGI->new;
if( $q->param('password') eq 'secret' ){
print ShowAWSTATS($q);
} else {
print ShowLoginForm($q);
}
}
where ShowLoginForm() prints a content header $q->header along with the html for a login form, and ShowAWSTATS prints a content header, and say, some html as provided by awstats.pl
Like Len Jaffe says, there is much much much more that needs to be done, so you want to use .htaccess (its either 3min with .htaccess or hours with anything else)
Is there a way in WWW::Mechanize or any Perl module to read on a file after accessing a website. For example, I clicked a button 'Receive', and a file (.txt) will appear containing a message. How will I be able to read the content? Answers are very much appreciated.. I've been working on this for days,, Also, I tried all the possibilities. Can anyone help? If you can give me an idea please? :)
Here is a part of my code:
...
my $username = "admin";<br>
my $password = "12345";<br>
my $url = "http://...do_gsm_sms.cgi";
my $mech = WWW::Mechanize->new(autocheck => 1, quiet => 0, agent_alias =>$login_agent, cookie_jar => $cookie_jar);
$mech->credentials($username, $password);<br>
$mech->get($url);
$mech->success() or die "Can't fetch the Requested page";<br>
print "OK! \n"; #This works <br>
$mech->form_number(1);
$mech->click()
;
After this, 'Downloads' dialog box will appear so I can save the file (but I can also set the default to open it immediately instead of saving). Question is, how can I read the content of this file?
..
I take you mean that the web site responds to the form submission by returning a non-HTML response (say a 'text/plain' file), that you wish to save.
I believe you want $mech->save_content( $filename )
Added:
First you need to submit the WWW:Mech's form submission, before saving the resulting (text) file.
The click is for clicking a button, whereas you want to submit a form, using $mech->submit() or $mech->submit_form( ... ).
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
my $username = "admin";
my $password = "12345";
my $login_agent = 'WWW::Mechanize login-agent';
my $cookie_jar;
#my $url = "http://localhost/cgi-bin/form_mech.pl";
my $url = "http://localhost/form_mech.html";
my $mech = WWW::Mechanize->new(autocheck => 1, quiet => 0,
agent_alias => $login_agent, cookie_jar => $cookie_jar
);
$mech->credentials($username, $password);
$mech->get($url);
$mech->success() or die "Can't fetch the Requested page";
print "OK! \n"; #This works
$mech->submit_form(
form_number => 1,
);
die "Submit failed" unless $mech->success;
$mech->save_content('out.txt');
After the click (assuming that's doing what it's supposed to), the returned data should be stored in your $mech object. You should be able to get the file data with $mech->content(),
perhaps after verifying success with $mech->status() and the type of response with $mech->content_type().
You may find it helpful to remember that WWW::Mechanize replaces the browser; anything a browser would have done, like bringing up a download window and saving a file, doesn't actually happen, but all the information the browser would have had is accessible through WWW::Mechanize's methods.
Dare I ask... have you tried this?
my $content = $mech->content();
Open the file (not 'Downloads' window) as if you were viewing it within your browser; you can save it later with a few lines of code.
Provided you have HTML::TreeBuilder installed:
my $textFile = $mech->content(format => "text");
should get you the text of the resulting window that opens.
Then open a filehandle to write your results in:
open my $fileHandle, ">", "results.txt";
print $fileHandle $textFile;
close $fileHandle;
I do this all the time with LWP, but I'm sure it's equally possible with Mech
I think where you might be going wrong is using Mech to request the page that has the button on it when you actually want to request the content from the page that the button causes to be sent to the browser when clicked.
What you need to do is review the html source of the page with the button that initiates the download and see what the Action associated with the button is. Most likely it will be a POST with some hidden fields or a URL to do a GET.
The Target URL of the Click has the stuff you actually want to get, not the URL of the page with the button on it.
For problems like this, you often have to investigate the complete chain of events that the browser handles. Use an HTTP sniffer tool to see everything the browser is doing until it gets to the file file. You then have to do the same thing in Mech.