Can I get some help on how to submit a POST with the necessary variables using WWW::Mechanize::Firefox? I've installed all the perl modules, and the firefox plugin and tested such that I can connect to a give host and get responses... my questions is how to submit a POST request. On the documentation Corion says he may never implement. This seems odd, I'm hoping I can use the inherited nature from Mechanize, but can't find any examples. A simple example would help me tremendously.
my $mech = WWW::Mechanize::Firefox->new();
$mech->allow( javascript =>1); # enable javascript
# http
$mech->get("http://www.example.com");
my $c = $mech->content;
Is there a mech->post() option I am simply missing?
many thanks in advance.
R
Normally you would just set the fields and submit the form like this:
$mech->get('http://www.website.com');
$mech->submit_form(
with_fields => {
user => 'me',
pass => 'secret',
}
);
get a page, fill out a form, submit the form
if you're going to be skipping the above steps by using post, you don't need mechanize firefox
Related
I am new to perl. I wish to create a perl program which sends request to a website and downloads the data. I read HTTP::Headers and HTTP::Request.
I wish to use HTTP::Request->new( 'POST', $URL, $Header, $PostData ).
My question is how can I determine header values and post data.
Thank you
I was creating a similar script sometime back. I think first you should capture the http request using the browser.
1) Add "HTTPFox" to firefox. It is very helpful.
2) Open HttpFox in a new window. Press start button. This will start capturing packets.
3) Open the website and follow the procedures to download the data.
Please ask if anything is not clear.
At my work I build a lot of wordpress sites and I also do a lot of cutting and pasting. In order to streamline this process I'm trying to make a crawler that can fill out and submit form information to wordpress. However, I can't get the crawler to operate correctly in the wordpress admin panel once I'm past the login.
I know it works to submit the login form because I've gotten the page back before. But this script doesn't seem to return the "settings" page, which is what I want. I've been trying to use this site as a guide: www.higherpass.com/Perl/Tutorials/Using-Www-mechanize/3/ for how to use mechanize but I could use some additional pointers for this. Here is my Perl script, I've tried a few variations but I just need to be pointed in the right direction.
Thanks!
use WWW::Mechanize;
my $m = WWW::Mechanize->new();
$url2 = 'http://www.moversbatonrougela.com/wp-admin/options-general.php';
$url = 'http://www.moversbatonrougela.com/wp-admin';
$m->get($url);
$m->form_name('loginform');
$m->set_fields('username' => 'user', 'password' => 'password');
$m->submit();
$response = $m->get($url2);
print $response->decoded_content();
Put the below lines of code just before $m->submit(); . Since WWW::Mechanize is a subclass of LWP::UserAgent you can use any of LWP's methods.
$m->add_handler("request_send", sub { shift->dump; return });
$m->add_handler("response_done", sub { shift->dump; return });
The above would enable logging in your code. Look out for the Request/Response return codes i.e. 200 (OK) or 302 (Redirect) etc. The URL request i.e. the $m->get() is probably getting redirected or the machine's ip is Blocked by the server. If its a redirect, then you can probably use $m->redirect_ok(); to follow the redirect URL, or in case you don't want to follow the redirect URL use $m->requests_redirectable (this is an LWP method). The logs should show something like below-
HTTP/1.1 200 OK
OR
HTTP/1.1 302 Found
If none of the above works, use an alternative of $m->submit(); like below and give it a try-
my $inputobject=$mech->current_form()->find_input( undef, 'submit' );
$m->click_button(input => $inputobject);
Getting the following error in a JavaScript link using perl - WWW::Mechanize.
Error GETing javascript:submt_os('2','contact%20info','contact%20info'):Protocol scheme 'javascript' is not supported
This is my code:
#!/usr/bin/perl
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$uri="http://tinyurl.com/76xv4ld";
$mech->get($uri);
# error on this link
$mech->follow_link( text => 'Contact Information');
print $mech->content();
Once I get the page, I want to click Contact Information.
Is there any other way to click Contact Information?
You can't follow a javascript link with WWW::Mechanize. Even if you had a javascript interpreter you'd need complete DOM support for anything non-trivial.
So - you need to script a web-browser. I use Selenium in my testing, which is quite bulky and requires java. You might want to investigate WWW::Mechanize::Firefox. I've not used it but it does provide a mechanize style interface to Firefox.
I'm trying to crawl this page using Perl LWP:
http://livingsocial.com/cities/86/deals/138811-hour-long-photo-session-cd-and-more
I had code that used to be able to handle living social, but it seems to have stopped working. Basically the idea was to crawl the page once, get its cookie, set the cookie in the UserAgent, and crawl it twice more. By doing this, you could get through the welcome page:
$response = $browser->get($url);
$cookie_jar->extract_cookies($response);
$browser->cookie_jar($cookie_jar);
$response = $browser->get($url);
$response = $browser->get($url);
This seems to have stopped working for normal LivingSocial pages, but still seems to work for LivinSocialEscapes. E.g.,:
http://livingsocial.com/escapes/148029-cook-islands-hotel-+-airfare
Any tips on how to get past the welcome page?
It looks like this page only works with a Javascript enabled browser (which LWP::UserAgent is not) You could try WWW::Mechanize::Firefox instead:
use WWW::Mechanize::Firefox;
my $mech = WWW::Mechanize::Firefox->new();
$mech->get($url);
Note that you must have Firefox and the mozrepl extension installed for this module to work.
I have built a CGI::Application currently running on local host and have used 2 authentication methods -
1. descried in http://www.perlmonks.org/?node_id=622071 by storing user password in database and
2. using LDAP credentials.
I was looking for a simple way to do google authentication but haven't found an easy way yet. Can someone point me in the right direction.
I looked at
1. Authen-GoogleAccount and
2. net-Google-FederatedLogin
but not enough documentation for either of these. Where do i start? Please let me know even if you have some pointer to doing this outside of cgi::application
This is the closest solution I could find. I am not a security expert, but I don't think websites serious about it would use this method. It uses WWW::Mechanize to authenticate using google email/password and then pull secure content out.
http://gregjessup.com/login-to-google-using-perl/
if $mech->get($url); returns error, authentication failed.
Here's the code I used for the Android Market for Developers (market.android.com/publish):
use WWW::Mechanize;
use HTTP::Cookies;
my $url = 'https://www.google.com/accounts/ServiceLogin';
my $username = 'username#gmail.com';
my $password = "PASSWORD";
my $mech = WWW::Mechanize->new();
$mech->cookie_jar(HTTP::Cookies->new());
$mech->get($url);
$mech->form_number(1);
$mech->field(Email => $username);
$mech->field(Passwd => $password);
$mech->click();
# Go to the next link, now that we are logged in.
$url = 'https://market.android.com/publish/Home';
$mech->get($url);
print $mech->content();
It's a small edit/cleanup of the link Prateek posted: http://gregjessup.com/login-to-google-using-perl.
I think it should be able to be used for most of Google's services that require you to be logged in.