At my work I build a lot of wordpress sites and I also do a lot of cutting and pasting. In order to streamline this process I'm trying to make a crawler that can fill out and submit form information to wordpress. However, I can't get the crawler to operate correctly in the wordpress admin panel once I'm past the login.
I know it works to submit the login form because I've gotten the page back before. But this script doesn't seem to return the "settings" page, which is what I want. I've been trying to use this site as a guide: www.higherpass.com/Perl/Tutorials/Using-Www-mechanize/3/ for how to use mechanize but I could use some additional pointers for this. Here is my Perl script, I've tried a few variations but I just need to be pointed in the right direction.
Thanks!
use WWW::Mechanize;
my $m = WWW::Mechanize->new();
$url2 = 'http://www.moversbatonrougela.com/wp-admin/options-general.php';
$url = 'http://www.moversbatonrougela.com/wp-admin';
$m->get($url);
$m->form_name('loginform');
$m->set_fields('username' => 'user', 'password' => 'password');
$m->submit();
$response = $m->get($url2);
print $response->decoded_content();
Put the below lines of code just before $m->submit(); . Since WWW::Mechanize is a subclass of LWP::UserAgent you can use any of LWP's methods.
$m->add_handler("request_send", sub { shift->dump; return });
$m->add_handler("response_done", sub { shift->dump; return });
The above would enable logging in your code. Look out for the Request/Response return codes i.e. 200 (OK) or 302 (Redirect) etc. The URL request i.e. the $m->get() is probably getting redirected or the machine's ip is Blocked by the server. If its a redirect, then you can probably use $m->redirect_ok(); to follow the redirect URL, or in case you don't want to follow the redirect URL use $m->requests_redirectable (this is an LWP method). The logs should show something like below-
HTTP/1.1 200 OK
OR
HTTP/1.1 302 Found
If none of the above works, use an alternative of $m->submit(); like below and give it a try-
my $inputobject=$mech->current_form()->find_input( undef, 'submit' );
$m->click_button(input => $inputobject);
Related
at Route / of my page, the user can see a button:
<button>click me</button>
A click on this button will open a new tab and do a post to the "post_route".
in the controller sub of the post_route, i want to check some things and then redirect the user to a different domain, let's just say for example to www.redirecturl.com.
I also need to send payload to www.redirecturl.com (post request).
the domain www.redirecturl.com will then check the value i sent to it and render a specific page.
Unfortunately it doesn't work as expected.
If i use i.e. $self->redirect_to('www.redirecturl.com');
i get a "redirect error" in my browser with the information "This problem can sometimes occur when cookies are disabled or denied.".
So... any idea how i can redirect with Mojolicious to a completely different domain and send some payload in the body?
Thank you in advance
The documentation for redirect_to() gives the following examples:
$c = $c->redirect_to('named', foo => 'bar');
$c = $c->redirect_to('named', {foo => 'bar'});
$c = $c->redirect_to('/index.html');
$c = $c->redirect_to('http://example.com/index.html');
From which, we can guess that if you want to redirect to a different domain, then you need to give it a complete URI - not just a domain name.
I am new to perl. I wish to create a perl program which sends request to a website and downloads the data. I read HTTP::Headers and HTTP::Request.
I wish to use HTTP::Request->new( 'POST', $URL, $Header, $PostData ).
My question is how can I determine header values and post data.
Thank you
I was creating a similar script sometime back. I think first you should capture the http request using the browser.
1) Add "HTTPFox" to firefox. It is very helpful.
2) Open HttpFox in a new window. Press start button. This will start capturing packets.
3) Open the website and follow the procedures to download the data.
Please ask if anything is not clear.
I am trying to login to a website using perl. I have tried all the options - LWP::Mechanize, LWP::UserAgent, etc. but still have not been able to login successfully. I get a response code of 200 which means successful but how will i move on to the next page? Any help will be appreciated.
Make sure you are using cookies with LWP
$ua->cookie_jar({ file => "$ENV{HOME}/.cookies.txt" });
after the login just have the $ua request the next page.
If the login redacts you to another page and you want to get that then use
$ua->requests_redirectable
For more info check out the docs at
http://search.cpan.org/~gaas/libwww-perl-6.04/lib/LWP/UserAgent.pm
I'm trying to crawl this page using Perl LWP:
http://livingsocial.com/cities/86/deals/138811-hour-long-photo-session-cd-and-more
I had code that used to be able to handle living social, but it seems to have stopped working. Basically the idea was to crawl the page once, get its cookie, set the cookie in the UserAgent, and crawl it twice more. By doing this, you could get through the welcome page:
$response = $browser->get($url);
$cookie_jar->extract_cookies($response);
$browser->cookie_jar($cookie_jar);
$response = $browser->get($url);
$response = $browser->get($url);
This seems to have stopped working for normal LivingSocial pages, but still seems to work for LivinSocialEscapes. E.g.,:
http://livingsocial.com/escapes/148029-cook-islands-hotel-+-airfare
Any tips on how to get past the welcome page?
It looks like this page only works with a Javascript enabled browser (which LWP::UserAgent is not) You could try WWW::Mechanize::Firefox instead:
use WWW::Mechanize::Firefox;
my $mech = WWW::Mechanize::Firefox->new();
$mech->get($url);
Note that you must have Firefox and the mozrepl extension installed for this module to work.
Can I get some help on how to submit a POST with the necessary variables using WWW::Mechanize::Firefox? I've installed all the perl modules, and the firefox plugin and tested such that I can connect to a give host and get responses... my questions is how to submit a POST request. On the documentation Corion says he may never implement. This seems odd, I'm hoping I can use the inherited nature from Mechanize, but can't find any examples. A simple example would help me tremendously.
my $mech = WWW::Mechanize::Firefox->new();
$mech->allow( javascript =>1); # enable javascript
# http
$mech->get("http://www.example.com");
my $c = $mech->content;
Is there a mech->post() option I am simply missing?
many thanks in advance.
R
Normally you would just set the fields and submit the form like this:
$mech->get('http://www.website.com');
$mech->submit_form(
with_fields => {
user => 'me',
pass => 'secret',
}
);
get a page, fill out a form, submit the form
if you're going to be skipping the above steps by using post, you don't need mechanize firefox