I have a webpage with 3 frames. The first frame has a form and when the form is submitted, the second frame loads some data. I need to be able to read the data in the second frame. What I have so far is this,
# Use WWW::Mechanize to download webpage
my $mechanize = WWW::Mechanize->new(
noproxy => 0,
stack_depth => 5,
autocheck => 1
);
$mechanize->proxy( https => undef );
my #frames;
eval{
my $me=$mechanize->get('link');
$me->is_success or die $me->status_line;
#frames = $mechanize->find_link( 'tag' => 'frame' ); # three frames
$me=$mechanize->get($frames[0]->url);
$me->is_success or die $me->status_line;
};
my $rb_value = 2000;
my $dt = '06/30/2011'
$mechanize->set_fields(
'idxevent' => $rb_value,
'mindate' => $dt
);
$mechanize->submit();
Now I need to retrieve the content of the second frame. What can I do for this?
Don't bother with the frameset, get the url of the frame holding the form directly and submit it. Get the result of $mechanize->submit() in a variable, and then you can access it by calling the content() method:
$result = $mechanize->submit();
print $result->content();
Mechanize does not care about the frameset and the submit target, it just gets the reply from the server so the same will apply for a normal frame-less layout.
You can find an example here
Follow the synopsis of WWW::Mechanize::Frames.
Related
I made a photobooth with Dancer some years ago and it worked fine.
Now I try to move this to Dancer2. However, It's not working anymore, because I have some infinite-loop.
Let's say my app looks like this:
package App;
use Dancer2;
# Photobox is my pm file with all the magic
use Photobox;
my $photobox = photobox->new();
get '/photo' => sub {
my $photo;
# Trigger cam and download photo. It returns me the filename of the photo.
$photo = $photobox->takePicture();
# Template photo is to show the photo
template 'photo',
{
'photo_filename' => $photo,
'redirect_uri' => "someURI"
};
}
takePicture() looks like this:
sub takePicture {
my $Objekt = shift;
my $return;
my $capture;
$return = `gphoto2 --auto-detect`;
if ($return =~ m/usb:/) {
$capture = `gphoto2 --capture-image-and-download --filename=$photoPath$filename`;
if (!-e $photoPath.$filename) {
return "no-photo-error.png";
}
else {
return $filename;
}
} else {
die "Camera not found: $return";
}
}
When I now call /photo, it'll result in an infinite loop. The browser is "frefreshing" all the time and my cam is shooting one photo after the other. But it never redirects to /showphoto.
It was working with Dancer(1) when I run the application just by perl app.pl from the bin directory. How I use Dancer2 und run it by using plackup app.psgi
I tried to put it into a before hook, but it changed nothing.
Update:
I figured out a way to work around this issue.
First I refactored my code a bit. Basic idea was to split the take photo and show photo operations into two different routes. This makes it easier to see what happens.
get '/takesinglephoto' => sub {
my $photo;
$photo = takePicture();
$single_photo=$photo;
redirect '/showsinglephoto';
;
get '/showsinglephoto' => sub {
set 'layout' => 'fotobox-main';
template 'fotobox_fotostrip',
{
'foto_filename' => $single_photo,
'redirect_uri' => "fotostrip",
'timer' => $timer,
'number' => 'blank'
};
};
And I moved the takePicture method just into my Dancer main App.pm.
Now I recognized from the log output, that the browser does not load the '/takesinglephoto' page once, but refreshes it every some secons. I think the reason is, that takePicture() takes some seconds to run and to return the output. And Dancer does not wait until it ends. With every reload, it triggers the takePicture() again and that causes the infinite-loop.
I worked around this by implementing a simple check to run takePicture() just once.
# define avariable set to 1 / true
my $do_stuff_once = 1;
get '/takesinglephoto' => sub {
my $photo;
# check if variable is true
if ($do_stuff_once == 1) {
$photo = takePicture();
$single_photo=$photo;
# set variable to false
$do_stuff_once = 0;
}
redirect '/showsinglephoto';
};
get '/showsinglephoto' => sub {
# set variable back to true
$do_stuff_once = 1;
set 'layout' => 'fotobox-main';
template 'fotobox_fotostrip',
{
'foto_filename' => $single_photo,
'redirect_uri' => "fotostrip",
'timer' => $timer,
'number' => 'blank'
};
};
Now it still refreshes /takesinglephoto, but it does not trigger takePicture() again and again and finally, when the method returns the photo filename, it redirects to /showsinglephoto.
I would call this a workaround. Is there a better way to solve this?
BR
Arne
I'm attempting to use a particular web service, and I can successfully perform the upload with the following command:
curl -X POST --header "Transfer-Encoding: chunked" -d #Downloads/file.pdf https://some.webservice/upload
I get back a json response indicate success.
However, I'm unable to figure out how to do the same with WWW::Mechanize.
$mech->post("https://" . $server . "/upload", Content_Type => 'multipart/form-data', Content => [upID => $upid, name => $dlfile, userID => 0, userK => 0, file_0 => [$dlfile]]);
This receives a similar json response with a big fat error message in it.
Do I need to explicitly set the Transfer-Encoding header first? Is there some other trick to it? Google's not shedding much light on this, Perlmonks neither, and the documentation's a little obtuse.
You can do it using HTTP::Request::StreamingUpload
my $starttime = time();
my $req = HTTP::Request::StreamingUpload->new(
POST => $url,
path => $file,
headers => HTTP::Headers->new(
'Transfer-Encoding' => 'chunked'
),
);
my $gen = $req->content;
die unless ref($gen) eq "CODE";
my $total = 0;
$req->content(sub {
my $chunk = &$gen();
$total += length($chunk);
print "\r$total / $size bytes ("
. int($total/$size*100)
. "%) sent, "
. int($total/1000/(time()-$starttime+1))
. " k / sec ";
return $chunk;
});
my $resp = $ua->request($req);
print "\n";
unless ($resp->is_success) {
die "Failed uploading the file: ", $resp->status_line;
}
my $con = $resp->content;
return $con;
Do you really need WWW::Mechanize? It is a subclass of LWP::UserAgent with additional functionality that gives browser-like functionality like filling in and submitting forms, clicking links, a page history with a "back" operation etc. If you don't need all of that then you may as well use LWP::UserAgent directly
Either way, the post method is inherited unchanged from LWP::UserAgent, and it's fine to use it directly as you have done
The way to send a chunked POST is to set the Content to a reference to a subroutine. The subroutine must return the next chunk of data each time it is called, and finally ann empty string or undef when there is no more to send
Is the data supposed to be a JSON string?
It's easiest to write a factory subroutine that returns a closure, like this
sub make_callback {
my ($data) = shift;
sub { substr($data, 0, 512, "") }
}
Then you can call post like this
my $payload = to_json(...);
$mech->post(
"https://$server/upload",
Content_Type => 'multipart/form-data',
Content => make_callback($payload)
);
Please be aware that all of this is untested
I am using Perl with the WWW::Mechanize module to submit a form to a webpage and save the result to a file. I know how to submit forms and save the data, but I can't save data after this six-second redirection.
After the form is submitted, the page is redirected to a page that says
Results should appear in this window in approximately 6 seconds...
and it is redirected again to the page with the result I want. My script can follow the first redirection, but not the second, and there is no link says something like "click here if not redirected".
Here is my script
use WWW::Mechanize;
my $mech = WWW::Mechanize->new(autocheck => 1);
$mech->get( "http://tempest.wellesley.edu/~btjaden/TargetRNA2/index.html");
$result = $mech->submit_form(
form_number => 1,
fields => {
text => 'Escherichia coli str. K-12 substr. MG1655',
sequence => '>RyhB' . "\n" .
'GCGATCAGGAAGACCCTCGCGGAGAACCTGAAAGCACGACATTGCTCACATTGCTTCCAGTATTACTTAGCCAGCCGGGTGCTGGCTTTT',
}
);
$mech->save_content(result);
What you need to do is extract the redirect URL and ran it manually:
Try this:
use WWW::Mechanize;
my $mech = WWW::Mechanize->new( autocheck => 1 );
$mech->get( "http://tempest.wellesley.edu/~btjaden/TargetRNA2/index.html");
$result = $mech->submit_form(
form_number => 1,
fields =>
{
text => 'Escherichia coli str. K-12 substr. MG1655',
sequence => '>RyhB GCGATCAGGAAGACCCTCGCGGAGAACCTGAAAGCACGACATTGCTCACATTGCTTCCAGTATTACTTAGCCAGCCGGGTGCTGGCTTTT',
}
);
my $content = $mech->content;
my $url1 = 'http://tempest.wellesley.edu/~btjaden/cgi-bin/';
my ($url2) = $content =~ /URL=(targetRNA2\.cgi?.+)?">/;
$mech->get($url1.$url2);
$mech->save_content(result);
WWW::Mechanize and meta refresh
Does the "6 seconds" contain something line the line below? [You may use save_content method of WWW::Machenize to save page to file]
<meta http-equiv="refresh" content="5; url=http://example.com/">
YES=>
Take a look at sources of WWW::Mechanize::Plugin::FollowMetaRedirect.
It shows how WWW::Mechanize may follow meta refresh with redirect.
It may quite likely solve your problem.
I'm getting an error No form defined at cqSubmitter.pl at line 33 which is the second set_fields method. Other times I get an Error POSTing http://micron.com Internal Server Error at line 39 , which corresponds to the last click_button line.
I'm not really sure what's going on, and why it's saying no form defined? The first half of the code which includes the first click_button method works fine and saves the correct page, but when I try set_fields for the second time, it errors out.
Anyone familiar with the Mechanize package realize what's going on here?
use Data::Dumper;
use HTTP::Request::Common qw(GET);
use WWW::Mechanize;
#Prepopulated information
my $types_ = "";
my $dept_ = "";
my $group_ = "";
#Create new WWW::Mechanize object
my $mech = WWW::Mechanize->new( 'ssl_opts' => { 'verify_hostname' => 0 } );
my $url = "http://f2prbrequest";
#Fetch URL or Die Tryin'
$mech ->get($url);
$fname = "user";
$pswd = "password";
#Login to ClearQuest form using credentials
$mech->set_fields(
USER => $fname
,PASSWORD => $pswd
);
$mech->click_button(
name => 'Submit'
);
#Set fields and actually fill out ClearQuest Form
$mech->set_fields(
types => $types_
,dept => $dept_
,group => $group_
);
$mech->click_button(
name => 'submit1'
);
$mech->save_content("clearQuestFilled.html");
I'm trying to submit a form by post method using WWW::Mechanize perl module.
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
...
$mech->get($url);
...
my $response = $mech->submit_form(
form_name => $name,
fields => {
$field_name => $field_value
},
button => 'Button'
);
$field_name is generally speaking a text field (though the type is not specified explicitly in the form), which has a preset value.
$field_name => $field_value in $mech->submit_form on whatever reason does not replace the value, instead $field_value is added into the form after the original value:
{submitted_field_value} = {original_value},{provided_value}
How to replace {original_value} with {provided_value} in the form to be submitted ?
What happens if you add this single line to your code before calling $mech->submit_form():
$mech->field( $name, [$field_value], 1 );
This makes sure that the first value is added, or overwritten if it already exists.
1 is the number parameter (or position index)
See the documentation of WWW::Mechanize:
$mech->field( $name, \#values, $number )
Given the name of a field, set its value to the value specified. [...]
The optional $number parameter is used to distinguish between two
fields with the same name. The fields are numbered from 1.
It's important to remember WWW::Mechanize is better thought of as a 'headless browser' as opposed to say LWP or curl, which only handle all the fiddly bits of http requests for you. Mech keeps its state as you do things.
You'll need to get the form by using $mech->forms or something similar (its best to decide from the documentation. I mean there so many ways to do it.), and then set the input field you want to change, using the field methods.
I guess the basic way to do this comes out as so:
$mech->form_name($name);
$mech->field($field_name, $field_value);
my $response = $mech->click('Button');
Should work. I believe it will also work if you get the field and directly use that (ie my $field = $mech->form_name($name); then use $field methods instead of $mech.
I managed to make it working at my will. Thanks Timbus and knb for your suggestions. Though my case may not be completely general (I know the preset value) but I'd share what I've found (by trails & errors).
my $mech = WWW::Mechanize->new();
$mech->get($url);
$mech->form_name( $name );
my $fields = $mech->form_name($name);
foreach my $k ( #{$fields->{inputs}}){
if ($k->{value} eq $default_value){
$k->{value}=$field_value;
}
}
my $response = $mech->click('Button_name');