I've run into an issue with mod_rewrite when submitting forms to our site perl scripts. If someone does a GET request on a page with a url such as http://www.example.com/us/florida/page-title, I rewrite that using the following rewrite rule which works correctly:
RewriteRule ^us/(.*)/(.*)$ /cgi-bin/script.pl?action=Display&state=$1&page=$2 [NC,L,QSA]
Now, if that page had a form on it I'd like to do a form post to the same url and have Mod Rewrite use the same rewrite rule to call the same script and invoke the same action. However, what's happening is that the rewrite rule is being triggered, the correct script is being called and all form POST variables are being posted, however, the rewritten parameters (action, state & page in this example) aren't being passed to the Perl script. I'm accessing these variables using the same Perl code for both the GET and POST requests:
use CGI;
$query = new CGI;
$action = $query->param('action');
$state = $query->param('state');
$page = $query->param('page');
I included the QSA flag since I figured that might resolve the issue but it didn't. If I do a POST directly to the script URL then everything works correctly. I'd appreciate any help in figuring out why this isn't currently working. Thanks in advance!
If you're doing a POST query, you need to use $query->url_param('action') etc. to get parameters from the query string. You don't need or benefit from the QSA modifier.
Change your script to:
use CGI;
use Data::Dumper;
my $query = CGI->new; # even though I'd rather call the object $cgi
print $query->header('text/plain'), Dumper($query);
and take a look at what is being passed to your script and update your question with that information.
Related
I am sending data from A.cgi to B.cgi. B.cgi updates the data in the database and is supposed to redirect back to A.cgi, at which point A.cgi should display the updated data. I added the following code to B.cgi to do the redirect, immediately after the database update:
$url = "http://Travel/cgi-bin/A.cgi/";
print "Location: $url\n\n";
exit();
After successfully updating the database, the page simply prints
Location: http://Travel/cgi-bin/A.cgi/
and stays on B.cgi, without ever getting redirected to A.cgi. How can I make the redirect work?
Location: is a header and headers must come before all ordinary output, that's probably your problem. But doing this manually is unneccessarly complicated anyways, you would be better of using the redirect function of CGI.pm
Use CGI's redirect method:
my $url = "http://Travel/cgi-bin/A.cgi";
my $q = CGI->new;
print $q->redirect($url);
I'm pretty new to Perl. While I just created a simple scripts to retrieve a file with
getstore($url, $file);
But how do I know whether the task is done correctly or the connection interrupted in the middle, or authentication failed, or whatever response. I searched all the web and I found some, like a response list, and some talking about useragent stuff, which I totally can't understand, especially the operator $ua->.
What I wish is to an explanation about that operator stuff (I don't even know what -> used for), and the RC code meaning, and finally, how to use it.
Its a lot of stuff so I appreciate any answer given, even just partially. And, thanks first for whoever will to help. =)
The LWP::Simple module is just that: quite simplistic. The documentation states that the getstore function returns the HTTP status code which we can save into a variable. There are also the is_success and is_error functions that tell us whether a certain return value is ok or not.
my $url = "http://www.example.com/";
my $filename = "some-file.html";
my $rc = getstore($url, $filename)
if (is_error($rc)) {
die "getstore of <$url> failed with $rc";
}
Of course, this doesn't catch errors with the file system.
The die throws a fatal exception that terminates the execution of your script and displays itself on the terminal. If you don't want to abort execution use warn.
The LWP::Simple functions provide high-level controls for common tasks. If you need more control over the requests, you have to manually create an LWP::UserAgent. An user agent (abbreviated ua) is a browser-like object that can make requests to servers. We have very detailed control over these requests, and can even modify the exact header fields.
The -> operator is a general dereference operator, which you'll use a lot when you need complex data structures. It is also used for method calls in object-oriented programming:
$object->method(#args);
would call the method on $object with the #args. We can also call methods on class names. To create a new object, usually the new method is used on the class name:
my $object = The::Class->new();
Methods are just like functions, except that you leave it to the class of the object to figure out which function exactly will be called.
The normal workflow with LWP::UserAgent looks like this:
use LWP::UserAgent; # load the class
my $ua = LWP::UserAgent->new();
We can also provide named arguments to the new method. Because these UA objects are robots, it is considered good manners to tell everybody who sent this Bot. We can do so with the from field:
my $ua = LWP::UserAgent->new(
from => 'ss-tangerine#example.com',
);
We could also change the timeout from the default three minutes. These options can also be set after we constructed a new $ua, so we can do
$ua->timeout(30); # half a minute
The $ua has methods for all the HTTP requests like get and post. To duplicate the behaviour of getstore, we first have to get the URL we are interested in:
my $url = "http://www.example.com/";
my $response = $ua->get($url);
The $response is an object too, and we can ask it whether it is_success:
$response->is_success or die $response->status_line;
So if execution flows past this statement, everything went fine. We can now access the content of the request. NB: use the decoded_content method, as this manages transfer encodings for us:
my $content = $response->decoded_content;
We can now print that to a file:
use autodie; # automatic error handling
open my $fh, ">", "some-file.html";
print {$fh} $content;
(when handling binary files on Windows: binmode $fh after opening the file, or use the ">:raw" open mode)
Done!
To learn about LWP::UserAgent, read the documentation. To learn about objects, read perlootut. You can also visit the perl tag on SO for some book suggestions.
How can I pass the variables from one perl webpage to the next, here is my example:
This is what I want passed from the first page, $data[0] and $data[2]
<a href="Month_entries.pl?month='$data[2]'&user='$data[0]'
style="text-decoration:none"
onclick="return popitup('Month_entries')">$busitotal2</a>
With it going to Month_entries.pl how to a call these variables in the new webpage(Month_entries)? what is this process called?
First, you should make sure that you are constructing the URI you actually want.
You probably don't want ' characters in the data
You problem should be protecting against XSS and broken data with URI::Encode.
Then it comes down to getting data from the query string.
How you do this depends on how you server and Perl are communicating.
If you are using Plack (which is generally a good idea for modern Perl), then see the code in the synopsis for Plack::Request:
my $app_or_middleware = sub {
my $env = shift;
my $req = Plack::Request->new($env);
my $path_info = $req->path_info;
# Change 'query' to whatever you called your key in the query string
my $query = $req->param('query');
my $res = $req->new_response(200);
$res->finalize;
};
If you are using a framework (such as Web::Simple, Catalyst or Dancer) then it will probably provide its own interface.
If you are using CGI, and using the CGI module, you would:
my $cgi = CGI->new();
my $ query = $cgi->param('query')
I want to fill in a web form with Perl. I am having trouble finding out the correct syntax to accomplish this. As in, how do I go to the URL, select the form, fill in the form, and then press enter to be sure it has been submitted?
Something like WWW::Mechanize::FormFiller?
WWW::Mechanize and its friends are the way to go. There are several examples in Spidering Hacks, but you'll also find plenty more by googling for the module name.
Good luck, :)
Start with WWW::Mechanize::Shell:
perl -MWWW::Mechanize::Shell -e shell
get http://some/page
fillout
...
submit
Afterwards, type "script", and save generated code as something.pl - and that's about it. It's done.
Request the form's action URL with Net::HTTP or something (can't recall the exact module), and include the forms fields as a GET/POST parameter (whichever the form calls for).
HTML::Form works nicely, too.
The synopsis of the module is an excellent example:
use HTML::Form;
$form = HTML::Form->parse($html, $base_uri);
$form->value(query => "Perl");
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$response = $ua->request($form->click);
I am very new to Perl and i am learning on the fly while i try to automate some projects for work. So far its has been a lot of fun.
I am working on generating a report for a customer. I can get this report from a web page i can access.
First i will need to fill a form with my user name, password and choose a server from a drop down list, and log in.
Second i need to click a link for the report section.
Third a need to fill a form to create the report.
Here is what i wrote so far:
my $mech = WWW::Mechanize->new();
my $url = 'http://X.X.X.X/Console/login/login.aspx';
$mech->get( $url );
$mech->submit_form(
form_number => 1,
fields =>{
'ctl00$ctl00$cphVeriCentre$cphLogin$txtUser' => 'someone',
'ctl00$ctl00$cphVeriCentre$cphLogin$txtPW' => '12345',
'ctl00$ctl00$cphVeriCentre$cphLogin$ddlServers' => 'Live',
button => 'Sign-In'
},
);
die unless ($mech->success);
$mech->dump_forms();
I dont understand why, but, after this i look at the what dump outputs and i see the code for the first login page, while i belive i should have reached the next page after my successful login.
Could there be something with a cookie that can effect me and the login attempt?
Anythings else i am doing wrong?
Appreciate you help,
Yaniv
This is several months after the fact, but I resolved the same issue based on a similar questions I asked. See Is it possible to automate postback from the client side? for more info.
I used Python's Mechanize instead or Perl, but the same principle applies.
Summarizing my earlier response:
ASP.NET pages need a hidden parameter called __EVENTTARGET in the form, which won't exist when you use mechanize normally.
When visited by a normal user, there is a __doPostBack('foo') function on these pages that gives the relevant value to __EVENTTARGET via a javascript onclick event on each of the links, but since mechanize doesn't use javascript you'll need to set these values yourself.
The python solution is below, but it shouldn't be too tough to adapt it to perl.
def add_event_target(form, target):
#Creates a new __EVENTTARGET control and adds the value specified
#.NET doesn't generate this in mechanize for some reason -- suspect maybe is
#normally generated by javascript or some useragent thing?
form.new_control('hidden','__EVENTTARGET',attrs = dict(name='__EVENTTARGET'))
form.set_all_readonly(False)
form["__EVENTTARGET"] = target
You can only mechanize stuff that you know. Before you write any more code, I suggest you use a tool like Firebug and inspect what is happening in your browser when you do this manually.
Of course there might be cookies that are used. Or maybe your forgot a hidden form parameter? Only you can tell.
EDIT:
WWW::Mechanize should take care of cookies without any further intervention.
You should always check whether the methods you called were successful. Does the first get() work?
It might be useful to take a look at the server logs to see what is actually requested and what HTTP status code is sent as a response.
If you are on Windows, use Fiddler to see what data is being sent when you perform this process manually, and then use Fiddler to compare it to the data captured when performed by your script.
In my experience, a web debugging proxy like Fiddler is more useful than Firebug when inspecting form posts.
I have found it very helpful to use Wireshark utility when writing web automation with WWW::Mechanize. It will help you in few ways:
Enable you realize whether your HTTP request was successful or not.
See the reason of failure on HTTP level.
Trace the exact data which you pass to the server and see what you receive back.
Just set an HTTP filter for the network traffic and start your Perl script.
The very short gist of aspx pages it that they hold all of the local session information within a couple of variables prefixed by "__" in the general aspxform. Usually this is a top level form and all form elements will be part of it, but I guess that can vary by implementation.
For the particular implementation I was dealing with I needed to worry about 2 of these state variables, specifically:
__VIEWSTATE
__EVENTVALIDATION.
Your goal is to make sure that these variables are submitted into the form you are submitting, since they might be part of that main form aspxform that I mentioned above, and you are probably submitting a different form than that.
When a browser loads up an aspx page a piece of javascript passes this session information along within the asp server/client interaction, but of course we don't have that luxury with perl mechanize, so you will need to manually post these yourself by adding the elements to the current form using mechanize.
In the case that I just solved I basically did this:
my $browser = WWW::Mechanize->new( );
# fetch the login page to get the initial session variables
my $login_page = 'http://www.example.com/login.aspx';
$response = $browser->get( $login_page);
# very short way to find the fields so you can add them to your post
$viewstate = ($browser->find_all_inputs( type => 'hidden', name => '__VIEWSTATE' ))[0]->value;
$validation = ($browser->find_all_inputs( type => 'hidden', name => '__EVENTVALIDATION' ))[0]->value;
# post back the formdata you need along with the session variables
$browser->post( $login_page, [ username => 'user', password => 'password, __VIEWSTATE => $viewstate, __EVENTVALIDATION => $validation ]);
# finally get back the content and make sure it looks right
print $response->content();