I inherited a legacy Perl script from an old server which is being removed. The script needs to be implemented on a new server. I've got it on the new server.
The script is pretty simple; it connects via expect & ssh to network devices and gathers data. For debugging purposes, I'm only working with the portion that gathers a list of the interfaces from the device.
The script on the new server always shows me a page within about 5 seconds of reloading it. Rarely, it includes the list of interfaces from the remote device. Most commonly, it contains all the HTML elements except the list of interfaces.
Now, on the old server, sometimes the script would take 20 seconds to output the data. That was fine.
Based on this, it seems that apache on the new server is displaying the data before the Perl script has finished returning its data, though that could certainly be incorrect.
Additional Information:
Unfortunately I cannot post any code - work policy. However, I'm pretty sure it's not a problem with expect. The expect portions are written as expect() or die('error msg') and I do not see the error messages. However, if I set the expect timeout to 0, then I do see the error messages.
The expect timeout value used in the script normally is 20 seconds ... but as I mentioned above, apache displays the static content from the script after about 5 seconds, and 95% of the time does not display the content that should retrieved from expect. Additionally, the script writes the expect content to a file on the drive - even when the page does not display it.
I just added my Troubleshooting Perl CGI scripts guide to Stackoverflow. :)
You might try CGI::Inspect. I haven't needed to try it myself, but I saw it demonstrated at YAPC, and it looked awesome.
Related
I have a CasperJS script which logs into our test website platform. Our website application produces dynamic data which is updated every second, and normally using a web browser the login is left running (as you would using webmail)
The script logs into the website as a user, waits five seconds for the page to populate with data and uses the this.capture function to grab a screen shot to confirm the details are correct.
What I want to do, is follow on from the login as I've noticed the CasperJS script does not stay logged in as our customers logins are persistent.
I want to do this because we are load testing a new proof of concept platform.
Does anyone know how I make CasperJS do this?
I also want to parse a csv list of username/passwords to simulate logins - I'm presuming that I have to do this via a shell script or get PhantomJS invoke each login sequentially?
(background: I'm not a web developer, but someone with 20 years of IT and Unix/Infrastructure - so I would class myself as an intermediate skill scripting)
Persistent login
This is what the --cookies-file commandline option is for. It stores all cookies in a file on disk and on subsequent invocations of the script will use the stored cookies to restore the session. So just run your script like this:
casperjs --cooies-file=cookies.txt yourScript.js
yourScript.js should be able to tell that are already logged in.
Multiple credentials
Your other problem can be solved in different ways, but none of them should be invoked with the --cookies-file option.
Since a CSV is a simple file format you can read it through the PhantomJS fs module and iterate over them with casper.eachThen. For each iteration, you would need to login, do your thing and don't forget to log out just in the same way you would do in a browser session.
Parse the CSV somehow in the shell and pass the pairs into CasperJS. Then you can access casper.cli to get the credentials to log in. With this option you don't need to log out, since each invocation runs in its own PhantomJS instance and doesn't share cookies.
This option can be combined with your first question, if that is what you want. Add on each invocation the option --cookies-file=cookies_<username>.txt, so you can run the shell script multiple times without logging in each time.
Load testing
If I understood correctly, then the web application is password protected. You would need to run a separate CasperJS process for each username/password pair. You should check the memory footprint for one script invocation and scale up. Memory is the primary limiting factor which you can calculate for your test machine, but CPU will also hit a limit somewhere.
PhantomJS/CasperJS instances are full browsers and are therefore much heavier than a slim webserver. So you will probably need multiple machines each with many instances that run your script to load test the webserver.
We have a perl web interface that I am currently working on to slowly convert to using Dancer 2 and PSGI instead of our slow old plain vanilla CGI model.
In our old model, we stored everything in sessions -- the history of what the users did, the call stacks, the data inputs, ........ you get the idea.
We do not want to do it that way anymore so that we can keep the sessions small and efficient. BUT, we'd still like to log just what the users have been doing (that way when an error gets reported we can see what they did to get to the error, what input(s) they put in, etc).
I looked at Logging on Dancer2 documentation, but this doesn't seem to quite get to what we need - this would only record Dancer2 messages + what other messages I put in.
This one that I found Dancer2::Logger doesn't seem to quite cut it either.
What other libraries could I use to do what I need? I seriously doubt that perl does NOT have somethign that does this so...
Just off the top of my head, I can think of Log::log4perl and Log::Dispatch, though there are myriad others.
You can use them to establish your own log files, separate from dancer's log.
As for the best way, most logging interfaces have the same api for logging, but differ in run-time instantiation, and configuration syntax. So read the docs on a few of them and maybe try a couple out on for size.
We use CGI.pm to help us handle file uploading on our website and through our API which is used by our Android & iPhone Apps. We recently noticed that CGI.pm seems to be returning no params for almost 50% of the files being uploaded via our iPhone App. We haven't seen a similar issue with those files being uploaded via our website.
We can't replicate the problem in testing but in production the cgi_error() method of CGI.pm isn't reporting any errors in those cases where the CGI.pm params are missing. We have confirmed that the iPhone App is always including the correct params when POSTing the files for upload.
Quick background on the setup. We have the application delpoyed on Amazon EC2 Servers which are being load balanced using the Amazon Elastic Load Balancers. We also have $CGI::POST_MAX=(1024*100000); so the POST max size is set to 100MB and we have confirmed that all uploads are under this limit.
I'm not sure where to go next. Any ideas on what the issue might be and how to resolve it would be great appreciated. Also helpful would be any ideas on how to identify the root of the issue so we can start troubleshooting.
Thanks in advance for your help!
The loss of params with no error logged is exactly the symptom of the CGI module encountering an error processing the POST data - such as POST exceeding $CGI::POST_MAX. Are you using CGI.pm in functional mode by calling param(), or object oriented mode calling $cgi->param()? Regarding cgi_error(), perldoc CGI warns: When using the function-oriented interface, errors may only occur the first time you call param(). Be ready for this!
As for debugging, if you suspect CGI.pm is masking errors from you, try looking at the CGI object before doing anything else:
use Data::Dumper;
my $cgi = CGI->new();
warn Dumper($cgi);
Within the dump of the CGI object you would see an error like this: '.cgi_error' => '413 Request entity too large' - which is what cgi_error() would return for POST_MAX exceeded.
Also, if using Modperl, be aware that CGI can old onto values such as $CGI::POST_MAX between requests to different apps. (But, since you are specifying POST_MAX in yours, this wouldn't appear to be your problem.)
We too are seeing this same behaviour with CGI.pm, although we had thought the problem was restricted to just IE. Solved by adding an $CGI::POST_MAX=5000000. Over kill as it is only a 50k file being passed back and for.
I'm attempting to move a web app we have (written in Perl) from an IIS6 server to an IIS7.5 server.
Everything seems to be parsing correctly, I'm just having some issues getting the app to actually work.
The app is basically a couple forms. You fill the first one out, click submit, it presents you with another form based on what checkboxes you selected (using includes and such).
I can get past the first form once... but then after that it stops working and pops up the generated error message. After looking into the code and such, it basically states that there aren't any checkboxes selected.
I know the app writes data into .dat files... (at what point, I'm not sure yet), but I don't see those being created. I've looked at file/directory permissions and seemingly I have MORE permissions on the new server than I did on the last. The user/group for the files/dirs are different though...
Would that have anything to do with it? Why would it pass me on to the next form, displaying the correct "modules" I checked the first time and then not any other time after that? (it seems to reset itself after a while)
I know this is complicated so if you have any questions for me, please ask and I'll answer to the best of my ability :).
Btw, total idiot when it comes to Perl.
EDIT AGAIN
I've removed the source as to not reveal any security vulnerabilities... Thanks for pointing that out.
I'm not sure what else to do to show exactly what's going on with this though :(.
I'd recommend verifying, step by step, that what you think is happening is really happening. Start by watching the HTTP request from your browser to the web server - are the arguments your second perl script expects actually being passed to the server? If not, you'll need to fix the first script.
(start edit)
There's lots of tools to watch the network traffic.
Wireshark will read the traffic as it passes over the network (you can run it on the sending or receiving system, or any system on the collision domain).
You can use a proxy server, like WebScarab (free), Burp, Paros, etc. You'll have to configure your browser to send traffic to the proxy server, which will then forward the requests to the server. These particular servers are intended to aid testing, in that you'll be able to mess with the requests as they go by (and much more)
As Sinan indicates, you can use browser addons like Fx LiveHttpHeaders, or Tamper Data, or Internet Explorer's developer kit (IIRC)
(end edit)
Next, you should print out all CGI arguments that the second perl script receives. That way, you'll know what the script really thinks it gets.
Then, you can enable verbose logging in IIS, so that it logs the full HTTP request.
This will get you closer to the source of the problem - you'll know if it's (a) the first script not creating correct HTML, resulting in an incomplete HTTP request from the browser, (b) the IIS server not receiving the CGI arguments for some odd reason, or (c) the arguments aren't getting from the IIS server and into the perl script (or, possibly, that the perl script is not correctly accessing the arguments).
Good luck!
What you need to do is clear.
There is a lot of weird excess baggage in the script. There seemed to be no subroutines. Just one long series of commands with global variables.
It is time to start refactoring.
Get one thing running at a time.
I saw HTML::Template there but you still had raw HTML mixed in with code. Separate code from presentation.
I would like to submit a form to a CGI script localy (w3c-markup-validator), but it is too slow using curl and apache, I want to use this CGI script more than 5,000 times in an another script.
and currently it takes more than one hour.
What should I do to give the form directly to the CGI script (I upload a file with curl)?
edit: It seems to be too complicated and time consuming for what I needed, so I waited 1 hour and a half, each time I needed to test my generated xhtml files.
In definitive I didn't test any of the answers below, so the question will remain open.
Depending on the details of the script you might be able to create a fake CGI environment using HTTP::Request::AsCGI and then sourcing the CGI script with the "do" operator. But when it comes to speed and maintainability your best bet would be to factor the important part of the script's work into its own module, and rewrite the CGI as a client of that module. That way you don't have to invoke it as a CGI -- the batch job you're talking about now would be just another program using the same module to do the same work, but without CGI or the webserver environment getting in the way.
OK, I looked at the source code for this thing and it is not easy extract the validation stuff from all the rest. So, here is what I would.
First, ditch curl. Starting a new process for each file you want to validate is not a good idea. You are going to need to write a driver script that takes a list of URL's and submits them to your local server running on localhost. In fact, you might later want to parallelize this because there will normally be a bunch of httpd processes alive anyway. Well, I get ahead of myself.
This script can use LWP because all you are doing is submitting some data to the CGI script on localhost and storing/processing results. You do not need full WWW::Mechanize functionality.
As for the validator CGI script, you should configure that as a mod_perl registry script. Make sure you preload all necessary libraries.
This should boost documents processed per second from 1.3 to something more palatable.
CGI is a pretty simple API. All it does is read data either from an environment variable (for GET requests) or from stdin (for POST requests). So all you need is to do is to set up the environment and call the script. See the docs for details.
If the script uses CGI.pm, you can run it from the command line by supplying the '-debug' switch (to CGI.pm, in the use statement.) That will then allow you to send the post variables on stdin. You may have to tweak the script a little to make this work.