I would like to submit a form to a CGI script localy (w3c-markup-validator), but it is too slow using curl and apache, I want to use this CGI script more than 5,000 times in an another script.
and currently it takes more than one hour.
What should I do to give the form directly to the CGI script (I upload a file with curl)?
edit: It seems to be too complicated and time consuming for what I needed, so I waited 1 hour and a half, each time I needed to test my generated xhtml files.
In definitive I didn't test any of the answers below, so the question will remain open.
Depending on the details of the script you might be able to create a fake CGI environment using HTTP::Request::AsCGI and then sourcing the CGI script with the "do" operator. But when it comes to speed and maintainability your best bet would be to factor the important part of the script's work into its own module, and rewrite the CGI as a client of that module. That way you don't have to invoke it as a CGI -- the batch job you're talking about now would be just another program using the same module to do the same work, but without CGI or the webserver environment getting in the way.
OK, I looked at the source code for this thing and it is not easy extract the validation stuff from all the rest. So, here is what I would.
First, ditch curl. Starting a new process for each file you want to validate is not a good idea. You are going to need to write a driver script that takes a list of URL's and submits them to your local server running on localhost. In fact, you might later want to parallelize this because there will normally be a bunch of httpd processes alive anyway. Well, I get ahead of myself.
This script can use LWP because all you are doing is submitting some data to the CGI script on localhost and storing/processing results. You do not need full WWW::Mechanize functionality.
As for the validator CGI script, you should configure that as a mod_perl registry script. Make sure you preload all necessary libraries.
This should boost documents processed per second from 1.3 to something more palatable.
CGI is a pretty simple API. All it does is read data either from an environment variable (for GET requests) or from stdin (for POST requests). So all you need is to do is to set up the environment and call the script. See the docs for details.
If the script uses CGI.pm, you can run it from the command line by supplying the '-debug' switch (to CGI.pm, in the use statement.) That will then allow you to send the post variables on stdin. You may have to tweak the script a little to make this work.
Related
I just need a plain static .html page form, to POST to a Powershell script.
I've seen plenty of Powershell Invoke-WebRequest cmdlet material, but where Powershell is always initiating the HTTP request (and then handling the HTTP response..)
Thank you!
The short answer is that you cannot POST directly to a PowerShell script. When you POST to a website you are passing arguments to the web server that are then being presented to code on that web server ( the target of your POST request ) that the web server is capable of executing. Web servers do not understand PowerShell ( unless Microsoft has implemented this, which a few quick googles suggests they haven't ).
That being said, your ultimate goal is likely that you want to consume data that you sourced from a form via a PowerShell script. You will need to implement a backend on the webserver to consume the POST request and pass it to the operating system level to be run via PowerShell. This is generally not a good idea but if you are doing it for an internal site to get something running quickly then so be it.
Here is the process to call a Powershell script from ASP.Net: http://jeffmurr.com/blog/?p=142
You could approach this problem in many other ways. You could write your backend site to save the data from the POST request to a file and come along and parse that file on a schedule with PowerShell. You could use a database in the same manor or you could create a trigger in the database to run the script each time a row is appended.
I suspect that if you work down one of these pathways you will ultimately find that the technology you are using on the backend ( like ASP.Net or PHP or JavaScript ) is capable of doing the work you need done and that you would have far less moving parts if you stuck with one of those. Don't be afraid to learn something new. Jumping to JavaScript from PowerShell is not that difficult.
And the world moves to fast. Here is a NodeJS-like implementation of a webserver in PowerShell.
https://gallery.technet.microsoft.com/scriptcenter/Powershell-Webserver-74dcf466
I’m trying to profile a website I have to work on in IIS, in Perl. The website uses Catalyst. I’m using Devel::NYTProf to profile it.
By default, the profile file is written in ./nytprof.out. I don’t have access to the command line used to launch the perl, nor to pass arguments (I use use Devel::NYTProf to enable profiling in my perl file).
But I can’t find the file… Do you have an idea where it’d be? How could I profile my website with NYTProf a better way?
I assume you mean IIS.
Have you checked the user the web server is running as has write permission to likely folders? It use to run as IANONUSR (IIRC) or similar, which had very controlled permissions for obvious reasons.
The IIS FastCGI module lets you set environment variables for the FastCGI processes, which should let you set out_file for NYTPROF. If all else fails you can hack Run.pm in NYTPROF and change the location that way, crude but at least you know where it is trying to write to.
I salute your efforts, I would probably just port the application to run under Linux. First time getting NYTProf working under Linux was hard enough, especially as the processes have to terminate normally, so the FastCGI processes got a method added to make them die when I fetched a specific URL, which I'd keep fetching till all the processes were dead.
That said NYTProf was well worth the effort on Linux, was able to track down a regular expression that was eating vast amounts of CPU, and didn't even need to be called 99.9% of the time it was. Experience on Windows was "fork" was a performance killer, but I think Microsoft fixed that somewhat since my IIS days.
I am searching for efficient ways of communication across two Perl
scripts. I have two scripts; Script 1 generates some data. I want my
Script 2 to be able to access that information.
The easiest/dumbest
way is to write the data generated by Script 1 as a file and read it
later using Script 2. Is there any other way than this? Can I store
the data in memory and make it available to Script 2 (of course with
support from my Linux )? Meaning malloc some data by Script 1 and make
Script 2 able to access it.
There is no guarantee that Script 2 will be run after Script 1. So
there should be some way to free that memory using a watchdog timer.
Let me reveal some more context. I am running these scripts on a web-server using CGI-Perl. So at the click of a button Script 1 is run and it generates a html web-page. Now the user can add some inputs to to this generated web-page and click a button on this new page.Now Script 2 should be able to read the data on new web-page.I can post the data back to web-server again but a more efficient way is to keep a copy of generated page in server also and make it available to script 2. Now, I would like to avoid writing down the generated page as a file. I was thinking of storing it in memory
This depends somewhat on your usage... one large set of data? Many small messages? Di you canre at all about data persistance? Is it TOTALLY asynchronous?
Some of the options are:
For any but the most high performace web sites, the best approach is to write our the HTML pages to files!. Unless the intrer-process communication is benchmarked to be the botttleneck in performance, don't both with any of the non-file solutions (shared memory, cache, intermediate server).
Specifically for two CGI scripts on the same server, if you run them under mod_perl or some other arrangement which shares Perl interpreter between 2 CGI processes, you can develop a package to serve as cache, which -with its package level variable - would be preserved in memory by mod_perl as long as mod_perl is running and can thus be used by a writer CGI process and a reader CGI process to communicate. Of course the usual synchronization/deadlock and persistance issues associated with reader/writer need to be considered.
As an alternative, use Apache::Session sessions to store inter-session data.
As you noted, shared memory. For example use IPC::ShareLite, IPC::Cache, or this solution from perlmonks.
Also, please check Chapter 16 Recipe 12 "Sharing Variables in Different Processes" from O'Reilly's "Perl Cookbook" (no link since non-pirated versions aren't online anywhere I know of)
Use a permanent medium. A file is one option. A database is another.
For async, use an intermediate messaging system (MQ, Tibco, something more lightweight). Probably a bit of an overkill in this scenario but a valid option to be aware of. This one is likely to be pretty stablem solid and optmized, but possibly not free and less flexible/tailored.
Or roll your own simple messaging system server - it's not THAT complicated for very simple one you seem to need.
Listen on one port for requests from first process to store data, listen on another port for requests from consumer process to send you that data, store the data in a storage area in memory and purge it when it expires using alarms or separate watcher child process).
You've tagged your question as "cgi". Are they both CGI programs? In that case, they can just talk to each other by making HTTP requests.
However, you'll have to tell a lot more about why you are trying to do this and what you need to accomplish for us to help you. It's certainly easy for Perl programs to communicate with each other in some fashion, but that doesn't mean it's the right answer for you.
When you have complex requirements for interaction among CGI programs, you probably want to move to a web framework that handles a lot of those details for you. Catalyst might be where'd you want to start. There's even a book for it.
I'm attempting to move a web app we have (written in Perl) from an IIS6 server to an IIS7.5 server.
Everything seems to be parsing correctly, I'm just having some issues getting the app to actually work.
The app is basically a couple forms. You fill the first one out, click submit, it presents you with another form based on what checkboxes you selected (using includes and such).
I can get past the first form once... but then after that it stops working and pops up the generated error message. After looking into the code and such, it basically states that there aren't any checkboxes selected.
I know the app writes data into .dat files... (at what point, I'm not sure yet), but I don't see those being created. I've looked at file/directory permissions and seemingly I have MORE permissions on the new server than I did on the last. The user/group for the files/dirs are different though...
Would that have anything to do with it? Why would it pass me on to the next form, displaying the correct "modules" I checked the first time and then not any other time after that? (it seems to reset itself after a while)
I know this is complicated so if you have any questions for me, please ask and I'll answer to the best of my ability :).
Btw, total idiot when it comes to Perl.
EDIT AGAIN
I've removed the source as to not reveal any security vulnerabilities... Thanks for pointing that out.
I'm not sure what else to do to show exactly what's going on with this though :(.
I'd recommend verifying, step by step, that what you think is happening is really happening. Start by watching the HTTP request from your browser to the web server - are the arguments your second perl script expects actually being passed to the server? If not, you'll need to fix the first script.
(start edit)
There's lots of tools to watch the network traffic.
Wireshark will read the traffic as it passes over the network (you can run it on the sending or receiving system, or any system on the collision domain).
You can use a proxy server, like WebScarab (free), Burp, Paros, etc. You'll have to configure your browser to send traffic to the proxy server, which will then forward the requests to the server. These particular servers are intended to aid testing, in that you'll be able to mess with the requests as they go by (and much more)
As Sinan indicates, you can use browser addons like Fx LiveHttpHeaders, or Tamper Data, or Internet Explorer's developer kit (IIRC)
(end edit)
Next, you should print out all CGI arguments that the second perl script receives. That way, you'll know what the script really thinks it gets.
Then, you can enable verbose logging in IIS, so that it logs the full HTTP request.
This will get you closer to the source of the problem - you'll know if it's (a) the first script not creating correct HTML, resulting in an incomplete HTTP request from the browser, (b) the IIS server not receiving the CGI arguments for some odd reason, or (c) the arguments aren't getting from the IIS server and into the perl script (or, possibly, that the perl script is not correctly accessing the arguments).
Good luck!
What you need to do is clear.
There is a lot of weird excess baggage in the script. There seemed to be no subroutines. Just one long series of commands with global variables.
It is time to start refactoring.
Get one thing running at a time.
I saw HTML::Template there but you still had raw HTML mixed in with code. Separate code from presentation.
I inherited a legacy Perl script from an old server which is being removed. The script needs to be implemented on a new server. I've got it on the new server.
The script is pretty simple; it connects via expect & ssh to network devices and gathers data. For debugging purposes, I'm only working with the portion that gathers a list of the interfaces from the device.
The script on the new server always shows me a page within about 5 seconds of reloading it. Rarely, it includes the list of interfaces from the remote device. Most commonly, it contains all the HTML elements except the list of interfaces.
Now, on the old server, sometimes the script would take 20 seconds to output the data. That was fine.
Based on this, it seems that apache on the new server is displaying the data before the Perl script has finished returning its data, though that could certainly be incorrect.
Additional Information:
Unfortunately I cannot post any code - work policy. However, I'm pretty sure it's not a problem with expect. The expect portions are written as expect() or die('error msg') and I do not see the error messages. However, if I set the expect timeout to 0, then I do see the error messages.
The expect timeout value used in the script normally is 20 seconds ... but as I mentioned above, apache displays the static content from the script after about 5 seconds, and 95% of the time does not display the content that should retrieved from expect. Additionally, the script writes the expect content to a file on the drive - even when the page does not display it.
I just added my Troubleshooting Perl CGI scripts guide to Stackoverflow. :)
You might try CGI::Inspect. I haven't needed to try it myself, but I saw it demonstrated at YAPC, and it looked awesome.