Uploading files smaller than 60kb works without a problem, but when I try to upload a larger file, the browser display this error:
An error occurred while reading the request
No error in the nginx error logs or any other useful information.
This is my nginx config for this particular script:
location ~ \.(pl|cgi)$ {
fastcgi_pass unix:/var/run/fcgiwrap.socket;
fastcgi_index index.cgi;
include fastcgi_params;
}
And this is a simple perl script I use instead of my original until I can get this working:
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "<pre>\n";
foreach $key (sort keys(%ENV)) {
print "$key = $ENV{$key}<p>";
}
print "</pre>\n";
How can I get nginx + fastcgi to work with larger file uploads?
I found the problem. Or at least a solution to the problem. I was using fcgiwrap as a wrapper, and apparently it has issues with larger files. So I switched to a different wrapper ( https://library.linode.com/web-servers/nginx/perl-fastcgi/centos-5 ) and everything works great now.
Related
I have a web server running nginx on http://192.168.x.x, and a perl module is installed using http://nginx.org/en/docs/http/ngx_http_perl_module.html
I need to do a GET request inside that module using LWP. This is what i have:
my $url = "http://192.168.x.x/app/test.php";
my $ua = LWP::UserAgent->new;
$ua->timeout(5); # muere a los 5 segundos sin respuesta.
$ua->default_header('Cookie' => "FOO=BAR;PATH=/");
my $resp = $ua->get($url);
if ($resp->is_success) {
my $message = $resp->decoded_content;
print STDERR "Received reply: $message\n";
} else {
print STDERR "HTTP GET error code: ", $resp->code, "\n";
print STDERR "HTTP GET error message: ", $resp->message, "\n";
};
Everytime i do that request i get error code 500 : read timeout. But if i use another server works just fine. The problem is when i use the same server where i run the script.
I think this is because nginx has only one worker thread. When you call your perl code inside nginx it blocks whole server. So one nginx can't process your blocking perl code and php handler at the same time. This is deadlock. You can try Nginx::HTTP which shouldn't block nginx while waiting http response. So, this will work, I think.
It would be great if someone can help. I really am stuck.
I am downloading the master files from SEC edgar and I got the script from—http://brage.bibsys.no/bi/bitstream/URN:NBN:no-bibsys_brage_38213/1/Norli_SRFE_2012.pdf (page 14..published now)
I get the error 404 master.gz not found
While debugging i made it paste the url and when i use the same in browser I can download the file. It is parsing the url correctly till QTR1 but after that it is not able to find the file when it actually exists ..please help.
1) for debugging reasons now I changed the code to 1995 (but later plan to add years 1995 to 2012)
2) It did not work for any file. When I said QTR1 abovr - I meant that the same code without the file name (just for testing ) -- ....full-index/1995/QTR1/ (without the file name) returns a status code OK but ...ftp.sec.gov/edgar/full-index/1995/QTR1/master.gz returns 404 file not found error. It does not work for any quarter.
I wasted so much time on this seemingly simple thing which is supposed to work but it is just not working.... could you copy past this and run..is it working for you?
The code below gets the master files from QTR folders. Pasting my code ::
—————-
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(250);
$ua->env_proxy;
for($year=1995; $year<1996; $year=$year+1){
for($i=1; $i<5; $i=$i+1){
$quarter = “QTR” . $i;
$filegrag = “ftp://ftp.sec.gov/edgar/full-index/” . $year . “/” . $quarter . “/master.gz”;
print $filegrag;
# This command gets the file from EDGAR
my $response = $ua->get($filegrag);
print $response;
print $response->status_line;
# Now just pipe the output to a file named appropriately
$filename = $year . $quarter . “master”;
open(MYOUTFILE, “>” . $filename);
if ($response->is_success) {
print MYOUTFILE $response->decoded_content;
}
else {
die $response->status_line;
}
close(MYOUTFILE);
}
}
I realized that there were some firewall issues that were causing the problem I had. Now things are fine.
I have a simple requirement of screen scraping a web-page (simple URL based reports) and direct the HTML response to an output file. The URL will however redirect to an authentication (HTTPS Login) page with "form based" authentication (no javascript) and upon authentication the report I am trying to view should show up in the $response (as HTML). Interestingly, my code is working just fine in a Windows machine, however the same code below is not working in AIX machine and it looks like the click_button() function call does nothing. I have tried click(), submit(), but none is working so instead of getting the actual report all I get is the logon screen in the HTML output file. Any ideas, what can be wrong?
use WWW::Mechanize;
use strict;
my $username = "admin";
my $password = "welcome1";
my $outpath = "/home/data/output";
my $fromday = 7;
my $url = "https://www.myreports.com/tax_report.php";
my $name = "tax_report";
my $outfile = "$outpath/$name.html";
my $mech = WWW::Mechanize->new(noproxy =>'0');
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$year += 1900;
$mon++; # since it will start from 0
$mday--; # yesterdays date (to day)
$fromday = $mday - $days; #(from day)
#Create URL extension for generating report with previous date
my $dt_range = "?Y&dc_date1=$mon%2F$fromday%2F$year&dc_date2=$mon%2F$mday%2F$year";
my $url = $url . $dt_range;
$mech->get($url);
$mech->field(login => "$username");
$mech->field(passwd => "$password");
$mech->add_handler("request_send", sub { shift->dump; return });
$mech->add_handler("response_done", sub { shift->dump; return });
$mech->click_button(value=>"Login now");
my $response = $mech->content();
print "Generating report: $name...\n";
open (OUT, ">>$outfile")|| die "Cannot create report file $outfile";
print OUT "$response";
close OUT;
The WWW::Mechanize version in both the Machines are same i.e. 1.54 but the Win machine perl version is 5.10.1 whereas the AIX machine's Perl version is 5.8.8.
Other Alternatives Used -
my $inputobject=$mech->current_form()->find_input( undef,'submit' );
print $inputobject->value . "\n";
$mech->click_button(input => $inputobject);
print $mech->status() . "\n";
The $inputobject shows the correct button element as in the HTML source and the second print returns a status of 200 which apparently stands for OK. But its still not working in the AIX machine.
UPDATE- It seems that the site I am trying to connect to has an un-trusted SSL certificate. I tried the program on three different machines Windows PC, Mac and AIX. On the Windows Machine the program works and I was able to login to the website through the browsers (Chrome, Firefox,IE). However in Mac, the login just won't work (through the browsers) and it shows an un-trusted certificate error (or warning!) this probably means the proxy settings are not set up, the Perl program won't work either. And lastly the AIX where the Perl is not working as well. Not sure how to bypass this un-trusted SSL certificate issue here. Any help will be appreciated.
UPDATE2: Included below lines of code in the script to see logging details and found that I was being re-directed (HTTP 302) since my IP was filtered by the server Firewall. Once the AIX ip was added to the server's firewall exception the script worked perfectly. The two lines below were the life saver-
$mech->add_handler("request_send", sub { shift->dump; return });
$mech->add_handler("response_done", sub { shift->dump; return });
Can you use the following line before my $mech = WWW::Mechanize->new(noproxy =>'0'); of your perl code and try again ?
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
I'm not of the Perl world, so some of this is new to me. I'm running Ubuntu Hardy LTS with apache2 and mod_fcgid packages installed. I'd like to get MT4 running under fcgid rather than mod-cgi (it seems to run OK with plain-old CGI).
I can't seem to get even a simple Perl script to run under fcgid. I created a simple "Hello World" app and included the code from this previous question to test if FCGI is running.
I named my script HelloWorld.fcgi (currently fcgid is set to handle .fcgi files only). Code:
#!/usr/bin/perl
use FCGI;
print "Content-type: text/html\n\n";
print "Hello world.\n\n";
my $request = FCGI::Request();
if ( $request->IsFastCGI ) {
print "we're running under FastCGI!\n";
} else {
print "plain old boring CGI\n";
}
When run from the command line, it prints "plain old boring..." When invoked via an http request to apache, I get a 500 Internal Server error and the output of the script is printed to the Apache error log:
Content-type: text/html
Hello world.
we're running under FastCGI!
[Wed Dec 03 22:26:19 2008] [warn] (104)Connection reset by peer: mod_fcgid: read data from fastcgi server error.
[Wed Dec 03 22:26:19 2008] [error] [client 70.23.221.171] Premature end of script headers: HelloWorld.fcgi
[Wed Dec 03 22:26:25 2008] [notice] mod_fcgid: process /www/mt/HelloWorld.fcgi(14189) exit(communication error), terminated by calling exit(), return code: 0
When I run the .cgi version of the same code, it works fine. Any idea why the output of the script is going to the error log? Apache config is the default mod_fcgid config plus, in a VirtualHost directive:
ServerName test1.example.com
DocumentRoot /www/example
<Directory /www/example>
AllowOverride None
AddHandler cgi-script .cgi
AddHandler fcgid-script .fcgi
Options +ExecCGI +Includes +FollowSymLinks
</Directory>
The problem is that the "Content-Type" header is sent outside of the request loop. You must print the "Content-Type" header for every request. If you move
print "Content-type: text/html\n\n";
to the top of the request loop it should fix the problem.
Also, you need to loop over the requests or you'll get no benefit, so following the first poster's example:
my $request = FCGI::Request();
while($request->Accept() >= 0) {
print("Content-type: text/html\n\n");
}
I use CGI::Fast more than FCGI, but the idea is the same, I think. The goal of fast cgi is to load the program once, and iterate in a loop for every request.
FCGI's man page says :
use FCGI;
my $count = 0;
my $request = FCGI::Request();
while($request->Accept() >= 0) {
print("Content-type: text/html\r\n\r\n", ++$count);
}
Which means, you have to Accept the request before being able to print anything back to the browser.
Movable Type uses CGI::Fast for FastCGI. The typical FastCGI script runs in a loop, as mat described. A loop that uses CGI::Fast would look like this:
#!/usr/bin/perl
use strict;
use CGI::Fast;
my $count = 0;
while (my $q = CGI::Fast->new) {
print("Content-Type: text/plain\n\n");
print("Process ID: $$; Count is: " . ++$count);
}
I tested this script on a server with the FCGI and CGI::Fast modules installed and count increments as you'd expect. If the process id changes, count will go back to 1 and then increment within that process. Each process has it's own variable space of course.
For MT, enabling FastCGI a matter of renaming (or symlinking) the cgi scripts to 'fcgi' (or making the handler for 'cgi' scripts fcgid, but that won't work for mt-xmlrpc.cgi which isn't FastCGI friendly yet). You'll also need to add some directives to your mt-config.cgi file so that it knows the new script names. Like this:
AdminScript mt.fcgi
CommentsScript mt-comments.fcgi
And so forth. More documentation specific to FastCGI and Movable Type is available on movabletype.org.
Anyway, based on your server's error logs, it looks like FCGI is working, and being invoked properly, but your script just isn't running in a loop, waiting for the next request to come along. So your test script did accomplish the task -- reporting whether FastCGI is configured or not. So now you should be able to reconfigure MT to use FastCGI.
I'm attempting to use an existing CAS server to authenticate login for a Perl CGI web script and am using the AuthCAS Perl module (v 1.3.1). I can connect to the CAS server to get the service ticket but when I try to connect to validate the ticket my script returns with the following error from the IO::Socket::SSL module:
500 Can't connect to [CAS Server]:443 (Bad hostname '[CAS Server]')
([CAS Server] substituted for real server name)
Symptoms/Tests:
If I type the generated URL for the authentication into the web browser's location bar it returns just fine with the expected XML snippet. So it is not a bad host name.
If I generate a script without using the AuthCAS module but using the IO::Socket::SSL module directly to query the CAS server for validation on the generated service ticket the Perl script will run fine from the command line but not in the browser.
If I add the AuthCAS module into the script in item 2, the script no longer works on the command line and still doesn't work in the browser.
Here is the bare-bones script that produces the error:
#!/usr/bin/perl
use strict;
use warnings;
use CGI;
use AuthCAS;
use CGI::Carp qw( fatalsToBrowser );
my $id = $ENV{QUERY_STRING};
my $q = new CGI;
my $target = "http://localhost/cgi-bin/testCAS.cgi";
my $cas = new AuthCAS(casUrl => 'https://cas_server/cas');
if ($id eq ""){
my $login_url = $cas->getServerLoginURL($target);
printf "Location: $login_url\n\n";
exit 0;
} else {
print $q->header();
print "CAS TEST<br>\n";
## When coming back from the CAS server a ticket is provided in the QUERY_STRING
print "QUERY_STRING = " . $id . "</br>\n";
## $ST should contain the received Service Ticket
my $ST = $q->param('ticket');
my $user = $cas->validateST($target, $ST); #### This is what fails
printf "Error: %s\n", &AuthCAS::get_errors() unless (defined $user);
}
Any ideas on where the conflict might be?
The error is coming from the line directly above the snippet Cebjyre quoted namely
$ssl_socket = new IO::Socket::SSL(%ssl_options);
namely the socket creation. All of the input parameters are correct. I had edited the module to put in debug statements and print out all the parameters just before that call and they are all fine. Looks like I'm going to have to dive deeper into the IO::Socket::SSL module.
As usually happens when I post questions like this, I found the problem. It turns out the Crypt::SSLeay module was not installed or at least not up to date. Of course the error messages didn't give me any clues. Updating it and all the problems go away and things are working fine now.
Well, from the module source it looks like that IO::Socket error is coming from get_https2
[...]
unless ($ssl_socket) {
$errors = sprintf "error %s unable to connect https://%s:%s/\n",&IO::Socket::SSL::errstr,$host,$port;
return undef;
}
[...]
which is called by callCAS, which is called by validateST.
One option is to temporarily edit the module file to put some debug statements in if you can, but if I had to guess, I'd say the casUrl you are supplying isn't matching up to the _parse_url regex properly - maybe you have three slashes after the https?