I'm debugging a problem with Apache, Fast-CGI and our Perl application.
The root cause is that the Perl application needs to exit and be reloaded if a certain operation was done.
Under CGI it is not a problem, as the application is being loaded for every request.
However, it seen that the Fast CGI protocol did not considered this case - there is not way to signal the WebServer that you are going to exit - you just exit.
But doing so, we ran into what seemed like a race condition on the mod_fastcgi side: it recognize that the programed exited, but then tries to send it requests. eventually it gives up and return error 500 to the browser.
I see similar error messages on the web, but these are about PHP exiting after 500 requests, and the solution is to tell fastcgi to limit requests-per-process to 500. This does not fit my problem, as I need to exit on certain request and not after fixed number of requests.
Info:
Apache module loading:
LoadModule fastcgi_module libexec/apache2/mod_fastcgi.so
Two log lines: the server detects that the app exited, but then trying to connect it
[Thu Jul 05 15:02:32 2012] [warn] FastCGI: (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" (pid 9277) terminated by calling exit with status '0'
[Thu Jul 05 15:02:32 2012] [warn] FastCGI: (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" has failed to remain running for 30 seconds given 3 attempts, its restart interval has been backed off to 600 seconds
Finally, giving up:
[Thu Jul 05 15:03:07 2012] [error] [client 127.0.0.1] FastCGI: comm with (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" aborted: (first read) idle timeout (30 sec), referer: http://localhost/~sfomberg/cgi-bin/mt/mt.cgi?__mode=cfg_plugins&blog_id=0&switched=1
[Thu Jul 05 15:03:07 2012] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi", referer: http://localhost/~sfomberg/cgi-bin/mt/mt.cgi?__mode=cfg_plugins&blog_id=0&switched=1
I tried to close the listening socket directly before ending the request, (the listening socket is fileno 0) but that didn't work.
thanks.
The problem was because mod_fastcgi was of an old version - 2.4.2
upgrading to 2.4.6 solved it.
Also, mod_fcgid does not have this problem.
Related
We are using SVN with ubuntu 14.04 and eclipse subversion and apache. It works fine when we commit small files. But when we try to commit large file, it gives following error.
eclipse error:
Some of selected resources were not committed.
Some of selected resources were not committed.
svn: E175002: Commit failed (details follow):
svn: E175002: Commit failed (details follow):
svn: E175002: can not read HTTP status line
svn: E175002: PUT request failed on '/svn/test/!svn/wrk/953b88fa-5601-0010-8146-c3b0661fb4b6/trunk/*/TokenManagerImpl.java'
apache error.log
[Mon Sep 05 19:12:18.533736 2016] [dav:error] [pid 26083:tid 140002512074496] (70007)The timeout specified has expired: [client 182.75.153.50:56725] Timeout reading the body (URI: /svn/test/!svn/wrk/953b88fa-5601-0010-8146-c3b0661fb4b6/trunk/*/TokenManagerImpl.java) [408, #0]
[Mon Sep 05 19:12:18.533851 2016] [dav:error] [pid 26083:tid 140002512074496] [client 182.75.153.50:56725] mod_dav_svn close_stream: error closing write stream [500, #185004]
[Mon Sep 05 19:12:18.533876 2016] [dav:error] [pid 26083:tid 140002512074496] [client 182.75.153.50:56725] Unexpected end of svndiff input [500, #185004]
Below is apche timeout configuration (apache.conf)
Timeout 300
KeepAlive On
MaxKeepAliveRequests 250
KeepAliveTimeout 20
LimitRequestBody 0
Below is reqtimeout.conf
<IfModule reqtimeout_module>
RequestReadTimeout header=200-400,minrate=5000
RequestReadTimeout body=1000,minrate=5000
</IfModule>
It seems that invalid RequestReadTimeout configuration is the root cause. You should not put a special timeout for body.
Apache Subversion transfers commit data as request body and it's size depends on the size of commit. Therefore, with the config you currently have, all commit operations that take more than ~1000 seconds will fail for your users.
Just in case someone else ends up here with:
svn commit via http
Large binary file ~80MB.
Transmitting file data .
svn: E185004: Commit failed (details follow):
svn: E185004: Unexpected end of svndiff input
Change the apache2 reqtimeout.conf file for the header as well.
(The symptom is the svn commit will croak after 40 seconds...)
RequestReadTimeout header=200-400,minrate=500
Fixed it for me. "Your mileage may vary..."
We use SugarCRM CE 6.5.16 on Centos 6.5.
I am getting this error :
Wed Apr 9 15:37:10 2014 [10389][1][ERROR] Unable to load custom logic file: include/SugarSearchEngine/SugarSearchEngineQueueManager.php
The real problem is that i dont receive emails from my inbound email.
They are all set up.I added the cron job to the crontab. Well actually I receive some emails, like 3 or 4 from 100. In the schedulers the job status is "running" and last successful run is "Never".
Every other scheduler job has a status "Done" and has last successful run.
I repaired inbound emails , scheduler jobs but with no effect.
The only thing i found is this :
http://suitecrm.com/forum/search?query=SugarSearchEngineQueueManager&searchdate=all&childforums=1
So I commented out this code and i no longer get the error but still I don't receive emails.
I don't know what else to do.
Please help me if you can !Thanks !
EDIT
I found that
"This file is only included in PRO version and it's useless in Community Edition.
Code Fix:
1. Comment code in /custom/Extension/application/Ext/LogicHooks/SugarFTSHooks.php
Do a Fast Rebuild from Administration (index.php?module=Administration&action=repair). This process will rebuild the piece of code that uses the SugarCRM to call inexistent file SugarSearchEngineQueueManager /custom/application/Ext/LogicHooks/logichooks.ext.php"
So I commented out the code and did the rebuild (yeah i did it before but now i know for sure that this file should not be in Sugarcrm CE )
The error doesn't show anymore but my scheduler still stays "running" and nothings happens , except:
When i did this (What Matthew Poer said) , I received 4 emails just like before... so something is causing a problem.
delete from job_queue where scheduler_id = 'THE_SCHEDULER_ID';
update schedulers set last_run = subdate(now(),360) where id = 'THE_SCHEDULER_ID';
EDIT 2:
This is from php error_log
[Sun Apr 13 03:34:27 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Apr 13 03:34:27 2014] [notice] Digest: done
[Sun Apr 13 03:34:33 2014] [notice] Apache/2.2.15 (Unix) DAV/2 mod_nss/2.2.15 NSS/3.14.0.0 Basic ECC PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Sun Apr 13 12:22:52 2014] [error] [client 122.155.18.51] File does not exist: /usr/share/phpMyAdmin/translators.html
[Sun Apr 13 13:45:31 2014] [error] [client 122.155.18.51] File does not exist: /usr/share/phpMyAdmin/translators.html
[Sun Apr 13 15:43:39 2014] [error] [client 66.249.66.74] File does not exist: /opt/otrs/var/httpd/htdocs/js/js-cache/ModuleJS_784dc12bf89d72db064caa6e8690168b.js
[Sun Apr 13 15:43:40 2014] [error] [client 66.249.66.74] File does not exist: /opt/otrs/var/httpd/htdocs/skins/Customer/default/css-cache/CommonCSS_b1f924c426a0e1a9f1553197a2ce25a4.css
[Sun Apr 13 15:43:41 2014] [error] [client 66.249.66.74] File does not exist: /opt/otrs/var/httpd/htdocs/js/js-cache/CommonJS_7f98ddff2f339e3b515f7901d82600bb.js
[Mon Apr 14 11:09:04 2014] [error] [client 192.168.10.1] PHP Warning: file_get_contents(): php_network_getaddresses: getaddrinfo failed: Name or service not known in /usr/share/phpMyAdmin/version_check.php on line 16, referer: http://support.expert-m.net/phpmyadmin/main.php?token=d2e60372f8b5d6d53f0c3c80a536be27
[Mon Apr 14 11:09:04 2014] [error] [client 192.168.10.1] PHP Warning: file_get_contents(http://www.phpmyadmin.net/home_page/version.json): failed to open stream: php_network_getaddresses: getaddrinfo failed: Name or service not known in /usr/share/phpMyAdmin/version_check.php on line 16, referer: http://support.expert-m.net/phpmyadmin/main.php?token=d2e60372f8b5d6d53f0c3c80a536be27
[Mon Apr 14 12:45:25 2014] [error] [client 178.235.72.68] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
This is from the logs folder of SugarCRM, error.log
[Mon Apr 14 08:58:59 2014] [error] [client 192.168.10.1] PHP Notice: Undefined index: 8854a79c-6171-036c-e7df-534548e8bc81 in /var/www/sugarcrm/public_html/modules/Emails/EmailUIAjax.php on line 879, referer: http://sugarcrm.support.expert-m.net/index.php?module=Emails&action=index&parentTab=All
[Mon Apr 14 08:58:59 2014] [error] [client 192.168.10.1] PHP Notice: Undefined index: 8854a79c-6171-036c-e7df-534548e8bc81 in /var/www/sugarcrm/public_html/modules/Emails/EmailUIAjax.php on line 880, referer: http://sugarcrm.support.expert-m.net/index.php?module=Emails&action=index&parentTab=All
[Mon Apr 14 11:22:17 2014] [error] [client 192.168.10.1] PHP Notice: Undefined index: 8854a79c-6171-036c-e7df-534548e8bc81 in /var/www/sugarcrm/public_html/modules/Emails/EmailUIAjax.php on line 879, referer: http://sugarcrm.support.expert-m.net/index.php?module=Emails&action=index&parentTab=All
[Mon Apr 14 11:22:17 2014] [error] [client 192.168.10.1] PHP Notice: Undefined index: 8854a79c-6171-036c-e7df-534548e8bc81 in /var/www/sugarcrm/public_html/modules/Emails/EmailUIAjax.php on line 880, referer: http://sugarcrm.support.expert-m.net/index.php?module=Emails&action=index&parentTab=All
[Mon Apr 14 11:24:47 2014] [error] [client 192.168.10.1] File does not exist: /var/www/sugarcrm/public_html/favicon.ico
I didn't paste all the info from the logs. There is more but the errors are repeating.
The file include/SugarSearchEngine/SugarSearchEngineQueueManager.php won't exist in your system because it's a Pro+ feature.
To reset a scheduler entry that got "stuck," delete the scheduler information from the job queue in the database and reset the last_run value. Find the ID of the scheduler from the URL within SugarCRM or by select id,name from schedulers. Once you have the ID of this scheduler, run this two queries:
delete from job_queue where scheduler_id = 'THE_SCHEDULER_ID';
update schedulers set last_run = subdate(now(),360) where id = 'THE_SCHEDULER_ID';
Running Puppet v2.7.14 on CEntOs 6 and also using Apache/Passenger instead of WEBrick. I was told that puppetmaster service is not required to be running (hence: chkconfig off puppetmaster) running when using httpd and passenger but in my case, if I don't start puppetmasterd manually, none of the agents can connect to the master. I can start httpd just fine and 'passenger' seems to start okay as well. This is my apache configuration file:
# /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module modules/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.12
PassengerRuby /usr/bin/ruby
#PassengerTempDir /var/run/rubygem-passenger
PassengerHighPerformance on
PassengerUseGlobalQueue on
PassengerMaxPoolSize 15
PassengerPoolIdleTime 150
PassengerMaxRequests 10000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off
</IfModule>
Upon restart, I see these in the httpd_error log:
[Sat Jun 09 04:06:47 2012] [notice] caught SIGTERM, shutting down
[Sat Jun 09 09:06:51 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Jun 09 09:06:51 2012] [notice] Digest: generating secret for digest authentication ...
[Sat Jun 09 09:06:51 2012] [notice] Digest: done
[Sat Jun 09 09:06:51 2012] [notice] Apache/2.2.15 (Unix) DAV/2 Phusion_Passenger/3.0.12 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
And passenger-status prints these info on the screen:
----------- General information -----------
max = 15
count = 0
active = 0
inactive = 0
Waiting on global queue: 0
----------- Application groups -----------
But still, as I said, none of my agents can actually talk to the master until I start puppetmasterd manually. Does anyone know what am I still missing? Or, is this the way it supposed too be? Cheers!!
It sounds like you may be missing an /etc/httpd/conf.d/puppetmaster.conf file that's based on https://github.com/puppetlabs/puppet/blob/master/ext/rack/files/apache2.conf
Without something like this, you're missing the glue that'll map port 8140 to rack-based pupeptmastd.
See http://docs.puppetlabs.com/guides/passenger.html
https://github.com/puppetlabs/puppet/tree/master/ext/rack
http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application_including_rails_gt_3
After a few days of banging head, now it's running. The main problem was with port number - the puppetmaster was running on different port than puppet agent trying to communicate on.
Another thing is: RackAutoDetect On must be executed before the dashboard vhost file. My So, I renamed passenger vhost file as: 00_passenger.conf to make sure it runs first. After that I get puppetmaster running using Apache/Passenger. Cheers!!
I have just purchased a license for Zend Studio 9. I have only a minimal amount of experience with the Zend framework, and no previous experience with Zend Studio. I am using http://framework.zend.com/manual/en/ as a tutorial on the framework and have browsed through the resources located at http://www.zend.com/en/products/studio/resources for help with the studio software.
My main problem is that after creating a new Zend project with zstudio, I'm not seeing the initial welcome message. Here are the steps I am using:
I've already installed the Zend Server and confirmed that web apps are working (made some test files, they all parsed correctly).
Create a new project with Zend Studio.
a. File->New->Local PHP Project
b. For location, I am using C:\Program Files\Zend\Apache2\htdocs.
c. For version I used the default "Zend Framework 1.11.11 (Built-in)"
I go to http://localhost:81/projectname. Instead of the default index controller being called, I just see my directory structure.
Addition info:
OS: Windows 7
PHP version: 5.3
ERROR LOGS:
>[Wed Nov 30 14:32:30 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
>[Wed Nov 30 14:32:30 2011] [warn] pid file C:/Program Files (x86)/Zend/Apache2/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
>[Wed Nov 30 14:32:30 2011] [notice] Digest: generating secret for digest authentication ...
>[Wed Nov 30 14:32:30 2011] [notice] Digest: done
>[Wed Nov 30 14:32:31 2011] [notice] Apache/2.2.16 (Win32) mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations
>[Wed Nov 30 14:32:31 2011] [notice] Server built: Aug 8 2010 16:45:53
>[Wed Nov 30 14:32:31 2011] [notice] Parent: Created child process 13788
>[Wed Nov 30 14:32:32 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
>[Wed Nov 30 14:32:32 2011] [notice] Digest: generating secret for digest authentication ...
>[Wed Nov 30 14:32:32 2011] [notice] Digest: done
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Child process is running
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Acquired the start mutex.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting 64 worker threads.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting thread to listen on port 10081.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting thread to listen on port 81.
If you navigate to http://localhost:81/projectname/index/index does the correct screen load?
If so:
Check that the .htaccess file in your public directory contains the correct rewrite rules for Zend Framework.
Check your httpd.conf file and make sure index.php is added to the DirectoryIndex directive.
I think the solution is going to be the second bullet, but let me know what you find and I can help further if that doesn't work. Make sure to restart apache after you make any changes to httpd.conf.
Otherwise, report any errors you see when you access the controller directly, and check Apache's error_log file to see if you get any errors.
I have a catalyst app running through fast cgi and the apache error logs are useless.
Example:
[Thu Oct 13 08:44:35 2011] [error] [client {IP}] FastCGI: server "/usr/local/www/handprints2/script/handprints2_fastcgi.pl" stderr: | -> handprints2::View::json->process | 0.000523s |, referer: https://[SERVER]/handprints2/
[Thu Oct 13 08:44:35 2011] [error] [client {IP}] FastCGI: server "/usr/local/www/handprints2/script/handprints2_fastcgi.pl" stderr: | /end | 0.000324s |, referer: https://[SERVER]handprints2/
[Thu Oct 13 08:44:35 2011] [error] [client {IP}] FastCGI: server "/usr/local/www/handprints2/script/handprints2_fastcgi.pl" stderr: '------------------------------------------------------------+-----------', referer: https://[SERVER]/handprints2/
Is there a way to fix this?
You can configure your own log feeds and format in apache using the TransferLog and LogFormat directives:
TransferLog /tmp/sample.log
LogFormat "bazinga -> %U"
See Apache 2.0 Logging Directives or Apache 1.3 Logging Directives
I had the same problem and didn't really find the Apache log config route that convenient.
This does the job pretty well though: https://metacpan.org/pod/Catalyst::Plugin::Log::Handler
Description from CPAN:
If your Catalyst project logs many messages, logging via standard
error to Apache's error log is not very clean: The log messages are
mixed with other web applications' noise; and especially if you use
mod_fastcgi, every line will be prepended with a long prefix.
An alternative is logging to a file. But then you have to make sure
that multiple processes won't corrupt the log file. The module
Log::Handler by Jonny Schulz does exactly this, because it supports
message-wise flocking.
This module is a wrapper for said Log::Handler.