I'm writing a web application for which I'm using Sphinx to search around a million documents.
The performance is excellent, with a typical query taking just 0.05 seconds, but if no queries are made for a few hours, it suddenly takes much longer - up to 1000x longer for a couple of queries, then returns to normal. The query log looks like this:
[Wed Mar 7 17:23:55.937 2012] 0.221 sec
[Wed Mar 7 17:32:00.726 2012] 0.012 sec
[Wed Mar 7 17:32:00.984 2012] 0.052 sec
[Wed Mar 7 17:32:01.416 2012] 0.222 sec
[Thu Mar 8 09:15:10.418 2012] 10.147 sec
[Thu Mar 8 09:16:00.560 2012] 48.262 sec
[Thu Mar 8 09:16:55.429 2012] 54.153 sec
[Thu Mar 8 09:17:54.454 2012] 0.012 sec
[Thu Mar 8 09:17:54.713 2012] 0.052 sec
[Thu Mar 8 09:17:55.141 2012] 0.218 sec
I'm guessing maybe my busy server is swapping Sphinx's memory when it is unused or something.
How can I resolve this?
I considered scripting fake queries every minute but that seems like quite an ugly hack.
How fast are the disks on this server?
I imagine this due to having to go back to disk for the indexes. Once accessed a few times, the OS will have cached the files.
It might be worth considering a SSD disk. A small SSD disk - big enough for sphinxes indexs - is relativly cheap nowadays.
If it really is memory swapping, that also suggests you have slow disks. But that is also something to address. Can you add more memory to server? (or even put the swap partition on the newly installed SSD disk :)
btw, find out if swapping is happening with something like Munin (or Cacti etc)
Related
I install Thinking sphinx 3 on Rails 3 app and new sphinx 2.1.1-beta, before this i have
Thinking sphinx 2 and sphinx 2.1.1-beta, and they work fine with add port: 3312 in sphinx.yml
But now when i add port: 3312 in thinking_sphinx.yml i get result
rake ts:start
Failed to start searchd daemon. Check myapp/log/development.searchd.log.
Failed to start searchd. Check the log files for more information.
development.searchd.log.
[Wed May 15 12:17:34.010 2013] [ 2480] binlog: replaying log /myapp/tmp/binlog/development/binlog.001
[Wed May 15 12:17:34.010 2013] [ 2480] binlog: replay stats: 0 rows in 0 commits; 0 updates; 0 indexes
[Wed May 15 12:17:34.010 2013] [ 2480] binlog: finished replaying /myapp/tmp/binlog/development/binlog.001; 0.0 MB in 0.000 sec
[Wed May 15 12:17:34.011 2013] [ 2480] binlog: finished replaying total 10 in 0.001 sec
[Wed May 15 12:17:34.012 2013] [ 2480] FATAL: listen() failed: Address already in use
[Wed May 15 12:17:34.455 2013] [ 2480] shutdown complete
[Wed May 15 12:17:34.469 2013] [ 2479] Child process 2480 has been finished, exit code 1. Watchdog finishes also. Good bye!
And wheh i delete port: 3312 in thinking_sphinx.yml i get
development.searchd.log.
[Wed May 15 12:23:44.040 2013] [ 2527] Child process 2528 has been forked
[Wed May 15 12:23:44.389 2013] [ 2528] listening on 127.0.0.1:9306
[Wed May 15 12:23:44.390 2013] [ 2528] bind() failed on 127.0.0.1, retrying...
[Wed May 15 12:24:20.446 2013] [ 2528] FATAL: bind() failed on 127.0.0.1: Address already in use
[Wed May 15 12:24:20.478 2013] [ 2527] Child process 2528 has been finished, exit code 1. Watchdog finishes also. Good bye!
The port setting is now mysql41 (as TS v3 uses Sphinx's Mysql41 protocol instead of the old binary protocol). Change it to that, make sure you don't have any old searchd processes running, and that should take care of things.
I am currently testing the gwan web server, I have a question about the default settings for the gwan worker, and the CPU Core detection.
Running gwan on a Xen server (which contains a 4-core Xeon CPU), the gwan.log file reports that only got a single core was detected:
'./gwan -w 4' used to override 1 x 1-Core CPU(s)
[Wed May 22 06:54:29 2013 GMT] using 1 workers 0[01]0
[Wed May 22 06:54:29 2013 GMT] among 2 threads 0[11]1
[Wed May 22 06:54:29 2013 GMT] (can't use more threads than CPU Cores)
[Wed May 22 06:54:29 2013 GMT] CPU: 1x Intel(R) Xeon(R) CPU E5506 # 2.13GHz
[Wed May 22 06:54:29 2013 GMT] 0 id: 0 0
[Wed May 22 06:54:29 2013 GMT] 1 id: 1 1
[Wed May 22 06:54:29 2013 GMT] 2 id: 2 2
[Wed May 22 06:54:29 2013 GMT] 3 id: 3 3
[Wed May 22 06:54:29 2013 GMT] CPU(s):1, Core(s)/CPU:1, Thread(s)/Core:2
Any idea?
Thanks!!
the gwan.log file reports that I only got a single core:
Xen, like many other hypervisors, is breaking the CPUID instruction and the /proc/cpuinfo Linux kernel structures (both of which are used by G-WAN to detect CPU Cores).
As you have seen, this is a real problem for multithreaded applications designed to scale on multicore.
'./gwan -w 4' used to override 1 x 1-Core CPU(s)
To avoid stupid manual overrides wasting memory and impacting performance, G-WAN checks that the user-supplied thread count is not greater than the actual CPU Core count.
This is why you have the warning: "(can't use more threads than CPU Cores)".
To bypass this protection and warning, you can use the -g ("God") mode:
./gwan -g -w 4
This command line switch is documented in the G-WAN executable help:
./gwan -h
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usage: gwan [-b -d -g -k -r -t -v -w] [argument]
(grouped options '-bd' are ignored, use '-b -d')
-b: use the TCP_DEFER_ACCEPT TCP option
(this is disabling the DoS shield!)
-d: daemon mode (uses an 'angel' process)
-d:group:user dumps 'root' privileges
-g: do not limit worker threads to Cores
(using more threads than physical Cores
will lower speed and waste resources)
-k: (gracefully) kill local gwan processes
using the pid files of the gwan folder
-r: run the specified C script (and exit)
for general-purpose code, not servlets
-t: store client requests in 'trace' file
(may be useful to debug/trace attacks)
-v: return version number and build date
-w: define the number of worker threads.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Available network interfaces (2):
127.0.0.1 192.168.0.11
I'm debugging a problem with Apache, Fast-CGI and our Perl application.
The root cause is that the Perl application needs to exit and be reloaded if a certain operation was done.
Under CGI it is not a problem, as the application is being loaded for every request.
However, it seen that the Fast CGI protocol did not considered this case - there is not way to signal the WebServer that you are going to exit - you just exit.
But doing so, we ran into what seemed like a race condition on the mod_fastcgi side: it recognize that the programed exited, but then tries to send it requests. eventually it gives up and return error 500 to the browser.
I see similar error messages on the web, but these are about PHP exiting after 500 requests, and the solution is to tell fastcgi to limit requests-per-process to 500. This does not fit my problem, as I need to exit on certain request and not after fixed number of requests.
Info:
Apache module loading:
LoadModule fastcgi_module libexec/apache2/mod_fastcgi.so
Two log lines: the server detects that the app exited, but then trying to connect it
[Thu Jul 05 15:02:32 2012] [warn] FastCGI: (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" (pid 9277) terminated by calling exit with status '0'
[Thu Jul 05 15:02:32 2012] [warn] FastCGI: (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" has failed to remain running for 30 seconds given 3 attempts, its restart interval has been backed off to 600 seconds
Finally, giving up:
[Thu Jul 05 15:03:07 2012] [error] [client 127.0.0.1] FastCGI: comm with (dynamic) server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi" aborted: (first read) idle timeout (30 sec), referer: http://localhost/~sfomberg/cgi-bin/mt/mt.cgi?__mode=cfg_plugins&blog_id=0&switched=1
[Thu Jul 05 15:03:07 2012] [error] [client 127.0.0.1] FastCGI: incomplete headers (0 bytes) received from server "/Users/sfomberg/Sites/cgi-bin/mt/mt.cgi", referer: http://localhost/~sfomberg/cgi-bin/mt/mt.cgi?__mode=cfg_plugins&blog_id=0&switched=1
I tried to close the listening socket directly before ending the request, (the listening socket is fileno 0) but that didn't work.
thanks.
The problem was because mod_fastcgi was of an old version - 2.4.2
upgrading to 2.4.6 solved it.
Also, mod_fcgid does not have this problem.
Running Puppet v2.7.14 on CEntOs 6 and also using Apache/Passenger instead of WEBrick. I was told that puppetmaster service is not required to be running (hence: chkconfig off puppetmaster) running when using httpd and passenger but in my case, if I don't start puppetmasterd manually, none of the agents can connect to the master. I can start httpd just fine and 'passenger' seems to start okay as well. This is my apache configuration file:
# /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module modules/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.12
PassengerRuby /usr/bin/ruby
#PassengerTempDir /var/run/rubygem-passenger
PassengerHighPerformance on
PassengerUseGlobalQueue on
PassengerMaxPoolSize 15
PassengerPoolIdleTime 150
PassengerMaxRequests 10000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off
</IfModule>
Upon restart, I see these in the httpd_error log:
[Sat Jun 09 04:06:47 2012] [notice] caught SIGTERM, shutting down
[Sat Jun 09 09:06:51 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Jun 09 09:06:51 2012] [notice] Digest: generating secret for digest authentication ...
[Sat Jun 09 09:06:51 2012] [notice] Digest: done
[Sat Jun 09 09:06:51 2012] [notice] Apache/2.2.15 (Unix) DAV/2 Phusion_Passenger/3.0.12 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
And passenger-status prints these info on the screen:
----------- General information -----------
max = 15
count = 0
active = 0
inactive = 0
Waiting on global queue: 0
----------- Application groups -----------
But still, as I said, none of my agents can actually talk to the master until I start puppetmasterd manually. Does anyone know what am I still missing? Or, is this the way it supposed too be? Cheers!!
It sounds like you may be missing an /etc/httpd/conf.d/puppetmaster.conf file that's based on https://github.com/puppetlabs/puppet/blob/master/ext/rack/files/apache2.conf
Without something like this, you're missing the glue that'll map port 8140 to rack-based pupeptmastd.
See http://docs.puppetlabs.com/guides/passenger.html
https://github.com/puppetlabs/puppet/tree/master/ext/rack
http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application_including_rails_gt_3
After a few days of banging head, now it's running. The main problem was with port number - the puppetmaster was running on different port than puppet agent trying to communicate on.
Another thing is: RackAutoDetect On must be executed before the dashboard vhost file. My So, I renamed passenger vhost file as: 00_passenger.conf to make sure it runs first. After that I get puppetmaster running using Apache/Passenger. Cheers!!
I have just purchased a license for Zend Studio 9. I have only a minimal amount of experience with the Zend framework, and no previous experience with Zend Studio. I am using http://framework.zend.com/manual/en/ as a tutorial on the framework and have browsed through the resources located at http://www.zend.com/en/products/studio/resources for help with the studio software.
My main problem is that after creating a new Zend project with zstudio, I'm not seeing the initial welcome message. Here are the steps I am using:
I've already installed the Zend Server and confirmed that web apps are working (made some test files, they all parsed correctly).
Create a new project with Zend Studio.
a. File->New->Local PHP Project
b. For location, I am using C:\Program Files\Zend\Apache2\htdocs.
c. For version I used the default "Zend Framework 1.11.11 (Built-in)"
I go to http://localhost:81/projectname. Instead of the default index controller being called, I just see my directory structure.
Addition info:
OS: Windows 7
PHP version: 5.3
ERROR LOGS:
>[Wed Nov 30 14:32:30 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
>[Wed Nov 30 14:32:30 2011] [warn] pid file C:/Program Files (x86)/Zend/Apache2/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
>[Wed Nov 30 14:32:30 2011] [notice] Digest: generating secret for digest authentication ...
>[Wed Nov 30 14:32:30 2011] [notice] Digest: done
>[Wed Nov 30 14:32:31 2011] [notice] Apache/2.2.16 (Win32) mod_ssl/2.2.16 OpenSSL/0.9.8o configured -- resuming normal operations
>[Wed Nov 30 14:32:31 2011] [notice] Server built: Aug 8 2010 16:45:53
>[Wed Nov 30 14:32:31 2011] [notice] Parent: Created child process 13788
>[Wed Nov 30 14:32:32 2011] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
>[Wed Nov 30 14:32:32 2011] [notice] Digest: generating secret for digest authentication ...
>[Wed Nov 30 14:32:32 2011] [notice] Digest: done
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Child process is running
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Acquired the start mutex.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting 64 worker threads.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting thread to listen on port 10081.
>[Wed Nov 30 14:32:33 2011] [notice] Child 13788: Starting thread to listen on port 81.
If you navigate to http://localhost:81/projectname/index/index does the correct screen load?
If so:
Check that the .htaccess file in your public directory contains the correct rewrite rules for Zend Framework.
Check your httpd.conf file and make sure index.php is added to the DirectoryIndex directive.
I think the solution is going to be the second bullet, but let me know what you find and I can help further if that doesn't work. Make sure to restart apache after you make any changes to httpd.conf.
Otherwise, report any errors you see when you access the controller directly, and check Apache's error_log file to see if you get any errors.