From some answer in SO I found that best practice to higher processing rate in celery is using it's concurency feature.
AFAIK celery use multiprocessing for concurency.
One of my task is for running a docker image with -p option.
So if there multiple process run this task,they will conflicting on choosing the host port.
So how to now that the current process is the 1st/2nd/3rd/4th process?
What I want to achieve is that (i.e):
1st process will use 127.0.0.1:8001
2nd process will use 127.0.0.1:8002
etc etc
Sincerely
-bino-
http://supervisord.org/configuration.html#program-x-section-values says you can use autorestart=true to restart on exit, but doesn't say how to give a maximum amount of restarts (within startsecs) before giving up. Is there a way to do this? Note: I'm not talking about the first startup, but about the event that a program crashes after, say, running fine for 10 days.
According to the docs, autorestart doesn't care about startretries:
autorestart controls whether supervisord will autorestart a program if
it exits after it has successfully started up (the process is in the
RUNNING state).
supervisord has a different restart mechanism for when the process is
starting up (the process is in the STARTING state). Retries during
process startup are controlled by startsecs and startretries.
You should use startretries as well, ex of program configuration:
[program:consumer_example]
command=command example
process_name=%(program_name)s_%(process_num)02d
numprocs=1
autostart=true
autorestart=true
startretries=10
user=USERNAME
As you can see I used startretries with 10, when you not inform into program it uses the default value (3).
I think that you need is to use the startretries parameter..
http://supervisord.org/configuration.html?highlight=startretries#program-x-section-example
best regards
From the bind9 man page, I understand that the named process starts one worker thread per CPU if it was able to determine the number of CPUs. If its unable to determine, a single worker thread is started.
My question is how does it calculate the number of CPUs? I presume by CPU, it means cores. The Linux machine I work is customized and has kernel 2.6.34 and does not support lscpu or nproc utilities. named is starting a single thread even if i give -n 4 option. Is there any other way to force named to start multiple threads?
Thanks in advance.
The application stack I want to dockerize shall run within a CentOS container. The installation procedure verifies kernel version to ensure application requirements are met. Currently it is detected using "uname ...".
However the application now detects the host kernel version, which is "UBUNTU ..." and not "CentOS" ..."
Is it possible to detect the container's kernel version?
Thanks.
In fact, the kernel is the same in the host and in the container. That's the very principle of containerization: the kernel is shared (because actually, a container is a collection of processes running on top of the host kernel, with special isolation properties).
Does that pose a problem for your application?
How have you set-up one or more worker scripts for queue-oriented systems?
How do you arrange to startup - and restart if necessary - worker scripts as required? (I'm thinking about such tools as init.d/, Ruby-based 'god', DJB's Daemontools, etc, etc)
I'm developing an asynchronous queue/worker system, in this case using PHP & BeanstalkdD (though the actual language and daemon isn't important). The tasks themselves are not too hard - encoding an array with the commands and parameters into JSON for transport through the Beanstalkd daemon, picking them up in a worker script to action them as required.
There are a number of other similar queue/worker setups out there, such as Starling, Gearman, Amazon's SQS and other more 'enterprise' oriented systems like IBM's MQ and RabbitMQ. If you run something like Gearman, or SQS - how do you start and control the worker pool? The questions is on the initial worker startup, and then being able to add additional extra workers, shutting them down at will (though I can send a message through the queue to shut them down - as long as some 'watcher' won't automatically restart them). This is not a PHP problem, it's about straight Unix processes of setting up one or more processes to run on startup, or adding more workers to the pool.
A bash script to loop a script is already in place - this calls the PHP script which then collects and runs tasks from the queue, occasionally exiting to be able to clean itself up (it can also pause a few seconds on failure, or via a planned event). This works fine, and building the worker processes on top of that won't be very hard at all.
Getting a good worker controller system is about flexibility, starting one or two automatically on a machine start, and being able to add a couple more from the command line when the queue is busy, shutting down the extras when no longer required.
I've been helping a friend who's working on a project that involves a Gearman-based queue that will dispatch various asynchronous jobs to various PHP and C daemons on a pool of several servers.
The workers have been designed to behave just like classic unix/linux daemons, thanks to simple shell scripts in /etc/init.d/, and commands like :
invoke-rc.d myWorker start|stop|restart|reload
This mechanism is simple and efficient. And as it relies on standard linux features, even people with a limited knowledge of your app can launch a daemon or stop one, if they know how it's called system-wise (aka "myWorker" in the above example).
Another advantage of this mechanism is it makes your workers pool management easy as well. You could have 10 daemons on your machine (myWorker1, myWorker2, ...) and have a "worker manager" start or stop them depending on the queue length. And as these commands can be run through ssh, you can easily manage several servers.
This solution may sound cheap, but if you build it with well-coded daemons and reliable management scripts, I don't see why it would be less efficient than big-bucks solutions, for any average (as in "non critical") project.
Real message queuing middleware like WebSphere MQ or MSMQ offer "triggers" where a service that is part of the MQM will start a worker when new messages are placed into a queue.
AFAIK, no "web service" queuing system can do that, by the nature of the beast. However I have only looked hard at SQS. There you have to poll the queue, and in Amazon's case overly eager polling is going to cost you some real $$.
I've recently been working on such a tool. It's not entirely finished (thought it should take more than a few more days before I hit something I could call 1.0) and clearly not ready for production yet, but the important part are already coded. Anybody can have a look at the code here: https://gitorious.org/workers_pool.
Supervisor is a good monitor tool. It includes a web UI where you can monitor and manage workers.
Here is a simple config file for a worker.
[program:demo]
command=php worker.php ; php command to run worker file
numprocs=2 ; number of processes
process_name=%(program_name)s_%(process_num)03d ; unique name for each process if numprocs > 1
directory=/var/www/demo/ ; directory containing worker file
stdout_logfile=/var/www/demo/worker.log ; log file location
autostart=true ; auto start program when supervisor starts
autorestart=true ; auto restart program if it exits