Supervisord sometimes starts celery, sometimes not - kubernetes

Question

I'm deploying my flask api on Kubernetes. The executed command when the container is started is the following:
supervisord -c /etc/supervisor/conf.d/celery.conf
gunicorn wsgi:app --bind=0.0.0.0:5000 --workers 1 --threads 12 --log-level=warning --access-logfile /var/log/gunicorn-access.log --error-logfile /var/log/gunicorn-error.log
You see above that I'm starting celery first with supervisor and after that I'm running the gunicorn server. Content of celery.conf:
[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
identifier = supervisor
directory = /tmp
nocleanup = true
[program:celery]
directory = /mydir/app
command = celery -A celery_worker.celery worker --loglevel=debug
When logged into my pods I can see that sometimes the process of starting celery is working (example in pod 1):
> more /tmp/supervisord.log
2021-06-08 18:19:46,460 CRIT Supervisor running as root (no user in config file)
2021-06-08 18:19:46,462 INFO daemonizing the supervisord process
2021-06-08 18:19:46,462 INFO set current directory: '/tmp'
2021-06-08 18:19:46,463 INFO supervisord started with pid 9
2021-06-08 18:19:47,469 INFO spawned: 'celery' with pid 15
2021-06-08 18:19:48,470 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Sometimes it's not (in pod 2):
> more /tmp/supervisord.log
2021-06-08 18:19:42,979 CRIT Supervisor running as root (no user in config file)
2021-06-08 18:19:42,988 INFO daemonizing the supervisord process
2021-06-08 18:19:42,988 INFO set current directory: '/tmp'
2021-06-08 18:19:42,989 INFO supervisord started with pid 9
2021-06-08 18:19:43,992 INFO spawned: 'celery' with pid 11
2021-06-08 18:19:44,994 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
>>>> 2021-06-08 18:19:58,642 INFO exited: celery (exit status 2; expected) <<<<<HERE
In my pod 1, a ps command shows the following:
> ps aux | grep celery
root 9 0.0 0.0 55308 16376 ? Ss 18:45 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/conf.d/celery.conf
root 23 2.2 0.8 2343684 352940 ? S 18:45 0:05 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 37 0.0 0.5 2341860 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 38 0.0 0.5 2341864 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 39 0.0 0.5 2341868 208716 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 40 0.0 0.5 2341872 208724 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 41 0.0 0.5 2341876 208728 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 42 0.0 0.5 2341880 208728 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 43 0.0 0.5 2341884 208736 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
root 44 0.0 0.5 2342836 211384 ? S 18:46 0:00 /usr/bin/python3 /usr/local/bin/celery -A celery_worker.celery worker --loglevel=debug
In my pod 2, I can see that supervisord/celery process is still there but I don't have all the individual /usr/local/bin/celery processes that I have in pod 1:
> ps aux | grep celery
root 9 0.0 0.0 55308 16296 ? Ss 18:19 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/conf.d/celery.conf
This behavior is not always the same. Sometimes when pods are restarted the two succeed to launch celery, sometimes none of them succeed. In this last scenario if I make a request to my API that is supposed to launch a celery task, I can see on my broker console (RabbitMQ) that a task is created but there is no message "activity" and nothing is written is my database table (the end result of my celery task).
If I start celery manually in my pods:
celery -A celery_worker.celery worker --loglevel=debug
everything works.
What could explain such a behavior?

Answer 1

Following the comments above, the best solution is to have two containers, the first having the entrypoint gunicorn and the other celery celery-worker. If the second is the same image as the first it works very well and I can scale on Kubernetes each container independently. The only thing is that the code sourcing is more difficult, each time I make a code change on the first I must apply the same changes manually on the second, maybe there is a better way to address this specific issue of the code sourcing.