Celery periodic tasks getting picked up too many times - celery

Question

I'm running Celery 3.0.24, and Celery-with-redis 3.0 in an app that uses periodic tasks to poll a json feed, and automate tasks based on it.
Celery is correctly putting out that a due task is occurring every 1 minute. However, it appears that the task is being picked up 2-3 times, which is resulting in duplicate and triplicate runs of the app behind it.
The problem typically doesn't occur for a day or a week, but then begins, and won't go away despite stopping and restarting the app.
Details:
Running it with Supervisor
Queue is on Redis (not RabbitMQ)
When it's running, the process (ps aux style) shows up as this:
[celeryd#sky04:MainProcess] -active- (worker --config celeryconfig --beat --schedule=/mnt/services/my_app/var/lib/celerybeat --loglevel=INFO --autoreload --app=my_app.queue.tasks --events --queues=my_app)
celerybeat.conf:
[program:celerybeat]
command=/mnt/services/my_app/bin/celery worker --config celeryconfig --beat --schedule=/mnt/services/my_app/var/lib/celerybeat --loglevel=INFO --autoreload --app=my_app.queue.tasks --events --queues=my_app
environment=PYTHONPATH=/mnt/services/my_app/conf
autostart=false
autorestart=true
startsecs=5
startretries=1
stopwaitsecs=300
numprocs=1
stopsignal=TERM
killasgroup=true
stdout_logfile=/mnt/services/my_app/var/log/celerybeat.log
stderr_logfile=/mnt/services/my_app/var/log/celerybeat.err
tasks.py contains this task:
#periodic_task(
run_every=datetime.timedelta(seconds=60),
name='tasks.my_app_fetch_and_parse_feed',
max_retries=0,
queue='my_app',
options={'queue': 'my_app'},
)
def my_app_fetch_and_parse_feed():
"""
Runs every minute and checks for
matches that need notifications sent.
"""
# some code that's getting run multiple times
pass
Any help and tips anyone could give on this would be greatly appreciated! I've troubleshooted through all my ideas on how to fix it. Thank you so much!
Added info - - -
The processes related to celery are:
$ ps xuf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
507 29554 0.0 0.0 12768 4828 pts/4 S+ 2013 0:00 -bash
507 22921 0.0 0.0 12920 5408 pts/0 S 19:22 0:00 -bash
507 25582 0.0 0.0 8584 812 pts/0 R+ 19:41 0:00 \_ ps xuf
507 13968 0.0 0.0 12804 5376 pts/1 S+ Feb04 0:00 -bash
507 7617 0.0 0.1 106536 12284 ? Ss 2013 60:06 python2.7 /mnt/services/my_app/bin/supervisord
507 23894 13.0 0.3 154644 25168 ? Rl 19:29 1:32 \_ [celeryd#sky03:MainProcess] -active- (worker --beat --schedule=/mnt/services/my_app/var/lib/celerybeat --loglevel=INFO --autoreload --app=my_app
507 23901 0.0 0.2 147852 22608 ? S 19:29 0:00 \_ [celerybeat]
507 23902 6.2 0.3 143476 26288 ? S 19:29 0:44 \_ [celeryd#sky03:PoolWorker-2]
507 23903 6.3 0.3 143980 26712 ? S 19:29 0:44 \_ [celeryd#sky03:PoolWorker-3]
Or for a more verbose output of all the processes related to celery on the box:
$ ps aux | grep celery
APP_TWO 22229 0.0 0.3 164808 26244 ? Sl 2013 2:01 python2.6 /mnt/services/APP_TWO-john/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E
APP_TWO 22266 0.0 0.3 153868 25536 ? S 2013 2:08 python2.6 /mnt/services/APP_TWO-john/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E
APP_TWO 22267 0.0 0.3 153312 24112 ? S 2013 2:05 python2.6 /mnt/services/APP_TWO-john/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E
APP_TWO 22000 0.0 0.0 3820 528 pts/2 S+ 2013 0:01 tail -n 30 -F var/log/celeryd.err
APP_FOUR 22055 0.0 0.0 3820 516 pts/3 S+ 2013 0:00 tail -F var/log/celeryd.err
APP_TWO 12190 0.0 0.3 159004 24316 ? Sl Jan06 0:53 python2.6 /mnt/services/APP_TWO/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E -Q APP_TWO
APP_TWO 12212 0.0 0.2 140736 20252 ? S Jan06 0:39 python2.6 /mnt/services/APP_TWO/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E -Q APP_TWO
APP_TWO 12215 0.0 0.2 140760 20260 ? S Jan06 0:48 python2.6 /mnt/services/APP_TWO/bin/celeryd --loglevel=INFO --autoreload -A APP_TWO.queue.tasks -E -Q APP_TWO
flume-ng 27903 0.0 0.0 3816 524 ? S Jan24 0:00 tail -F /mnt/services/APP_TWO/var/log/celeryd.err
flume-ng 27904 0.0 0.0 3816 524 ? S Jan24 0:00 tail -F /mnt/services/APP_FOUR/var/log/celeryd.log
flume-ng 27927 0.0 0.0 3820 576 ? S Jan24 0:00 tail -F /mnt/services/APP_THREE/var/log/celeryd.err
flume-ng 27945 0.0 0.0 3812 564 ? S Jan24 0:00 tail -F /mnt/services/APP_THREE/var/log/celerybeat.err
flume-ng 27951 0.0 0.0 3812 564 ? S Jan24 0:00 tail -F /mnt/services/MY_APP/var/log/celeryd.log
flume-ng 27967 0.0 0.0 3816 580 ? S Jan24 0:00 tail -F /mnt/services/APP_THREE/var/log/celeryd.log
flume-ng 27969 0.0 0.0 3820 528 ? S Jan24 0:00 tail -F /mnt/services/MY_APP/var/log/celerybeat.log
flume-ng 27970 0.0 0.0 3820 528 ? S Jan24 0:00 tail -F /mnt/services/APP_FOUR/var/log/celeryd.err
flume-ng 27974 0.0 0.0 3816 568 ? S Jan24 0:00 tail -F /mnt/services/APP_THREE/var/log/celerybeat.log
flume-ng 27977 0.0 0.0 3812 564 ? S Jan24 0:00 tail -F /mnt/services/MY_APP/var/log/celeryd.err
flume-ng 28050 0.0 0.0 3816 520 ? S Jan24 0:00 tail -F /mnt/services/APP_TWO/var/log/celeryd.log
508 9256 0.0 0.3 197348 29076 ? Sl Feb08 0:04 python2.7 /mnt/services/APP_THREE/bin/celery worker -B -Q APP_THREE --loglevel=INFO --autoreload -A APP_THREE.queue.tasks -E
508 9264 0.0 0.3 200584 27656 ? S Feb08 0:00 python2.7 /mnt/services/APP_THREE/bin/celery worker -B -Q APP_THREE --loglevel=INFO --autoreload -A APP_THREE.queue.tasks -E
508 9265 0.0 0.3 202092 28060 ? S Feb08 0:48 python2.7 /mnt/services/APP_THREE/bin/celery worker -B -Q APP_THREE --loglevel=INFO --autoreload -A APP_THREE.queue.tasks -E
508 9266 0.0 0.3 206420 29880 ? S Feb08 0:46 python2.7 /mnt/services/APP_THREE/bin/celery worker -B -Q APP_THREE --loglevel=INFO --autoreload -A APP_THREE.queue.tasks -E
APP_FOUR 14512 0.0 0.3 153144 23736 ? Sl 18:23 0:00 python2.7 /mnt/services/APP_FOUR/bin/celeryd --loglevel=INFO --autoreload -A APP_FOUR.queue.tasks -E -Q APP_FOUR
APP_FOUR 14545 0.0 0.2 136212 19868 ? S 18:23 0:00 python2.7 /mnt/services/APP_FOUR/bin/celeryd --loglevel=INFO --autoreload -A APP_FOUR.queue.tasks -E -Q APP_FOUR
APP_FOUR 14547 0.0 0.2 136232 19872 ? S 18:23 0:00 python2.7 /mnt/services/APP_FOUR/bin/celeryd --loglevel=INFO --autoreload -A APP_FOUR.queue.tasks -E -Q APP_FOUR
507 23894 14.6 0.3 154644 25168 ? Sl 19:29 2:08 [celeryd#sky03:MainProcess] -active- (worker --beat --schedule=/mnt/services/MY_APP/var/lib/celerybeat --loglevel=INFO --autoreload --app=MY_APP.queue.tasks --events --queues=MY_APP)
507 23901 0.0 0.2 147852 22640 ? S 19:29 0:00 [celerybeat]
507 23902 6.1 0.3 143500 26312 ? S 19:29 0:53 [celeryd#sky03:PoolWorker-2]
507 23903 6.1 0.3 143660 26452 ? S 19:29 0:54 [celeryd#sky03:PoolWorker-3]
507 25859 0.0 0.0 6040 676 pts/0 S+ 19:43 0:00 grep celery

Answer 1

Upon conversation with the lovely folks in the Celery IRC (ionelmc), this is likely happening because more than one instance of beat is running on my machine.
You can see when you 'ps aux | grep celery' on the box, my_app is using beat, and so is APP_THREE. I should be able to resolve it by shutting one of these down.
http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html?highlight=#cmdoption-celery-worker-B

Answer 2

Also, it may have been that the --schedule parameter wasn't set on one instance of beat, but it was on the other. Or, it could have been that the redis queues on both were using db 0. I haven't definitively found the fix, but the --schedule parameter one is working currently.

Answer 3

I have the same issue with beat, it turns out that I had incorrect beat database file permissions.
I used docker-compose installation with plugged local database file as volume.
I ran beat and locally and from docker, from different users (custom local and root in docker).
It seems that once I first ran beat locally, docker installation wasn't able to read the database, cause it was owned by local user.