I am playing with demo cases with celery tutorial. However, the results were shown disabled when I start the task app. as below. Any idea?
celery --app=plmtcheck worker -l info
Then I see:
- ** ---------- .> app: plmtcheck:0x7f9fd2fdf160
- ** ---------- .> transport: amqp://guest#localhost:5672//
- ** ---------- .> results: disabled
I have seen the result is ready.
[2014-05-05 16:16:55,382: INFO/MainProcess] Connected to amqp://guest#127.0.0.1:5672//
[2014-05-05 16:16:55,389: INFO/MainProcess] mingle: searching for neighbors
[2014-05-05 16:16:56,401: INFO/MainProcess] mingle: all alone
[2014-05-05 16:16:56,422: WARNING/MainProcess] celery#D-NYC-00552088-Linux ready.
[2014-05-05 16:17:27,726: INFO/MainProcess] Received task: plmtcheck.add[7ea5a501-1085-48b7-8f7e-dac8ac2c5377]
[2014-05-05 16:17:27,759: INFO/MainProcess] Task plmtcheck.add[7ea5a501-1085-48b7-8f7e-dac8ac2c5377] succeeded in 0.032166894000056345s: 37
My code is simply
from celery import Celery
app = Celery('plmtcheck', backend='amqp', broker='amqp://')
#app.task
def add(x, y):
return x + y
if __name__ == '__main__':
app.worker_main()
install the django-celery-results package then add it in settings.py at
INSTALLED_APPS = [
...
'django_celery_results',
...
]
This will do what you intend to do
Related
How should I handle SoftTImeLimitException with group of celery tasks?
i have task
> #shared_task(bind=True, priority=2, autoretry_for=(EasySNMPError,),
> soft_time_limit=10, retry_kwargs={'max_retries': 3, 'countdown': 2},
> acks_late=True) def discover_interface(self, interface_id: int) -> dict:
> logger.info(f'start disco interface {interface_id}')
> p = probes.DiscoverSNMP()
> try:
> with utils.Timer() as t1:
> p.discover_interface(interface_id=interface_id)
> logger.info(f'stop disco interface {interface_id}')
> except SoftTimeLimitExceeded:
> p.stats['message'] = 'soft time limit exceeded'
> return p.stats
when i run it i get soft time limit exception.
log from celery
[2021-02-24 16:30:20,298: WARNING/MainProcess] Soft time limit (10s) exceeded for pollers.tasks.discover_interface[6b166746-95c4-458c-bb63-318bdb588d00]
[2021-02-24 16:30:20,374: ERROR/MainProcess] Process 'ForkPoolWorker-2' pid:19585 exited with 'signal 11 (SIGSEGV)'
[2021-02-24 16:30:20,393: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Traceback (most recent call last):
File "/home/kolekcjoner/miniconda3/envs/kolekcjoner/lib/python3.6/site-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).
execution in python console
job = group([tasks.discover_interface.s(1870361)]).apply_async()
In[45]: job.join_native()
Traceback (most recent call last):
File "/home/kolekcjoner/miniconda3/envs/kolekcjoner/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-45-9330abace970>", line 1, in <module>
job.join_native()
File "/home/kolekcjoner/miniconda3/envs/kolekcjoner/lib/python3.6/site-packages/celery/result.py", line 818, in join_native
raise value
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).
how should i change it to NOT have exception in job.join_native(), but message 'soft time limit exceeded'returned from task?
I've some tasks with manually configured routes and 3 workers which were configured to consume tasks from specific queue. But only one worker consuming all of the tasks and I've no idea how to fix this issue.
My celeryconfig.py
class CeleryConfig:
enable_utc = True
timezone = 'UTC'
imports = ('events.tasks')
broker_url = Config.BROKER_URL
broker_transport_options = {'visibility_timeout': 10800} # 3H
worker_hijack_root_logger = False
task_protocol = 2
task_ignore_result = True
task_publish_retry_policy = {'max_retries': 3, 'interval_start': 0, 'interval_step': 0.2, 'interval_max': 0.2}
task_time_limit = 30 # sec
task_soft_time_limit = 15 # sec
task_default_queue = 'low'
task_default_exchange = 'low'
task_default_routing_key = 'low'
task_queues = (
Queue('daily', Exchange('daily'), routing_key='daily'),
Queue('high', Exchange('high'), routing_key='high'),
Queue('normal', Exchange('normal'), routing_key='normal'),
Queue('low', Exchange('low'), routing_key='low'),
Queue('service', Exchange('service'), routing_key='service'),
Queue('award', Exchange('award'), routing_key='award'),
)
task_route = {
# -- SCHEDULE QUEUE --
base_path.format(task='refresh_rank'): {'queue': 'daily'}
# -- HIGH QUEUE --
base_path.format(task='execute_order'): {'queue': 'high'},
# -- NORMAL QUEUE --
base_path.format(task='calculate_cost'): {'queue': 'normal'},
# -- SERVICE QUEUE --
base_path.format(task='send_pin'): {'queue': 'service'},
# -- LOW QUEUE
base_path.format(task='invite_to_tournament'): {'queue': 'low'},
# -- AWARD QUEUE
base_path.format(task='get_lesson_award'): {'queue': 'award'},
# -- TEST TASK
worker_concurrency = multiprocessing.cpu_count() * 2 + 1
worker_prefetch_multiplier = 1 #
worker_max_tasks_per_child = 1
worker_max_memory_per_child = 90000 # 90MB
beat_max_loop_interval = 60 * 5 # 5 min
I run workers in a docker, part of my stack.yml
version: "3.7"
services:
worker_high:
command: celery worker -l debug -A runcelery.celery -Q high -n worker.high#%h
worker_normal:
command: celery worker -l debug -A runcelery.celery -Q normal,award,service,low -n worker.normal#%h
worker_schedule:
command: celery worker -l debug -A runcelery.celery -Q daily -n worker.schedule#%h
beat:
command: celery beat -l debug -A runcelery.celery
flower:
command: flower -l debug -A runcelery.celery --port=5555
broker:
image: redis:5.0-alpine
I thought that my config is right and run command correct too, but docker logs and flower shown that only worker.normal consume all tasks.
I
Update
Here is part of task.py:
def refresh_rank_in_tournaments():
logger.debug(f'Start task refresh_rank_in_tournaments')
return AnalyticBackgroundManager.refresh_tournaments_rank()
base_path is shortcut for full task path:
base_path = 'events.tasks.{task}'
execute_order task code:
#celery.task(bind=True, default_retry_delay=5)
def execute_order(self, private_id, **kwargs):
try:
return OrderBackgroundManager.execute_order(private_id, **kwargs)
except IEXException as exc:
raise self.retry(exc=exc)
This task will call in a view as tasks.execute_order.delay(id)
Your worker.normal is subscribed to the normal,award,service,low queues. Furthermore, the low queue is the default one, so every task that does not have explicitly set queue will be executed on worker.normal.
I'm running a task every second, and it seems celery doesn't actually perform the task every second.
I guess celery might be a good scheduler for every 1 minute task, but might not be adequte for every second task.
Here's the picture which illustrates what I mean.
I'm using the following options
'schedule': 1.0,
'args': [],
'options': {
'expires': 3
}
And I'm using celery 4.0.0
Yes, Celery actually handles times as low as 1 second, and possibly lower since it takes a float. See this entry of periodic tasks in the docs http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html:
from celery import Celery
from celery.schedules import crontab
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
# Calls test('hello') every 10 seconds.
sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')
# Calls test('world') every 30 seconds
sender.add_periodic_task(30.0, test.s('world'), expires=10)
# Executes every Monday morning at 7:30 a.m.
sender.add_periodic_task(
crontab(hour=7, minute=30, day_of_week=1),
test.s('Happy Mondays!'),
)
#app.task
def test(arg):
print(arg)
A better written example can be found 1/3 the way down https://github.com/celery/celery/issues/3589:
# file: tasks.py
from celery import Celery
celery = Celery('tasks', broker='pyamqp://guest#localhost//')
#celery.task
def add(x, y):
return x + y
#celery.on_after_configure.connect
def add_periodic(**kwargs):
celery.add_periodic_task(10.0, add.s(2,3), name='add every 10')
So sender is the actual Celery broker, i.e. app = Celery()
I want to check whether thread will be preserved during os.fork.
This is what the thread is started before os.fork.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import multiprocessing
from threading import Timer
import time
i = [0]
def work():
while True:
time.sleep(1)
i[0] += 1
print i[0]
print multiprocessing.current_process()
if __name__ == '__main__':
task = Timer(1, work)
task.daemon = True
task.start()
time.sleep(2)
print 'pre fork'
if os.fork():
# parent process
else:
# child process
multiprocessing.current_process().name = "Secondary"
time.sleep(2)
print 'post fork'
time.sleep(1000)
The output of this program:
pre fork
1
<_MainProcess(MainProcess, started)>
2
<_MainProcess(MainProcess, started)>
post fork
post fork
3
<_MainProcess(MainProcess, started)>
4
<_MainProcess(MainProcess, started)>
5
<_MainProcess(MainProcess, started)>
6
<_MainProcess(MainProcess, started)>
7
<_MainProcess(MainProcess, started)>
==========================================================================
This is what the thread is started after os.fork.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import multiprocessing
from threading import Timer
import time
i = [0]
def work():
while True:
time.sleep(1)
i[0] += 1
print i[0]
print multiprocessing.current_process()
if __name__ == '__main__':
task = Timer(1, work)
task.daemon = True
time.sleep(2)
print 'pre fork'
if os.fork():
# parent process
task.start()
else:
# sub process
multiprocessing.current_process().name = "Secondary"
task.start()
time.sleep(2)
print 'post fork'
time.sleep(1000)
The output is like this:
pre fork
post fork
post fork
1
1
<_MainProcess(Secondary, started)>
<_MainProcess(MainProcess, started)>
2
2
<_MainProcess(MainProcess, started)>
<_MainProcess(Secondary, started)>
Summary
It seems that when we start the thread after os.fork then every thing will be ok and both the parent and child process will run the thread.
When we start the thread before os.fork, then according to the output, it seems that the thread is not come up with os.fork, i.e., it is not executed in the child process.
The question is how should this happen and what is the reason for it.
Suddenly, i come up with an explanation and the answer is No, thread will no be preserved during os.fork.
os.fork only do a copy-on-write.
parent and child process both have the same code.
However, if we start a thread before os.fork, then there no start logic to child process.
So, if we want a thread exist in child process then we can only start it after we have os.fork it.
:)
Note: This has some similar with gunicorn --preload argument.
'worker_collect' is the celery package.
All the tasks are routing into the same queue.
I'm executing 2 tasks.
The task 'collect_data' is calling a task 'do_request'.
I put a sleep(5) in order to force the timeout.
When the task 'do_request' timeout of fail I'm trying to call the task 'fetch_result'
The function 'fetch_result' is empty but afterwards it should get the result of the task 'do_request'.
My question :
Where do I have to catch the exception raised 'TimeLimitExceeded' in order to call the function 'fetch_result' ?
Here is the code :
from __future__ import absolute_import
from time import sleep
from worker_collect.celery import app
from celery.utils.log import get_task_logger
from celery.exceptions import TimeLimitExceeded
#app.task(time_limit=3 , max_retries=None)
def fetch_result():
logger = get_task_logger(__name__)
logger.info('-------------------')
logger.info('----- FETCH -------')
logger.info('-------------------')
logger.info('-------------------')
#app.task(time_limit=3 , max_retries=None)
def do_request():
logger = get_task_logger(__name__)
logger.info('---------------------')
logger.info('----- DO REQUEST ----')
logger.info('---------------------')
logger.info('---------------------')
sleep(5)
return 'do_request DONE'
#app.task(time_limit=8 , max_retries=None)
def collect_data():
logger = get_task_logger(__name__)
logger.info('---------------------')
logger.info('-----COLLECT DATA----')
logger.info('---------------------')
do_request.apply_async(link=fetch_result.s(), link_error=fetch_result.s())
logger.info('---------------------')
logger.info('--END COLLECT DATA---')
logger.info('---------------------')
Here are the celery logs
[2015-02-13 15:18:08,275: INFO/Beat] beat: Starting...
[2015-02-13 15:18:08,301: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
[2015-02-13 15:18:08,315: INFO/MainProcess] mingle: searching for neighbors
[2015-02-13 15:18:09,328: INFO/MainProcess] mingle: all alone
[2015-02-13 15:18:09,366: WARNING/MainProcess] celery#collector ready.
[2015-02-13 15:18:09,416: INFO/MainProcess] Received task: worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]
[2015-02-13 15:18:09,419: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: ---------------------
[2015-02-13 15:18:09,420: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: -----COLLECT DATA----
[2015-02-13 15:18:09,421: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: ---------------------
[2015-02-13 15:18:09,469: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: ---------------------
[2015-02-13 15:18:09,470: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: --END COLLECT DATA---
[2015-02-13 15:18:09,470: INFO/Worker-5] worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283]: ---------------------
[2015-02-13 15:18:09,470: INFO/MainProcess] Received task: worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]
[2015-02-13 15:18:09,472: INFO/Worker-2] worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]: ---------------------
[2015-02-13 15:18:09,472: INFO/Worker-2] worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]: ----- DO REQUEST ----
[2015-02-13 15:18:09,472: INFO/Worker-2] worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]: ---------------------
[2015-02-13 15:18:09,473: INFO/Worker-2] worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]: ---------------------
[2015-02-13 15:18:09,475: INFO/MainProcess] Task worker_collect.tasks.collect_data[1df333f3-cc34-4d10-ad1b-181432bf2283] succeeded in 0.0564397389971s: None
[2015-02-13 15:18:12,478: ERROR/MainProcess] Task worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c] raised unexpected: TimeLimitExceeded(3,)
Traceback (most recent call last):
File "/home/bschom/celery_env/local/lib/python2.7/site-packages/billiard/pool.py", line 639, in on_hard_timeout
raise TimeLimitExceeded(job._timeout)
TimeLimitExceeded: TimeLimitExceeded(3,)
[2015-02-13 15:18:12,479: ERROR/MainProcess] Hard time limit (3s) exceeded for worker_collect.tasks.do_request[333e3002-c014-416f-a85a-ca839bd5376c]
[2015-02-13 15:18:12,587: ERROR/MainProcess] Process 'Worker-2' pid:14159 exited with 'signal 9 (SIGKILL)'
Thanks in advance.