We have this inside celery beat scheduler:
"task_name": {
"task": "utils.tasks.task_name",
"schedule": crontab(minute="*/05"),
}
so this will run every five minutes but I want task_name to run only once. I can not say:
"schedule": crontab(minute="1", hour="1")
because the task_name run time can be any time.
I have added a condition to the task so it will not continue after 10 minutes. Do you think it is possible for it to run only once instead of checking the task constantly?
I would say you need to add some kind of locking to your code, here's an example:
LOCKS = {}
#task(bind=True)
def task_name(self):
lock_id = self.name
if LOCKS.get(lock_id) is None:
LOCKS[lock_id] = True
return your_function()
logger.debug("You can run this only once")
Related
I'm trying to write integration test for the celery tasks and have a test class like below
#pytest.mark.usefixtures('celery_session_app')
#pytest.mark.usefixtures('celery_session_worker')
#pytest.mark.usefixtures('mongodb')
class TestIntegration:
def test_delete_collection_from_mongodb(self, x, y):
results = delete_collection_from_mongodb(x, y).delay()
assert results.get(timeout=20) == 20
And in my conftest I have the following fixtures
#pytest.fixture(scope='session')
def celery_config():
return {
'broker_url': RABBITMQ_BROKER_URL,
'shutdown_timeout': 30,
}
#pytest.fixture(scope='session')
def celery_worker_parameters():
return {
'queues': (....),
}
#pytest.fixture(scope='session')
def celery_enable_logging():
return True
However, when I run the test it times out. Stacktrace:
task_id = '6009db28-637b-4447-a2c5-c0bdb3c03981', timeout = 10.0, interval = 0.5
no_ack = True, on_interval = <promise#0x7fcfaac01d30>
def wait_for(self, task_id,
timeout=None, interval=0.5, no_ack=True, on_interval=None):
"""Wait for task and return its result.
If the task raises an exception, this exception
will be re-raised by :func:`wait_for`.
Raises:
celery.exceptions.TimeoutError:
If `timeout` is not :const:`None`, and the operation
takes longer than `timeout` seconds.
"""
self._ensure_not_eager()
time_elapsed = 0.0
while 1:
meta = self.get_task_meta(task_id)
if meta['status'] in states.READY_STATES:
return meta
if on_interval:
on_interval()
# avoid hammering the CPU checking status.
time.sleep(interval)
time_elapsed += interval
if timeout and time_elapsed >= timeout:
> raise TimeoutError('The operation timed out.')
E celery.exceptions.TimeoutError: The operation timed out.
/venv/lib/python3.9/site-packages/celery/backends/base.py:792: TimeoutError
I've also tried to set the result backend to RPC, Redis, or cache+memory, and it still times out. Any idea what I'm missing?
Depending on the celery version you are using it could be this issue which has been fixed since 5.0.4. It was root-caused to the cache backend, though I see you've tried other backends with the same result.
Is it possible your backend and/or broker arent up when you run the tests? I also saw this timeout error when testing in a docker setup with the broker, backend, and test code all running in separate containers. Blowing away all the containers & rebuilding fixed it for me, so it was probably some weird state.
The github issue offers this minimum fixture & test case that might be worth trying out to see if you can get that to work.
Ivé been playing with two tasks only to learn the basics of FreeRTOS but i have some confusing things happening I can not understand.
I have two Tasks with the same priority:
xTaskCreate( vTaskLED1, "LED1", configMINIMAL_STACK_SIZE, NULL, tskIDLE_PRIORITY + 1, NULL );
xTaskCreate( vTaskLED2, "LED2", configMINIMAL_STACK_SIZE, NULL, tskIDLE_PRIORITY + 1, NULL );
Then, in each task i just turn on the corresponding led in order to know thich task is running.
// Task 1 to be created.
void vTaskLED1(void *pvParameters)
{
for( ;; )
{
GPIO_PortToggle(BOARD_INITPINS_LED1_GPIO, BOARD_INITPINS_LED1_GPIO_PIN_MASK);
// Task code goes here.
}
}
// Task 2 to be created.
void vTaskLED2(void *pvParameters)
{
for( ;; )
{
GPIO_PortToggle(BOARD_INITPINS_LED2_GPIO, BOARD_INITPINS_LED2_GPIO_PIN_MASK);
// Task code goes here.
}
}
And as I understand, as I have configUSE_PREEMPTION On and configUSE_TIME_SLICING Off, at this point the only running task would be LED1 since I never blocked or suspended this task.
Then I tested adding a TaskYELD() at the end of task 1 (LED1), this resoulted in just task 2 running all time (because this is never blocked or suspended) after one run of task 1.
Ath this moment everything is ok, but when I started testing with vTaskDelay( 1000 / portTICK_PERIOD_MS ); in order to block task 1, I don't get why this task is running even if task 2 is never blocked or suspended.
I can't understand why at the first two cases Task 1 was always READY but never ran and why in the third case ir runs even if there is a second Task running that never gets blocked or suspended.
For example i have the following class. How i can prevent execution of get_entity task if create_entity task was not executed?
class MyTaskSequence(TaskSequence):
#seq_task(1)
def create_entity(self):
self.round += 1
with self.client.post('/entities', json={}, catch_response=True) as resp:
if resp.status_code != HTTPStatus.CREATED:
resp.failure()
# how to stop other tasks for that run?
self.entity_id = resp.json()['data']['entity_id']
#seq_task(2)
def get_entity(self):
# It is being always executed,
# but it should not be run if create_entity task failed
resp = self.client.get(f'/entities/{self.entity_id}')
...
I found TaskSet.interrupt method in documentation, but does not allow to cancel root TaskSet. I tried to make parent TaskSet for my task sequence, so TaskSet.interrupt works.
class MyTaskSet(TaskSet):
tasks = {MyTaskSequence: 10}
But now i see that all results in ui are cleared after i call interrupt!
I just need to skip dependent tasks in this sequence. I need the results.
The easiest way to solve this is just to use a single #task with multiple requests inside it. Then, if a request fails just do a return after resp.failure()
Might self.interrupt() be what you are looking for?
See https://docs.locust.io/en/latest/writing-a-locustfile.html#interrupting-a-taskset for reference.
Why not using on_start(self): which runs once whenever a locust created, it can set a global which can be checked whether the locust executes the tasks
class MyTaskSequence(TaskSequence):
entity_created = false
def on_start(self):
self.round += 1
with self.client.post('/entities', json={}, catch_response=True) as resp:
if resp.status_code != HTTPStatus.CREATED:
self.entity_created = true
resp.failure()
self.entity_id = resp.json()['data']['entity_id']
#seq_task(2)
def get_entity(self):
if self.entity_created:
resp = self.client.get(f'/entities/{self.entity_id}')
...
I need following workflow for my celery tasks.
when taskA finishes with success I want to execute taskB.
I know there is signal #task_success but this returns only task's result, and I need access to parameters of previous task's arguments. So I decided for code like these:
#app.task
def taskA(arg):
# not cool, but... https://github.com/celery/celery/issues/3797
from shopify.tasks import taskA
taskA(arg)
#task_postrun.connect
def fetch_taskA_success_handler(sender=None, **kwargs):
from gcp.tasks import taskB
if kwargs.get('state') == 'SUCCESS':
taskB.apply_async((kwargs.get('args')[0], ))
The problem is the taskB seems to be executed in some endless loop many, many times instead only once.
This way it works correctly:
#app.task
def taskA(arg):
# not cool, but... https://github.com/celery/celery/issues/3797
# otherwise it won't added in periodic tasks
from shopify.tasks import taskA
return taskA(arg)
#task_postrun.connect
def taskA_success_handler(sender=None, state=None, **kwargs):
resource_name = kwargs.get('kwargs', {}).get('resource_name')
if resource_name and state == 'SUCCESS':
if sender.name == 'shopify.tasks.taskA':
from gcp.tasks import taskB
taskB.apply_async(kwargs={
'resource_name': resource_name
})
just for reference:
celery==4.1.0
Django==2.0
django-celery-beat==1.1.0
django-celery-results==1.0.1
flower==0.9.2
amqp==2.2.2
Python 3.6
I'm using Jenkins Rest API to build and schedule job.
The problem that i schedule one job for the Week-end but it execute it several times (Same job executed every minute).
For the rest of the week the job is executed only once, so if there any GUI options to empty the week-end job list ?
you can use the following groovy script to clean all ( or part of your queue ....)
this example delete all jobs that start with a specific branch name
import jenkins.model.*
def branchName = build.environment.get("GIT_BRANCH_NAME")
println "=========before clean the queue ... =="
def q = Jenkins.instance.queue
q.items.each {
println("${it.task.name}:")
}
q.items.findAll { it.task.name.startsWith(branchName) }.each { q.cancel(it.task) }
println "=========after clean the queue ... =="
q = Jenkins.instance.queue
q.items.each {
println("${it.task.name}:")
}