Within a class I have a have a asyncio loop with is created with run_until_complete and the argument return_when. The functions within tasks are running as long as the application is running
I would like to to create a test with Pytest that validate the following situation:
task asyncfoobar or asynctesting is finished for some reason
the loop should stop running and the finally statement will be called.
When testing the application for this situation, it works as expected.
A test is preferred to easily validate it keeps working correct.
Pytest is used for this. How can this be done?
Catching the log line at the finally statement?
Snippets of the code that need to be tested:
def asyncfoobar:
try:
....
finally:
return
def asynctesting(a,b):
while True:
....
await asyncio.sleep(10)
class Do_something:
def start
try:
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.loop)
self.tasks=[
self.loop.create_task(asyncfoobar(config)),
asyncio.get_event_loop().create_task(asynctesting(a, b), )
]
self.loop.run_until_complete(
asyncio.wait(self.tasks,
return_when=asyncio.FIRST_COMPLETED)
)
finally:
logging.info("We are stopping")
return
Related
I receive an error RuntimeError: Event loop is closed each time when i try to make more than one async call function inside my test. I already tried to use all suggestions on stackoverflow to rewrite event_loop fixture but nothing works. I wonder what i'm missing
Run test command: python -m pytest tests/ --asyncio-mode=auto
requirements.txt
pytest==7.1.2
pytest-asyncio==0.18.3
pytest-html==3.1.1
pytest-metadata==2.0.1
test.py
async def test_user(test_client_fast_api):
assert 200 == 200
request_first = test_client_fast_api.post( # works fine
"/first_route",
)
request_second = test_client_fast_api.post( # recieve RuntimeError: Event loop is closed
"/second_route",
)
conftest.py
#pytest.fixture()
def event_loop():
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
yield loop
loop.close()
It took me all afternoon to solve this problem.
I also try to succeed from other people's code, here is my code.
Add a file conftest.py to the directory where the test script is placed.
And write the following code.
import pytest
from main import app
from httpx import AsyncClient
#pytest.fixture(scope="session")
def anyio_backend():
return "asyncio"
#pytest.fixture(scope="session")
async def client():
async with AsyncClient(app=app, base_url="http://test") as client:
print("Client is ready")
yield client
And then write a test script test_xxx.py.
import pytest
from httpx import AsyncClient
#pytest.mark.anyio
async def test_run_not_exists_schedule(client: AsyncClient):
response = await client.get("/schedule/list")
assert response.status_code == 200
schedules = response.json()["data"]["schedules"]
schedules_exists = [i["id"] for i in schedules]
not_exists_id = max(schedules_exists) + 1
request_body = {"id": not_exists_id}
response = await client.put("/schedule/run_cycle", data=request_body)
assert response.status_code != 200
#pytest.mark.anyio
async def test_run_adfasdfw(client: AsyncClient):
response = await client.get("/schedule/list")
assert response.status_code == 200
schedules = response.json()["data"]["schedules"]
schedules_exists = [i["id"] for i in schedules]
not_exists_id = max(schedules_exists) + 1
request_body = {"id": not_exists_id}
response = await client.put("/schedule/run_cycle", data=request_body)
assert response.status_code != 200
This is the real test code for my own project. You can change it to your own.Finally, run in the project's terminal python -m pytest.If all goes well, it should be ok's.This may involve libraries that need to be installed.
pytest
httpx
Yeah wow I had a similar afternoon to your experience #Bai Jinge
This is the event loop fixture and TestClient pattern that worked for me:
from asyncio import get_event_loop
from unittest import TestCase
from async_asgi_testclient import TestClient
#pytest.fixture(scope="module")
def event_loop():
loop = get_event_loop()
yield loop
#pytest.mark.asyncio
async def test_example_test_case(self):
async with TestClient(app) as async_client:
response = await async_client.get(
"/api/v1/example",
query_string={"example": "param"},
)
assert response.status_code == HTTP_200_OK
Ref to relevant GitHub issue: https://github.com/tiangolo/fastapi/issues/2006#issuecomment-689611040
Please note - I could NOT figure our how to use Class based tests. Neither unittest.TestCase or asynctest.case.TestCase would work for me. pytest-asyncio docs (here) state that:
Test classes subclassing the standard unittest library are not supported, users are recommended to use unittest.IsolatedAsyncioTestCase or an async framework such as asynctest.
I have a block of try-exception code and I want to test if the block works properly? I wonder if there is a way to test this block with a unit test using pylint? This is the snipent I have:
class data_type(Enum):
SESSIONS = 0
RUNS = 1
def find_filter(data_type)
try:
if data_type.name == 'SESSIONS':
return {}
elif data_type.name == 'RUNS':
#Filter out rerun jobs
filter = {
"filter":{
"attName":"is_rerun",
"operand":"IN",
"#c":".InFilter",
"values":["FALSE"]
}
}
return filter
except AttributeError as erro:
loghandler.critical('Error in creating count query: {}'.format(erro))
I want to test if the data_type is not a valid enum member, the exception catches it before the execution goes to if statement. For example if:
find_filter('')
is called I like to see that the except AttributeError as erro captures the error and the error is logged.
Is there a way to do that?
PS: I know with pytest.raises() I can test if a raised error is really raised? But as far as I understand pytest.raises() does not work for testing the exception block of a try-exception case.
Any suggestion please?
Thanks.
Is it possible to schedule a task in celery from another task?
I've got this Python script:
import logging
from celery import Celery
from datetime import datetime
logger = logging.getLogger(__name__)
app = Celery('app', backend='amqp://', broker='pyamqp://guest#localhost:5672//')
#app.task()
def add(x, y):
result = x + y
logger.info(f'Add: {x} + {y} = {result}')
return result
#app.task()
def setPeriodicTask():
#option 1
app.add_periodic_task(10, add.s(30, 1))
#option 2
app.conf.beat_schedule = {
'add-every-5-seconds': {
'task': 'app.add',
'schedule': 5.0,
'args': (now.hour, now.second)
}
}
logger.info(f'setPeriodicTask succeeded')
return 1
When I call the add task, it works OK.
If I call the setPeriodicTask task it does not throw any error but the add task is not scheduled.
I've tried both options, none is working:
add_periodic_task
modify the beat_schedule
If I add this code to my Python script (as I've seen in the celery docs):
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(5.0, add.s(10, 1))
I can see the add task running scheduled as expected. So celery and celery beat seem to be working fine.
But I want to enable/disable the task on demand.
Is it possible? And if so, what am I doing wrong?
In case someone else faces this issue.
I ended up using a database with a similar approach as the one mention in the django -celery-beat doc: django-celery-beat - Database-backed Periodic Tasks
Slightly off topic question here but I'm looking to mock the calls to the sender.add_periodic_task(5.0, add.s(10, 1)) and I can not find anything that would tell me where it is
I'm using #mock.patch("celery.add_periodic_task")
any ideas would be welcome
import pytest
from ddt import ddt, data, unpack
from unittest import mock
from django.test import TestCase
from myapp.celery import setup_periodic_tasks
class CeleryTasksTest(TestCase):
#mock.patch("celery.add_periodic_task")
#mock.patch("os.environ.get")
#data(
{"env_to_test":"local", "number_of_tasks":0},
{"env_to_test":"TEST", "number_of_tasks":0},
{"env_to_test":"STAGING", "number_of_tasks":5},
{"env_to_test":"LIVE", "number_of_tasks":1},
)
#unpack
def test_setup_periodic_tasks_per_environment(self, mock_os_environ_get, mock_add_periodic_task, env_to_test, number_of_tasks):
setup_periodic_tasks([])
mock_os_environ_get.return_value=env_to_test
self.assertEqual(mock_add_periodic_task.call_count, number_of_tasks)
For example i have the following class. How i can prevent execution of get_entity task if create_entity task was not executed?
class MyTaskSequence(TaskSequence):
#seq_task(1)
def create_entity(self):
self.round += 1
with self.client.post('/entities', json={}, catch_response=True) as resp:
if resp.status_code != HTTPStatus.CREATED:
resp.failure()
# how to stop other tasks for that run?
self.entity_id = resp.json()['data']['entity_id']
#seq_task(2)
def get_entity(self):
# It is being always executed,
# but it should not be run if create_entity task failed
resp = self.client.get(f'/entities/{self.entity_id}')
...
I found TaskSet.interrupt method in documentation, but does not allow to cancel root TaskSet. I tried to make parent TaskSet for my task sequence, so TaskSet.interrupt works.
class MyTaskSet(TaskSet):
tasks = {MyTaskSequence: 10}
But now i see that all results in ui are cleared after i call interrupt!
I just need to skip dependent tasks in this sequence. I need the results.
The easiest way to solve this is just to use a single #task with multiple requests inside it. Then, if a request fails just do a return after resp.failure()
Might self.interrupt() be what you are looking for?
See https://docs.locust.io/en/latest/writing-a-locustfile.html#interrupting-a-taskset for reference.
Why not using on_start(self): which runs once whenever a locust created, it can set a global which can be checked whether the locust executes the tasks
class MyTaskSequence(TaskSequence):
entity_created = false
def on_start(self):
self.round += 1
with self.client.post('/entities', json={}, catch_response=True) as resp:
if resp.status_code != HTTPStatus.CREATED:
self.entity_created = true
resp.failure()
self.entity_id = resp.json()['data']['entity_id']
#seq_task(2)
def get_entity(self):
if self.entity_created:
resp = self.client.get(f'/entities/{self.entity_id}')
...
i'm trying to get a clean and gracefull shutdown, and for some reason, it wont execute. iv'e tried:
sys addShutdownHook{
logger.warn("SHUTTING DOWN...")
// irrelevant logic here...
}
and also:
Runtime.getRuntime.addShutdownHook(ThreadOperations.delayOnThread{
logger.warn("SHUTTING DOWN...")
// irrelevant logic here...
}
)
where ThreadOperations.delayOnThread definition is:
object ThreadOperations {
def startOnThread(body: =>Unit) : Thread = {
onThread(true, body)
}
def delayOnThread(body: =>Unit) : Thread = {
onThread(false, body)
}
private def onThread(runNow : Boolean, body: =>Unit) : Thread = {
val t=new Thread {
override def run=body
}
if(runNow){t.start}
t
}
// more irrelevant operations...
}
but when i run my program (executable jar, double activation), the hook does not start. so what am i doing wrong? what is the right way to add a shutdown hook in scala? is it in any way related to the fact i'm using double activation?
double activation is done like that:
object Gate extends App {
val givenArgs = if(args.isEmpty){
Array("run")
}else{
args
}
val jar = Main.getClass.getProtectionDomain().getCodeSource().getLocation().getFile;
val dir = jar.dropRight(jar.split(System.getProperty("file.separator")).last.length + 1)
val arguments = Seq("java", "-cp", jar, "boot.Main") ++ givenArgs.toSeq
Process(arguments, new java.io.File(dir)).run();
}
(scala version: 2.9.2 )
thanks.
In your second attempt, your shutdown hook you seems to just create a thread and never start it (so it just gets garbage collected and does nothing). Did I miss something? (EDIT: yes I did, see comment. My bad).
In the first attempt, the problem might just be that the underlying log has some caching, and the application exits before the log is flushed.
Solved it.
For some reason, I thought that run as opposed to ! would detach the process. It actually hangs on because there are open streams left to the Process, which is returned from run (or maybe it just hangs for another reason, 'cause exec doesn't hang, but returns a Process with open streams to and from the child process, much like run). For this reason, the original process was still alive, and I accidentally sent the signals to it. Of course, it did not contain a handler, or a shutdown hook, so nothing happened.
The solution was to use Runtime.getRuntime.exec(arguments.toArray) instead of Process(arguments, new java.io.File(dir)).run();, close the streams in the Gate object, and send the ^C signal to the right process.