I am new to pytest. When is #pytest.hookimpl executes? And what is the complete usage of it? I have tried with logs. For (hookwrapper=true), it is printing, 3 sets of after and before yield for a single test.
pytest uses #pytest.hookimpl just to label hook methods. (So #pytest.hookimpl is executed when pytest collects the hook method.)
If you read the source code of pytest, you can find these codes:
def normalize_hookimpl_opts(opts):
opts.setdefault("tryfirst", False)
opts.setdefault("trylast", False)
opts.setdefault("hookwrapper", False)
opts.setdefault("optionalhook", False)
It means pytest will label the hook method with #pytest.hookimpl(tryfirst=False, trylast=False, hookwrapper=False, optionalhook=False) by default. Pytest will treat these hook methods in different ways according to this label(decorator) when executing them.
Take the hookwrapper parameter for example. If the hook method is labeled as hookwrapper=True, pytest will execute the part before yield first and then execute other same type hook methods. After these methods executed, the part after yield will be executed. (This feature is just like pytest fixtures.)
One usage of #pytest.hookimpl(hookwrapper=True) is that you can calculate the total cost time of some hook methods.
(Here, the example code will calculate the tests collect time.)
#pytest.hookimpl(hookwrapper=True)
def pytest_collection(session):
collect_timeout = 5
collect_begin_time = time.time()
yield
collect_end_time = time.time()
c_time = collect_end_time - collect_begin_time
if c_time > collect_timeout:
raise Exception('Collection timeout.')
Related
Is Pytest allows to skip not a whole test (function) but a part of code inside the function?
What I want (usage example):
def test_fill(my_dict: dict):
assert all(v is None for v is my_dict.values()):
my_dict.fill()
# Temporary check for "foo" values
assert all(v is not None for v is my_dict.values()):
# Should skip the code below
pytest.mark.skip(reason='Need values setup')
# The real checks with exact values are here (skipped for now)
assert my_dict['key_1'] = 1 # Part of future test
assert my_dict['key_2'] = 10 # Part of future test
assert my_dict['key_3'] = 100 # Part of future test
How pytest.mark.skip supposed to work:
It may raise exception and quietly catch it.
And I will see it in the final test results output like in case of regular skipping.
Surely I can easily comment it, place in if branch, or skip the whole test with #pytest.mark.skip decorator,
but this will be not reflected in the tests output and it's easily to forgot about this weak test.
Skipping a test while inside a test makes sense, if the information needed to decide if to skip a test is only available inside the test. This can easily be done using pytest.skip:
def test_something():
if not some_condition():
pytest.skip("Condition not fullfilled")
# do the test
This will skip the test the same way a pytest.mark.skipIf decorator will do, e.g. mark the test as skipped in the output and display the given skip reason.
In most cases (e.g. when the skip condition can be defined outside of the test) the decorator version can be used. From the documentation:
It is better to use the pytest.mark.skipif marker when possible to declare a test to be skipped under certain conditions like mismatching platforms or dependencies.
For the sake of completeness: in unittest this is also possible by using TestCase.skipTest.
I am using pytest and would like to invoke a test function for a number of objects returned by a server, and for a number of servers.
The servers are defined in a YAML file and those definitions are provided as parametrization to a fixture "server_connection" that returns a Connection object for a single server. Due to the parametrization, it causes the test function to be invoked once for each server.
I am able to do this with a loop in the test function: There is a second fixture "server_objects" that takes a "server_connection" fixture as input and returns a list of server objects. The pytest test function then takes that second fixture and executes the actual test in a loop through the server objects.
Here is that code:
import pytest
SD_LIST = ... # read list of server definitions from YAML file
#pytest.fixture(
params=SD_LIST,
scope='module'
)
def server_connection(request):
server_definition = request.param
return Connection(server_definition.url, ...)
#pytest.fixture(
scope='module'
)
def server_objects(request, server_connection):
return server_connection.get_objects()
def test_object_foo(server_objects):
for server_object in server_objects:
# Perform test for a single server object:
assert server_object == 'foo'
However, the disadvantage is of course that a test failure causes the entire test function to end.
What I want to happen instead is that the test function is invoked for each single server object, so that a test failure for one object does not prevent the tests on the other objects. Ideally, I'd like to have a fixture that provides a single server object, that I can pass to the test function:
...
#pytest.fixture(
scope='module'
)
def server_object(request, server_connection):
server_objects = server_connection.get_objects()
# TBD: Some magic to parametrize this fixture with server_objects
def test_object_foo(server_object):
# Perform test for a single server object:
assert server_object == 'foo'
I have read through all pytest docs regarding fixtures but did not find a way to do this.
I know about pytest hooks and have used e.g. pytest_generate_tests() before, but I did not find a way how pytest_generate_tests() can access the values of other fixtures.
Any ideas?
Update: Let me add that I also did search SO for this, but did not find an answer. I specifically looked at:
pytest fixture of fixtures
How to parametrize a Pytest fixture
py.test: Pass a parameter to a fixture function
initing a pytest fixture with a parameter
I have some tests which I would like to repeat a number of times. I tried the pytest-repeat plugin
pip3 install pytest-repeat
import pytest
#pytest.mark.repeat(2)
class TestDemo():
def test_demo1(self):
pass
def test_demo2(self):
pass
This works
test_class_repeat.py::TestDemo::test_demo1[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo1[2/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[2/2] PASSED
Except that I want an interleaved order running all tests, and the run all tests again
test_class_repeat.py::TestDemo::test_demo1[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[1/2] PASSED
test_class_repeat.py::TestDemo::test_demo1[2/2] PASSED
test_class_repeat.py::TestDemo::test_demo2[2/2] PASSED
Is there a simple way to do this?
Well, not a very clean solution, naively just define a test function that executes the interleaved tests while skipping the definition itself, and apply repeat on that:
import pytest
#pytest.mark.skip(reason='Definition only')
class TestDemo():
def test_demo1(self):
print('In Test 1')
assert 1 == 1
def test_demo2(self):
print('In Test 2')
assert 2 == 2
#pytest.mark.repeat(2)
def test_all():
demo = TestDemo()
demo.test_demo1()
demo.test_demo2()
Execution (in jupyter notebook) gives:
Test.py::TestDemo::test_demo1 SKIPPED
Test.py::TestDemo::test_demo2 SKIPPED
Test.py::test_all[1/2]
In Test 1
In Test 2
PASSED
TestProject.py::test_all[2/2]
In Test 1
In Test 2
PASSED
Side note: if one of the two nested tests does not pass test_all does not pass, something desired from interleaved tests?
You can use pytest-flakefinder package by dropbox. It repeats tests after the run is complete.
Usage: py.test --flake-finder --flake-runs=runs.
This can be done using the pytest.mark.parametrize if the functions have a parameter. Below is an example.
import pytest
iter_list = [1,2,3]
#pytest.mark.parametrize('param1', iter_list, scope = 'class')
class TestDemo():
def test_demo1(self, param1):
pass
def test_demo2(self, param1):
pass
I have some py.test tests that have multiple dependent and parameterized fixtures and I want to measure the time taken by each fixture. However, in the logs with --durations it only shows time for setup for actual tests, but doesn't give me a breakdown of how long each individual fixture took.
Here is a concrete example of how to do this:
import logging
import time
import pytest
logger = logging.getLogger(__name__)
#pytest.hookimpl(hookwrapper=True)
def pytest_fixture_setup(fixturedef, request):
start = time.time()
yield
end = time.time()
logger.info(
'pytest_fixture_setup'
f', request={request}'
f', time={end - start}'
)
With output similar to:
2018-10-29 20:43:18,783 - INFO pytest_fixture_setup, request=<SubRequest 'some_data_source' for <Function 'test_ruleset_customer_to_campaign'>>, time=3.4723987579345703
The magic is hookwrapper:
pytest plugins can implement hook wrappers which wrap the execution of other hook implementations. A hook wrapper is a generator function which yields exactly once. When pytest invokes hooks it first executes hook wrappers and passes the same arguments as to the regular hooks.
One fairly important gotcha that I ran into is that the conftest.py has to be in your project's root folder in order to pick up the pytest_fixture_setup hook.
There isn't anything builtin for that, but you can easily implement yourself by using the new pytest_fixture_setup hook in a conftest.py file.
I am building some tests for python3 code using py.test. The code accesses a Postgresql Database using aiopg (Asyncio based interface to postgres).
My main expectations:
Every test case should have access to a new asyncio event loop.
A test that runs too long will stop with a timeout exception.
Every test case should have access to a database connection.
I don't want to repeat myself when writing the test cases.
Using py.test fixtures I can get pretty close to what I want, but I still have to repeat myself a bit in every asynchronous test case.
This is how my code looks like:
#pytest.fixture(scope='function')
def tloop(request):
# This fixture is responsible for getting a new event loop
# for every test, and close it when the test ends.
...
def run_timeout(cor,loop,timeout=ASYNC_TEST_TIMEOUT):
"""
Run a given coroutine with timeout.
"""
task_with_timeout = asyncio.wait_for(cor,timeout)
try:
loop.run_until_complete(task_with_timeout)
except futures.TimeoutError:
# Timeout:
raise ExceptAsyncTestTimeout()
#pytest.fixture(scope='module')
def clean_test_db(request):
# Empty the test database.
...
#pytest.fixture(scope='function')
def udb(request,clean_test_db,tloop):
# Obtain a connection to the database using aiopg
# (That's why we need tloop here).
...
# An example for a test:
def test_insert_user(tloop,udb):
#asyncio.coroutine
def insert_user():
# Do user insertion here ...
yield from udb.insert_new_user(...
...
run_timeout(insert_user(),tloop)
I can live with the solution that I have so far, but it can get cumbersome to define an inner coroutine and add the run_timeout line for every asynchronous test that I write.
I want my tests to look somewhat like this:
#some_magic_decorator
def test_insert_user(udb):
# Do user insertion here ...
yield from udb.insert_new_user(...
...
I attempted to create such a decorator in some elegant way, but failed. More generally, if my test looks like:
#some_magic_decorator
def my_test(arg1,arg2,...,arg_n):
...
Then the produced function (After the decorator is applied) should be:
def my_test_wrapper(tloop,arg1,arg2,...,arg_n):
run_timeout(my_test(),tloop)
Note that some of my tests use other fixtures (besides udb for example), and those fixtures must show up as arguments to the produced function, or else py.test will not invoke them.
I tried using both wrapt and decorator python modules to create such a magic decorator, however it seems like both of those modules help me create a function with a signature identical to my_test, which is not a good solution in this case.
This can probably solved using eval or a similar hack, but I was wondering if there is something elegant that I'm missing here.
I’m currently trying to solve a similar problem. Here’s what I’ve come up with so far. It seems to work but needs some clean-up:
# tests/test_foo.py
import asyncio
#asyncio.coroutine
def test_coro(loop):
yield from asyncio.sleep(0.1)
assert 0
# tests/conftest.py
import asyncio
#pytest.yield_fixture
def loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
yield loop
loop.close()
def pytest_pycollect_makeitem(collector, name, obj):
"""Collect asyncio coroutines as normal functions, not as generators."""
if asyncio.iscoroutinefunction(obj):
return list(collector._genfunctions(name, obj))
def pytest_pyfunc_call(pyfuncitem):
"""If ``pyfuncitem.obj`` is an asyncio coroutinefunction, execute it via
the event loop instead of calling it directly."""
testfunction = pyfuncitem.obj
if not asyncio.iscoroutinefunction(testfunction):
return
# Copied from _pytest/python.py:pytest_pyfunc_call()
funcargs = pyfuncitem.funcargs
testargs = {}
for arg in pyfuncitem._fixtureinfo.argnames:
testargs[arg] = funcargs[arg]
coro = testfunction(**testargs) # Will no execute the test yet!
# Run the coro in the event loop
loop = testargs.get('loop', asyncio.get_event_loop())
loop.run_until_complete(coro)
return True # TODO: What to return here?
So I basically let pytest collect asyncio coroutines like normal functions. I also intercept text exectuion for functions. If the to-be-tested function is a coroutine, I execute it in the event loop. It works with or without a fixture creating a new event loop instance per test.
Edit: According to Ronny Pfannschmidt, something like this will be added to pytest after the 2.7 release. :-)
Every test case should have access to a new asyncio event loop.
The test suite of asyncio uses unittest.TestCase. It uses setUp() method to create a new event loop. addCleanup(loop.close) is close automatically the event loop, even on error.
Sorry, I don't know how to write this with py.test if you don't want to use TestCase. But if I remember correctly, py.test supports unittest.TestCase.
A test that runs too long will stop with a timeout exception.
You can use loop.call_later() with a function which raises a BaseException as a watch dog.