Can py.test supports multiple -k options?
Each testcase belongs to a particular group such as _eventnotification or _interface, etc.
Is it possible to run test cases that belong to either one or both at the same time?
ie, run testcases that has _eventnotification or _interface in the name at the same time.
I tried the following and only the testcases with _interface were executed.
If that is not supported, is there another way to do this?
py.test -k "_eventnotification" -k "_interface"
The bad news: pytest-2.3.3 does not support it.
The good news: i took your question as an opportunity to finally enhance "-k" behaviour, so that you can use "not", "or", "end" etc, see the [extended -k example][1]. It works now like "-m" except that it matches on (substrings of) test names, not markers. You can use this in-development pytest version with "pip install -i http://pypi.testrun.org -U pytest".
Related
I already know that we can use marks and then call pytest with -m to execute only certain tests
my question is: is there a way to mark a test so that that test is not executed without adding any -m when calling pytest?
EDIT:
I am thinking something like:
mark the test with a special (I don't know if that exists, that is why this question) mark the test thespecialtest.py as
#pytest.mark.notselect
then running the tests like always: pytest will exclude that test.
If I want to run that test specifically I can do explicitly pytest thescpecialtest.py
I know that the best and easiest way would be just to use -m in calling pytest but I want to ask if there is an option where this would not be necessary
-m option is probably the most comfortable to use for this use case.
However, you can choose what tests to run based on names with -k option, which is basically the same as -m option, but you select test cases based on their names, rather than marks.
Another option is you can change test discovery process, so you can for example tell pytest to collect and execute only functions that comply to a certain name patters, e.g. you add to your pytest.ini:
[pytest]
python_functions = *_check
which tells pytest to collect and execute only functions that comply to this glob pattern. You can do this with classes and files as well.
I am using pytest with tox. I can run some of my tests with a keyword like this:
pytest -k <keyword> path/to/tests
Now it would be really convenient to be able to do this also with tox, as the environments there are clean and different python versions can be tested. However the nearest thing I have found is:
tox -- path/to/tests/test_very_specific_name.py:TestClass.test_func
This is not easy to type, so I rather just run tox without arguments and wait 2 minutes for everything to finish.
Is there a way to run single tests based on keywords with tox? I tried:
tox -- -k <keyword>
This results in a huge list of import errors. It doesn't seem to be able to find any of my local includes. Is this supposed to work?
I figured it out thanks to the comment by phd.
Everything on the command line after -- can be used in tox.ini as {posargs}. I was using that wrong. My tox.ini now has a line like this:
commands = py.test {posargs} <test_folder>
Now it works perfectly with:
tox -- -k <keyword>
I have a test-suit as usual for Perl projects, containing a lib and a t directory. The tests in t are structured through subdirectories. So I run them using:
prove -Ilib -r t/
So far nothing special, and afaik quite a standard way of testing in Perl.
Since it is the assumption, that this is the standard way of testing, I'd like to make sure that the following applies:
"If you run prove -r on t, you have tested everything that is there to test".
This is very important, since otherwise you can never be sure that you really called all the tests and the stuff is fine. Somebody calling the above would then maybe - not knowing so - just call a part of the available tests, leaving some behind. Quite annoying... tests that are not run, are of no help. It should be as easy and predictive as possible for developers to call all the tests! It is a bad thing when you have to look up how to run the rest of the test-suit. You might not know about it, or might not do it anyway.
So here comes my problem: I have to integrate some Tests using pgTAP which kindly provides the tool pg_prove. Now I have to make two commandos to do the testing. Additionally to running prove -Ilib -r I also have to run something like e.g. pg_prove -S schema=customerX -U dbuser -d dbname t/pgTAP/*.sql. The problem is not that big if you call the tests automatically from cron or what ever. But it really decreases the chance that we lazy developers run all test tests during our busy days.
So I wonder what would be the best approach to implement the tests in such a way that prove will also include those tests. Is it, that I have to create some .t-files which wrap the whole thing (and how?)? Are there any tricks I can do with the whole Harness stuff on CPAN? Would a simple test_all.sh in the root-dir, including both commandos, do the best job, even if it breaks the assumptions I made above?
So my question in short is: Can I run all tests, including pgTAP with prove? If not, is there a best practice for solving my problem?
Thanks a lot.
Yes. In fact, pg_prove just passes everything off to prove. Assuming your pgTAP tests end in .sql, you can run all your tests like this:
prove -lr --ext .sql --ext .t \
--source pgTAP \
--pgtap-option dbname=dbname \
--pgtap-option username=dbuser \
--pgtap-option suffix=.pg \
--pgtap-option set=schema=customerX
If you use Module::Build, you can also have ./Build test run all the tests, too, as I've done for circle.
See the TAP::Parser::SourceHandler::pgTAP documentation for details.
I'm familiar with the command py.test -k string for select all tests that contains the string in their name and run it.
What I want is to select tests with more than one string parameter like an OR logical selection. For example, let's say that I have 3 tests:
test_should_connect
test_should_execute
test_should_return
And let's say I just want to run test_should_connect and test_should_return only.
I've looked for an answer in py.test documentation, and to do that I should use the following command:
py.test -k "connect or return"
But, this doesn't work =/
py.test -k "connect or return" should work. Are you using pytest==2.3.4? Could you paste the output from "py.test" otherwise along with the tests?
I'm trying to pass some parameters to a test script through the prove command. Based on some old threads where is option was hashed out, and the content's of my prove's man page (quoted below), using :: before the options should work, but prove is still parsing the test script's option as it's own:
$ prove -v t/040-unit-object-test.t :: --unit 270149
Unknown option: unit
Here the relevant part of the man page:
Arguments to Tests
It is possible to supply arguments to tests. To do so separate them
from prove's own arguments with the arisdottle, '::'. For example
prove -v t/mytest.t :: --url http://example.com
would run t/mytest.t with the options '--url http://example.com'. When
running multiple tests they will each receive the same arguments.
I tried using sudo to run the option as root, but got the same error.
I am using the Getopt::Long module to get the options, and the options work when I run the test script using plain old perl, but then my SKIP and TODO blocks don't work.
I would rather not have to use the --exec workaround.
I'm running Perl 5.10 on Mac OS 10.6.
The output of prove -V is:
prove v2.64, using Test::Harness v3.25 and Perl v5.10.0
Well, you must have two versions of prove in your $PATH, an old one from 2006 from before App::Prove existed I suggest you delete that one, and re-install App::Prove