I am trying to run 2 test files using xdist with 2 gateways (-n=2). Each test file contains tests which are user permission specific. While running the test with pytest and pytest-xdist, I noticed some of the test fail randomly. It is happening because some of the tests from file1 getting executed by a different gw. So, if [gw0] was running most of the tests from file0, sometimes, [gw0] also executes some tests from file1 which causes the failure.
I am trying to find out if there is a way I can force/ask xdist to execute a specific file or perhaps if there is a way to assign a file to a gw?
pytest test_*.py -n=2 -s -v
also tried:
pytest test_*.py -n=2 -s -v --dist=loadfile
Assuming your file for running parallel tests is correctly distributed (properly receiving PYTEST_XDIST_WORKER and PYTEST_XDIST_WORKER_COUNT environment variables), you only need to run:
pytest test_*.py --tx '2*popen' --dist=loadfile --dist=each
Related
Background
Trigger legacy non-python testcases from pytest. Since these testcases are categorized as testsuites, from pytest perspective we'll be doing an ssh on a remote machine and trigger a testsuite. So from pytest's point of view it is a single testcase, but actually it would be a bunch executing on remote machine.
Requirement
The testsuite will generate a testreport which we'll SCP back to the pytest machine. I wish to parse the testreport and report the PASS/FAIL for each testcase from pytest
I have been looking into example but still can't get my head around on how would I trigger the test case with SSH and parse the testreport(XML/JSON) and generate pytest report
Any suggestions ?
Update:
I have been able to parse the yaml file to generate the terminal report(pytest_terminal_summary) for my testcases. But I would like that pytest also reports the number of testcases failed/passed.
Can you try pytest test.py -v --junitxml="result.xml"
You can also generate html result using pytest file.py -sv --html report.html
I am not understanding how to properly run a simple test(feature file and python file)
with the library pytest-bdd.
From the official documentation, I can't understand what command to issue to run a test.
I tried using pytest command, but I saw the NO test ran.
Do I need to use another library behave to run a feature file?
I figured out trying for 2 days,that ,
for running a pytest-bdd test, there are certain requirements, at least in my view.
put both the feature file and python file in the same directory (maybe this can be changed with configuration files)
the python file name needs to start with test_
the python file needs to contain a method of which name will start with test_
the method starting with test_ , need to be assigned to the #scenario sentence
to run the test, issue pytest command in the same directory(maybe it is also configurable)
After issuing you will only see the method with the name starting with test_ has passed, but all the tests actually ran. To test, you can assert False in any #when or #then annotated method, it will throw errors.
The system contained : pytest-bdd==3.0.2 (copied from pip freeze output)
Features files and python files can be placed in different folders using the bdd_features_base_dir hook provided by pytest-bdd; I think it is better having features files in different folders too.
Here you can see a working example (a simple hello world BDD test):
https://github.com/davidemoro/pytest-play-docker/tree/master/tests
https://github.com/davidemoro/pytest-play-docker/blob/master/tests/pytest.ini (see bdd_features_base_dir in [pytest] section)
https://github.com/davidemoro/pytest-play-docker/tree/master/tests/bdd
If you want to try out pytest-bdd without installation you can use Docker. Create a folder with inside your pytest BDD files and if you want a separate features folder targeted in bdd_features_base_dir and run:
docker run --rm -it -v $(pwd):/src davidemoro/pytest-play:latest
I've found out, that in the python file you don't have to put:
the method starting with test_ , need to be assigned to the #scenario sentence
You can just add: scenarios("") - to allow the tests to be started, which are using steps defined in this specific python file.
Remember to import scenarios!: from pytest_bdd import scenarios
Example:
Code example
Command..
pytest -v path_to_test_file.py
Things to note here..
Check format of feature file as filename.feature
Always __init__ modules, otherwise test-runner will not find test files
Glue right step definitions to test function
Add feature in features module
If you are using python3 execute test with python3
So,
python3 -m pytest -v path_to_test_file.py
Documentation
https://pytest-bdd.readthedocs.io/en/stable/#
I am using Eclipse+Maven based Robot Framework with Java implementation of SeleniumLibrary.
I could execute tests in sauce labs but it executes only on one VM. Has anyone achieved parallel execution of robot tests in Sauce Labs say in multiple VMs? Or can anyone guide to achieve this? Thanks in advance.
This is what I am using to run on multiple concurrent VM's on saucelabs. I have a 1-click batch file that uses start pybot to invoke parallel execution. Example:
ECHO starting parallel run on saucelabs.com
cd c:\base\dir\script
ECHO Win7/Chrome40:
start pybot -v REMOTE_URL:http://user:key#ondemand.saucelabs.com:80/wd/hub -T -d results/Win7Chrome40 -v DESIRED_CAPABILITIES:"name:Win7 + Chrome40, platform:Windows 7, browserName:chrome, version:40" tests/test.robot
ECHO Win8/IE11
start pybot -v REMOTE_URL:http://user:key#ondemand.saucelabs.com:80/wd/hub -T -d results/Win8IE11 -v DESIRED_CAPABILITIES:"name:Win8 + IE11, platform:Windows 8.1, browserName:internet explorer, version:11" tests/test.robot
-T tells pybot to not overwrite result logs but create a timestamped log for each run
-d specifies where the results will go
Works like a charm!
A parallel executor for Robot Framework tests. With Pabot you can split one execution into multiple and save test execution time.
https://github.com/mkorpela/pabot
I am using pytest with tox. I can run some of my tests with a keyword like this:
pytest -k <keyword> path/to/tests
Now it would be really convenient to be able to do this also with tox, as the environments there are clean and different python versions can be tested. However the nearest thing I have found is:
tox -- path/to/tests/test_very_specific_name.py:TestClass.test_func
This is not easy to type, so I rather just run tox without arguments and wait 2 minutes for everything to finish.
Is there a way to run single tests based on keywords with tox? I tried:
tox -- -k <keyword>
This results in a huge list of import errors. It doesn't seem to be able to find any of my local includes. Is this supposed to work?
I figured it out thanks to the comment by phd.
Everything on the command line after -- can be used in tox.ini as {posargs}. I was using that wrong. My tox.ini now has a line like this:
commands = py.test {posargs} <test_folder>
Now it works perfectly with:
tox -- -k <keyword>
I would like to use tox to run py.test on a project which needs additional setup in addition to installing packages into the virtualenv. After creating the virtualenv and installing dependencies, some commands need to be run.
Specifically I'm talking about setting up a node and npm environment using nodeenv:
nodeenv --prebuilt -p
I see that tox allows me to provide a custom command used for installing dependencies by setting install_command in tox.ini. But I don't think this is what I want because that replaces the command (I assume pip) used to install dependencies.
I thought about using a py.test fixture with session scope to handle setting up nodeenv but that seems hacky to me as I don't want this to happen when py.test is run directly, not via tox.
What is the least insane way of achieving this?
You can do all necessary setup after the creation of the virtualenv and the dependency installation in commands. Yes, it says "the commands to be called for testing." but if you need to do extra work to prepare for testing you can just do it right there.
It works through whatever you throw at it in the order it is given - e.g.:
[testenv:someenv]
deps =
nodeenv
pytest
flexmock
commands =
nodeenv --prebuilt -p
; ... and whatever else you might need to do
py.test path/to/my/tests
If you have commands/scripts or whatever else that produces the right result but it returns a non zero exit status you can ignore that by prepending - (as in - naughty-command).
If you need more steps to happen you can wrap them in a little (Python) script and call that script instead as outlined in https://stackoverflow.com/a/47834447/2626627.
There is also an issue to add the ability to use more than one install command: https://github.com/tox-dev/tox/issues/715 is implemented.
I had the same issue, and as it was important for me to be able to create the environment without invoking the tests (via --notest), I wanted the install to happen in the install phase and not the run phase, so I did something slightly differently. First, I created a create-env script:
#!/usr/bin/env sh
set -e
pip install $#
nodeenv --prebuilt --python-virtualenv --node=8.2.1
Made it executable, Then in tox.ini:
[tox]
skipsdist = True
[testenv]
install_command = ./create-env {opts} {packages}
deps = nodeenv
commands = node --version
This complete example runs and outputs the following:
$ tox
python create: .../.tox/python
python installdeps: nodeenv
python installed: nodeenv==1.3.0
python runtests: PYTHONHASHSEED='1150209523'
python runtests: commands[0] | node --version
v8.2.1
_____________________________________________________________________ summary ______________________________________________________________________
python: commands succeeded
congratulations :)
This approach has the downside that it would only work on Unix.
In tox 715, I propose the possibility of native support for multiple install commands.