I have the following tox.ini file:
[tox]
envlist = py310, flake8
isolated_build = True
[testenv]
skip_install = True
deps = -rtest_requirements.txt
passenv = *
commands =
pytest {posargs} --teamcity
[testenv:flake8]
deps = flake8
skip_install = True
commands = flake8 tests/
On teamcity, I run my python test through tox from within a script build step where I call the following shell script
#! /bin/sh
python -m tox .
Now, there is one red test that I want to mute. When I mute it, however, teamcity makes my build red even though it marked my test as muted, like so:
The problem is well-known, since 11 years, as reported here.
How can I modify my command in my tox.ini file to make my build green again? I don't want to mark my python test with the skip tag. I don't want to change the tox command from
commands =
pytest {posargs} --teamcity
to
commands =
- pytest {posargs} --teamcity
because that will just ignore any error that might happen during the pytest run (like "Internal error happened while executing tests", or "No tests were collected" for example).
Ideally, I would like to call
commands =
pytest {posargs} --teamcity || [ $? = 1 ]
but apparently tox does not understand the symbol ||.
What can I do?
You can call a custom shell script in your commands section, and there you can do whatever you want, including using ||.
e.g.
commands = my_custom_script.sh
Related
I'm trying to implement danger-swift but I don't know where I can access test results and flag failing ones from Bitrise.
I'm using a plugin for danger-swift called DangerXCodeSummary but I don't know where Bitrise stores test results from xcode-test#2.
DangerFile:
import Danger
import DangerXCodeSummary
import Foundation
let danger = Danger()
let testReportPath = "??" // What's the path for the test results?
XCodeSummary(filePath: testReportPath).report()
Bitrise script:
...
UnitTests:
before_run:
- _ensure_dependencies
after_run:
- _add_build_result_to_pr
steps:
- xcode-test#2: {} # What's the file path for the test results?
- deploy-to-bitrise-io#1: {}
envs:
- ots:
is_expand: false
BITRISE_PROJECT_PATH: MyApp.xcodeproj
- opts:
is_expand: false
BITRISE_SCHEME: AppTests
description: Unit Tests running at every commit.
...
_add_build_result_to_pr:
steps:
- script#1:
title: Commenting on the PR
is_always_run: true
inputs:
- content: |-
#!/usr/bin/env bash
# fail if any commands fails
set -e
echo "################### DANGER ######################"
echo "Install Danger"
brew install danger/tap/danger-swift
echo "Run danger"
danger-swift ci
echo "#################################################"
You can see what outputs the step generates on the Workflow Editor UI, and alternatively in the Step's repository in step.yml. For Xcode Test step specifically: https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L287
Am I right that you're looking for the xcresult test output? If you are, you can read it from the BITRISE_XCRESULT_PATH environment variable (https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L296), which is the output of Xcode Test (once Xcode Test is finished, it sets this environment variable to the xcresult's path). Keep in mind this is in .xcresult format (the official Xcode test result format).
I'm having trouble implementing a toy example that runs pytest within .gitlab-ci.yml
gitlab_ci is a repo containing a single file test_hello.py
gitlab_ci/
test_hello.py
test_hello.py
# test_hello.py
import pytest
def hello():
print("hello")
def hello_test():
assert hello() == 'hello'
.gitlab-ci.yml
# .gitlab-ci.yml
pytest:
image: python:3.6
script:
- apt-get update -q -y
- pip install pytest
- pytest # if this is removed, the job outputs 'Success'
CI/CD terminal output
$ pytest
=== test session starts ===
platform linux -- Python 3.6.9, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
rootdir: /builds/kunov/gitlab_ci
collected 0 items
=== no tests ran in 0.02s ===
ERROR: Job failed: exit code 1
I'm not sure why the test did not run... pytest does not seem to recognize test_hello.py
Solution
Put the python file inside the newly creared tests folder:
gitlab_ci/
.gitlab-ci.yml
tests/
test_hello.py
Modify gitlab-ci.yml in the following manner:
# .gitlab-ci.yml
pytest:
image: python:3.6
script:
- apt-get update -q -y
- pip install pytest
- pwd
- ls -l
- export PYTHONPATH="$PYTHONPATH:."
- python -c "import sys;print(sys.path)"
- pytest
And test_hello.py would stay the same as before.
This blog post mentions a similar pipeline, but:
However, this did not work as pytest was unable to find the ‘bild’ module (ie. the source code) to test.
The problem encountered here is that the ‘bild’ module is not able to be found by the test_*.py files, as the top-level directory of the project was not being specified in the system path:
pytest:
stage: Test
script:
- pwd
- ls -l
- export PYTHONPATH="$PYTHONPATH:."
- python -c "import sys;print(sys.path)"
- pytest
The OP kunov confirms in the comments:
It works now! I put the single file inside a newly created folder called 'test'
Manipulation of the PYTHONPATH variable is considered by some to be a bad practice (see e.g., this answer on stackoverflow or this Level Up Coding post). While this is possible not a huge issue in the scope of a GitLab CI job, here is a solution based on Alberto Mardegan's comment at the mentioned blog post without the need to fiddle with PYTHONPATH (also somewhat cleaner):
pytest:
stage: Test
script:
- pwd
- ls -l
- python -m pytest
Why does this work? From the pytest docs:
You can invoke testing through the Python interpreter from the command
line:
python -m pytest [...]
This is almost equivalent to invoking the
command line script pytest [...] directly, except that calling via
python will also add the current directory to sys.path.
test_hello.py
def test_hello():#func name must start with "test_",not "hello_test"
this is my first time using tox to create a python package. I didn't underestimate this task and read myself a little bit into how setuptools and pyscaffold works, what does what and why and so on, got some yt-videos to grasp how to get it on.
But, guess what, it doesn't work, and i have absolutely no clue why.
This is what i did:
putup --tox river
then i placed my sources under src/
and some tests under tests/
so this is my folder structure so far:
river/
src/
dispatcher/
__init__.py
updater.py
river/
__init__.py
dispatcher_config.py
dispatcher.py
flow.py
logger.py
log_config.yml
tests/
__init__.py
test_cases.py
AUTHORS.rst
CHANGELOG.rst
LICENSE.txt
README.rst
setup.cfg
setup.py
tox.ini
log_config.yml
all i want to achieve for now is getting my tests running properly.
tox.ini :
[tox]
minversion = 2.4
envlist = default
[testenv]
setenv = TOXINIDIR = {toxinidir}
sitepackages = True
commands =
python --version
pytest
deps = pytest
setup.cfg - almost unchanged :
# This file is used to configure your project.
# Read more about the various options under:
# http://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files
[metadata]
name = river
description = Add a short description here!
author = Kristian Jülfs
author-email = kristian.juelfs#...
license = mit
long-description = file: README.rst
long-description-content-type = text/x-rst; charset=UTF-8
url = https://github.com/pyscaffold/pyscaffold/
project-urls =
Documentation = https://pyscaffold.org/
# Change if running only on Windows, Mac or Linux (comma-separated)
platforms = any
# Add here all kinds of additional classifiers as defined under
# https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers =
Development Status :: 4 - Beta
Programming Language :: Python
[options]
zip_safe = False
packages = find:
include_package_data = True
package_dir =
=src
# DON'T CHANGE THE FOLLOWING LINE! IT WILL BE UPDATED BY PYSCAFFOLD!
setup_requires = pyscaffold>=3.2a0,<3.3a0
# Add here dependencies of your project (semicolon/line-separated), e.g.
# install_requires = numpy; scipy
# The usage of test_requires is discouraged, see `Dependency Management` docs
# tests_require = pytest; pytest-cov
# Require a specific Python version, e.g. Python 2.7 or >= 3.4
# python_requires = >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*
[options.packages.find]
where = src
exclude =
tests
[options.extras_require]
# Add here additional requirements for extra features, to install with:
# `pip install river[PDF]` like:
# PDF = ReportLab; RXP
# Add here test requirements (semicolon/line-separated)
testing =
pytest
pytest-cov
# flake8
[options.entry_points]
# Add here console scripts like:
# console_scripts =
# script_name = river.module:function
# For example:
# console_scripts =
# fibonacci = river.skeleton:run
# And any other entry points, for example:
# pyscaffold.cli =
# awesome = pyscaffoldext.awesome.extension:AwesomeExtension
[test]
# py.test options when running `python setup.py test`
#addopts = --verbose
extras = True
[aliases]
dists = bdist_wheel
[bdist_wheel]
# Use this option if your package is pure-python
universal = 1
[build_sphinx]
source_dir = docs
build_dir = build/sphinx
[devpi:upload]
# Options for the devpi: PyPI server and packaging tool
# VCS export must be deactivated since we are using setuptools-scm
no-vcs = 1
formats = bdist_wheel
[tool:pytest]
addopts = --verbose
norecursedirs =
dist
build
.tox
[flake8]
# Some sane defaults for the code style checker flake8
exclude =
.tox
build
dist
.eggs
docs/conf.py
[pyscaffold]
# PyScaffold's parameters when the project was created.
# This will be used when updating. Do not change!
version = 3.2.1
package = river
extensions =
tox
alrighti so far. Whatever i'm missing, i don't get it.
when i start tox on verbose,
tox -r -vvv
i get this (cut)
...
creating '/tmp/pip-wheel-rb7acy7z/river-0.0.post0.dev1+gc371b6b.dirty-py2.py3-none-any.whl' and adding '.' to it
adding 'dispatcher/__init__.py'
adding 'dispatcher/updater.py'
adding 'river/__init__.py'
adding 'river/dispatcher.py'
adding 'river/dispatcher_config.py'
adding 'river/flow.py'
adding 'river/logger.py'
adding 'river-0.0.post0.dev1+gc371b6b.dirty.dist-info/top_level.txt'
adding 'river-0.0.post0.dev1+gc371b6b.dirty.dist-info/WHEEL'
adding 'river-0.0.post0.dev1+gc371b6b.dirty.dist-info/METADATA'
adding 'river-0.0.post0.dev1+gc371b6b.dirty.dist-info/RECORD'
removing build/bdist.freebsd-12.0-RELEASE-p7-amd64/wheel
done
Stored in directory: /home/kjuelf/.cache/pip/wheels/f7/c7/76/923a5b579b9178351cdbe053f020f660101c03b78a4085d281
Removing source in /tmp/pip-req-build-jhc29i8z
Successfully built river
Installing collected packages: river
Successfully installed river-0.0.post0.dev1+gc371b6b.dirty
Cleaning up...
...
default start: run-test-pre
default run-test-pre: PYTHONHASHSEED='4259061550'
default finish: run-test-pre after 0.00 seconds
default start: run-test
default run-test: commands[0] | python --version
setting PATH=/usr/home/kjuelf/infra/river/.tox/default/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/kjuelf/bin
[11307] /usr/home/kjuelf/infra/river$ /usr/home/kjuelf/infra/river/.tox/default/bin/python --version
Python 3.6.9
default run-test: commands[1] | pytest
setting PATH=/usr/home/kjuelf/infra/river/.tox/default/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/kjuelf/bin
WARNING: test command found but not installed in testenv
cmd: /usr/local/bin/pytest
env: /usr/home/kjuelf/infra/river/.tox/default
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
DEPRECATION WARNING: this will be an error in tox 4 and above!
[11308] /usr/home/kjuelf/infra/river$ /usr/local/bin/pytest
============================================================ test session starts ============================================================
platform freebsd12 -- Python 3.6.9, pytest-4.5.0, py-1.8.0, pluggy-0.12.0 -- /usr/local/bin/python3.6
cachedir: .tox/default/.pytest_cache
rootdir: /usr/home/kjuelf/infra/river, inifile: setup.cfg
plugins: cov-2.7.1, flake8-1.0.4
collected 0 items / 1 errors
================================================================== ERRORS ===================================================================
___________________________________________________ ERROR collecting tests/test_cases.py ____________________________________________________
/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py:359: in get_provider
module = sys.modules[moduleOrReq]
E KeyError: 'river'
During handling of the above exception, another exception occurred:
src/river/logger.py:18: in <module>
config = resource_string("river", "log_config.yml")
/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py:1156: in resource_string
return get_provider(package_or_requirement).get_resource_string(
/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py:361: in get_provider
__import__(moduleOrReq)
E ModuleNotFoundError: No module named 'river'
...
i mean, first
Successfully built river
then
E ModuleNotFoundError: No module named 'river'
in pkg_resources/__ init __.py - KeyError: 'river'
i can't get a clue whats wrong with pkg_resources here - where is my stuff gone suddenly o_0 ??!
I also don't get what that warning means:
WARNING: test command found but not installed in testenv
it's there, pytest works, and the dependency is set in tox.ini
lost in weirdness
I'm trying to debug a python codebase that uses tox for unit tests. One of the failing tests is proving difficult due to figure out, and I'd like to use pudb to step through the code.
At first thought, one would think to just pip install pudb then in the unit test code add in import pudb and pudb.settrace(). But that results in a ModuleNotFoundError:
> import pudb
>E ModuleNotFoundError: No module named 'pudb'
>tests/mytest.py:130: ModuleNotFoundError
> ERROR: InvocationError for command '/Users/me/myproject/.tox/py3/bin/pytest tests' (exited with code 1)
Noticing the .tox project folder leads one to realize there's a site-packages folder within tox, which makes sense since the point of tox is to manage testing under different virtualenv scenarios. This also means there's a tox.ini configuration file, with a deps section that may look like this:
[tox]
envlist = lint, py3
[testenv]
deps =
pytest
commands = pytest tests
adding pudb to the deps list should solve the ModuleNotFoundError, but leads to another error:
self = <_pytest.capture.DontReadFromInput object at 0x103bd2b00>
def fileno(self):
> raise UnsupportedOperation("redirected stdin is pseudofile, "
"has no fileno()")
E io.UnsupportedOperation: redirected stdin is pseudofile, has no fileno()
.tox/py3/lib/python3.6/site-packages/_pytest/capture.py:583: UnsupportedOperation
So, I'm stuck at this point. Is it not possible to use pudb instead of pdb within Tox?
There's a package called pytest-pudb which overrides the pudb entry points within an automated test environment like tox to successfully jump into the debugger.
To use it, just make your tox.ini file have both the pudb and pytest-pudb entries in its testenv dependencies, similar to this:
[tox]
envlist = lint, py3
[testenv]
deps =
pytest
pudb
pytest-pudb
commands = pytest tests
Using the original PDB (not PUDB) could work too. At least it works on Django and Nose testers. Without changing tox.ini, simply add a pdb breakpoint wherever you need, with:
import pdb; pdb.set_trace()
Then, when it get to that breakpoint, you can use the regular PDB commands:
w - print stacktrace
s - step into
n - step over
c - continue
p - print an argument value
a - print arguments of current function
I've had no problem passing arguments to a python program called volatility in the past using sys.process._. However, I tried to add another argument and the program won't run.
Here is the code that worked:
psScan = Try(s"python vol.py -f $memFile --profile=$os psscan".!!.trim).getOrElse("Failed")
This code also worked:
s"python vol.py -f $memFile --profile=$os svcscan --verbose"
I'm not sure why the program won't accept the additional argument. Here are the different things I've tried that won't work.
psScan = Try(s"python vol.py -f $memFile --profile=$os --kdbg=$kdbg psscan".!!.trim)
psScan = Try(s"python vol.py -f $memFile --profile=$os -g $kdbg psscan".!!.trim)
psScan = Seq("python", "vol.py", "-f", memFile, s"--profile=$os", "--kdbg=$kdbg", "psscan").!!.trim
psScan = Seq("python", "vol.py", "-f", memFile, s"--profile=$os", "-g", kdbg, "psscan").!!.trim
I also tried to create an external environmental variable from the command line with "export VOLATILITY..", but that failed. (I know trying that was a hail mary).