I'm having trouble implementing a toy example that runs pytest within .gitlab-ci.yml
gitlab_ci is a repo containing a single file test_hello.py
gitlab_ci/
test_hello.py
test_hello.py
# test_hello.py
import pytest
def hello():
print("hello")
def hello_test():
assert hello() == 'hello'
.gitlab-ci.yml
# .gitlab-ci.yml
pytest:
image: python:3.6
script:
- apt-get update -q -y
- pip install pytest
- pytest # if this is removed, the job outputs 'Success'
CI/CD terminal output
$ pytest
=== test session starts ===
platform linux -- Python 3.6.9, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
rootdir: /builds/kunov/gitlab_ci
collected 0 items
=== no tests ran in 0.02s ===
ERROR: Job failed: exit code 1
I'm not sure why the test did not run... pytest does not seem to recognize test_hello.py
Solution
Put the python file inside the newly creared tests folder:
gitlab_ci/
.gitlab-ci.yml
tests/
test_hello.py
Modify gitlab-ci.yml in the following manner:
# .gitlab-ci.yml
pytest:
image: python:3.6
script:
- apt-get update -q -y
- pip install pytest
- pwd
- ls -l
- export PYTHONPATH="$PYTHONPATH:."
- python -c "import sys;print(sys.path)"
- pytest
And test_hello.py would stay the same as before.
This blog post mentions a similar pipeline, but:
However, this did not work as pytest was unable to find the ‘bild’ module (ie. the source code) to test.
The problem encountered here is that the ‘bild’ module is not able to be found by the test_*.py files, as the top-level directory of the project was not being specified in the system path:
pytest:
stage: Test
script:
- pwd
- ls -l
- export PYTHONPATH="$PYTHONPATH:."
- python -c "import sys;print(sys.path)"
- pytest
The OP kunov confirms in the comments:
It works now! I put the single file inside a newly created folder called 'test'
Manipulation of the PYTHONPATH variable is considered by some to be a bad practice (see e.g., this answer on stackoverflow or this Level Up Coding post). While this is possible not a huge issue in the scope of a GitLab CI job, here is a solution based on Alberto Mardegan's comment at the mentioned blog post without the need to fiddle with PYTHONPATH (also somewhat cleaner):
pytest:
stage: Test
script:
- pwd
- ls -l
- python -m pytest
Why does this work? From the pytest docs:
You can invoke testing through the Python interpreter from the command
line:
python -m pytest [...]
This is almost equivalent to invoking the
command line script pytest [...] directly, except that calling via
python will also add the current directory to sys.path.
test_hello.py
def test_hello():#func name must start with "test_",not "hello_test"
Related
I'm trying to implement danger-swift but I don't know where I can access test results and flag failing ones from Bitrise.
I'm using a plugin for danger-swift called DangerXCodeSummary but I don't know where Bitrise stores test results from xcode-test#2.
DangerFile:
import Danger
import DangerXCodeSummary
import Foundation
let danger = Danger()
let testReportPath = "??" // What's the path for the test results?
XCodeSummary(filePath: testReportPath).report()
Bitrise script:
...
UnitTests:
before_run:
- _ensure_dependencies
after_run:
- _add_build_result_to_pr
steps:
- xcode-test#2: {} # What's the file path for the test results?
- deploy-to-bitrise-io#1: {}
envs:
- ots:
is_expand: false
BITRISE_PROJECT_PATH: MyApp.xcodeproj
- opts:
is_expand: false
BITRISE_SCHEME: AppTests
description: Unit Tests running at every commit.
...
_add_build_result_to_pr:
steps:
- script#1:
title: Commenting on the PR
is_always_run: true
inputs:
- content: |-
#!/usr/bin/env bash
# fail if any commands fails
set -e
echo "################### DANGER ######################"
echo "Install Danger"
brew install danger/tap/danger-swift
echo "Run danger"
danger-swift ci
echo "#################################################"
You can see what outputs the step generates on the Workflow Editor UI, and alternatively in the Step's repository in step.yml. For Xcode Test step specifically: https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L287
Am I right that you're looking for the xcresult test output? If you are, you can read it from the BITRISE_XCRESULT_PATH environment variable (https://github.com/bitrise-steplib/steps-xcode-test/blob/deb39d7e9e055a22f33550ed3110fb3c71beeb79/step.yml#L296), which is the output of Xcode Test (once Xcode Test is finished, it sets this environment variable to the xcresult's path). Keep in mind this is in .xcresult format (the official Xcode test result format).
I use /home/runner/work/SIESTAstepper/SIESTAstepper/coverage.xml but I think the path is not correct. My report is not uploading on GitHub Actions.
Full file.
The coverage file is located at /home/runner/work/SIESTAstepper/SIESTAstepper/coverage.xml. In general, it is /home/runner/work/<project>/<project>/coverage.xml.
I solved it with the following code.
- name: Generate Report
run: |
pip install codecov
pip install pytest-cov
pytest --cov=./ --cov-report=xml
codecov
- name: Upload coverage to Codecov
uses: codecov/codecov-action#v3.1.0
with:
token: ${{ secrets.CODECOV_TOKEN }}
directory: ./coverage/reports/
env_vars: OS,PYTHON
files: /home/runner/work/SIESTAstepper/SIESTAstepper/coverage.xml
flags: tests
Full code is here.
From your last run, check if the "Upload coverage to CodeCov" is actually executed.
The workflow seems to stop before that, in the test with pytest step:
/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/distutils/file_util.py:44: DistutilsFileError
=========================== short test summary info ============================
FAILED tests/__main__.py::test_merge_ani - distutils.errors.DistutilsFileErro...
FAILED tests/tests.py::test_merge_ani - distutils.errors.DistutilsFileError: ...
========================= 2 failed, 30 passed in 4.23s =========================
If it does not stop before that, you need to check if the file is indeed created, considering the error message is:
None of the following appear to exist as files: ./coverage.xml
I have the following tox.ini file:
[tox]
envlist = py310, flake8
isolated_build = True
[testenv]
skip_install = True
deps = -rtest_requirements.txt
passenv = *
commands =
pytest {posargs} --teamcity
[testenv:flake8]
deps = flake8
skip_install = True
commands = flake8 tests/
On teamcity, I run my python test through tox from within a script build step where I call the following shell script
#! /bin/sh
python -m tox .
Now, there is one red test that I want to mute. When I mute it, however, teamcity makes my build red even though it marked my test as muted, like so:
The problem is well-known, since 11 years, as reported here.
How can I modify my command in my tox.ini file to make my build green again? I don't want to mark my python test with the skip tag. I don't want to change the tox command from
commands =
pytest {posargs} --teamcity
to
commands =
- pytest {posargs} --teamcity
because that will just ignore any error that might happen during the pytest run (like "Internal error happened while executing tests", or "No tests were collected" for example).
Ideally, I would like to call
commands =
pytest {posargs} --teamcity || [ $? = 1 ]
but apparently tox does not understand the symbol ||.
What can I do?
You can call a custom shell script in your commands section, and there you can do whatever you want, including using ||.
e.g.
commands = my_custom_script.sh
My ultimate goal is to have tests run automatically anytime a container is updated. For example, if update /api, it should sync the changes between local and the container. After that it should automatically run the tests... ultimately.
I'm starting out with Hello World! though per the example:
# DevSpace --version = 5.16.0
version: v1beta11
...
hooks:
- command: |
echo Hello World!
container:
imageSelector: ${APP-NAME}/${API-DEV}
events: ["after:initialSync:${API}"]
...
I've tried all of the following and don't get the desired behavior:
stop:sync:${API}
restart:sync:${name}
after:initialSync:${API}
devCommand:after:sync
At best I can just get Hello World! to print on the initial run of devspace dev -b, but nothing after I make changes to the files for /api which causes files to sync.
Suggestions?
You will need a post-sync hook for this, which is separate from the DevSpace lifecycle hooks. You can define it with the dev.sync directly and it looks like this:
dev:
sync:
- imageSelector: john/devbackend
onUpload:
execRemote:
onBatch:
command: bash
args:
- -c
- "echo 'Hello World!' && other commands..."
More information in the docs: https://devspace.sh/cli/docs/configuration/development/file-synchronization#onupload
The MetaCPAN Travis CI coverage builds are quite slow. See https://travis-ci.org/metacpan/metacpan-web/builds/238884497 This is likely in part because we're not successfully ignoring the /local folder that gets created by Carton as part of our build. See https://coveralls.io/builds/11809290
We're using perl-helpers to help with our Travis configuration. I thought I should be able to use the DEVEL_COVER_OPTIONS environment variable in order to fix this, but I guess I don't have the correct incantation. I've included the entire config below because a few snippets out of context seemed misleading.
language: perl
perl:
- "5.22"
matrix:
fast_finish: true
allow_failures:
- env: COVERAGE=1 USE_CPANFILE_SNAPSHOT=true
- env: USE_CPANFILE_SNAPSHOT=false HARNESS_VERBOSE=1
env:
global:
# Carton --deployment only works on the same version of perl
# that the snapshot was built from.
- DEPLOYMENT_PERL_VERSION=5.22
- DEVEL_COVER_OPTIONS="-ignore ^local/"
matrix:
# Get one passing run with coverage and one passing run with Test::Vars
# checks. If run together they more than double the build time.
- COVERAGE=1 USE_CPANFILE_SNAPSHOT=true
- USE_CPANFILE_SNAPSHOT=false HARNESS_VERBOSE=1
- USE_CPANFILE_SNAPSHOT=true
before_install:
- git clone git://github.com/travis-perl/helpers ~/travis-perl-helpers
- source ~/travis-perl-helpers/init
- npm install -g less js-beautify
# Pre-install from backpan to avoid upgrade breakage.
- cpanm -n http://cpan.metacpan.org/authors/id/M/ML/MLEHMANN/common-sense-3.6.tar.gz
- cpanm -n App::cpm Carton
install:
- cpan-install --coverage # installs converage prereqs, if enabled
- 'cpm install `test "${USE_CPANFILE_SNAPSHOT}" = "false" && echo " --resolver metadb" || echo " --resolver snapshot"`'
before_script:
- coverage-setup
script:
# Devel::Cover isn't in the cpanfile
# but if it's installed into the global dirs this should work.
- carton exec prove -lr -j$(test-jobs) t
after_success:
- coverage-report
notifications:
email:
recipients:
- olaf#seekrit.com
on_success: change
on_failure: always
irc: "irc.perl.org#metacpan-travis"
# Use newer travis infrastructure.
sudo: false
cache:
directories:
- local
The syntax for the Devel::Cover options on the command line is weird. You need to put stuff comma-separated. At least when you use PERL5OPT.
DEVEL_COVER_OPTIONS="-ignore,^local/"
See for example https://github.com/simbabque/AWS-S3/blob/master/.travis.yml#L26, where it's a whole lot of stuff with commas.
PERL5OPT=-MDevel::Cover=-ignore,"t/",+ignore,"prove",-coverage,statement,branch,condition,path,subroutine prove -lrs t