GitLab CI - How to verify that Unity3D Tests actually passed - unity3d

Question

Here is my gitlab-ci.yml:
variables:
GIT_STRATEGY: fetch
GIT_CLONE_PATH: $CI_BUILDS_DIR
GIT_DEPTH: 10
GIT_CLEAN_FLAGS: none
stages:
- test
- build
unit-test:
script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\'2019.4.3f1'\\Editor\\Unity.exe \
-runTests \
-batchmode \
-projectPath . \
-logFile ./log.txt \
-testResults ./unit-tests.xml"
stage: test
tags:
- unity
unity-build:
stage: build
script: echo 'Building...'
tags:
- unity
Here is the output of the pipeline:
Running with gitlab-runner 13.1.1 (6fbc7474)
on Desktop ryBW4ftU
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:01
Running on DESKTOP-G62KPLQ...
Getting source from Git repository
00:06
Fetching changes with git depth set to 10...
Reinitialized existing Git repository in X:/ScarsOFHonor_CI/.git/
Checking out 8ca0d362 as master...
Encountered 2 file(s) that should have been pointers, but weren't:
ProjectSettings/EditorSettings.asset
ProjectSettings/XRSettings.asset
git-lfs/2.4.2 (GitHub; windows amd64; go 1.8.3; git 6f4b2e98)
Skipping Git submodules setup
Executing "step_script" stage of the job script
01:18
$ C:\'Program Files'\Unity\Hub\Editor\'2019.4.3f1'\Editor\Unity.exe -runTests -batchmode -projectPath . -logFile ./log.txt -testResults ./unit-tests.xml
. is not a valid directory name. Please make sure there are no unallowed characters in the name.
(Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 218)
. is not a valid directory name. Please make sure there are no unallowed characters in the name.
(Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 218)
Job succeeded
As you can see pipeline passed with success. However that should not be true. I intentionally created an pushed a failing test.
See the content of unit-tests.xml:
<?xml version="1.0" encoding="utf-8"?>
<test-run id="2" testcasecount="1" result="Failed(Child)" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0" engine-version="3.5.0.0" clr-version="4.0.30319.42000" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.105692">
<test-suite type="TestSuite" id="1007" name="ScarsOFHonor" fullname="ScarsOFHonor" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.105692" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
<properties />
<failure>
<message><![CDATA[One or more child tests had errors]]></message>
</failure>
<test-suite type="Assembly" id="1010" name="EditMode.dll" fullname="X:/ScarsOFHonor_CI/Library/ScriptAssemblies/EditMode.dll" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.079181" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
<properties>
<property name="_PID" value="27144" />
<property name="_APPDOMAIN" value="Unity Child Domain" />
<property name="platform" value="EditMode" />
</properties>
<failure>
<message><![CDATA[One or more child tests had errors]]></message>
</failure>
<test-suite type="TestSuite" id="1011" name="Tests" fullname="Tests" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.076239" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
<properties />
<failure>
<message><![CDATA[One or more child tests had errors]]></message>
</failure>
<test-suite type="TestFixture" id="1008" name="DummyTest" fullname="Tests.DummyTest" classname="Tests.DummyTest" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.067615" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
<properties />
<failure>
<message><![CDATA[One or more child tests had errors]]></message>
</failure>
<test-case id="1009" name="DummyTestSimplePasses" fullname="Tests.DummyTest.DummyTestSimplePasses" methodname="DummyTestSimplePasses" classname="Tests.DummyTest" runstate="Runnable" seed="785721540" result="Failed" start-time="2020-07-17 11:40:32Z" end-time="2020-07-17 11:40:32Z" duration="0.032807" asserts="0">
<properties />
<failure>
<message><![CDATA[ Expected: 1
But was: 2
]]></message>
<stack-trace><![CDATA[at Tests.DummyTest.DummyTestSimplePasses () [0x00001] in X:\ScarsOFHonor_CI\Assets\Tests\EditMode\DummyTest.cs:15
]]></stack-trace>
</failure>
</test-case>
</test-suite>
</test-suite>
</test-suite>
</test-suite>
</test-run>
As you can see there is a failing test. Why then my pipeline passes ?
Is there any way I can print my results in the pipeline itself and mark the pipeline as failed when actually I have failing test?

Answer 1

Hm a bit (actually extremely :D ) dirty but you could simply check if the result file contains the word <failure> using find like e.g.
find "<failure>" ./unit-tests.xml > nul && exit 1 || exit 0
> nul don't print to the console
so if the word <failure> was found it exits with failure (1) otherwise with success (0)
Not a yml expert but I think you could e.g. add it to
script: "C:\\'Program Files'\\Unity\\Hub\\Editor\\'2019.4.3f1'\\Editor\\Unity.exe -runTests -batchmode -projectPath . -logFile ./log.txt -testResults ./unit-tests.xml && find '<failure>' ./unit-tests.xml > nul && exit 1 || exit 0"