I have a test-complete project which is run via azure-devops, setup for CI. This project is linked to cucumber for the steps, as well as pushing results back up for reporting (we already have this setup and working for selenium, visual studio). The problem I am having is that I have multiple test runs in cucumber that are run using this single test-complete project.
When running manually there is no problem because we can manually switch between what test runs on cucumber our project currently pushes too. But with azure-devops that is not an option.
We currently create a .trx report that we use to bring results back into azure-devops but this report is not compatible with cucumber. I ran /ListLoggers to show all loggers available but none of these seem useable for pushing to cucumber using hiptest-publisher:
/ListLoggers
The following Test Logger Add-Ins are available:
Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameLogger
Uri: logger://Microsoft/TestPlatform/Extensions/Blame/v1
FriendlyName: Blame
Microsoft.VisualStudio.TestPlatform.Extensions.HtmlLogger.HtmlLogger
Uri: logger://Microsoft/TestPlatform/HtmlLogger/v1
FriendlyName: Html
Microsoft.VisualStudio.TestPlatform.Extensions.TrxLogger.TrxLogger
Uri: logger://Microsoft/TestPlatform/TrxLogger/v1
FriendlyName: Trx
This is the part of the release pipeline that runs the tests:
I filter out tests to-be-run like so FullyQualifiedName~SmokeTest. So the question remains, how can I send results back up to cucumber, to specific test runs, from the single project via azure-devops without changing the test-run linked to in the test-complete project manually?
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
For my application I wrote a custom integration test that starts the application, performs important functionality and tests many calculations. The test is part of the application's source control.
The details of every exception are written to the Trace or as an exception.
I would like to be able - on my Azure DevOps build server - to start this custom test after the build has finished.
I will adapt my test application then to mail the results to some developers.
For me this suits my needs better then test frameworks I have seen.
Currently I start Test in Debug Mode in Visual Studio (not integrated yet). It is a Console application, executed by 'Test.exe' on the command prompt. I will change that to Windows application (unattended execution, without console).
Question: how do I start my custom test on the Azure build server (who calls it Pipeline) after the build has finished? Which build step do I have to use?
Note that the integration test is also a build artifact of the same build.
It all depends what external dependencies you have if you will be able to reach them from Azure DevOps agent but since you are on self hosted you have all cards in your hands. If this is Console app you can simply call it. Let's assume you copied it to Build.ArtifactStagingDirectory you can run it as:
- pwsh: Test.exe
workingDirectory: '$(Build.ArtifactStagingDirectory)'
if you want to do as part of the same stage when you produced that file. If you want to make this in another stage then you need to first download pipeline/build artifact and call it as above (you will get this probably in a different folder, so that needs to be changed).
I have a dot net core version: '3.0.100', building on 'ubuntu-latest' and am trying to get code coverage reports.
I have been using Coverlet to generate Cobertura files which can then use the PublishCodeCoverageResults#1 to publish to the Devops pipelines code coverage viewer.
I have used one stage name build for building the project and generating artifacts and published them.
In another stage named unit tests i will be downloading the artifacts and using the artifacts of previous build.
But when I run unit tests by specifying the working directory where these artifacts were downloaded
Everything went fine up to this point,but when it is generating the code coverage reports in the unit test job it is generating NaN% value.
enter image description here
I have tried all possible ways playing with working path directory and restoring the nuget packages.But No Luck None worked for me. I have even searched for this issue,I found one but that didn't worked me
When i tried to run the build and unit tests in one stage itself it successfully generated the code coverage reports,but placing that in another stage and downloading the previous build artifacts made me run into the issue.
How would I achieve this ?
Right now I'm creating a build with Ubuntu shell commands in VSTS to copy the code from VSTS to an external AWS static server which has already installed npm and protractor. Then the execution is triggered from VSTS build using same ubuntu shell commands and the final HTML reports are being saved in that server itself.
Any other way to bring back the results folder into VSTS? or to create any dashboard to show that HTML reports?
You can use the protractor-trx-reporter package to create a trx report and then after your test is finished, pull the results back into a folder in Azure Devops*(VSTS). Use the "publish test results" task to make the results viewable.
(https://www.npmjs.com/package/protractor-trx-reporter)