I have a pipeline that runs on pull requests and executes tests , it also produces an xml execution report (it is a node project and tests run via jest) , and coverage report that I use for sonarcloud. However , the developers want to be able to see the tests results on each pr and see what tests failed in particular, so my question is whether it is possible to somehow export and display those results in github or sonarcloud
Please refer to the following two methods to get the test cases metric in sonarcloud:
You can add sonar.testExecutionReportPaths in Additional Properties option in Prepare Analysis Configuration task. About sonar.coverageReportPaths
For SonarQube to pick up and analyse the content of the report file, you need to inform the sonar scanner. About the location of the XML file. To do so, edit the sonarqube.properties file. Please refer to this.
coverage reporting:
sonar.testExecutionReportPaths=reports/xxx/xxx.xml
In addition, you can also refer to this similar case.
Related
I have all my test cases in Azure Devops Test Plans and now I'm executing them manually against a build.
How do I view manual test results by build?
I can view and filter test results by build on the "Runs" page but I would prefer to see test results grouped by build not filtered to a single build.
Any options for this?
I had hoped the the Test Plans - Progress Report would provide this but I don't see a way.
Maybe there is a way to group all my manual tests executions into a Test Run?
I researched and attempted with the related Azure DevOps REST API and Azure CLI, however, I did not find any available way.
It seems that we can only directly filter the test results to the build run level, and cannot to the build pipeline level.
Maybe, you can indirectly get the associated build pipeline via the following way:
Use the REST API "Runs - Query" to list the test runs associated with a specified build pipeline.
Use the REST API "Results - List" in a loop to list the test results of each test run.
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
I have a automation script which built in Java/selenium. I want to run my test cases from VSTS by triggering my script and also want to get the result back in VSTS. Can anybody give me the path how can I make that happen. Also where should I keep by project?
I was doing research on it. But doesn't make sense to me
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/continuous-test-selenium?view=azure-devops
run my test cases from VSTS by triggering my script and also want to
get the result back in VSTS
For this issue ,there is a blog that gives a path to do this , please refer to it to see if it helps .
In this path , you need to do stuff both to your project, and to your vsts build definition. The jest-junit package is needed in the project and then create a build with three tasks(“npm”,“Publish Test Results”, “Publish Code Coverage Results” ).
I have a requirement of settings code coverage percentage for my unit test cases running in VSTS through the build.yml file.
Right now the build is passing for any code coverage percentage that's coming. I want to restrict this and fail the build if the percentage is not 70%.
Can you help me with this?
You need to publish the test results (and code coverage is in that) and then use another task or release gate to check those results. You can do this with SonarQube Quality gate (needs a SonarQube server) or you can try the marketplace.
For example this task: https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
You can also check it yourself (using PowerShell for example) via the API: https://learn.microsoft.com/en-us/rest/api/vsts/test/code%20coverage?view=vsts-rest-5.0