We are trying to Run tests in parallel using the Visual Studio Test task based on below link but we noticed "Run Unit tests" step failing due to packages directory was not found. Trying to contact Azure DevOps team to get some help here.
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops
We tried to publish the test files as artifacts from the first job. And download the test artifacts to the second job. But we found the test files were located separately and there were many test data files needed to be published too. It was hard to publish all the test files to azure devops sever. And the vstest task failed with either assemblies not found error or some testcases just failed.
We also tried configuring a single job running in parallel with both build tasks and vstest task. It turned out the publish build artifacts task didnot work properly in parallel jobs.
After many times try and working together with #NAGAJU. The workaround we found is to configure two agent jobs to separate the publish artifacts task and Vstest task.
The first agent job with build tasks and publish artifacts tasks runs in an single agent. The second job with build tasks and Vstest task to run the tests in multiple agents in parallel.
Related
I have a dot net core version: '3.0.100', building on 'ubuntu-latest' and am trying to get code coverage reports.
I have been using Coverlet to generate Cobertura files which can then use the PublishCodeCoverageResults#1 to publish to the Devops pipelines code coverage viewer.
I have used one stage name build for building the project and generating artifacts and published them.
In another stage named unit tests i will be downloading the artifacts and using the artifacts of previous build.
But when I run unit tests by specifying the working directory where these artifacts were downloaded
Everything went fine up to this point,but when it is generating the code coverage reports in the unit test job it is generating NaN% value.
enter image description here
I have tried all possible ways playing with working path directory and restoring the nuget packages.But No Luck None worked for me. I have even searched for this issue,I found one but that didn't worked me
When i tried to run the build and unit tests in one stage itself it successfully generated the code coverage reports,but placing that in another stage and downloading the previous build artifacts made me run into the issue.
How would I achieve this ?
I am trying to run xUnit tests using the Azure CI pipeline, despite the unit tests running successfully in Visual Studio 2017, but on Azure DevOps no results. Below are screenshots for the vs solution and build and test tasks in the Azure pipeline: '
Test project
The solution in the visual studio
The solution in the pipeline
Build task
Test task
Test task results
What is the problem with my setup?
Please change your test files and replace this line:
**\bin\$(BuildConfiguration)\**\*test*.dll
with this:
**\bin\$(BuildConfiguration)\**\UnitTesting.dll
or this:
**\bin\$(BuildConfiguration)\UnitTesting.dll
Please try to apply this pattern to your folder structure in you bin folder.
I have a staged release pipeline that deploys to Development and then to Staging.
I want my Integration Test project to run during release pipeline only before deployment to staging. How do I do that? is there a specific test task that I will be able to hook into before the staging deployment? like a pre-deploy task or something?
In order to run Integration Tests in your release pipeline. You should include your Test projects or test assembly dll files in the artifacts published in your build pipeline. So that your Integration test projects are accessible to the test tasks in your release pipeline.
To Include your test files in the artifacts. You can add a second publish build artifacts task in your Build Pipeline. Specify Path to publish to the location of your test files. Fore below example.
I add a another publish build artifacts task to publish all the test files that needed to run the tests.
Then I specify the Path to publish field to folder that contains all the test files.
After the build pipeline completed. You will see the Test folder being added to the build artifacts and published to the azure devops pipeline server.
Now you can run your tests in your release pipeline by adding the VsTest task or other test tasks in your release pipeline. The release pipeline will download your artifacts to
folder $(System.DefaultWorkingDirectory)(eg. C:\agent_work\r1\a).
For above example you will find the test files in folder $(System.DefaultWorkingDirectory)\artifact_alias\Test and the artifacts to be deployed in $(System.DefaultWorkingDirectory)\artifact_alias\drop.
Then you can either create a new stage with vstest task before deployment stage as #Kontekst mentioned, or just add the vstest task on the top of your deployment stage before your deployment task.
In above screenshot I add the Vstest task before the Azure deployment task and point the Search folder to the Test folder where the test files reside.
Update:
You can find the artifacts-alias in the field shown in below screenshot.
The main idea is to publish the test files in the artifacts in build pipeline and then add the test tasks in the release pipeline to run the tests.
I suppose you are using C# unit tests.
Add new stage "Integration tests" right before stage "Deployment to Staging".
Add task installing VSTest Platform. It will be cached in agent folder.
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer?view=azure-devops
Add to your tests projects proper unit tests adapter NuGet package, so that VSTest Platform will be able to find unit tests in dll assemblies, for example for NUnit tests it will be "NUnit 3 Test Adapter".
Add VSTest task specyfing:
a) Test files -> for example "***IntegrationTests.dll"
b) Test platform version -> "Installed by Tools Installer"
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops
All test results returned by VSTest task will be integrated with Azure DevOps services.
I Love both answers by Levi Lu-MSFT and Kontekst!
My team used to do it like that and it got the job done.
However if you'd like more "fancier" reporting and if like me, you were looking for something out of the box that Microsoft was offering specifically for integration tests or automated tests then Azure DevOps' Test Plan feature might be what you're looking for.
Test Plans really seem to be catering for actual web automated tests but you can basically run any tests that you can fit in a release pipeline.
It integrates well with Visual Studio Test Manager.
Here's the link to the documentation: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
And here's some screenshots in case the link fails in the future:
In Team City, tests are updated as they pass or fail which is useful for long running tests (you might abort the run).
You can also publish arificts while the build is running. e.g.
##teamcity[publishArtifacts 'alogOrScreenshot']
Can you do this in Azure DevOps, ideally using the VS Runner?
Azure DevOps has support for in-progress view of Test data as soon as its published instead of awaiting completion of session. More details # https://learn.microsoft.com/en-us/azure/devops/pipelines/test/review-continuous-test-results-after-build?view=azure-devops#view-execution-of-in-progress-tests.
To add, Junit / Xunit / NUnit and Ctest formats are supported with Publish test results task. https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
I have configured a private Build agent/self hosted VM in VSTS.
I want to copy all test results for each test case to my self hosted VM. Currently for this I have created Build Pipeline.
In publish test task,has Test result files e.g. .xml file and Search folder as Default source repository path.
Test case snapshot:
How can I specify the path of attachment in test case and copy/download artifacts to VM?
Also after processing the attachment, I want to copy the output back to test results.
How can this be achieved?
Any help highly appreciated.
Thank you
All necessary files are on agent machine (build/deployment agent), so if you run test during release, the test result files will be on that agent machine.
On the other hand, if you can’t use VsTest task to run test, then you can publish test result through Publish Test Result task.
The VsTest task will publish the results automatically.
If you're not using VsTest for whatever reason, then using deployment groups would be easier. With deployment groups, you don't need to copy files -- the deployment runs directly on the target machines with no intermediary agent machine necessary.