I run CI/CD pipeline on Azure DevOps and generate Junit xml files with test results. By the way, I use Cypress framework for my tests.
Is there any way to ingest Junit results into TestRail 7.5 and publish test results in "Test Runs and Results" tab in TestRail?
Actually, what I would like to achieve is to generate new runs on TestRail and push the test results after each run on pipeline.
Can anyone that has done it previously help me, please?
You might want to have a look at Railflow which provides exact functionality you want. It is a commercial tool but it provides free support for JUnit report uploading into TestRail.
P.S. I'm developer at Railflow
Related
I want to populate test plan with result of tests that are done by our CI pipeline on github. I am thinking of using a release pipeline like this. (Azure DevOps Test Plans - Synchronize JUnit test results in Runs with test cases on Tests Plan)
But as the author was not able to populate test plans with it. I am looking for a better solution.
As I need the project report for the test plan.
This seems a pretty straightforward thing to do, but I cannot find relevant information. In Visual Studio is very easy to calculate code metrics for all projects and I would like to do the same during a build pipeline in Azure DevOps.
Has anyone done something like this?
In azure devops, you can review code coverage results. The results can be viewed and downloaded on the Code coverage tab.
Publish Code Coverage Results publishes code coverage results to
Azure Pipelines or TFS, which were produced by a build in Cobertura
or JaCoCo format.
Built-in tasks such as Visual Studio Test, .NET Core, Ant, Maven,
Gulp, Grunt, and Gradle provide the option to publish code coverage
data to the pipeline.
Here are some documents you can refer to:
Review code coverage results
Azure DevOps and the Code Coverage
In addition, you can get code analysis through SonarCloud integrated with Azure devops. SonarCloud is a cloud-based code quality and security service.
Here is the lab you can follow.
You can generate code metrics by adding the Microsoft.CodeAnalysis.Metrics nuget and building with the parameter that target the "Metrics" build, this will store the results in an xml file.
msbuild /t:Metrics
If you're building a solution with multiple projects, you have to add the nuget to every project otherwise the build will fail because the Metrics target won't be present for the other projects.
The build will produce one xml file per project.
You can also use the metrics command line tool from the roslyn analyzer repository.
See https://learn.microsoft.com/en-us/visualstudio/code-quality/how-to-generate-code-metrics-data?view=vs-2022
I've created CI&CD on Azure repo, and it is doing code deployment on App service.
Now created Automation that opens the browsers and do actions and generates reports for test cases. (C#+Selenium+report package).
Now I need to the setup Test plan, I know, needs to run headless testing, my the questions are
need to do without buying a Test plan package, then how we should do? (I have a subscription but not with test plans)
How to do using Azure Test plans (any recent blog pls, and no MS docs links).
any help appreciated
As far as I know, you can execute Selenium tests directly in the pipeline(Without test plan).
Here are the steps:
Add Visual Studio Test Task to run the Selenium tests and the test result will be output to the Test tab(Build Pipeline or Release Pipeline).
You also could use Publish Test Results task to publish test result.
Here is a blog about the detailed steps: Automating Selenium Tests in Azure Pipelines.
For headless testing,
You could refer to UI testing considerations.
By the way, since you have already created CI/CD, you can also consider integrating UI testing steps into CI/CD.
If you still want to configure test plan, you could refer to this doc.
As noted here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/review-continuous-test-results-after-build?view=azure-devops#tests-tab
If you use the Visual Studio Test task to run tests, diagnostic output logged from tests (using any of Console.WriteLine, Trace.WriteLine or TestContext.WriteLine methods), will appear as an attachment for a failed test.
How can I enforce Azure Devops pipelines to publish attachments for successful tests?
Update
riQQ pointed to the another stackoverflow question:Azure DevOps: Console output for green tests is missing
Proposed solution doesn't help me, because they solved using TestContext. We are using xUnit, and there is no TestContext for xUnit.
How can I enforce Azure Devops pipelines to publish attachments for
successful tests?
Sorry but as I know this is not supported behavior for VSTest Task. It's by design that only failed tests should have the log as attachment. And after my check, no extensions in Marketplace extends this kind of option.
So if you do want this option, you may consider using rest api like what riQQ suggests above. Or you can post a new feature request in DC forum, it it gets enough votes, the team would consider it seriously.
Hope it helps :)
I am in the process of configuring Azure DevOps CI\CD pipeline for my WPF desktop application. UI test cases of the WPF application are written using TestStack.White framework. Is it possible to use test cases written usingTestStack.White framework in DevOps CI\CD pipeline? If yes, how can I configure TestStack.White UI test cases in CI\CD pipeline? How can I publish test results from my TestStack.White test cases to DevOps? Could you point me to a code example ?
You can run TestStack.White UI test in CI\CD pipeline and below is the configuration about build pipeline.
According to the code sample, I think you can get this sample to test. https://github.com/xingyu217/TestStackDemo