I have all my test cases in Azure Devops Test Plans and now I'm executing them manually against a build.
How do I view manual test results by build?
I can view and filter test results by build on the "Runs" page but I would prefer to see test results grouped by build not filtered to a single build.
Any options for this?
I had hoped the the Test Plans - Progress Report would provide this but I don't see a way.
Maybe there is a way to group all my manual tests executions into a Test Run?
I researched and attempted with the related Azure DevOps REST API and Azure CLI, however, I did not find any available way.
It seems that we can only directly filter the test results to the build run level, and cannot to the build pipeline level.
Maybe, you can indirectly get the associated build pipeline via the following way:
Use the REST API "Runs - Query" to list the test runs associated with a specified build pipeline.
Use the REST API "Results - List" in a loop to list the test results of each test run.
Related
I want to populate test plan with result of tests that are done by our CI pipeline on github. I am thinking of using a release pipeline like this. (Azure DevOps Test Plans - Synchronize JUnit test results in Runs with test cases on Tests Plan)
But as the author was not able to populate test plans with it. I am looking for a better solution.
As I need the project report for the test plan.
I'm new on DevOps Azure. I'm trying to link my automated tests for Azure DevOps platform, but outcome is always "In progress". I don't get failed or passed for any test. I created my tests in Visual Studio C#.
I created app and CheckIn it in Visual Studio and it's showed in Repos File, also my test folder is shown too. If I do changes in my code it's updated in Repos. So it is fine.
Then, I created Pipeline:TestMirjana-.NET Desktop-CI
and build is successful for this one. Than, I created Release linked for this pipeline:
and it is successful too.
I linked my test case for Visual Studio:
also linked Test Plan Settings for these pipeline and release
but when I run my test case Outcome is always "In progress", no other results shown (not failed or passed)
validation is good
but Outcome is always None
.
So, I'm not sure why it's not working properly.
Maybe I didn't link something properly, can you please support?
Many thanks in advance.
The 'In Progress' result means it has some automation associated but there are no incoming results (having the required test configuration).
Try to run the vstest task in the Release pipeline (instead of the build pipeline) and use "Select tests using=Test plan" and pick the Test plan where you expect to see the results from automated tests. It works for me.
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
I have a use case to run test project to run test cases in specific sequence with in TFS release pipeline.
I see that Ordered test and Playlist are not an option any more what would be the process for this.
I'm not sure you will be able to do it in a scheduled manner. The only way I think the ordering in a test plan is honored is when you execute it from the test plan directly. There is some specific configuration you'd have to do on the pipeline: mostly making sure that it is setup to select the tests using "Test run". Make sure you don't have the phase setup to use multiple agents.
This Microsoft documentation covers it I think:
https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
I am a fairly new engineer (about 1 year) and I was tasked with figuring out a solution to shift performance testing as far left as possible.. to fit into a DevOps model. What I am trying to achieve is:
- Add additional scripting into the existing Newman to record the response time of EACH request in the collection.
- Generate a report each time the pipeline (testing for pass or fail of API scripts) is invoked to run, and save it in ADO along with the pass/fail results.
I do NOT want to have to create a collection for each individual API, but rather have large collections run daily (ex; Web services API's, app API's, UI API's)
ANY direction or help would be super helpful, I've googled a lot but just can't come up with a lightweight solution that isn't going to negatively impact the overall testing or dev process.
In azure devops pipeline, you can use Newman the cli Companion for Postman to analyze the response time by install extension Newman the cli Companion for Postman first:
In the configuration of task:
Add your collection url which get from Postman into the
Collection Source Type:
Then fill your local Newman path and the html report export
path in the task.
Note: Don't forget install Newman locally first.
After you run the pipeline, the task will run Newman and export the
report with .html to the path you specified. As the below pic
shown(this is my report), you can see the response time and the
results of all APIs in the collection.
Since you want the report saved in ADO repos, I' afraid you need to use git command to push local .html report to Azure Devops.