I have a use case to run test project to run test cases in specific sequence with in TFS release pipeline.
I see that Ordered test and Playlist are not an option any more what would be the process for this.
I'm not sure you will be able to do it in a scheduled manner. The only way I think the ordering in a test plan is honored is when you execute it from the test plan directly. There is some specific configuration you'd have to do on the pipeline: mostly making sure that it is setup to select the tests using "Test run". Make sure you don't have the phase setup to use multiple agents.
This Microsoft documentation covers it I think:
https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
Related
I have all my test cases in Azure Devops Test Plans and now I'm executing them manually against a build.
How do I view manual test results by build?
I can view and filter test results by build on the "Runs" page but I would prefer to see test results grouped by build not filtered to a single build.
Any options for this?
I had hoped the the Test Plans - Progress Report would provide this but I don't see a way.
Maybe there is a way to group all my manual tests executions into a Test Run?
I researched and attempted with the related Azure DevOps REST API and Azure CLI, however, I did not find any available way.
It seems that we can only directly filter the test results to the build run level, and cannot to the build pipeline level.
Maybe, you can indirectly get the associated build pipeline via the following way:
Use the REST API "Runs - Query" to list the test runs associated with a specified build pipeline.
Use the REST API "Results - List" in a loop to list the test results of each test run.
I'm currently using cTest to publish test results from google test into azure.
Is there any way i can decorate my test so it comes up already linked to a requirement. without manually having to link it in azure?
Edit:
I'm using gtest recordproperty to add data to the xml output of the test case. I have tried with #1234( as in a commit), 1234 and several combinations. I cannot find anything in the documentation.
Currently, you could refer to this doc: Run automated tests from test plans to automate test cases in your test plans and run them directly from Azure Test Plans. And then open the Test Plans page and select the test plan to see the status of your tests if tests are updated after test execution is complete.
In addition, if you turn on below “Automatically link new work in this build” option in build pipeline page, and then you can link work item to Git commit, the new build queued by this commit will link to this work item when it completes successfully.
Also, you could enable “Create work item on failure” option to create a new work item when this build is failed.
BTW, you could link work items to deployments refer to this doc.
Whenever multiple builds are running on the same pipeline only the first build completes and the rest error.
I would the builds to run sequentially instead of in parallel like they are now, so that if two developers check-in at about the same time, the second build will also complete.
When editing the pipeline in the execution plan under the Parallelism heading I have the radio button set to "none" and they still seem to try to run in parallel anyway.
Can anyone suggest how to solve this issue?
enter image description here
It seems you have multiple build agents. Assuming you are using self-hosted build agents, you could specify certain demands of the agent to use only one agent. In this way, if the agent is not free, the build will keep waiting. To use a particular agent, add a demand of Agent.Name equals agentname, check the screenshot below. Agent name can be found in capabilities of the agent.
I have a build process in which I have a couple of Tests Tasks. Some of them may become quite time consuming when they all run and most of the time, most tests are not expected.
Still, I would like to have ALL these tests run on a scheduled trigger.
I know I could simply clone the pipeline and use one for gating with impacted tests only and the other one for schedule with all tests but as an OO developer, I don't like this.
I already tried linking the checkbox parameter to a process variable and modifying it using PowerShell but failed to have it work (How can I modify a process variable using Powershell in a Azure build pipeline).
Isn't there any other way of doing this?
You may be able to do this by setting the following condition on the test tasks that you'd only like to run during the scheduled build:
eq(variables['Build.Reason'], 'Schedule')
See here for a list of predefined variables (search for 'Build.Reason'):
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml
See here for more information on expressions:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
It looks like functionality for this is now built in. According to the docs a variable can be set which will cause all tests to be run:
By setting a build variable. Even after TIA has been enabled in the VSTest task, it can be disabled for a specific build by setting the variable DisableTestImpactAnalysis to true. This override will force TIA to run all tests for that build. In subsequent builds, TIA will go back to optimized test selection
This question concerns use of the Jenkins Workflow plugin and "synchronizing" a stage amongst independent jobs.
We have a generic workflow for multiple projects with steps:
build project
push project to test environment
run (long) end-to-end test suite
push project to production
Step 3 runs a long time. If multiple projects are built and pushed to the test environment within the same window of time, we'd like to only run once the end-to-end test suite.
Can we have the jobs some how synchronize on step 3?
The desired orchestration can be achieved by make Step 3 a build action. I.e.
build end-to-end-tests
Where end-to-end-tests is a job dedicated to running the slow end-to-end tests.
Adding a Quiet period to end-to-end-tests supports the goal of "collecting" projects updated within a time period to end-to-end test. That is, if project A and B are pushed to the test environment with Quiet period seconds, then end-to-end-tests runs only once.
JENKINS-30269 might be helpful, but your use case is indeed subtly different from the usual one that RFE would solve; you really seem to need a cross-job stage, which is not currently possible though in principle such a step could be written. In the meantime, a downstream deployment job is probably the most reasonable workaround.