Azure DevOps: Run tests parallely on agents in agentpool with different run settings - azure-devops

Question

We have setup a agentpool with 3 agents tagged to it for running tests in parallel. We would like to use various input values for .runsettings file to override test run parameters (overrideTestrunParameters) & distribute our test runs on various agents. e.g.,
Lets assume that the agentpool P1 have associated agents A1, A2, A3.
We need agent A1 to configure a test run parameter executeTests = Functionality1, agent A2 to configure a test run parameter executeTests = Functionality2 etc.,
Please let us know if it is possible to use executionPlan with options Multiagent or Multi Configuration to achieve it.

Answer 1

So if I did not misunderstand, what you want is run the tests with multiple-configuration into multi-agents?
If yes, I'd better suggest you could apply with matrix in pipeline to achieve what you want.
*Note: Matrix is the new feature that only support YAML pipeline. If you want to make use matrix in your side, you had to use YAML to configure your pipeline.*
For how to apply matrix in this scenario, you could refer to below simple sample:
strategy:
matrix:
execTest1:
agentname: "Agent-V1"
executeTests: "Functionality1"
execTest2:
agentname: "Agent-V2"
executeTests: "Functionality2"
execTest3:
agentname: "Agent-V3"
executeTests: "Functionality3"
maxParallel: 3
pool:
name: '{pool name}'
demand:
- agent-name -equals $(agentname)
...
...
With such YAML definition, it can run the job at same time and with different configuration. Also, different configuration run onto the specified agent.
Note: Please ensure your project support parallel consuming.
For more details, see this.

Answer 2

I was able to find a solution for my case here by doing below
Add a variable group in the pipeline named executeTests & assigning names, values for the respective variable group as Functionality1, Functionality2 etc.,
Added multiple agent jobs in the same pipeline & assigned the Override test run parameters with -(test.runsetting variable) $(Functionality1) etc across agents A1, A2, A3
The above does run tests parallelly based on the settings available at each agent job

Answer 3

Using different runsettings or even override settings is not supported. The test task expects it to be consistent across all the agents. It will use whichever is configured for the first to start the test task. For example, if you were to pass an override variable $(Agent.Name), it would use the first agent name regardless of which agent picked it up.
The only way we found to manage this was to handle it in our test framework code. Instead of loading from runsettings, we set environment variables on the agent in a step prior to the test task. Then our test framework will load from the environment variable.