Using Azure DevOps, I am creating a release pipeline in which I would like to have one stage having 1 job with 5 steps. First three steps are same task types but with different variables, and I would like to make them parallel so some flow would go like:
Job
parallel: step1, step2, step3
then: step4 (after all 3 parallel steps succeeded/failed)
then: step5 (after step4 is done)
This is the current job setup
I am not sure how to set up Control Options - run this task for all of these steps. I would need somehow to set the first three to run immediately (maybe Custom condition "always()") and step 4 and step 5 to run sequentially after previous steps are done.
Step 5 can be: Even if the previous task has failed unless the deployment was cancelled, but I am not sure if I set the same setting for step 4 will it consider only step 3 as previous or all three previous (step1 - step3) tasks.
Also for Execution plan Parallelism I guess it's ok to set multi-agent to three since I would have max 3 steps executing in parallel overall.
parallel: step1, step2, step3
If you have 5 task in one agent job, and just want to run previous three tasks parallel first, I'm afraid to say that this does not supported in Azure Devops.
While you put several tasks in one agent job, that's means the task will and must running in order. And also, as you mentioned in the second pic, the specified in Multi-agent is used for run agent job parallel, rather than run task parallel.
Fortunately, until now, there has been such suggestion raised by other user in our official Developer Community. And there are many users has the same demand with you. You can vote and comment there. Our Product Group team will take your suggestions kindly. If it has enough votes which means its high priority, and the team will consider it seriously.
Related
We are running into the following issue:
We have a job in our pipeline that runs tests. The number of tests need to be distributed over 4 agents to run optimal. It can happen that only one agent is available and the job will start to run all the load on that specific agent, which can then time-out because it takes too long for other agents to become available in time to share in the load.
In essence, if we run with 4 agents, the job will run with optimal efficiency.
My question: is it possible to let a job wait for a specific number of agents to become available before starting the tasks in the job?
That`s not possible through out-of-box features.... But you may create a simple PowerShell script that will query your agents statuses: https://learn.microsoft.com/en-us/rest/api/azure/devops/distributedtask/agents/list?view=azure-devops-rest-7.1
and use includeAssignedRequest
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/agents?includeAssignedRequest={includeAssignedRequest}&api-version=7.1-preview.1
if you see assignedRequest, your build agent is busy...
We're using SonarQube for tests, and there's one token it uses, as long as one pipeline is running, it goes fine, but if I run different pipelines (all of them have E2E tests as final jobs), they all fail, because they keep calling a token that expires as soon as its used by one pipeline (job). Would it be possible to have -all- pipelines pause at job "x" if they detect some pipeline running job "x" already? The jobs have same names across all pipelines. Yes, I know this is solved by just running one pipeline at a time, but that's not what my devs wanna do.
The best way to make jobs run one by one is set demands for your agent job to run on a specific self-hosted agent. Just as below, set a user-defined capabilities for the self-hosted agent and then require run on the agent by setting demands in agent job.
In this way, agent jobs will only run on this agent. Build will run one by one until the previous one complete.
Besides, you could control if a job should run by defining approvals and checks. By using Invoke REST API check to make a call to a REST API such as Gets a list of builds, and define the success criteria as build count is zero, then, next build starts.
I am using Windows Self hosted agent for my Azure DevOps pipelines. Currently the pipelines are executed sequentially. If more than one pipelines triggered from different ADO projects, then it has to wait in queue to get the agent. In order to execute the pipeline in parallel, I came to know from some tutorials if we increase the paid parallel jobs for self hosted agent under billing section of Organization setting. Is my understanding correct? If so what are the precautionary steps I need to take. Do we have any control of when the pipelines to be executed in parallel?
Thanks.
In order to run self-hosted parallel jobs, you need to purchase parallel jobs and register several self-hosted agents.
For parallel jobs, you can register any number of self-hosted agents in your organization. If you want to run 3 jobs in parallel, then you must register at least 3 self-hosted agents in one agent pool. DevOps charges based on the number of jobs you want to run at a time, not the number of agents registered. There are no time limits on self-hosted jobs. For private projects, you can have one job and one additional job for each active Visual Studio Enterprise subscriber who is a member of your organization.
About how to purchase parallel jobs, please refer to Buy parallel jobs.
For how to control the use of parallel jobs, please refer to the following:
For classic pipeline, you can specify when to run the job through dependencies and Run this job in Additional options in the agent job. Then the pipeline will run in sequence according to your settings.
For YAML pipeline, you can specify the conditions under which the job should run with "dependsOn" and "condition".
For example:
For more info about conditions, please refer to Specify conditions
If you don't specify a specific order, the jobs will run in parallel based on the parallel jobs you purchased.
I don't know if my experience can help. I'll try. I started a new job and we use self-hosted TFS / Azure DevOps. I am changing our build process to create 3 product SKUs (it uses conditional compilation). Let's call them Good, Better & Best.
I edited the Build definition. First I switched to the Variables tab. I created a Process variable named SKUs and set it to Good,Better,Best. The commas are important.
Next I switched to the Tasks tab. I located the Agent Phase. Mine was called Phase 1. Select it. On the right, under Parallelism, I selected Multi-configuration. In the Multipliers text field I entered SKUs. I set Maximum number of agents to 3.
What I don't yet know is the TFS back-end administration and options that the company purchased beforehand.
I have a pipeline with 6 tasks in one job, this job runs on agent01(self-hosted), there are two users, user1 and user2(more privileges) on agent01.
My requirement is, my pipeline job running on agent01 must use user1 for task 1,2,3 and use2 for task 4,5,6. How to achieve this ? please suggest.
How to switch to a different user on same agent to run a particular
task in AzureDevOps pipeline
I'm afraid this is impossible to achieve. When pipeline is running, in the Initialize job step, the user has been determined, and we cannot assign another user to the tasks.
As Shayki suggested in the comment, you can split the job into two jobs and use different agents to run.
Lets say that after every deployment I want to execute some system tests (putting a message and check if it reaches its destination).
What are my options?
I looked at post-deployment gates and the idea would be to invoke an azure function that would trigger the success, wait some time and then "assert". Is this the right way? What about timeouts since I'm going to wait (possibly for a minute or two).
In your release pipeline, you can create a first stage for deployment, and a second one for system tests.
In the system tests stage, you can choose the pre-deployment conditions to suit your requirements: e.g. after stage (and select the first deployment stage)
Optionally you can add pre-deployment approvals so somebody needs to manually approve this.