I'm learning about azure pipelines. By default you get 1 free parallel job for x number of minutes.
A pipeline contains number of tasks. And atleast 1 job. All the tasks in the pipeline (across multiple jobs) run on the same agent?
Does 1 parallel job means 1 pipeline execution containing 2 or more jobs? or only 2 jobs?
No. Each job will run on a new agent.
1 parallel job means that one job can run at a time. Two parallel jobs means that two parallel jobs can run at a time, each on a separate agent. And so on.
Related
We are running into the following issue:
We have a job in our pipeline that runs tests. The number of tests need to be distributed over 4 agents to run optimal. It can happen that only one agent is available and the job will start to run all the load on that specific agent, which can then time-out because it takes too long for other agents to become available in time to share in the load.
In essence, if we run with 4 agents, the job will run with optimal efficiency.
My question: is it possible to let a job wait for a specific number of agents to become available before starting the tasks in the job?
That`s not possible through out-of-box features.... But you may create a simple PowerShell script that will query your agents statuses: https://learn.microsoft.com/en-us/rest/api/azure/devops/distributedtask/agents/list?view=azure-devops-rest-7.1
and use includeAssignedRequest
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/agents?includeAssignedRequest={includeAssignedRequest}&api-version=7.1-preview.1
if you see assignedRequest, your build agent is busy...
I have a pipeline with 6 tasks in one job, this job runs on agent01(self-hosted), there are two users, user1 and user2(more privileges) on agent01.
My requirement is, my pipeline job running on agent01 must use user1 for task 1,2,3 and use2 for task 4,5,6. How to achieve this ? please suggest.
How to switch to a different user on same agent to run a particular
task in AzureDevOps pipeline
I'm afraid this is impossible to achieve. When pipeline is running, in the Initialize job step, the user has been determined, and we cannot assign another user to the tasks.
As Shayki suggested in the comment, you can split the job into two jobs and use different agents to run.
We have deployed Azure devops Self-hosted Agent in our own agentpool.
We have also purchased 4 parallel job per agent , but when our build runs, it runs only one job at a time and other pipeline/release jobs have to wait.
How can we run 4 parallel pipeline/release jobs ?
How can we run 4 parallel pipeline/release jobs ?
According to your screenshot, you seem to only register one self-hosted agent, and running a parallel job will consume one agent, so if you want to run 4 jobs in parallel, you need to register at least 4 self-hosted agents.
Running parallel jobs on the same agent can be achieved by container jobs
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops-2019#multiple-jobs-with-agent-pools-on-a-single-hosted-agent
There is a small gotcha around being logged out when multiple jobs are running at the same time and a Docker workaround suggested here
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops-2019#multiple-jobs-with-agent-pools-on-a-single-hosted-agent
Using Azure DevOps, I am creating a release pipeline in which I would like to have one stage having 1 job with 5 steps. First three steps are same task types but with different variables, and I would like to make them parallel so some flow would go like:
Job
parallel: step1, step2, step3
then: step4 (after all 3 parallel steps succeeded/failed)
then: step5 (after step4 is done)
This is the current job setup
I am not sure how to set up Control Options - run this task for all of these steps. I would need somehow to set the first three to run immediately (maybe Custom condition "always()") and step 4 and step 5 to run sequentially after previous steps are done.
Step 5 can be: Even if the previous task has failed unless the deployment was cancelled, but I am not sure if I set the same setting for step 4 will it consider only step 3 as previous or all three previous (step1 - step3) tasks.
Also for Execution plan Parallelism I guess it's ok to set multi-agent to three since I would have max 3 steps executing in parallel overall.
parallel: step1, step2, step3
If you have 5 task in one agent job, and just want to run previous three tasks parallel first, I'm afraid to say that this does not supported in Azure Devops.
While you put several tasks in one agent job, that's means the task will and must running in order. And also, as you mentioned in the second pic, the specified in Multi-agent is used for run agent job parallel, rather than run task parallel.
Fortunately, until now, there has been such suggestion raised by other user in our official Developer Community. And there are many users has the same demand with you. You can vote and comment there. Our Product Group team will take your suggestions kindly. If it has enough votes which means its high priority, and the team will consider it seriously.
I have few questions about running tasks in parallel in Azure Batch. Per the official documentation, "Azure Batch allows you to set maximum tasks per node up to four times (4x) the number of node cores."
Is there a setup other than specifying the max tasks per node when creating a pool, that needs to be done (to the code) to be able to run parallel tasks with batch?
So if I am understanding this correctly, if I have a Standard_D1_v2 machine with 1 core, I can run up to 4 concurrent tasks running in parallel in it. Is that right? If yes, I ran some tests and I am quite not sure about the behavior that I got. In a pool of D1_v2 machines set up to run 1 task per node, I get about 16 min for my job execution time. Then, using the same applications and same parameters with the only change being a new pool with same setup, also D1_v2, except running 4 tasks per node, I still get a job execution time of about 15 min. There wasn't any improvement in the job execution time for running tasks in parallel. What could be happening? What am I missing here?
I ran a test with a pool of D3_v2 machines with 4 cores, set up to run 2 tasks per core for a total of 8 tasks per node, and another test with a pool (same number of machines as previous one) of D2_v2 machines with 2 cores, set up to run 2 tasks per core for a total of 4 parallel tasks per node. The run time/ job execution time for both these tests were the same. Isn't there supposed to be an improvement considering that 8 tasks are running per node in the first test versus 4 tasks per node in the second test? If yes, what could be a reason why I'm not getting this improvement?
No. Although you may want to look into the task scheduling policy, compute node fill type to control how your tasks are distributed amongst nodes in your pool.
How many tasks are in your job? Are your tasks compute-bound? If so, you won't see any improvement (perhaps even end-to-end performance degradation).
Batch merely schedules the tasks concurrently on the node. If the command/process that you're running utilizes all of the cores on the machine and is compute-bound, you won't see an improvement. You should double check your tasks start and end times within the job and the node execution info to see if they are actually being scheduled concurrently on the same node.