What are the differences between an Agent Job and a Deployment Group Job in Azure DevOps? What are the reason to create one or the other?
What are the differences between an Agent Job and a Deployment Group
Job in Azure DevOps?
Agent job:
Run steps on an agent which in an agent pool.
Deployment group jobs:
Run on machines in a deployment group.
These are the definition of them. You can see that, the fundamental difference between them is that the target when running the job are different.
For agent job, it can only run on one target at a time (unless set up parallel to run on multiple targets at a time, but parallel is essentially multiple jobs). And deployment group job is, since the deployment group is multiple machines are bound in a group, it can run a job on multiple machines at a time.
In the usage scenario, Agent job can used in both Build and Release pipeline. But for Deployment agent job, it can only be used in Release pipeline for application/project deploy.
What are the reason to create one or the other?
In build pipeline, it should no doubt that you can only use Agent job (or Agentless) job.
I think what you concerned should be the usage in Release pipeline. As I mentioned above, these different jobs can all be used in Release pipeline, and they can all be used for project deployed.
But in terms of specific use, it depends on the task you will use and the number of target servers you want to deploy to.
Agent job:
If your deployment target server number less than 5 objects, and need to deploy to multiple machines at the same time, you can set up a parallel job for Agent job. The Agent job may take a little longer time than Deployment group job. But because the number of deployed targets is not too much, the difference is not obvious.
Deployment group job:
For medium and large companies, the Deployment target objects are generally more than 10, even 100. It is most appropriate to use Deployment group job, because it can be deployed on different machines in one job.
In release, recommend you use Deployment group job if you have multiple targets to deploy to:
Related
I am using Windows Self hosted agent for my Azure DevOps pipelines. Currently the pipelines are executed sequentially. If more than one pipelines triggered from different ADO projects, then it has to wait in queue to get the agent. In order to execute the pipeline in parallel, I came to know from some tutorials if we increase the paid parallel jobs for self hosted agent under billing section of Organization setting. Is my understanding correct? If so what are the precautionary steps I need to take. Do we have any control of when the pipelines to be executed in parallel?
Thanks.
In order to run self-hosted parallel jobs, you need to purchase parallel jobs and register several self-hosted agents.
For parallel jobs, you can register any number of self-hosted agents in your organization. If you want to run 3 jobs in parallel, then you must register at least 3 self-hosted agents in one agent pool. DevOps charges based on the number of jobs you want to run at a time, not the number of agents registered. There are no time limits on self-hosted jobs. For private projects, you can have one job and one additional job for each active Visual Studio Enterprise subscriber who is a member of your organization.
About how to purchase parallel jobs, please refer to Buy parallel jobs.
For how to control the use of parallel jobs, please refer to the following:
For classic pipeline, you can specify when to run the job through dependencies and Run this job in Additional options in the agent job. Then the pipeline will run in sequence according to your settings.
For YAML pipeline, you can specify the conditions under which the job should run with "dependsOn" and "condition".
For example:
For more info about conditions, please refer to Specify conditions
If you don't specify a specific order, the jobs will run in parallel based on the parallel jobs you purchased.
I don't know if my experience can help. I'll try. I started a new job and we use self-hosted TFS / Azure DevOps. I am changing our build process to create 3 product SKUs (it uses conditional compilation). Let's call them Good, Better & Best.
I edited the Build definition. First I switched to the Variables tab. I created a Process variable named SKUs and set it to Good,Better,Best. The commas are important.
Next I switched to the Tasks tab. I located the Agent Phase. Mine was called Phase 1. Select it. On the right, under Parallelism, I selected Multi-configuration. In the Multipliers text field I entered SKUs. I set Maximum number of agents to 3.
What I don't yet know is the TFS back-end administration and options that the company purchased beforehand.
I have multiple local on-prem agents that can run a particular deployment group job (for the purpose of load balancing). I want only the first available one to run the job and not all of them. Which setting of the Deployment Group Job I can use to do that? My only options seems to be "Multiple" and "Single at a time" both of which run the jobs on all servers matching the Required Tags.
I want only the first available one to run the job and not all of them. Which setting of the Deployment Group Job I can use to do that?
There is an option for the Release pipeline with Deployment Group job Required tags：
Then we just need to add the tag to the machine, which we want to deploy:
Now, the release pipeline will run a Deployment Group Job in one of the matching servers.
Update:
I do have the required tag, but on two agent servers. I want the Job
to pick only one of the two matching servers and run the job on it
instead of running it on both.
As workaround, you could create a private agent pool and add two or more agents in it. Those agents are deployed on different machines. With this way, the pipeline will executed on one of the agent.
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops&viewFallbackFrom=vsts
I'm recreating our very complicated deployment process in Azure DevOps.
Currently, our live environment is composed of 4 VMs behind a load balancer.
The strategy I thought about is kind of blue-green deployment, just that I'll be working on half (2) of the VMs each time.
It's not like rolling deployment since there will not be 2 versions that run simultaneously.
As soon as I finish deploying the first half and I make sure it's up and running, I'll connect it and immediately disconnect the other half to deploy the new version to it too.
I can't do blue-green for now by the book (deploying a complete separate set of VMs in the new version) since unfortunately IPs are hard-coded. The live environment will be just fine with 2 VMs only for the deployment time.
My problem is how to configure a release pipeline in Azure DevOps for this kind of solution (I use the GUI version, not YAML)?
(these are stages)
DbConversionDev > FirstHalfDev > SecondHalfDev > DbConversionQA > FirstHalfQA > SecondHalfQA > DbConversionProd > FirstHalfProd > SecondHalfProd
Maybe the db conversion stage can be parallel.
It's seems complicated and hard to maintain.
Any suggestions?
Thanks in advance :)
what you could try for example is to create deployment groups, e.g. First Group and Second Group - register your agents/VMs to the groups, and in the release pipeline, create Deployment Group job for the First Group, then add the steps to verify the deployment was successful and up/running, and in the same pipeline create another Deployment Group job for the second Deployment Group. (Note: You can create an agentless job for any manual intervention)
EDIT
Here is the complete solution as a summary of the conversation:
Create one Deployment Group for each Environment (e.g. Dev, QA, Prod) and tag the first half of the Agents/VMs in a Deployment Group as blue and the other half as green.
In the release pipeline, create a Stage for each Environment (e.g. Dev, QA, Prod) and in one Stage create 2 Deployment Group Jobs for the corresponding Deployment Group (Dev Stage -> Dev Deployment Group), each Job with a different Tag: blue and green. This will make sure, that in one job half of your VMs will be affected by the deployment.
For reusability purposes, group your Tasks in Task Groups to reuse them in your Deployment Group Jobs.
I have a build with multiple jobs, where they depend on each other's output. But I also have multiple agents, which gives me the following issue:
If Agent1 runs Job1, Agent2 runs Job2, and Job3 requires the output from both Job1 and Job2, I can't access the files from just one agent, since they are located on different machines.
How do I make my jobs able to download the output of other agents?
I looked for the workspace on MS Docs, but it doesn't describe how to handle this scenario.
To add more details on top of JukkaK's answer.
I looked for the workspace on MS Docs, but it doesn't describe how to
handle this scenario.
The workspace is something corresponding to agents. No sure which kind of agent do you use, but different agents have different OS instance, so the content under same path(workspace) in one agent should be quite different from that in another agent.
So workspace is not the approach for you needs.
How do I make my jobs able to download the output of other agents?
You can use Publish Artifacts+Download Artifacts combination to do what you need. See this:
You can place Publish build Artifacts task as the last task of agent job1 and job2. Then add a Download buil Artifacts as the first one of agent job3.
And make sure agent job3 depends on agent job1 and job2 like this:
In this way, the output from agent job1 and job2 can be installed in agent job3's machine for further usage. Hope it helps.
Pipeline artifacts in multi-stage pipelines would be a perfect match for this, if the current features available with multi-stage pipelines otherwise satisfy your needs.
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml
If not, the best I can come up with is directing the the jobs to same agent by adding a capability to agent and adding a demand to the pool-assignment (or by creating your own pool). With Deployment group agents, adding tags is a handy way to direct jobs to a certain agent in deployment group, but haven't found anything similar on build agents.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml
I'm trying to re-use a Deployment Group Job and execute it for each member of a ClientID list, but it is not working. The name of the stage will also be used in my tasks.
As described on this page...
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=classic
I've create two variables:
ClientID
client1, client2, client3
Multipliers
ClientID
The stage only has a Deployment Group Job and not an Agent Job because after the deploy I have tasks to run on the self-hosted servers. I need to stop and remove a service, copy files, register and restart a service.
There are no other options on the Stage or the Deployment Group Job for multipliers. (There is the "multiple" options for parallel deployments to servers within the group but that is unrelated as far as I know.)
If I add an Agent Job then I can see that it has the "Multi-configuration" and "Multipliers" options that other articles mention. On a deploy the agent job executes multiple times as expected but not the deployment group job.
The article above talks about "Multi-configuration deployments", so I'm I wrong to think the Deployment Group Job should work that way?
Any help is appreciated. Thanks.
I am afraid that using multipliers in the deployment groups is not feasible.
According to the definition of Multi-configuration in the agent job : each configuration will use a single agent.
But, in effect, a deployment group is just another grouping of agents, much like an agent pool.It could not assign a single agent to each configuration.
So I think that multipliers are not available in the deployment groups.If you want to use multipliers, then you need to use agent job.