After reading the very short official documents about the azure pipeline agent, I am getting very confused.
What exactly is an azure pipeline agent?
What is an agent job?
What's the relationship between agent and VM?
What's the relationship between agent job and VM? For each agent， one VM will be temporally assigned to it and will be back to the pool after the agent job finished?
If 2 different agent jobs run by 2 agents need the same running environment and the VM is agent job dependent. How should I retain the first agent job's running environment after it's finished running? Recreated again?
If each agent needs a VM, why create this concept? why not just directly use the VM or container?
Pipeline agent is machine where your build is performed. An agent is installable software that runs one job at a time.
Agent job is a set of steps which is recognized as execution boundary. Each job runs on an agent. All of the steps run together on the same agent.
From that perspective you can distinguish two kind of jobs - the onces installed on VM and onces instaled on container.
Agent job runs on agent which can be installed on VM. VM's are not assigned. Agents are assigned. There is agent pool, not VM pool.
I don't understand this one. Agents after finishing their job are going back to pool.
You may have more agents on on VM for instance one agents is installed on VM and few others as containers.
Please take a look here. You will find explanation for these concepts.
Related
Is there any description about the algorithm by which Azure DevOps selects the next free agent?
Our scenario is that we will run multiple agents on a single vm and we will have multiple such VMs in our VMSS that we will manually scale in and out (first based on cron schedules).
In order to best utilize the available VMs we would like to make sure that the jobs are evenly distributed on all VMs.
Meaning first run only one job (on the respective agent) per VM. If there are more jobs than VMs than share the jobs so that 2 jobs per VM are processed. Then continue like that until all agents are busy.
What I want to avoid is that the first VM is loaded with jobs until its full and then the next VM.
So I am asking myself how I can influence the selection mechanism to find the next free agent.
Thank you
PS. Such a “load balancing” is already inbuilt in Jenkins.
I have a release pipeline composed of several stages
The tfs server have only one worker.
When i start a release, the worker run stages randomly.
The problem is : if someone else start another pipeline, sometimes the worker take the other pipeline before getting back to the next stage.
Is there a way to lock the worker on the entire release pipeline ?
When running a pipeline, a job is the unit of scale. This means each job can potentially run on a different agent.
Each job runs on an agent. A job represents an execution boundary of a set of steps. All of the steps run together on the same agent.
Next to that,
A stage is a logical boundary in the pipeline. It can be used to mark separation of concerns (for example, Build, QA, and production).
More information: Azure Pipelines - Key concepts.
You should also have a look at the Pipeline run sequence. It clearly explains how the entire pipeline-process works.
Whenever Azure Pipelines needs to run a job, it will ask the pool for an agent.
If the availability of the agent is the issue, you might want to add more agents to the pool. If needed, you can even run multiple agents on the same machine.
Although multiple agents can be installed per machine, we strongly suggest to only install one agent per machine. Installing two or more agents may adversely affect performance and the result of your pipelines.
I have set up a multi-agent job for an Azure Release Pipeline. There are two agents in the agent pool. The job needs to be executed by every agent in the agent pool.
The settings shown in the above schedule two agent jobs whenever a release is triggered. If both agents are idle while the deployment starts, everything works as expected and both agents execute the job. But, as soon as one agent is busy at that time the behavior becomes unexpected and both jobs are executed by the same agent consecutively.
How can I ensure that every agent of the agent pool is executing the defined agent job?
This is as designed. If one agent pool has two agents A and B, while A is busy, the build job will running with B. So in pool, if some agents are busy, the pipeline will running with other free and available agent.
The precondition of Parallel job is that you must has enough agents free and available. That's why as soon as one agent is busy at that time the behavior becomes unexpected.
I have installed agent on VM and configured a CI build pipeline. The pipeline is triggered and works perfectly fine.
Now I want to use same build pipeline, same agent, but different VM. Is this possible?
How will the execution happen for builds and on which VM will the source be copied?
Thank you.
Like the others I'm also not sure what you're trying to do and also think that the same agent across multiple machines is not possible.
But if you have to alternate or choose easily between VMs, you could set up for each of your VMs (used for this special scenario) an individual agent queue with one agent in that pool. That way you can choose the agent pool at queue time via the agent queue dropdown field. But that would only work if you're triggering manually, not in a typical CI scenario. In that case you would have to edit the definition to enforce any particular VM each time you want to swap VMs.
NO. These private agents are supposed to have a unique name and are assigned to an Agent Pool/Queue. They are polling up to VSTS/Azure Devops server if they have a job to do. Then they execute it. If you clone a machine with the same private build agent, then theoretically the agent that picks it up will execute the job, but that is theoretic. I really don't know how the Agent Queues will handle this.
It depends on what you want to do.
If you want to spread the workload, like 2 build servers and have builds go to whichever build server isn't busy, then you would create 1 Agent Pool/Queue. Create a Private Agent on one server and register it to that Pool, then on the second server un-register the agent and then re-register the agent add it to the SAME pool.
If you want to do work on 2 servers at the exact same time, like a deployment to 2 servers at the same time, then you would create a 'Deployment Group' and add both servers to that. You would unregister both agents from the Agent Pool/Queue. From your 'Deployment Group' copy the PowerShell script snippet and run it on each machine. This way you can use this in your Release Pipeline and deployments in parallel, which take less time to do deployments.
You could set up a variable in the pipeline so you can specify the name of the VM at build-time.
Also, once you have one or more agents, you would add them to an app pool. When builds are run, it will choose one agent from the pool and use that.
I'd like to enhance the release definition so that I don't need to have a separate environment that only starts an Azure VM.
If we take a scenario where we have a Test, Beta, Production environments. The client wants the application to be installed in Beta and Production on their local network. We internally want a Test environment to run E2E tests against, allow for non-technical folks to exercise the app without needing VPN access to the customer beta environment, etc.
So here we have Environment followed by where the Agent is running:
Test - Azure VM
Beta - Client machine
Production - Client machine
How we've solved this is to install the VSTS Agent on a machine at the client, which allows us to target that agent queue in the Beta and Production environments defined for that release. Then we typically build an Azure VM and target that agent queue for the Test environment.
We don't want to run that Azure VM 24/7/365. However if it's not running, then it can't respond to requests from Release Management.
What I've done is to create a environment named Start Test VM and Stop Test VM that use the Azure Resource Group Deployment to start and stop the VM. Those 2 additional environments can have their agent queue set to Hosted.
I'd like to figure out how to combine the first 3 environments into a logical Test instead of having to create 3 release management environments.
Start Test VM - Hosted
Test - Azure VM
Stop Test VM - Hosted
Beta - Client machine
Production - Client machine
The problem is that can be rather ugly and confusing when handing this over to one of our PM's or even myself when I circle back around 3 months later and think, "What the hell is this environment? Oh it's just there to start/stop the VM."
Options:
Stay with status quo - keep it like it is, it can't be fixed
We could open up a port on the Azure VM and use Powershell remoting. Then run on the Hosted agent or on an on-premise agent to start the VM, then deploy the application, then stop the VM. - we really dislike this because the deployment would not be the same as the client on-premise deploy. We'd like each environments' tasks to be the same, just with different variables.
You can use "Azure PowerShell" and "Azure SQL Database Deployment" tasks to configure your Azure VM and SQL or call other script to run on the Azure VM.
There isn't any way to set the agent for tasks. You can submit a feature request on VSTS User Voice for this.
And another way to reduce the environment is that: if you deploy every build linked to the release, then you can add "Start Test VM" task into your build definition to start the VM when build is successful and add "Stop Test VM" task into "Beta" environment.
What we've currently settled on is to continue with having an environment that isn't really what I would consider an environment, but more of a stage in the release pipeline that starts and/or shuts down a VM. We run that on a hosted agent so it can start the VM and make sure to check Skip artifacts download on the environment.
For a continuous integration build, we set a chain so the VM gets started, CI environment gets kicked off and then VM gets stopped. The remaining environments are then manually deployed up the chain as desired.
So here's an example:
Start CI VM
CI
Stop CI VM
Beta
Production
And here's an image of how it looks in Release Management as of 2016.06.27:
I put single quotes around environment because I think I agree with this user voice request in that, it's really more of a stage in the release pipeline. Much like database development, the logical and the physical don't necessarily map 1 to 1.