We pay for a number of Microsoft hosted build agents in Azure pipelines. We have a lot of build pipelines, where many of them do jobs in parallel.
Are there any metrics I can use to see the utilization of the build agents and even more interesting, how many jobs are in queue for a free build agent?
Since this would be for the whole Azure Devops instance the Dashboard feature doesn't seems to be appropriate because it only seems to hold project specific metrics.
Got to your Organization Settings-Parallel jobs blade. This will give you the ability to view the jobs in progress.
As for metrics there is a public preview just came out for this; however, I do not have it available yet.
Agent pool usage data is sampled and aggregated by the Analytics service every 10 mins. The number of jobs is plotted based on the max number of running jobs for the specified interval of time.
This feature is enabled by default. To try it out, follow the guidance
below.
Within project settings, navigate to the pipelines “Agent pools” tab
From the agent pool, select a pool (e.g., Azure Pipelines) Within the
pool, select the “Analytics” tab
Related
Is there any description about the algorithm by which Azure DevOps selects the next free agent?
Our scenario is that we will run multiple agents on a single vm and we will have multiple such VMs in our VMSS that we will manually scale in and out (first based on cron schedules).
In order to best utilize the available VMs we would like to make sure that the jobs are evenly distributed on all VMs.
Meaning first run only one job (on the respective agent) per VM. If there are more jobs than VMs than share the jobs so that 2 jobs per VM are processed. Then continue like that until all agents are busy.
What I want to avoid is that the first VM is loaded with jobs until its full and then the next VM.
So I am asking myself how I can influence the selection mechanism to find the next free agent.
Thank you
PS. Such a “load balancing” is already inbuilt in Jenkins.
I am using Windows Self hosted agent for my Azure DevOps pipelines. Currently the pipelines are executed sequentially. If more than one pipelines triggered from different ADO projects, then it has to wait in queue to get the agent. In order to execute the pipeline in parallel, I came to know from some tutorials if we increase the paid parallel jobs for self hosted agent under billing section of Organization setting. Is my understanding correct? If so what are the precautionary steps I need to take. Do we have any control of when the pipelines to be executed in parallel?
Thanks.
In order to run self-hosted parallel jobs, you need to purchase parallel jobs and register several self-hosted agents.
For parallel jobs, you can register any number of self-hosted agents in your organization. If you want to run 3 jobs in parallel, then you must register at least 3 self-hosted agents in one agent pool. DevOps charges based on the number of jobs you want to run at a time, not the number of agents registered. There are no time limits on self-hosted jobs. For private projects, you can have one job and one additional job for each active Visual Studio Enterprise subscriber who is a member of your organization.
About how to purchase parallel jobs, please refer to Buy parallel jobs.
For how to control the use of parallel jobs, please refer to the following:
For classic pipeline, you can specify when to run the job through dependencies and Run this job in Additional options in the agent job. Then the pipeline will run in sequence according to your settings.
For YAML pipeline, you can specify the conditions under which the job should run with "dependsOn" and "condition".
For example:
For more info about conditions, please refer to Specify conditions
If you don't specify a specific order, the jobs will run in parallel based on the parallel jobs you purchased.
I don't know if my experience can help. I'll try. I started a new job and we use self-hosted TFS / Azure DevOps. I am changing our build process to create 3 product SKUs (it uses conditional compilation). Let's call them Good, Better & Best.
I edited the Build definition. First I switched to the Variables tab. I created a Process variable named SKUs and set it to Good,Better,Best. The commas are important.
Next I switched to the Tasks tab. I located the Agent Phase. Mine was called Phase 1. Select it. On the right, under Parallelism, I selected Multi-configuration. In the Multipliers text field I entered SKUs. I set Maximum number of agents to 3.
What I don't yet know is the TFS back-end administration and options that the company purchased beforehand.
Cheers!
Maybe some of you already have done something similar.
We created a dedicated, self hosted AZ DevOps Agent Pool in one of our subscriptions with terraform.
So terraform being terraform and DevOps doing its magic with the agent pools, any major update on the scale set for now results to a recreation of the scale set with corresponding downtime. We know about the necessary ignore_changes lifecycle changes which would probably prevent that, but they are not yet implemented.
So my question is: has anyone experience how AZ DevOps reacts when you change the target Scale Set of a running Agent Pool?
Meaning just changing the target ScaleSet via the Azure DevOps Portal.
A little downtime is fine with we but we would really love to being able to deploy the new infra running parallel to the old agent set and then switch via the portal. Like a standard Blue/Green deployment scheme.
Also having a fallback to the old agent pool would be a major bonus.
As long as an Agent Pools doesn't support more than 1 scale sets that seemed to be the most viable solution.
Anyone here ever tried anything like this?
Thanks!
To answer my own question:
We just pulled the plug and switched over to a new Scale Set.
The downtime is immediately, because DevOps scales the "old" scale set to 0 right after.
After approximately 10-15 minutes Azure DevOps started to scale the new instances up and added them to the agent pool.
So in a nutshell: Blue/Green deployment of the Scale Set worked basically. You can schedule new jobs while the agents are down but running at the time of the switch are terminated as the instances are deleted right away
I am trying to invoke the multiple releases definition using REST API.Also enabled multiple agents for each Agent job. But even after triggering multiple releases the second release is in Queue and not at all starting. Is there any way to start the deployment parallelly from a single release defition.
Parallel jobs have different restrictions depending on the agents you use and your project is public or private.
Microsoft-hosted agent
If your jobs run on the pool of Microsoft-hosted agents. Microsoft provides a free tier of service by default in every organization:
Public project: 10 free Microsoft-hosted parallel jobs that can run
for up to 360 minutes (6 hours) each time, with no overall time limit
per month.
Private project: One free job that can run for up to 60 minutes each
time, until you've used 1,800 minutes (30 hours) per month.
Note:When you purchase your first Microsoft-hosted parallel job, the number of parallel jobs you have in the organization is still 1. To be able to run two jobs concurrently, you will need to purchase two parallel jobs if you are currently on the free tier. The first purchase only removes the time limits on the first job.
Self-hosted agent
To use self-hosted parallel jobs, you need to deploy self-hosted agents on your machines. You can register any number of these self-hosted agents in your organization. Microsoft charges based on the number of jobs you want to run at a time, not the number of agents registered.
Public project: Unlimited parallel jobs.
Private project: One self-hosted parallel job. Additionally, for each
active Visual Studio Enterprise subscriber who is a member of your
organization, you get one additional self-hosted parallel job.
For private project,when the free tier is no longer sufficient, you can pay for additional capacity per parallel job. Buy self-hosted parallel jobs.There are no time limits on self-hosted jobs.
These are stated in the document, you can refer to the link in Daniel's comment for details.
I have several self-hosted Azure DevOps agents (each installed on a dedicated on-prem server) and I need to perform reoccurring maintenance on them (i.e. patching, etc.). Is there's a good way to define those maintenance windows within Azure DevOps so that server admins could do their job without worrying to interrupt any ongoing build/release task?
There seem to be a setting related to configuring reoccurring maintenance (Organization Settings -> Agent Pools -> <Pool Name>-> Settings [tab]) but it seems as if it would apply to the whole pool and it's hard to tell which of the agents will be considered offline at which time slot.
Unfortunately I couldn't find any documentation about it and not sure if there's something that Azure DevOps would also be doing on the agent machines (i.e. running cleanup, updating agents and so on)
Currently, the process involves a person with admin permissions in Azure DevOps to disable an agent allowing a server admin to perform regular maintenance and to re-enable it back when server admin is done. It would be great if a server admin could not involve an Azure DevOps Admin every time for such routines.
Due to the fact that you have your own Azure Pipelines agents, then the maintenance should be easier and you will have total control of either having automatic maintenance or not. If you use Microsoft's hosted agents, you could not update the hosted agents from Microsoft because these agents are maintained by Microsoft exclusively.
The best way to do this is by having more than one agent on one machine instance then organize the agents on one pool. If you have multiple pools, then you can configure Azure DevOps to have different maintenance window schedule on each pool to have different time, and give some time to download and configure itself.
For example, I usually configure the maintenance window on weekend days such as Sunday early morning once a month on certain date. And for any pools I have I gave them intervals of 40 minutes on each pool to have maintenance to give enough time for the agent to download, update and restart itself.
Please consult these documentation further for detailed explanation and use cases:
For Azure DevOps Server:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops-2019
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops-2019
For Azure DevOps Service (on cloud TFS, formerly Visual Studio Team Services):
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops