I have several self-hosted Azure DevOps agents (each installed on a dedicated on-prem server) and I need to perform reoccurring maintenance on them (i.e. patching, etc.). Is there's a good way to define those maintenance windows within Azure DevOps so that server admins could do their job without worrying to interrupt any ongoing build/release task?
There seem to be a setting related to configuring reoccurring maintenance (Organization Settings -> Agent Pools -> <Pool Name>-> Settings [tab]) but it seems as if it would apply to the whole pool and it's hard to tell which of the agents will be considered offline at which time slot.
Unfortunately I couldn't find any documentation about it and not sure if there's something that Azure DevOps would also be doing on the agent machines (i.e. running cleanup, updating agents and so on)
Currently, the process involves a person with admin permissions in Azure DevOps to disable an agent allowing a server admin to perform regular maintenance and to re-enable it back when server admin is done. It would be great if a server admin could not involve an Azure DevOps Admin every time for such routines.
Due to the fact that you have your own Azure Pipelines agents, then the maintenance should be easier and you will have total control of either having automatic maintenance or not. If you use Microsoft's hosted agents, you could not update the hosted agents from Microsoft because these agents are maintained by Microsoft exclusively.
The best way to do this is by having more than one agent on one machine instance then organize the agents on one pool. If you have multiple pools, then you can configure Azure DevOps to have different maintenance window schedule on each pool to have different time, and give some time to download and configure itself.
For example, I usually configure the maintenance window on weekend days such as Sunday early morning once a month on certain date. And for any pools I have I gave them intervals of 40 minutes on each pool to have maintenance to give enough time for the agent to download, update and restart itself.
Please consult these documentation further for detailed explanation and use cases:
For Azure DevOps Server:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops-2019
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops-2019
For Azure DevOps Service (on cloud TFS, formerly Visual Studio Team Services):
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops
Related
I am using Windows Self hosted agent for my Azure DevOps pipelines. Currently the pipelines are executed sequentially. If more than one pipelines triggered from different ADO projects, then it has to wait in queue to get the agent. In order to execute the pipeline in parallel, I came to know from some tutorials if we increase the paid parallel jobs for self hosted agent under billing section of Organization setting. Is my understanding correct? If so what are the precautionary steps I need to take. Do we have any control of when the pipelines to be executed in parallel?
Thanks.
In order to run self-hosted parallel jobs, you need to purchase parallel jobs and register several self-hosted agents.
For parallel jobs, you can register any number of self-hosted agents in your organization. If you want to run 3 jobs in parallel, then you must register at least 3 self-hosted agents in one agent pool. DevOps charges based on the number of jobs you want to run at a time, not the number of agents registered. There are no time limits on self-hosted jobs. For private projects, you can have one job and one additional job for each active Visual Studio Enterprise subscriber who is a member of your organization.
About how to purchase parallel jobs, please refer to Buy parallel jobs.
For how to control the use of parallel jobs, please refer to the following:
For classic pipeline, you can specify when to run the job through dependencies and Run this job in Additional options in the agent job. Then the pipeline will run in sequence according to your settings.
For YAML pipeline, you can specify the conditions under which the job should run with "dependsOn" and "condition".
For example:
For more info about conditions, please refer to Specify conditions
If you don't specify a specific order, the jobs will run in parallel based on the parallel jobs you purchased.
I don't know if my experience can help. I'll try. I started a new job and we use self-hosted TFS / Azure DevOps. I am changing our build process to create 3 product SKUs (it uses conditional compilation). Let's call them Good, Better & Best.
I edited the Build definition. First I switched to the Variables tab. I created a Process variable named SKUs and set it to Good,Better,Best. The commas are important.
Next I switched to the Tasks tab. I located the Agent Phase. Mine was called Phase 1. Select it. On the right, under Parallelism, I selected Multi-configuration. In the Multipliers text field I entered SKUs. I set Maximum number of agents to 3.
What I don't yet know is the TFS back-end administration and options that the company purchased beforehand.
We pay for a number of Microsoft hosted build agents in Azure pipelines. We have a lot of build pipelines, where many of them do jobs in parallel.
Are there any metrics I can use to see the utilization of the build agents and even more interesting, how many jobs are in queue for a free build agent?
Since this would be for the whole Azure Devops instance the Dashboard feature doesn't seems to be appropriate because it only seems to hold project specific metrics.
Got to your Organization Settings-Parallel jobs blade. This will give you the ability to view the jobs in progress.
As for metrics there is a public preview just came out for this; however, I do not have it available yet.
Agent pool usage data is sampled and aggregated by the Analytics service every 10 mins. The number of jobs is plotted based on the max number of running jobs for the specified interval of time.
This feature is enabled by default. To try it out, follow the guidance
below.
Within project settings, navigate to the pipelines “Agent pools” tab
From the agent pool, select a pool (e.g., Azure Pipelines) Within the
pool, select the “Analytics” tab
Is it possible to run multiple Azure Self-hosted build/deploy agents and multiple deployment agents on one server? Also, can these agents service more than one organisation or even multiple Azure AD Tenants?
I do realise the consequences with the server straining under IO bottlenecks and the like, these agents will probably never have to manage more than 3 projects being build and/or deployed at a time, but the sources can be from different projects in different organisations or possibly Tenants.
I have deployed my Deployment Agents to the servers and they function fine with a Microsoft-hosted build agent (my question is about ONE of these servers, it would apply to all of them eventually), but I am afraid to now start deploying the build agents to the same servers now.
This approach is very Do-able and is actually really cost-effective if you do not have continuous deployments or your virtual machine has the IO capacity to handle the planed traffic.
Understand the basics of an Agent. What exactly happens when you host a Windows Agent is that it creates a Windows Service which would run internally a separate new process and perform the actions for the agent.
Since these are independent processes, they are not at all impacted by the operations of other agents. As long as you are not trying to access the same files/resources this approach is actually a great approach and we should surely try this.
I have installed agent on VM and configured a CI build pipeline. The pipeline is triggered and works perfectly fine.
Now I want to use same build pipeline, same agent, but different VM. Is this possible?
How will the execution happen for builds and on which VM will the source be copied?
Thank you.
Like the others I'm also not sure what you're trying to do and also think that the same agent across multiple machines is not possible.
But if you have to alternate or choose easily between VMs, you could set up for each of your VMs (used for this special scenario) an individual agent queue with one agent in that pool. That way you can choose the agent pool at queue time via the agent queue dropdown field. But that would only work if you're triggering manually, not in a typical CI scenario. In that case you would have to edit the definition to enforce any particular VM each time you want to swap VMs.
NO. These private agents are supposed to have a unique name and are assigned to an Agent Pool/Queue. They are polling up to VSTS/Azure Devops server if they have a job to do. Then they execute it. If you clone a machine with the same private build agent, then theoretically the agent that picks it up will execute the job, but that is theoretic. I really don't know how the Agent Queues will handle this.
It depends on what you want to do.
If you want to spread the workload, like 2 build servers and have builds go to whichever build server isn't busy, then you would create 1 Agent Pool/Queue. Create a Private Agent on one server and register it to that Pool, then on the second server un-register the agent and then re-register the agent add it to the SAME pool.
If you want to do work on 2 servers at the exact same time, like a deployment to 2 servers at the same time, then you would create a 'Deployment Group' and add both servers to that. You would unregister both agents from the Agent Pool/Queue. From your 'Deployment Group' copy the PowerShell script snippet and run it on each machine. This way you can use this in your Release Pipeline and deployments in parallel, which take less time to do deployments.
You could set up a variable in the pipeline so you can specify the name of the VM at build-time.
Also, once you have one or more agents, you would add them to an app pool. When builds are run, it will choose one agent from the pool and use that.
I'd like to enhance the release definition so that I don't need to have a separate environment that only starts an Azure VM.
If we take a scenario where we have a Test, Beta, Production environments. The client wants the application to be installed in Beta and Production on their local network. We internally want a Test environment to run E2E tests against, allow for non-technical folks to exercise the app without needing VPN access to the customer beta environment, etc.
So here we have Environment followed by where the Agent is running:
Test - Azure VM
Beta - Client machine
Production - Client machine
How we've solved this is to install the VSTS Agent on a machine at the client, which allows us to target that agent queue in the Beta and Production environments defined for that release. Then we typically build an Azure VM and target that agent queue for the Test environment.
We don't want to run that Azure VM 24/7/365. However if it's not running, then it can't respond to requests from Release Management.
What I've done is to create a environment named Start Test VM and Stop Test VM that use the Azure Resource Group Deployment to start and stop the VM. Those 2 additional environments can have their agent queue set to Hosted.
I'd like to figure out how to combine the first 3 environments into a logical Test instead of having to create 3 release management environments.
Start Test VM - Hosted
Test - Azure VM
Stop Test VM - Hosted
Beta - Client machine
Production - Client machine
The problem is that can be rather ugly and confusing when handing this over to one of our PM's or even myself when I circle back around 3 months later and think, "What the hell is this environment? Oh it's just there to start/stop the VM."
Options:
Stay with status quo - keep it like it is, it can't be fixed
We could open up a port on the Azure VM and use Powershell remoting. Then run on the Hosted agent or on an on-premise agent to start the VM, then deploy the application, then stop the VM. - we really dislike this because the deployment would not be the same as the client on-premise deploy. We'd like each environments' tasks to be the same, just with different variables.
You can use "Azure PowerShell" and "Azure SQL Database Deployment" tasks to configure your Azure VM and SQL or call other script to run on the Azure VM.
There isn't any way to set the agent for tasks. You can submit a feature request on VSTS User Voice for this.
And another way to reduce the environment is that: if you deploy every build linked to the release, then you can add "Start Test VM" task into your build definition to start the VM when build is successful and add "Stop Test VM" task into "Beta" environment.
What we've currently settled on is to continue with having an environment that isn't really what I would consider an environment, but more of a stage in the release pipeline that starts and/or shuts down a VM. We run that on a hosted agent so it can start the VM and make sure to check Skip artifacts download on the environment.
For a continuous integration build, we set a chain so the VM gets started, CI environment gets kicked off and then VM gets stopped. The remaining environments are then manually deployed up the chain as desired.
So here's an example:
Start CI VM
CI
Stop CI VM
Beta
Production
And here's an image of how it looks in Release Management as of 2016.06.27:
I put single quotes around environment because I think I agree with this user voice request in that, it's really more of a stage in the release pipeline. Much like database development, the logical and the physical don't necessarily map 1 to 1.