I'm fresh in the area of CD/CI and I need to implement it in an old project from my company. From what I have read we have a couple of options like Azure DevOps or Team City, I chose these two options because most of our projects are built in Microsoft technologies.
I have been reading for a while, but I cannot grasp the proper definition of a Build Agent; also, I found this old question, but the answer is unclear:
In Team foundation server what is build agent and controller?
Further, I read different documentation:
Azure Pipelines agents
Build Agent
And their definitions are the following ones:
An agent is installable software that runs one job at a time.
Microsoft
A TeamCity Build Agent is a piece of software that listens for the
commands from the TeamCity server and starts the actual build
processes. JetBrains
However, I cannot understand exactly their role or purpose, do they build the Test, UAT and Production pipelines in parallel to see if the compilations were successful? Like here:
Or what do they do? Since the solution has multiple projects inside maybe 8 or 10.
You can take the example that I gave in the comments below:
Let's suppose you create a project in Azure DevOps for your new CRM
for a Dentist with a Debug, UAT, and Release environments plus a team
of 5 devs. What would represent these agents?
Thanks for any explanation.
It's analogous to a human "agent" who has different skills. Think of Build Agent as computer process that has certain capabilities to perform a build.
Some agents can perform certain jobs (e.g. build Apple specific programs), while other agents are more general purpose. Sometimes a computer can have multiple agents that can work in parallel, other times computers only have a single agent assigned to them.
Edit - Added the following to address additional questions:
Agents can be "local" which means they are on the server where the build software (e.g., Azure Pipelines, Bamboo, Team City). They can also be "remote" or on a different computer. A remote agent may be needed to build Apple specific software as this software often needs a Mac to compile.
Extending the human "agent" with different skills analogy, agents can be assigned jobs. So one agent may be assigned building software in your pipeline while another agent is busy handling deployments to different environments. Since each "agent" can only do a single job at a time, more agents can speed up build pipelines by allowing parallel jobs.
In Azure Devops,there is a left Navigation called pipelines where you need to create a build pipeline(with certain tasks) which actually requires an agent to perform the tasks.
In general,build agent/agent is a hosted machine with necessary capabilities(in case of Azure devops) used to run the predefined tasks as per the build pipeline setup to build the source code and make it available for deployment
Related
I am unable to find any documentation on wiki which details approach for SSH/RDP into the ADO agents. Will be good to know the approach for both BTL/ATL agents.
The Microsoft hosted agents? You don't. You can't. They are assigned when the pipeline is queued and immediately deprovisioned after the pipeline finishes running.
Your own on-premise/self-hosted agents? The same way you'd SSH/RDP into any other machine. If you're having trouble with that, that's something for you to discuss with your networking / infrastructure team.
If you are using a Microsoft hosted then you can not per the documentation. The agent is created on a one-time instance and then torn down as soon as the build is completed.
Microsoft-hosted agents do not offer:
The ability to remotely connect.
The ability to drop artifacts to a UNC file share.
The ability to join machines directly to your corporate network.
The ability to get bigger or more powerful build machines.
The ability to pre-load custom software. You can install software during a pipeline run, such as through tool installer tasks or in a
script.
Potential performance advantages that you might get by using self-hosted agents that might start and run builds faster.
The ability to run XAML builds.
If you are using a self-hosted machine running the agent, then you'd just SSH/RDP into the server like any other and check the work folder.
I'm running Azure pipelines on a Windows self-hosted agent. One of my pipelines can do both a 32-bit build and a 64-bit build. I want to use the matrix and maxParallel capabilities to do both builds at once, on the same agent, to save time.
This isn't possible, because the 32-bit build and the 64-bit build both write to the registry, and whoever gets there second, errors out.
The obvious solution is to get a second Azure VM and run a second self-hosted agent on that VM. But I want to see if I can run the two build tasks as two different users, on the theory that they will then write to their own HKCU and not clobber each other.
This would require the default pipeline to trigger a second pipeline, or perhaps run a template, and run it as a different user.
Can this be done?
OTHER USEFUL INFO:
On an Azure DevOps skill-level scale of Beginner-Intermediate-Expert, I'm smack in the middle of Intermediate. Still learning.
The build step uses the built-in VSBuild task.
You can trigger a second pipeline (i.e. by using Trigger Build Task), but pipelines don't have a concept of running as a user - they run on an agent. That agent runs as specific user and it would be tricky to try and execute code as a different user.
Running a second self-hosted agent is a good direction. You don't necessarily need another VM - you could run another agent on the same machine, but as a different user, using different work directory.
You could use agent capabilities and demands to fine tune which kind of build runs on which agent.
We use Azure DevOps Server 2020 on prem. We are not in a position to move to Services. We are not in a position to use agents in Azure, because these agent would not be able to talk to the server, which is behind the corporate firewall.
So, we are stuck with on-prem agents and this suffocates us. Our app is a monolith in the worst sense of the word. Its PR build takes time and is extremely hungry on resources. Currently we have 2-3 PR builds running concurrently on the same machine and it takes eternity. In short - it is very bad.
On the other hands, developers have pretty powerful machines. Of course, I do not want my machine to serve PR builds of others, but if it could run my own PR build, that would be fantastic.
Alas, I do not see how this can be done. So, my question - is it possible to configure Azure DevOps Server 2020 so that PR validation build for a PR submitted by Alice would run on the build agent running on Alice's machine and only if there is no such agent would it pick up a "standard" build agent running on a build server?
Clarification
Although the question speaks about Alice, but there are also Bob and Charlie and Deepak. About 300+ developers. So, the solution should not assume there is only one developer working on the code.
is it possible to configure Azure DevOps Server 2020 so that PR validation build for a PR submitted by Alice would run on the build agent running on Alice's machine
We could install the self-hosted agent in the Alice's machine, open Organization Settings->Agent pools->select the agent->click the tab Capabilities->search the field Agent.ComputerName, then open PR build definition and add Demands, such as below.
Then the PR build will run with the specified agent, you could check this doc: Specify demands
for more details.
In addition, if the PR build does not find the matching agent, the build will not run with another agent.
Update1
Each developers have their owner agent on their owner’s machine, right?
Check this doc: predefined variables, it contain the variable Build.RequestedFor and Build.RequestedForEmail, if the pipeline is pull request trigger, it will show the pull request creator name and email, add this to the pipeline Demands. Such as
And then, let all developers add capabilities to their agents, such as TriggerName = {their email or display name}. Since I am using the variable Build.RequestedForEmail, in my agent, I add the capabilities:
Then if I create a pull request, the build will run via my own machine agent.
Update 2
For example, the original PR build name is A. Open it and add default demand(you could add anything)
Add new build definition name B, add task power shell and enter power shell script to call below REST API to update the A definition
Definitions - Get
GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=6.1-preview.7
Definitions - Update
PUT https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=6.1-preview.7
Then add build B as pull request build.
Update3
is it possible to configure Azure DevOps Server 2020 so that PR validation build for a PR submitted by Alice would run on the build agent running on Alice's machine and only if there is no such agent would it pick up a "standard" build agent running on a build server?
If the "real" targets the build agent initially, then what happens when the auxiliary one determines the dev has an individual agent?
The real build definition has demands, If the build runs by default demands, it will run the standard build agent.
The auxiliary one, we need to add condition to run the power shell script to update the real build demands. if Alice's machine has matching agent, it will update the real build demands, and then the real build will run the Alice's machine agent.
I'm a developer whose transitioning into Devops. By observation, I've noticed that a lot of dev shops have started using Octopus Deploy and Azure Devops Services (AzDo, formerly VSTS), or they are starting new projects to setup devops ci/cd pipelines AND they spec to use both tools.
I've been through some quick training for both tools and though they aren't perfectly the same, AzDo seems to offer all of the same features as Octopus Deploy.
So, my question is if a company is already using AzDo for much of their version control, or anything CI/CD pipeline-related, why would you use Octopus? What benefit does it offer to use Octopus for your build and deploys to AzDo?
Note, I am very, very new to Devops. I'm just asking because at the "10,000 feet view" there doesn't seem to be any reason for Octopus if you're already using AzDo. I mention Octopus Deploy by name because I see it come up frequently. However, I assume there could be other tools that serve the same purpose of automatic build and deploying that might also integrate with AzDo. However, AzDo offers build and deploy built in one. Why split out the work?
Let me preface why I like to both build and deploy with VSTS:
Same permissioning end to end
Line of sight from end to end build and deployment
Reasons I favor Octopus Deploy over VSTS Release:
Ability to upload packages/artifacts
External ones that are maybe one off packages to get deployed for a specific release
Target Definition
When you create Targets or servers you are deploying to, you are able to add a target to one or multiple environments and assign tags/roles to a target. What does this mean? More flexible server definition rather than defining strict Agents to a pool or servers to a Deployment Group, you can allow a target to span multiple (ie: a testing server that spans your Dev and Test environments and only gets triggered on steps that are defined for that role). I realize you can accomplish similar things to this in VSTS but in my opinion it's far more cumbersome.
Variable Definition
Variables can be grouped at a global level and grouped by a specific pipeline/process (that part is similar to VSTS). Variables can also be grouped or scoped by environments or roles (above) so you are able to have different variable values per role per environment; both super granular and flexible. Places this comes in handy is if you have a backend server with a connection string and maybe 2 content delivery nodes (role - content delivery) that get slightly different values than the backend server. At the moment, I do not know (other than creating new environments) how one would accomplish the same in VSTS.
Process Definition
All of the above comes together in the process definition of Octopus Deploy. The super flexible and granular variables and target definition allows you to focus on the actual deployment process rather than getting hung up on the nuances of the UI and its limitations. One example would be defining a process where the first step would be taking something out of a load balancer from a central server, step two deploy code to delivery server one, step three put back in lb, step 4 take out node two from lb called from a central server, step 5 deploy code to node two, and last step, back into load balancer. I realize it's a very simple hypothetical, but within Octopus Deploy, it's one steady process filtered to execute on specific roles, within VSTS you would have to break that down into different agent phases and probably pipelines.
Above are really the biggest points I see to use Octopus Deploy over VSTS Release. Now why would someone use VSTS to build and OD to release/deploy? There are a lot of different factors that go into it, some are corporate drivers like having an enterprise git client that has permissions handled thru MSDN. Sometimes it's a project management driver of having work items tied tightly to commit and builds, but with the added flexibility that OD brings to the table for free/minimal cost.
Hoping this help shine a little light into maybe why some people are crossing streams and using both VSTS and OD.
A lot of good points have been made already, but it really comes down to what you need. I would venture a lot of us started using Octopus before Release Management was really a thing.
We use VSTS for all our source control and builds and then all our deployments are handled through Octopus.
When we started evaluating tools, VSTS had nothing for deployments. Even now, they are still playing catch up to Octopus in feature set.
If you are doing true multi-tenanted and multi-environment deployments, I don't think VSTS really compares. We are using Octopus with around 30 tenants, some on Azure, some on premise. We deploy a mix of web and desktop apps. We are even using Octopus to deploy some legacy VB6 and winforms applications.
Multi-Tenancy (critical for us)
VSTS added Deployment Groups a while ago which sound pretty similar to Octopus Environments before multi-tenancy was implemented. Before Octopus had true multi-tenancy (it's been around a while now), people would work around it by creating different environments per tenant, like "CustomerA - Dev", "CustomerA - Prod", etc. Now you just have your Dev/Test/Prod environments and each tenant can have variables scoped to those individual environments.
Support
Documentation is excellent and it's really easy to get up and running.
The few times I've needed to contact someone at Octopus, they've answered very quickly and knowledgeably.
Usability
Having the Octopus dashboard giving us an overview of all our projects is amazing. I don't know of anyway to do this in VSTS, without going into each individual project.
Octopus works great on a mobile device for checking deployment status and even starting new deployments.
Community
Octopus works with their customers to understand what they want and they often release draft RFCs and have several times completely changed course based on customer feedback.
If we know what sort of applications you are deploying, and to what kinds of environments, we would be able to better tailor our responses.
The features you see today in VSTS weren't there a few years ago, so there might be an historical reason.
But I want to state here some non-opinionated reasons that may suggest an organization to opt for different tools instead of one.
Separate responsibility and access levels
Multiple CI tools in dev teams (orgs that are using also Jenkins or TeamCity or else) and need to standardize and control deployments
An org needs a feature available only in Octopus (maybe Multi-tenancy)
Octopus does a great job of focusing on deployments. Features reach octopus before vsts, support is local and responsive. That, and you never run out of build/release minutes!
Seriously though, I just like to support smaller companies where possible and if all features were equal, I'd still pick them.
The big reason in the past was that TFS On prem and early VSTS did NOT support non-Microsoft (.Net) code very well if at all. You could utilize the source control and work features of TFS and then use octopus/Jenkins etc... as the build release parts to cover code that TFS didn't really know what to do with.
Also the release pipelines used to be very simplistic and not that useful where the other products were all plugin based and could do (almost) anything you needed them to. Most of that has changed so that VSTS is much better at working with Non-Microsoft code bases then it used to be. Over time integrations get created inside a companies walls and undoing those decisions can be more painful then just having "too many" tools. Also I feel like there is just more people out there familiar with those tools since they have been mature longer and cover a larger part of the development world then VSTS has in the past.
To fully implement CD you need both. VSTS runs tests and is a build server. OD isn’t. VSTS is light on sophisticated application installations. And if you are provisioning environments, IaC style, you need Terraform in addition. Don’t try to shoehorn everything into a single tool. DevOps requires a whole ecosystem. The reasons are not historical.
I have a Windows VM that hosts a VSTS build agent. Due to the number and length of builds that are running I would like to know whether multiple build agents can be hosted on one computer? That would allow a dedicated agent for slow builds, and a dedicated agent for quick builds.
https://www.visualstudio.com/en-us/docs/build/admin/agents/v2-windows
Yes you can run multiple agents in a single VM.
Make two directories say Agent1 and Agent2, extract the agent in each one of them and configure them with different names against your VSTS/TFS account.
It should work out of the box.
We run 4 agent jobs per machine concurrently with no issues. As mentioned above, should work out of the box. Just make sure you clean up directories. We have a script to do it every night
Yes, this works, I did the following:
Created a PAT for agent installation needs
Downloaded agent binaries from the agent creation page
Unpacked the archive contents into 2 different directories ("c:\ado-build-agents\agent1" and "c:\ado-build-agents\agent2")
Ran "config.cmd" and followed configuration instructions, provided by it.
Updated pipelines to build the agent pool, which those agents reside in ("Default" in my case)
To test the setup - triggered all 15 pipelines, that I had. As the result I was able to see two pipelines running at the same time, while others were in the "Queued" state (according to my expectations).
I will be also testing out how resources are consumed by the agents to try to understand if I should deploy more agents on the build machine.