We are using Microsoft Azure DevOps to build and run tests.
We have 10 Microsoft provided agents and use YAML (defaultVmImage: 'windows-latest') to build and run tests.
Sometimes the tests take 40mins. Sometimes they take 1hour 40mins!
Why is there such a difference? Surely it's not noise neighbours? Are they in different geo locations?
(example AGENT_MACHINENAME "fv-az689")
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#capabilities-and-limitations
Could we detect a slow agent and fail the build early?
Collection perf info hasn't helped https://github.com/microsoft/azure-pipelines-tasks/blob/master/ci/
- task: DotNetCoreCLI#2
displayName: 'dotnet test'
inputs:
command: test
projects: '**/*.Tests/**/*.Tests.dll'
arguments: --verbosity detailed --filter "$(testFilter)"
Related
I'm converting to a full YAML AzDO pipeline and need to wait for manual validation for certain stages of my pipeline. Added the new ManualValidation task into a serverless job, however it fails immediately with no details about why. I did add a Delay task in there as well (just as a sanity check to make sure my serverless job was actually running successfully), and it runs fine.
- job: waitForValidation
displayName: Wait for external validation
pool: Server
timeoutInMinutes: 4320 # job times out in 3 days
steps:
- task: Delay#1
inputs:
delayForMinutes: '1'
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
me#email.com
you#email.com
instructions: 'Please validate deployment can continue and resume'
onTimeout: 'reject'
These are the docs I'm using:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/manual-validation?view=azure-devops&tabs=yaml
I also dropped into the GitHub project just to make sure the task is still version 0 (it is).
Suggestions on why this might be failing and/or ways I can get some more details in the pipeline about WHY it failed?
Turns out we are actually using AzDO Server, not AzDO Services (thanks, Microsoft for naming them so similarly) and this task is not yet available in the Server version :(
For anyone also frustrated by this lack of functionality on-prem, here’s the documentation on using Deployment Jobs and some about Environments
We are able to get most of the functionality we were looking for this way, thou it does require setting up environments.
I have a pipeline I created in Azure DevOps that builds an Angular application and runs some tests on it. I separated the pipeline into two jobs, Build and Test. The Build job completes successfully. The Test job checks out the code from Git again even though the Build job already did it. The Test job needs the files created in the Build job in order to run successfully like the npm packages.
Here is my YAML file:
trigger:
- develop
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
system.debug: false
stages:
- stage: Client
pool:
name: Windows
jobs:
- job: Build
displayName: Build Angular
steps:
- template: templates/angularprodbuild.yml
- job: Test
displayName: Run Unit and Cypress Tests
dependsOn: Build
steps:
- template: templates/angularlinttest.yml
- template: templates/angularunittest.yml
- template: templates/cypresstest.yml
My agent pool is declared at the stage level so both jobs would be using the same agent. Also I added a dependsOn to the Test job to ensure the same agent would be used. After checking logs, the same agent is in fact used.
How can I get the Test job to use the files that were created in the Build job and not checkout the code again? I'm using Angular 11 and Azure DevOps Server 2020 if that helps.
use files checked out from previous job in another job in an Azure pipeline
If you are using a self-hosted agent, by default, none of the workspace are cleaned in between two consecutive jobs. As a result, you can do incremental builds and deployments, provided that tasks are implemented to make use of that.
So, we could use - checkout: none in the next job to skip checking out the same code in the Build job:
- job: Test
displayName: Run Unit and Cypress Tests
dependsOn: Build
steps:
- checkout: none
- template: templates/angularlinttest.yml
But just as Bo Søborg Petersen said, DependsOn does not ensure that the same agent is used. You need add a User Capability to that specific build agent then in the build definition you put that capability as a demand:
pool:
name: string
demands: string | [ string ]
Please check this document How to send TFS build to a specific agent or server for some more info.
In the test job, we could use predefined variables like $(System.DefaultWorkingDirectory) to access the files for Node and npm.
On the other hand, if you are using the Hosted agent, we need use PublishBuildArtifacts task to publish Artifact to the azure artifacts, so that we could use the DownloadBuildArtifacts task to download the artifacts in the next job:
jobs:
- job: Build
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: npm test
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(System.DefaultWorkingDirectory)'
artifactName: WebSite
# download the artifact and deploy it only if the build job succeeded
- job: Deploy
pool:
vmImage: 'ubuntu-16.04'
steps:
- checkout: none #skip checking out the default repository resource
- task: DownloadBuildArtifacts#0
displayName: 'Download Build Artifacts'
inputs:
artifactName: WebSite
downloadPath: $(System.DefaultWorkingDirectory)
You could check Official documents and examples for some more details.
Assume that the agent is cleaned between jobs, so to access the files, you need to create an artifact during the build job and then download it during the test job.
Also, DependsOn does not ensure that the same agent is used, only that the second job runs after the first job.
Also you can set the second job to not checkout the code with "-checkout: none"
I am getting following error when I try to run my pipeline on self hosted agent (2.175.2)
##[error]No agent found in pool Default which satisfies the following demand: node.js. All demands: vstest, node.js, Agent.Version -gtVersion 2.163.1
The tasks in my pipeline are
- task: DownloadBuildArtifacts#0
- task: PowerShell#2
- task: CopyFiles#2
- task: PublishBuildArtifacts#1
- task: AppCenterDistribute#3
- task: VSTest#2
- task: store-publish#0
No agent found in pool Default which satisfies the following demand: node.js.
From the error message, node.js is a necessary demand to run your pipeline. But it seems that the agents in the Default Agent Pool doesn't have this Capability.
You can navigate to Organization Settings -> Agent Pools -> Default -> Target Agent -> Capabilities to check the information.
Here are two methods to solve this issue:
You could manually install node.js on the machine where the agent is installed. Then restart the build agent. Restarting the build agent will capture the added Capabilities.
In Azure Devops Pipeline, you could add the task: Node.js tool installer
This task can install node.js in the agents of the Default agent pool.
Today i'm setting up azure devops to checkout how it can help us in our build/release process. It is a slow process I have to say especially because al my jobs are queued and I don't know why. I have two pipelines which do basically the same thing. But one is made with the classic editor and one with YAML.
# Xamarin.Android
# Build a Xamarin.Android project.
# Add steps that test, sign, and distribute an app, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/xamarin
trigger:
- master
schedules:
- cron: "0 3 * * Mon-Fri"
displayName: M-F 3:00 AM (UTC) daily build
branches:
include:
- master
pool:
vmImage: 'macos-latest'
variables:
buildConfiguration: 'Release'
outputDirectory: '$(build.binariesDirectory)/$(buildConfiguration)'
steps:
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '**/*.sln'
- task: XamarinAndroid#1
inputs:
projectFile: '**/*droid*.csproj'
outputDirectory: '$(outputDirectory)'
configuration: '$(buildConfiguration)'
The log of the job itself doesn't say very much:
Pool: Azure Pipelines
Image: macos-latest
Queued: Today at 15:13 [manage parallel jobs]
The agent request is not running because all potential agents are running other requests. Current position in queue: 3
Job preparation parameters
2 queue time variables used
system.debug : true
agent.diagnostic : true
I don't know what the problem is of the queued jobs.... The project itself is just the template when you create a new xamarin forms project.
Also as a side note, if the build succeeds where does azure put the apk file?
Thanks in advance!
After investigation, there is a recently event of availability degradation of Azure DevOps, which affected these services, and it has been resolved now. This could affect customers in
Europe. If you want to know more information, please click here: Hosted Pools Availability Degradation in Europe
Our engineers are currently investigating an event impacting Azure
DevOps hosted pools in Europe. The event is being triaged and we will
post an update as soon as we know more.
The issue is now fully mitigated. Our engineers will be investigating
this further to learn from and reduce the risk of potential
recurrences. We apologize for the impact this had on our customers.
About the second part, agree with Krzysztof Madej . After the build succeeds, you need to publish the file as artifact for deployment.
I had this the same, so I assume that this is global issue. Maybe related to this:
From March 24th - 26th, 2020 many customers in Europe and the United
Kingdom experienced delays in their builds and releases targeting our
hosted Windows and Linux agents. This incident was caused by VM
capacity constraints arising from the global health pandemic that led
to increased machine reimage times and then increased wait times for
available agents. Many customers experienced significant delays in
their pipelines over multiple days. We sincerely apologize for the
impact of this incident.
I know that this is related to March, but could appear again. Just a guess.
Part 2
Since you build your app you must publish your artifact (apk file). You can use Publish Build Artifacts task
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: $(outputDirectory)
artifactName: MyBuildOutputs
I landed on this page stumped as to why I was getting no error message in the pipeline.
Turns out when I had changed the pipeline yaml file name, I had accidently set the pipeline status to 'Paused'
I'm setting up my Azure Devops Pipelines, and have a build that requires some fairly lengthy setup steps to run. These need to run before other tasks, which can be run in parallel.
However, I can only see this being done by specifying jobs, which would require to do these lengthy steps each time. Ie:
jobs:
- job: Run1
steps:
- task: Long running setup task
- task: Run taskA
- job: Run2
- task: Long running setup task
- task: Run taskB
Is there a way to have this long running task run, and have task A/B depend on that environment without running them sequentially? Ideally it'd be something like:
-job
steps:
-task: Long running setup
-task: Parallel: taskA
-task: Parallel: taskB
Or have the previous jobs take a container/image snapshot and reuse if that's possible?
Short answer, you can't.
Tasks within a job cannot run in parallel as they run on the same agent and the environments can't be "snapshotted" by Azure Devops to be re-utilized by other jobs in parallel later. But jobs can run in parallel as they can be scheduled on different agents, so setup will run twice but in parallel. So there is a trade-off between time and resource usage which you will need to decide on based on your requirements.
There is another solution though, based on how much you would like to invest in this:
If your "setup" doesn't change that often and you are willing to host your own agents. Then you could run a separate "setup + agent" build which creates a docker image of your agents, pushes it out to your azure container registry and then deploys this image to your self-hosted agents (Azure Kubernetes Service) cluster. Then Task A and Task B can easily run in parallel with the assumption that the environment they are running in (agent + setup docker image) is always ready. This is exactly my setup.
See : Azure DevOps Docker
An update to this - a docker image as suggested by #dparker, while it is probably a better way, was a bit OTT for myself. Instead I used a pipeline artifact to cache some build / setup files. And then each other job was made dependent on this setup job.
This obviously doesn't sound great, but it works fine to get the performance optimizations I was after.
Eg Job1 would include this:
- task: PublishPipelineArtifact#0
inputs:
artifactName: 'Setup-Build'
targetPath: '$(buildDir)'
And Job2 to X would download this as an artifact:
- task: DownloadPipelineArtifact#1
displayName: 'Download Setup'
inputs:
targetPath: '$(buildDir)'
artifactName: 'Setup-Build'
Additionally there is the option to use caching, but this didn't quite fit in with my scenario. I'd recommend you make a call between artifacts and caching:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops