I'm setting up my Azure Devops Pipelines, and have a build that requires some fairly lengthy setup steps to run. These need to run before other tasks, which can be run in parallel.
However, I can only see this being done by specifying jobs, which would require to do these lengthy steps each time. Ie:
jobs:
- job: Run1
steps:
- task: Long running setup task
- task: Run taskA
- job: Run2
- task: Long running setup task
- task: Run taskB
Is there a way to have this long running task run, and have task A/B depend on that environment without running them sequentially? Ideally it'd be something like:
-job
steps:
-task: Long running setup
-task: Parallel: taskA
-task: Parallel: taskB
Or have the previous jobs take a container/image snapshot and reuse if that's possible?
Short answer, you can't.
Tasks within a job cannot run in parallel as they run on the same agent and the environments can't be "snapshotted" by Azure Devops to be re-utilized by other jobs in parallel later. But jobs can run in parallel as they can be scheduled on different agents, so setup will run twice but in parallel. So there is a trade-off between time and resource usage which you will need to decide on based on your requirements.
There is another solution though, based on how much you would like to invest in this:
If your "setup" doesn't change that often and you are willing to host your own agents. Then you could run a separate "setup + agent" build which creates a docker image of your agents, pushes it out to your azure container registry and then deploys this image to your self-hosted agents (Azure Kubernetes Service) cluster. Then Task A and Task B can easily run in parallel with the assumption that the environment they are running in (agent + setup docker image) is always ready. This is exactly my setup.
See : Azure DevOps Docker
An update to this - a docker image as suggested by #dparker, while it is probably a better way, was a bit OTT for myself. Instead I used a pipeline artifact to cache some build / setup files. And then each other job was made dependent on this setup job.
This obviously doesn't sound great, but it works fine to get the performance optimizations I was after.
Eg Job1 would include this:
- task: PublishPipelineArtifact#0
inputs:
artifactName: 'Setup-Build'
targetPath: '$(buildDir)'
And Job2 to X would download this as an artifact:
- task: DownloadPipelineArtifact#1
displayName: 'Download Setup'
inputs:
targetPath: '$(buildDir)'
artifactName: 'Setup-Build'
Additionally there is the option to use caching, but this didn't quite fit in with my scenario. I'd recommend you make a call between artifacts and caching:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops
Related
I'm converting to a full YAML AzDO pipeline and need to wait for manual validation for certain stages of my pipeline. Added the new ManualValidation task into a serverless job, however it fails immediately with no details about why. I did add a Delay task in there as well (just as a sanity check to make sure my serverless job was actually running successfully), and it runs fine.
- job: waitForValidation
displayName: Wait for external validation
pool: Server
timeoutInMinutes: 4320 # job times out in 3 days
steps:
- task: Delay#1
inputs:
delayForMinutes: '1'
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
me#email.com
you#email.com
instructions: 'Please validate deployment can continue and resume'
onTimeout: 'reject'
These are the docs I'm using:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/manual-validation?view=azure-devops&tabs=yaml
I also dropped into the GitHub project just to make sure the task is still version 0 (it is).
Suggestions on why this might be failing and/or ways I can get some more details in the pipeline about WHY it failed?
Turns out we are actually using AzDO Server, not AzDO Services (thanks, Microsoft for naming them so similarly) and this task is not yet available in the Server version :(
For anyone also frustrated by this lack of functionality on-prem, here’s the documentation on using Deployment Jobs and some about Environments
We are able to get most of the functionality we were looking for this way, thou it does require setting up environments.
With the classic Azure DevOps release pipeline our release flow was very easy to setup.
We had a build pipeline running many times during the day. On success it deployed to our development environment. Every night the latest successful deployment to dev was released to our test environment (running automated tests for hours), before it deployed to UAT. But often we also need to deploy to test during the day, if we have a new change which needs to go directly into test or UAT. The classic pipelines allowed us to skip a stage, or deploy if the previous was only partly successful.
1) Development - automatic
2) Test - nightly or manually
3) UAT - nightly or manually
4) Staging - manual approval
5) Production - manual approval
With the multi-stage pipelines the same flow seems to be very difficult to do. At least when it comes to making it as a single deployment pipeline. The first part is fine. We can have our build trigger the development deployment. But how can we hold back the release to the test environment until 0:30am, while still retain the ability to also release it manually? If I created a separate test environment pipeline, then it could work if it had no triggers, but a schedule.
Same with the UAT, as we also need the flexibility to manually run UAT deployments, then it would also need to go into its own pipeline. Releases to our staging and production environment we "gate" with manual approvals, which is fine.
While this technically could work, if we split the deployment pipeline into multiple pipelines it really gets difficult to manage "a release". Not to say that it kind of goes against the whole multi-stage pipeline principle if we create a separate pipeline per stage.
But with this being so easy to setup like this in the classic pipelines, then I cannot really imaging that other companies have not run into the same limitations. Is it just me who cannot see the light, or cannot this really not be done with multi-stage pipelines?
manually run UAT deployments
We could add Azure DevOps Multi-Stage Pipelines Approval Strategies in the yaml build.
Steps:
Open the tab Environments and click the button New environment-> Click the button approvals and checks-> My environment name is TEST.
Then use it in the yaml pipeline(just a sample):
trigger: none
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: A
jobs:
- deployment: 'MyDeployment'
displayName: MyDeployment
environment: 'TEST'
- job: A1
steps:
- script: echo "##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- stage: B
condition: and(succeeded(), ne(stageDependencies.A.A1.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
Result:
We could also configure schedule Trigger and use them in the multi-stage pipelines.
Note: The schedule trigger and Approval Strategies are using in the stage level.
For scheduled jobs: you can use something like this in your YAML:
(Copied from Microsoft documentation)
schedules:
- cron: string # cron syntax defining a schedule
displayName: string # friendly name given to a specific schedule
branches:
include: [ string ] # which branches the schedule applies to
exclude: [ string ] # which branches to exclude from the schedule
always: boolean # whether to always run the pipeline or only if there have been source code changes since the last successful scheduled run. The default is false.
For manual jobs, you can use the Create Release button to create and deploy a release manually. Do note that sometimes this can create a conflict with the schedule. Also, to "hold back a release" put an approver on the release, and then when approving, defer the release:
noting that it's in UTC, and it defaults to tomorrow - you can change it to any time after now.
We are using Microsoft Azure DevOps to build and run tests.
We have 10 Microsoft provided agents and use YAML (defaultVmImage: 'windows-latest') to build and run tests.
Sometimes the tests take 40mins. Sometimes they take 1hour 40mins!
Why is there such a difference? Surely it's not noise neighbours? Are they in different geo locations?
(example AGENT_MACHINENAME "fv-az689")
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#capabilities-and-limitations
Could we detect a slow agent and fail the build early?
Collection perf info hasn't helped https://github.com/microsoft/azure-pipelines-tasks/blob/master/ci/
- task: DotNetCoreCLI#2
displayName: 'dotnet test'
inputs:
command: test
projects: '**/*.Tests/**/*.Tests.dll'
arguments: --verbosity detailed --filter "$(testFilter)"
Dear assorted Developers,
in azure pipeline's container jobs, for every job containers get pulled from registry, even if the same container is used for multiple jobs.
Of course in case the images are really small, this is no problem, but in case anyone is intending to build with the same image which is covering the vscode local development - this can use up more time than the actual build.
So has anyone solved caching the container?
Here is an example:
# in this example, all jobs use the same container.
# in stage 1, the jobs are started serial, so job 2 only starts if
# job 1 is done -> and the image is downloaded for both jobs independently
# in stage 2, the jobs are started in parallel,
# and the image is downloaded for both jobs in the stage independently
trigger:
batch: true
branches:
include:
- "*"
resources:
containers:
- container: ubuntu
image: ubuntu:18.04
stages:
- stage: STAGE1
jobs:
- job: PrintInfoStage1Job1
container: ubuntu
steps:
- script: |
echo "THIS IS STAGE 1, JOB 1"
displayName: "JOB 1"
- job: PrintInfoStage1Job2
dependsOn: PrintInfoStage1Job1
container: ubuntu
steps:
- script: |
echo "THIS IS STAGE 1, JOB 2"
displayName: "JOB 2"
- stage: STAGE2
dependsOn: STAGE1
jobs:
- job: PrintInfoStage2Job1
dependsOn: []
container: ubuntu
steps:
- script: |
echo "THIS IS THE STAGE 2, JOB 1"
displayName: "JOB 1"
- job: PrintInfoStage2Job2
container: ubuntu
dependsOn: []
steps:
- script: |
echo "THIS IS THE STAGE 2, JOB 2"
displayName: "JOB 2"
Azure DevOps Container Jobs: Cache Container for multiple Jobs?
Initially, our design and develop idea is mostly considering for the security and consistency reasons, it should be a fresh image each time. Now, we have received many feature request about hoping support cache image which same with yours from lots of developers. Now, considering the disadvantage of this design idea, it would let developers wasting too much time to wait for the image pulled down. If the image can be cached, it can greatly improve the efficiency of the build.
Now, the bulk of the actual caching work about this feature has been developed done by our Azure Artifacts Team. Since the latest process I got from that team is before we can release this feature in azure devops, there are some work we need to do, which about around security to make sure that the cache can't be used as an attack vector. Once this is done we will launch a customer preview. It would be deployed recently.
Please see our Roadmap: Speed up pipeline with caching to track its develop and release process. You can also track this blog which published by the azure artifacts PM. Also, you can follow and monitor this PR.
Until now, there's no much better work around to improve this. Even use the Cache task to perform its thing in combination with the Docker save/load respective operations pretty much matched that of downloading the base image/layers from a public registry.
I will still monitor this feature develop process. Once the PR finished and the feature code deployed to all regions, even it released as a preview feature, I will update this answer to let you and other SO users know.
I have a pipeline with 2 stages - a build/test stage, and a Teardown stage that cleans up external resources after the build/test stage. The teardown stage depends on some state information that gets generated in the build/test stage. I'm trying to use Azure hosted agents to do this. The problem is that the way I have it now, each stage deploys a new agent, so I lose the state I need for the teardown stage.
My pipeline looks something like this:
trigger:
- master
stages:
- stage: Build_stage
jobs:
- job: Build_job
pool:
vmImage: 'ubuntu-latest'
steps:
- task: InstallSomeTool#
- script: invoke someTool
- script: run some test
- stage: Teardown_stage
condition: always()
jobs:
- job: Teardown_job
pool:
vmImage: 'ubuntu-latest'
steps:
- script: invoke SomeTool --cleanup
The teardown stage fails because it's a brand new agent that knows nothing about the state created by the previous invoke someTool script.
I'm trying to do it this way because the Build stage creates some resources externally that I want to be cleaned up every time, even if the Build stage fails.
Is it possible to have an Azure hosted build agent persist between
pipeline stages?
No, you can't. The hosted agent are all randomly assigned by server. You could not use any script or command to specify a specific one.
Since you said that the Build_Stage will create some resources externally, so that you want to execute clean up to clean it.
In fact, for this, you can execute this clean up command as the last steps in Build_Stage. If this, whether using hosted or private agent will not affect what you want.