I am exploring Azure Pipeline As Code and would like to understand how to make use of "deploymentMode" for validating and deploying ARM templates for each Azure environments.
I already have Release Pipelines created in Azure DevOps via Visual Builder for deployment tasks with one main ARM template and multiple paramater JSON files corresponding to each environment in Azure. Each of those pipeline has two stages. One for validation of ARM templates and Second for deployment.
I am now trying to converting those release pipelines to Azure Pipeline as Code in YAML format and would like to create one YAML file consolidating deployment validation tasks (deploymentMode: 'Validation') for each environment first followed by actual deployment (deploymentMode: 'Incremental').
1) Is it a right strategy for carrying out Azure DevOps Pipeline As code for a multi environment release cycle?
2) Will the YAML have two stages (one for validation and another one for deployment) and each stage having many tasks (each task for one environment)?
3) Do I need to create each Azure Environment first in 'Environments' section under Pipelines and configure the virtual machine for managing the deployment of various environments via YAML file?
Thanks.
According to your requirements, you could configure virtual machines for each azure environments in the Azure Pipeline -> Environments. Then you could reference the environments in Yaml code.
Here are the steps, you could refer to them.
Step1: Configure virtual machine for each Azure Environments.
Note: If the virtual machines are under the same environment, you need to add tags for each virtual machine. Tags can be used to distinguish virtual machines in the same environment.
Step2: You could create the Yaml file and add multiple stages (e.g. validation stage and deployment stage) in it. Each stage can use the environments and contain multiple tasks.
Here is an example:
trigger:
- master
stages:
- stage: validation
jobs:
- deployment: validation
displayName: validation ARM
environment:
name: testmachine
resourceType: VirtualMachine
tags: tag
strategy:
runOnce:
deploy:
steps:
- task: AzureResourceManagerTemplateDeployment#3
...
- task:
...
- stage: deployment
jobs:
- deployment: deployment
displayName: deploy
environment:
name: testmachine
resourceType: VirtualMachine
tags: tag
strategy:
runOnce:
deploy:
steps:
- task: AzureResourceManagerTemplateDeployment#3
...
- task:
...
Here are the docs about using multiple stages and virtual machines.
Hope this helps.
Related
In AzureDevops yaml pipeline, can we define multiple node pools to the same stage. For example we have vm [vmpool] based and docker [dockerpool] based build agents and they are belongs to separate pools. But Some our pipeline stages can be run in any of these pools and where as some pipelines stages need to be run specific pools. So looking for a way to multiple nodepools for the stages where we can run in both the pools.
Secondly, ca we define the precedence to the stages like, first need to check the available vms in the vmpool, if no vms are free to schedule, then schedul the dockerpool.
By going through the docs, I couldn't find any helpful information on this.
You can use template for each pool if across multiple pools in azure pipelines. A single step can be defined in one file and used multiple places in another file.
Please refer to doc: step.template
For example:
# File: steps/build.yml
steps:
- script: npm install
- script: npm test
Across multiple pools:
# File: azure-pipeline.yml
stages:
- stage : stage1
jobs:
- job: run_in_pool_1
pool:
name: vmpool
steps:
- template: steps/build.yml # Template reference
- job: run_in_pool_2
pool:
name: dockerpool
steps:
- template: steps/build.yml # Template reference
If you want to define the precedence to the stages like : check the available vms in the vmpool, you can use demand command to make sure the capabilities of agents.Please refer to:Demands
for example:
pool:
name: MyPool
demands:
- myCustomCapability # exists check for myCustomCapability
- Agent.Version -equals 2.144.0 # equals check for Agent.Version 2.144.0
I have the following problem: I have specified a pipeline yaml which runs on a self-hosted agent for the first stage. For the second stage I haven't declared an agent. It should default to a Microsoft-hosted agent (and does for all other pipelines with the same syntax and use-case). However, when initializing the agent at execution time, the chosen agent is of the self-hosted variety.
I've tried creating a fork, deleting the repo and re-initializing it. Both to no avail.
When I specify an agent for the stage, like so (as described in this ms documentation):
pool:
vmImage: 'ubuntu-latest'
I get the following message:
There was a resource authorization issue: "The pipeline is not valid. Could not find a pool with name Azure Pipelines. The pool does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
When I click authorize resources, I get the message that it was successful. However, the authorization issue keeps returning.
azure-pipeline.yaml
resources:
repositories:
- repository: templates
type: git
name: azure-devops-reusable-tasks
variables:
- group: mijnverzekeringsvoorwaarden-api-service_ONT # for deployments (build scripts only target ONT)
- template: add-variable-branch-name.yml#templates
name: 1.0.0$(branchName)-$(Build.BuildId)
stages:
- stage: Build
displayName: Build
jobs:
- job: Build
displayName: Build
pool:
name: Default
demands: companyName.Docker
steps:
- template: maven.yml#templates
- publish: $(System.DefaultWorkingDirectory)
artifact: mijnverzekeringsvoorwaarden-api-service
- template: publish-surefire-test-results.yml#templates
- template: publish-jacoco-code-coverage.yml#templates
- template: git-set-version-tag.yml#templates
- stage: Scan
dependsOn: Build
jobs:
- job: Scan
displayName: Dependency scan
steps:
- template: owasp-scan.yml#templates
And this is the template in question being called for the second stage. The one that is supposed to run on a MS-hosted machine:
owasp-scan.yml#templates
steps:
- download: current
- task: dependency-check-build-task#6
displayName: Owasp dependency check
inputs:
projectName: $(Build.Repository.Name)
scanPath: $(Pipeline.Workspace)
format: 'HTML'
Any insight as to why it is defaulting to the wrong pool for this particular pipeline?
Update:
Added a screen shot of the agent pools and agents in the microsoft hosted pool.
Tried referencing the agent pool with the name given in the screenshot, like so:
pool:
name: 'Hosted Ubuntu 1604'
vmImage: 'ubuntu-latest'
This gives the following error:
##[error]No agent found in pool Hosted Ubuntu 1604 which satisfies the following demand: ubuntu-latest. All demands: ubuntu-latest, Agent.Version -gtVersion 2.188.2
When using:
pool:
name: 'Hosted Ubuntu 1604'
vmImage: 'Hosted Ubuntu 1604 2'
I get: Encountered error(s) while parsing pipeline YAML: /azure-pipelines.yml (Line: 37, Col: 20): Invalid demand 'Hosted Ubuntu 1604 2'. The demand should be in the format '<NAME>' to test existence, and '<NAME> -equals <VALUE>' to test for a specific value. For example, 'VISUALSTUDIO' or 'agent.os -equals Windows_NT'.
Update 2:
I compared projects. Another project did have the Azure Pipelines agent pool. I approved this project (the one which has the issue) to also have access to this pool. When I explicitly define it in the yaml, it uses this (cloud hosted) pool. However, when I provide no pool information it keeps defaulting to the self-hosted variant. To re-iterate, this only happens for this particular pipeline.
Any insight as to why it is defaulting to the wrong pool for this particular pipeline?
You could try to change the value in the Default agent pool for YAML to Azure Pipelines in the UI:
The problem has been circumvented. When explicitly defining the agent from the agent pool, with correct nomenclature, it does pick an agent from this pool.
However, this does not explain why this behavior is exhibited for this pipeline, while all other pipelines pick an agent from the same pool without defining this. These default to this (the same) pool.
In this specific case the hack is defining the pool like so:
pool:
name: 'Hosted Ubuntu 1604'
With the classic Azure DevOps release pipeline our release flow was very easy to setup.
We had a build pipeline running many times during the day. On success it deployed to our development environment. Every night the latest successful deployment to dev was released to our test environment (running automated tests for hours), before it deployed to UAT. But often we also need to deploy to test during the day, if we have a new change which needs to go directly into test or UAT. The classic pipelines allowed us to skip a stage, or deploy if the previous was only partly successful.
1) Development - automatic
2) Test - nightly or manually
3) UAT - nightly or manually
4) Staging - manual approval
5) Production - manual approval
With the multi-stage pipelines the same flow seems to be very difficult to do. At least when it comes to making it as a single deployment pipeline. The first part is fine. We can have our build trigger the development deployment. But how can we hold back the release to the test environment until 0:30am, while still retain the ability to also release it manually? If I created a separate test environment pipeline, then it could work if it had no triggers, but a schedule.
Same with the UAT, as we also need the flexibility to manually run UAT deployments, then it would also need to go into its own pipeline. Releases to our staging and production environment we "gate" with manual approvals, which is fine.
While this technically could work, if we split the deployment pipeline into multiple pipelines it really gets difficult to manage "a release". Not to say that it kind of goes against the whole multi-stage pipeline principle if we create a separate pipeline per stage.
But with this being so easy to setup like this in the classic pipelines, then I cannot really imaging that other companies have not run into the same limitations. Is it just me who cannot see the light, or cannot this really not be done with multi-stage pipelines?
manually run UAT deployments
We could add Azure DevOps Multi-Stage Pipelines Approval Strategies in the yaml build.
Steps:
Open the tab Environments and click the button New environment-> Click the button approvals and checks-> My environment name is TEST.
Then use it in the yaml pipeline(just a sample):
trigger: none
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: A
jobs:
- deployment: 'MyDeployment'
displayName: MyDeployment
environment: 'TEST'
- job: A1
steps:
- script: echo "##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- stage: B
condition: and(succeeded(), ne(stageDependencies.A.A1.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
Result:
We could also configure schedule Trigger and use them in the multi-stage pipelines.
Note: The schedule trigger and Approval Strategies are using in the stage level.
For scheduled jobs: you can use something like this in your YAML:
(Copied from Microsoft documentation)
schedules:
- cron: string # cron syntax defining a schedule
displayName: string # friendly name given to a specific schedule
branches:
include: [ string ] # which branches the schedule applies to
exclude: [ string ] # which branches to exclude from the schedule
always: boolean # whether to always run the pipeline or only if there have been source code changes since the last successful scheduled run. The default is false.
For manual jobs, you can use the Create Release button to create and deploy a release manually. Do note that sometimes this can create a conflict with the schedule. Also, to "hold back a release" put an approver on the release, and then when approving, defer the release:
noting that it's in UTC, and it defaults to tomorrow - you can change it to any time after now.
I've got a single self-hosted agent. Its used as a kind of deployment agent.
All release versions of our software gets build by this agent and then copied to a network location.
Question: Is there a way I can utilize both the agent from the 'azure-pipelines' Microsoft hosted pool and my own self-hosted pool in my pipelines?
EDIT
Unfortunately this is not possible at the moment.
This is why you should upvote the feature request:
https://developercommunity.visualstudio.com/t/allow-agent-pools-to-contain-microsoft-hosted-and/396893
This is not possible. There is a ticket on developer community asking for similar functionality but it is already closed.
There is another ticket Allow agent pools to contain Microsoft hosted and self-hosted agents which refer to similar case, it is open but MS is silent there.
Which benefits do you want to achieve?
Basically, you can use several agent pools in one build/release definition. You just split your definition into several jobs and assign the needed agent pool to the corresponding job.
If you want to dynamically assign different pools from one pipeline to do the same build steps, we can not do that (as Krzysztof mentioned).
You can do hacky thing and use multiple jobs/stages. Jobs/stages will use different pools. You just need to skip depending if it is release version. Note that pipeline skeleton is not tested.
variables:
${{ eq(variables['Build.SourceBranch'], 'release/*') }}:
release_build: True
stages:
- stage: normal
condition: eq(variables['release_build'], False)
pool:
vmImage: 'windows-latest'
jobs:
- job: Builds
steps:
- template: build.yaml
- stage: release
condition: eq(variables['release_build'], True)
pool: My-agent
jobs:
- job: Builds
steps:
- template: build.yaml
I have the following YML file for my pipeline:
trigger: none
stages:
# Other stages here...
- stage: Release
jobs:
- deployment: Staging
environment: staging
strategy:
runOnce:
deploy:
steps:
- download: none
- task: DownloadBuildArtifacts#0
# ...
- task: AzureRmWebAppDeployment#4
displayName: Deploy in staging
# ...
- deployment: Production
environment: prod
dependsOn: Staging
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureAppServiceManage#0
displayName: Swap stg-prod slots
# ...
Based on this, to give more context, my thinking is to have 2 stages: the first one is to build my application, the second one is to release in staging (QA) and to production next.
The environment "prod" though, has a check (or approval, whatever you want to call it).
I'm not sure if I'm encountering a bug or not, but what is happening is that when stage 1 completes (the build phase), the release phase of stage 2 is blocked and waiting for approval even considering that "staging" has not any check enabled (only prod).
The easiest workaround is to create different stages, one for staging and one for production, but the thing is that it's not matching my expected behaviour. I'm expecting that the deployment for the job staging completes successfully, then the job "production" waits for the approval.
Do you have any suggestion regarding this? Is this a bug?
Checks (approvals) for a deployment job are blocking the entire stage
Sorry for any inconvenience.
Personally, This behavior is by designed at this moment.
As the document state:
Approvals in multi-stage YAML pipelines
We continue to improve multi-stage YAML pipelines, we now let you add manual approvals to
these pipelines. Infrastructure owners can protect their environments
and seek manual approvals before a stage in any pipeline deploys to
them.
This feature is designed based on stage not environment, so it block the whole stage.
As I test, I could reproduce this issue as you. But your request is reasonable (Personally), this feature should be designed based on environment.
You could add your request for this feature on our UserVoice site (https://developercommunity.visualstudio.com/content/idea/post.html?space=21 ), which is our main forum for product suggestions. Thank you for helping us build a better Azure DevOps.
Hope this helps.