I am using the new Azure DevOps Yaml multi stage pipeline functionality
I've got an Azure DevOps yaml pipeline file for which I want to use templates. I would like the pipeline to checkout self and another repository.
For some reason, self repo has been checked out when this runs, but the repo: pipelines is not being checked out and therefore the job fails (because some of the file dependencies it requires are not there.
Here is an excerpt from my template:
resources:
repositories:
- repository: self
- repository: pipelines
name: vstsproject/pipelines
type: git
source: pipelines
variables:
# Container registry service connection established during pipeline creation
imageRepository: 'vstsprojectweb'
dockerfilePath: '$(Build.SourcesDirectory)/src/Dockerfile.CI'
BuildConfiguration: 'Release'
tag: '$(Build.BuildId)'
stages:
- stage: 'PRD'
jobs:
- template: update-connection-string-db.yml#pipelines
parameters:
resourceGroup: 'application-DEV'
DBSearchString: '###dbservername###'
What is it that I am doing wrong?
I have referred to this microsoft documentation.
You don't need to reference the linked repo in the resources (i.e. self), and if it is the only repo, then it is checked out by default in jobs (not deployment jobs), but if you have additional repos, then you need to check them out manually (with -checkout: <name_of_repo>).
So just do (PS: Cleaned up a little, assumed that the repo is in the same project):
resources:
repositories:
- repository: pipelines
source: pipelines
variables:
# Container registry service connection established during pipeline creation
imageRepository: 'vstsprojectweb'
dockerfilePath: '$(Build.SourcesDirectory)/src/Dockerfile.CI'
BuildConfiguration: 'Release'
tag: '$(Build.BuildId)'
stages:
- stage: 'PRD'
jobs:
- checkout: self
- checkout: pipelines
- template: update-connection-string-db.yml#pipelines
parameters:
resourceGroup: 'application-DEV'
DBSearchString: '###dbservername###'
I ended up putting everything into the same repo and then checking out self in a job.
That worked for me.
jobs:
- job: dbconnectionstring
displayName: 'db connection string'
pool: Windows
steps:
- checkout: self
- template: templates/update-connection-string-db.yml
Related
My issue
Yaml Devops propose the extends bloc. This is the kind of sample one use:
# main.yaml
trigger: none
extends:
template: ./extends-pipeline1.yaml
And:
# extends-pipeline1.yaml
stages:
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
Works fine, but this works as well without extends:
# main.yaml
trigger: none
stages:
- template: extends-pipeline1.yaml
What I did
I read the release note:
Release Note
My question
What is the benefit to use extends?
Thanks
What is the benefit to use extends?
There are two benefits of this feature.
First, 'extends' is more free than 'stages'.
When you use stages, then the pipeline will expect stages in the template, you must write the stages part in the template, otherwise the validation step will not be passed.
For example, when you use stages, your pipeline must be like this:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- template: extends-pipeline1.yaml
extends-pipeline1.yaml
stages: #You must have this section in the template.
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
But if you use extends, your pipeline could be like this instead of must writing stages section in template:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
extends:
template: extends-pipeline1.yaml
extends-pipeline1.yaml
jobs:
- job:
steps:
- script: echo Build solution
Second, 'extends' has a feature named Required template.(Daniel had already mentioned this.)
If you set the 'required template' of the service connection, then the service connection will only been able to use in specific YAML template file which been 'extends'.
No matter directly use the service connection via main YAML file or use the service connection in the YAML file referred via 'stages'-template will both make the pipeline failed.
This feature can help you increase the security and this feature is exclusive to the 'extends' feature.
I have a separate pipeline in Azure Devops for sonar analysis, which scans another repo in Azure Devops, however, it is only scanning the master branch, I would like it to scan specific branches, ideally, all feature branches or just our test branch.
I've tried a few things but unable to get this working, here is part of my sonar pipeline code (removed/changed any names/sensitive info):
resources:
repositories:
- repository: platform-code
type: git
name: platform-code
#pipelines:
# - pipeline: platform-build
# source: platform-build
# trigger:
# branches:
# - feature/awesomeaddition
#trigger:
# branches:
# include:
# - master
# - features/*
stages:
- stage: Sonarqube
variables:
- template: variables.yaml
jobs:
- job: SonarqubeScan
steps:
- checkout: self
- checkout: platform-code
- task: SonarQubePrepare#4
displayName: 'SonarQube Prepare'
inputs:
SonarQube: ${{ variables.SonarQubeServiceConnectionName }}
scannerMode: 'CLI'
configMode: 'manual'
cliProjectKey: ${{ variables.cliProjectKey }}
cliProjectName: ${{ variables.cliProjectName }}
cliSources: '$(Build.SourcesDirectory)/platform-build'
extraProperties: |
sonar.branch.name='feature/awesomeaddition'
I am building out a new yaml-based build process, provisioning via terraform, using Azure GIT repos. The release pipeline uses a yaml file (below) and I know all of the rest of it works because the whole deployment works except the deploy just drops the .zip file in the site/wwwroot directory instead of unzipping it. As far as I can tell it is all about the last two lines in the deploy section at the very bottom of the file.
Also, the package and zip themselves are fine, as I was able to manually deploy the code via Azure CLI in VS Code and it works.
Do I need something else to get the release to unzip the zip contents into the wwwroot dir?
name: Deploy $(Date:yyyyMMdd)$(Rev:.r) # build numbering format
trigger: none
resources:
repositories:
- repository: templates
type: git
name: repo/branch
ref: refs/heads/main
pipelines:
- pipeline: build
project: ADOProject
source: buildpipelinename
trigger:
branches:
- dev
variables:
- name: tier_prefix
value: D
- name: env_identifier
value: dev
- name: workstream
value: liehadmin
- name: region_prefix
value: C
- name: rgp_identifier
value: OURRGP
- group: terraform (non-prod)
stages:
- stage: infrastructure # our terraform methods are called from here
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule', 'ResourceTrigger'))
displayName: UPDATE INFRASTRUCTURE
jobs:
- template: 'core/terraform.yaml#templates'
parameters:
tier_prefix: $(tier_prefix)
tf_key: $(workstream)-$(env_identifier)
module_name: 'web_1tier1region_S'
variable_file_path: "$(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/pipelines/$(workstream)/$(workstream).$(env_identifier).tfvars.json"
- stage: deploy
dependsOn: [infrastructure]
displayName: DEPLOY APP
jobs:
# Track deployments on the environment.
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'windows-latest'
# Creates an environment if it doesn't exist.
environment: 'site-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- bash: ls $BUILD_ARTIFACTSTAGINGDIRECTORY
- bash: ls $PIPELINE_WORKSPACE
- task: AzureRmWebAppDeployment#4
displayName: 'deploy $(tier_prefix)-$(region_prefix)-$(workstream)-UI-01'
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'subname'
# SlotName: 'slot'
appType: 'webApp'
WebAppName: '$(tier_prefix)-$(region_prefix)-$(workstream)-UI-01'
ResourceGroupName: '$(region_prefix)-Y-Z-$(rgp_identifier)-$(tier_prefix)-XXX-XX'
# DeployToSlotOrASEFlag: true
Package: '$(Pipeline.Workspace)/build/repo_name'
enableCustomDeployment: true
deploymentMethod: 'zipDeploy'
I might have found the fix:
UseWebDeploy: true
enableCustomDeployment: true
deploymentMethod: 'runFromPackage'
Now, if you read the Microsoft documentation, this is weird, since it lists enableCustomDeployment as an argument alias for UseWebDeploy, but for some reason using both worked; my code deployed to site/wwwroot.
I too struggled with this and found this:
Go to App Service -> Configuration
Add a new Application Setting: WEBSITE_RUN_FROM_PACKAGE = 1
Go to App Service -> Console. Run the command dir
You should find, that the contents of your zip file is now shown, and that the site works.
You can read more here: https://github.com/projectkudu/kudu/wiki/WEBSITE_RUN_FROM_PACKAGE-and-WEBSITE_CONTENTAZUREFILECONNECTIONSTRING-Best-Practices
My issue
I have an Azure devops project with a template on another repository than main one's.
I try to load the template but I get this:
/azure-pipelines.yml: File /TemplateRepository/build.yml not found in
repository
https://dev.azure.com/XXXXX/TestAzureDevops/_git/TemplateRepository
branch refs/heads/main
What I did
build.yml is my template into repository: TemplateRepository.
azure-pipelines.yml is my main file.
this is my template:
parameters:
- name: 'name'
default: 'name'
type: 'string'
steps:
- script: echo ${{parameters.name}}
This is my main:
pool:
vmimage: 'windows-latest'
parameters:
- name: contexts
type: object
default: [{
name: macOS,
pool: 'macOS-latest',
sign: false
},
{
name: Windows,
pool: 'windows-latest',
sign: true
}]
resources:
repositories:
- repository: Tuto-Ressources
ref: main
type: git
name: TemplateRepository
stages:
- stage: BUILD
jobs:
- job: test
steps:
- checkout: self
- checkout: Tuto-Ressources
- script: dir
displayName: Dir
- ${{ each context in parameters.contexts }}:
- template: .\TemplateRepository\build.yml#Tuto-Ressources
parameters:
name: ${{context.name}}
pool: ${{context.pool}}
sign: ${{context.sign}}
buildSteps:
- checkout: self
- checkout: Tuto-Ressources
- bash: echo "Test module"
What I tested
If I remove the template lines from main script so I just have script: dir I can see my checkout on the VM and build.yml into \TemplateRepository directory.
So there is no reason it can't find my template.
I checked the branch name, it's main and I have no other one. I can see the file from the Devops Portal
What I need
I would like to understand what happen and what I can do.
I know there are artifact, but in this situation I don't understand whay it is not working like this because I don't change job neither stage.
The template line should reference the template file by its path relative to TemplateRepository. If your build template resides at \build.yml in TemplateRepository, then you should just do:
- template: build.yml#Tuto-Ressources
Also, you don't need to do - checkout: Tuto-Ressources - the pipeline engine will know to fetch the template file from referenced repository.
Question: how do you setup CI/CD in YAML pipelines for following context.
branches
master
release/{ALPHABETICAL NAME} ex. release/Albert next release is release/Bertrand and so on.
environments
accept: everything that's pushed on master
test: latest release ex. release/Bertrand
sandbox: latest release -1 (here we can test hotfixes) ex. release/Albert
live: latest release -1 (with hotfixes)
Closest solution
build: creates project artifacts
build.yml
trigger:
- master
- release/*
pool:
vmImage: 'ubuntu-latest'
steps:
- powershell: |
New-Item -Path . -Name "testfile1.txt" -ItemType "file" -Value "This is a text string."
- publish: $(Pipeline.workspace)
artifact: testArtifact
release-phase1: deploys master branch to accept
release-phase1.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- master
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
release-phase2: deploys release branch to test
release-phase2.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- release/current
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
release-phase3: deploys release-1 branch to sandbox and after manual approval to live
release-phase3.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- release/previous
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
Reasons why this solutions doesn't fulfill our needs:
the names of the release branches aren't static.
we should be able to run release-phase3.yml pipeline without running a build on this branch firts. It should download artifacts from the latest build of that branch. Which is not the case.
SHORT ON PURPOSE
Since you have multiple branches (master and releases branches), different branch is built and deploy to different environment. So you can try having the CI build yaml pipeline in each branch and put the CD deployment yaml pipeline in a template yaml in master branch.(You have to have the build yaml file in each branch to get the code in this branch built. You can check this thread).
Below is a simple example.
In master branch
There are azure-pipelines.yml and a template-deploy.yml. In azure-pipelines.yml the Environment value will be passed as a parameter to template-deploy.yml. So that the build will be deployed to its corresponding environment.
azure-pipelines.yml:
trigger:
- master
- release/*
pool:
vmImage: 'windows-latest'
resources:
repositories:
- repository: deploy
type: git
name: {project name}
jobs:
- job: Build
steps:
- script: echo "start build job"
- template: template-deploy.yml#deploy
parameters:
envir: "prod"
template-deploy.yml:
parameters:
envir: ""
jobs:
- deployment: DeployWeb
environment: '${{parameters.envir}}'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
In the release branches
You can define its individual ci build yaml like below example:
azure-pipelines.yml in release-phase2 branch:
pool:
vmImage: 'windows-latest'
resources:
repositories:
- repository: deploy
type: git
name: {project name}
jobs:
- job: Build
steps:
- script: echo "start build job"
- template: template-deploy.yml#deploy
parameters:
envir: "test"