In my YAML pipeline I have a deployment job:
- stage: deployment_to_development
jobs:
- deployment: deployment_to_development
displayName: Deploy to Development
environment: Development
variables:
- template: migrate-vars.yml
strategy:
runOnce:
deploy:
steps:
- template: migrate-data.yml
The deployment template is a simple DbUp task:
steps:
- task: UpdateDatabaseWithDbUp#2
displayName: Migrate data
inputs:
ConnectionString: 'Data Source=$(DatabaseServer);Initial Catalog=$(DatabaseName);Integrated Security=SSPI'
ScriptPath: '$(Pipeline.Workspace)\data-migrations'
JournalSchemaName: dbo
JournalTableName: _SchemaVersions
LogScriptOutput: true
IncludeSubfolders: true
Order: FolderStructure
TransactionStrategy: SingleTransaction
The variables template defines the server and db name:
variables:
DatabaseServer: 'server'
DatabaseName: 'dbName'
Instances: '_1,_2'
This all works fine for a single database. However, I wish to repeat the task for each instance defined in the Instances variable, i.e for databases named dbName_1, dbName_2. This seemingly simple feat is proving difficult.
I have tried passing the instances as a parameter array and iterating them using
parameters:
param: []
steps:
- ${{each p in parameters.param}}:
but the 'p' variable isn't evaluated in the task.
There have been many more futile attempts. I must be missing something very obvious here. What is it?
I can't test UpdateDatabaseWithDbUp#2 but I have sth what explain how you can achieve your goal. First define template.yaml
parameters:
- name: 'instances'
type: object
default: {}
- name: 'server'
type: string
default: ''
steps:
- ${{ each instance in parameters.instances }}:
- script: echo ${{ parameters.server }}:${{ instance }}
then reuse this template in your build:
steps:
- template: template.yaml
parameters:
instances:
- test1
- test2
server: someServer
And here is the result:
Related
I have a deployment pipeline in Azure DevOps which deploys database changes to a list of databases. Rather than having these run sequentially I would like to run them in parallel. The rolling deployment strategy supports running in parallel but I don't know how to pass variables in this configuration. For Jobs there is a Matrix option to pass variables to different executions. However I can't find an equivalent to use for deployment.
Here is the relevant pipeline portion
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
- group: LibraryData
parameters:
- name: 'qaDatabases'
type: object
default:
- databaseSet:
databases: ['DB1','DB1-A']
- databaseSet:
databases: ['DB2','DB2-A']
- databaseSet:
databases: ['DB3','DB3-A']
#Build details skipped, working fine
- stage: DeployToQA
jobs:
- deployment: DeployQA
environment:
name: QA
resourceType: VirtualMachine
tags: Database
strategy:
rolling:
maxParallel: 2
deploy:
steps:
- ${{ each databaseSet in parameters.qaDatabases }}:
- template: Pipeline-Templates/DBDeploy.yml
parameters:
DatabaseServer: "$(lib-QADBServer)"
DBName: ${{ databaseSet.databases[0] }}
DBaName: ${{ databaseSet.databases[1] }}
I have two Azure DevOps .yml-based templates, one referring to another. I'm finding that variables specified in the one being referred to do not expand when I need them to in a conditional in the template referring to that template. Here is how they look:
Parent template:
jobs:
- deployment:
displayName: Release
variables:
- template: variables.yml
environment:
name: QA
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
...
#this copies to xxMaster
- task: CopyFiles#2
#condition: eq('${{ parameters.appName }}', 'all')
condition: and(eq(variables['Build.SourceBranchName'], 'master'), eq('${{ configuration }}', '$(productionConfiguration)'))
inputs:
SourceFolder: 'C:\DevOps\$(Build.BuildNumber)\Content\D_C\a\1\s\xxTexas\obj\Release\Package\PackageTmp'
Contents: '**\*.*'
OverWrite: true
TargetFolder: 'C:\Production\Web Sites\xxMaster'
Child Template:
variables:
- name: solution
value: '**/*.sln'
- name: buildPlatform
value: 'Any CPU'
- name: buildConfiguration
value: "${{ parameters.configuration }}"
- name: productionConfiguration
value: 'Horsie'
In this case, when the conditional should work and ${{ configuration }} equals 'Horsie', the YAML is evaluated and comes up with
and(eq(variables['Build.SourceBranchName'], 'master'), eq('Horsie', '$(productionConfiguration)'))
That is, $(productionConfiguration), the variable specified in the referred-to template, is never expanded into 'Horsie.' What should I do to make it so the variable from that sub-template is expanded and my conditional works?
From your YAML sample, the cause of this issue is the format of the variable:$(productionConfiguration) in condition.
You need to use the format: variables['productionConfiguration'] to call the pipeline variable.
Here is an example:
stages:
- stage: deploy
jobs:
- deployment: DeployWeb
displayName: deploy Web App
variables:
- template: template.yml
pool:
vmImage: 'Ubuntu-latest'
# creates an environment if it doesn't exist
environment: 'smarthotel-dev'
strategy:
runOnce:
deploy:
steps:
- script: echo Hello world
condition: and(eq(variables['Build.SourceBranchName'], 'main'), eq('${{ configuration }}', variables['productionConfiguration']))
I have declared a variable group Agile-Connections as below and the group do not have any restriction for any Pipeline:
I am using another template called vars.yml to store some other variables:
variables:
- group: Agile-Connections
- name: extensions_dir
value: /apps/agile/product/agile936/integration/sdk/extensions
- name: properties_dir
value: /apps/agile/product/Properties
- name: build_name
value: RestrictPreliminaryBOMPX.jar
- name: resource_name
value: RestrictPreliminaryBOMPX.properties
My Azure Pipeline looks like below, which is calling a deploy.yml template, and I am passing two parameters (connection, environment) from azure pipeline.yml to deploy.yml.
Below is my azure-pipeline.yml:
trigger:
- None
pool:
name: AgentBuildAgile
stages:
- template: templates/build.yml
- stage: DEV_Deployment
variables:
- template: templates/vars.yml
jobs:
- job:
steps:
- script:
echo $(Dev-mnode1)
- template: templates/deploy.yml
parameters:
connection: $(Dev-mnode1)
environment: 'DEV'
Below is my deploy.yml:
parameters:
- name: connection
- name: environment
jobs:
- deployment:
variables:
- template: vars.yml
environment: ${{ parameters.environment }}
displayName: Deploy to ${{ parameters.environment }}
strategy:
runOnce:
deploy:
steps:
- script:
echo Initiating Deployment ${{ parameters.connection }}
- template: copy-artifact.yml
parameters:
connection: ${{ parameters.connection }}
# - template: copy-resources.yml
# parameters:
# connection: ${{ parameters.connection }}
From my deploy.yml I am passing a parameter connection further to another template called copy-artifact.yml, which is below:
parameters:
- name: connection
jobs:
- job:
variables:
- template: vars.yml
displayName: 'Copy jar'
steps:
# - script:
# echo ${{ parameters.connection }}
- task: SSH#0
displayName: 'Task - Backup Existing jar file'
inputs:
sshEndpoint: ${{ parameters.connection }}
runOptions: inline
inline:
if [[ -f ${{ variables.extensions_dir }}/${{ variables.build_name }} ]]; then mv ${{ variables.extensions_dir }}/${{ variables.build_name }} ${{ variables.extensions_dir }}/${{ variables.build_name }}_"`date +"%d%m%Y%H%M%S"`"; echo "Successfully Backed up the existing jar"; fi
Now when I run my pipeline I am getting error message :
The pipeline is not valid. Job Job3: Step SSH input sshEndpoint references service connection $(Dev-mnode1) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
When I print the value of $(Dev-mnode1) using the commented out lines in copy-artifacts.yml file, it prints fine (Dev11 Connection) but when I try to use that as service connection for my ssh task, it gives me the above error.
Also, there is a service connection Dev11 Connection in my project and all the pipelines are allowed to use that service connection.
The pipeline is not valid. Job Job3: Step SSH input sshEndpoint references service connection $(Dev-mnode1) which could not be found. The service connection does not exist or has not been authorized for use.
From the error message, the parameter not get the variable value when expand the template.
Test the same sample and can reproduce the issue.
When you define the variable template at stage level, the variables in template will expand at runtime and the paramter:connection will be expand at compile time. So the correct value cannot be passed to parameter.
To solve this issue, you can define the variable template at root level.
Refer to the sample:
trigger:
- None
pool:
name: AgentBuildAgile
variables:
- template: templates/vars.yml
stages:
- template: templates/build.yml
- stage: DEV_Deployment
jobs:
- job:
steps:
- script:
echo $(Dev-mnode1)
- template: templates/deploy.yml
parameters:
connection: $(Dev-mnode1)
environment: 'DEV'
I've got yaml pipeline that referes some templates.
I have variable group linked to this main yaml file and I want to pass one of its variables to the template.
It's simple when I want to "just use it" as bellow:
example:
stages:
- stage: Deployment
variables:
- group: My_group_variables
jobs:
- template: /templates/jobs/myJobTemplate.yml
parameters:
someParameter: $(variable_from_my_variable_group)
myJobTemplate.yml:
parameters:
- name: someParameter
default: ''
jobs:
- job: Myjob
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: cat ${{ parameters.someParameter}}
It does not cooperate when I want to have parameters validation like:
parameters:
- name: environmentName
type: string
values:
- Development
- Test
- UAT
- Production
Or when I want to use "service connection" name as a variable.
...
- task: KubernetesManifest#0
displayName: Deploy to Kubernetes cluster
inputs:
action: 'deploy'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: ${{ parameters.kubernetesNamespace }}
manifests: ${{ variables.manifestFile }}
...
Does anyone know how should I use those variables with pre-validated parameters or service connections?
It's most probably an issue with the time of resolving values. Pre-defined parameters and service connections names are checked on compile-time, while values from $() are resolved during runtime.
I cannot use extends and variables in this template.
Maybe someone has a pattern for those kinds of usage?
I have a requirement for a couple of job templates but I can’t get the conditions and dependencies configured to run how I want. I have the basic yaml below:
parameters:
- name: environment
displayName: Environment
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
stages:
- stage: app
displayName: App
jobs:
- deployment: kickoff
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: Write-Host "Run kickoff tasks"
- template: undeploy.yml
parameters:
environment: ${{ parameters.environment }}
action: ${{ parameters.action }}
- template: deploy.yml
parameters:
environment: ${{ parameters.environment }}
action: ${{ parameters.action }}
The requirements I have are the below:
Undeploy.yml: if Action = ‘Deploy’, does not run. If Action = ‘Undeploy’, only run this template. Action = ‘Redeploy’, run both templates but undeploy.yml must run first.
Deploy.yml: If Action = ‘Deploy’, only run this template. If Action = ‘Undeploy’, does not run. If Action = ‘Redeploy’, run both templates but deploy.yml must run second.
Closest I can get to is setting the templates yaml as below:
undeploy.yml
parameters:
- name: environment
default: environmentToDeployTo
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
jobs:
- deployment: undeploy
dependsOn: kickoff
condition: ne ('${{parameters.action}}', 'deploy')
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
#undeploy steps here#
deploy.yml
parameters:
- name: environment
default: environmentToDeployTo
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
jobs:
- deployment: dploy
dependsOn: undeploy
condition: ne ('${{parameters.action}}', 'undeploy')
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
#dploy steps here#
This results only affects deploy.yml but it is not able to run in isolation - the undeploy.yml job template will have to run beforehand. However, without the "dependsOn", I can see no other way to ensure that undeploy.yml runs first.
Is there anyway to achieve this as per requirements outlined above?
Thanks in advance
dependsOn takes the name of a stage or job as input, not the name of a template file. Seeing more of your templates would be useful, but if you could use the name of the stage or job in dependsOn, you might get the result you're looking for.
Think of the expanded template as your guide here - you can refer to a stage or job that's declared in a different template, so long as it expands to the right place when the pipeline runs.