set and refer to variables in yaml - azure-devops

I have the following yml code that sets and refers to some variables as follows:
<one.yml>
- task: AzurePowerShell#4
displayName: 'Copy functions templates'
inputs:
azureSubscription: ${{parameters.serviceConnection}}
ScriptPath: ${{ parameters.root }}/Scripts/ReleaseManagement/CopyChildTemplatesToContainer.ps1
ScriptArguments: '-resourceGroupName ''${{ parameters.solutionAbbreviation}}-data-${{ parameters.environmentAbbreviation}}''
name: copyFunctionsTemplates
- powershell: |
Write-Host "##vso[task.setvariable variable=data_containerSASToken;isOutput=true]$(copyFunctionsTemplates.containerSASToken)"
Write-Host "##vso[task.setvariable variable=data_containerEndPoint;isOutput=true]$(copyFunctionsTemplates.containerEndPoint)"
displayName: 'set data output variables'
name: dataVariables
<two.yml>
stages:
- deployment: ${{ parameters.stageName }}_DeployResources
displayName: ${{ parameters.stageName }}_DeployResources
- stage: ${{ parameters.stageName }}
dependsOn: ${{ parameters.dependsOn }}
condition: ${{ parameters.condition }}
jobs:
- deployment: ${{ parameters.stageName }}_DeployResources
displayName: ${{ parameters.stageName }}_DeployResources
steps:
- template: one.yml
jobs:
- job: ${{ parameters.stageName }}_DeployFunctions
dependsOn: ${{ parameters.stageName }}_DeployResources
variables:
data_containerEndPoint: $[ dependencies.DeployResources.outputs['DeployResources.dataVariables.data_containerEndPoint'] ]
data_containerSASToken: $[ dependencies.DeployResources.outputs['DeployResources.dataVariables.data_containerSASToken'] ]
steps:
- ${{ each func in parameters.functionApps }}:
- template: three.yml
<three.yml>
steps:
- task: AzureResourceGroupDeployment#2
displayName: 'deploy ${{ parameters.name }} data resources'
inputs:
azureSubscription: ${{parameters.serviceConnection}}
resourceGroupName: ${{parameters.solutionAbbreviation}}-data-${{parameters.environmentAbbreviation}}
location: ${{parameters.location}}
csmFile: ${{ parameters.root }}/functions_arm_templates/${{ parameters.name }}/Infrastructure/data/template.json
csmParametersFile: ${{ parameters.root }}/functions_arm_templates/${{ parameters.name }}/Infrastructure/data/parameters/parameters.${{parameters.environmentAbbreviation}}.json
overrideParameters: -environmentAbbreviation "${{parameters.environmentAbbreviation}}"
-tenantId "${{parameters.tenantId}}"
-solutionAbbreviation "${{parameters.solutionAbbreviation}}"
-containerBaseUrl "$(data_containerEndPoint)functions/${{ parameters.name }}/Infrastructure/data/"
-containerSasToken "$(data_containerSASToken)"
deploymentMode: 'Incremental'
On enabling the debug mode while running pipeline, I see values printed for data_containerSASToken and data_containerEndPoint from the task 'Copy functions templates' however I see empty values from the task 'deploy data resources'. What am I missing?

Your problem may be in when you retrieve the output from the previous job:
data_containerEndPoint: $[ dependencies.DeployResources.outputs['DeployResources.dataVariables.data_containerEndPoint'] ]
That's looking for a prior job called DeployResources, but the prior job is actually called {{ parameters.stageName }}_DeployResources.

Related

Not able to set variable based on parameters in YAML

I'm trying to build a new yaml file that reads keyvault secrets based on the parameters at the runtime and declared variables with the condition as per the parameters, but this isn't working.
- name: azure_subscription
displayName: " Select subscription "
type: string
default: "service-connection-dev"
values:
- 'service-connection-dev'
- 'service-connection-sit'
- 'service-connection-tes'
- 'service-connection-prd'
variables:
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-sit') }}:
name: key_vault
value: 'core-kv-sit'
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-dev') }}:
name: key_vault
value: 'core-kv-dev'
stages:
- stage: Validate
${{ if eq(parameters.azure_subscription, 'service-connection-dev') }}:
pool:
name: agent-pool-win-dev
${{ if eq(parameters.azure_subscription, 'service-connection-sit') }}:
pool:
name: agent-pool-win-sit
jobs:
- job: Validate
steps:
- task: AzureKeyVault#2
inputs:
KeyVaultName: "${{variables.key_vault}}"
SecretsFilter: "*"
RunAsPreJob: false
azureSubscription: ${{ parameters.azure_subscription }}
I've tried using variables inside jobs, but that is also not working. Can someone please help?
Also, I'll have to declare 2 more variables as per the parameters input, Is it possible ?
Thanks in advance
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-sit') }}:
You're using a literal value of "${{ parameters.azure_subscription }}" for the left side of the comparison. The comparison should just be parameters.azure_subscription.
So: - ${{ if eq(parameters.azure_subscription, 'service-connection-sit') }}:

How to use a dependsOn for the previous loop in Azure Devops YAML pipeline while using a each argument

I have a pipeline setup that is very basic and using the below template
parameters:
- name: countries
type: object
default:
- country: 'uk'
envs: ['development', 'preproduction', 'production']
- country: 'us'
envs: ['development', 'preproduction', 'production']
stages:
- ${{ each c in parameters.countries }}:
- stage: ${{ c.country }}
displayName: ${{ c.country }}
# some variable here to control if to use [] to deploy to all stages or just one
dependsOn: []
jobs:
- ${{ each env in c.envs }}:
- job: ${{ c.country }}_${{ env }}
steps:
- script: echo "Hello world"
condition: contains(variables['System.JobDisplayName'], 'production')
- script: echo "Hello world 2"
I want to have it so it only deploys to the new job loop in the pipeline if the previous job loop has succeeded. I can't seem to figure out a way of doing this. Any help will be appreicated. Thanks!
I managed to achieve this with the below job spec
- job: ${{ c.country }}_${{ env }}
${{if eq(env, 'preproduction')}}:
condition: eq(dependencies.${{ c.country }}_development.result, 'Succeeded')
dependsOn: ${{ c.country }}_development
${{if eq(env, 'production')}}:
condition: eq(dependencies.${{ c.country }}_preproduction.result, 'Succeeded')
dependsOn: ${{ c.country }}_preproduction
steps:

AzureDevops Pipeline Templates From Folder

I am trying to use templates in my pipeline that are in the same project folder but on Validate yam and run pipeline gives me an error
/templates/transform-settings.yml (Line: 1, Col: 1): A sequence was not expected
Here is the part of azure-pipelines.yml and template:
imagePullSecret: ' fd bfgbgf '
dockerfilePath: '**/Dockerfile'
vmImageName: 'ubuntu-latest'
testVar: 'test'
tag: '$(Build.BuildId)'
ConnectionStrings.DefaultConnection: ""
stages:
- template: templates/transform-settings.yml
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
.... and template:
- stage: TransformFiles
displayName: TransformFiles
variables:
- ${{ if eq(variables['Build.SourceBranchName'], 'development') }}:
- group: dev-secrets
- name: testVar
value: 'dev'
- name: ConnectionStrings.DefaultConnection
value: $(psql-conn-str-dev)
- ${{ if eq(variables['Build.SourceBranchName'], 'qa') }}:
- group: qa-secrets
- name: testVar
value: 'qa'
jobs:
- job: Transform_AppSettings
steps:
- bash: echo "===== Transforming appsettings.json for $(variables['Build.SourceBranchName']) environment ====="
displayName: 'File Transform'
- task: FileTransform#1
inputs:
folderPath: '$(System.DefaultWorkingDirectory)'
fileType: 'json'
targetFiles: 'appsettings.json'
- upload: appsettings.json
artifact: appsettings
/templates/transform-settings.yml (Line: 1, Col: 1): A sequence was not expected
Based on your yaml sample, the issue is related to the format of the YAML template.
To solve this issue, you need to add the stages: field at the top of the template YAML file.
For example:
azure-pipelines.yml
stages:
- template: templates/transform-settings.yml
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
transform-settings.yml
stages:
- stage: TransformFiles
displayName: TransformFiles
variables:
- ${{ if eq(variables['Build.SourceBranchName'], 'development') }}:
- group: dev-secrets
- name: testVar
value: 'dev'
- name: ConnectionStrings.DefaultConnection
value: $(psql-conn-str-dev)
- ${{ if eq(variables['Build.SourceBranchName'], 'qa') }}:
- group: qa-secrets
- name: testVar
value: 'qa'
jobs:
- job: Transform_AppSettings
steps:
- bash: echo "===== Transforming appsettings.json for $(variables['Build.SourceBranchName']) environment ====="
displayName: 'File Transform'
- task: FileTransform#1
inputs:
folderPath: '$(System.DefaultWorkingDirectory)'
fileType: 'json'
targetFiles: 'appsettings.json'
- upload: appsettings.json
artifact: appsettings
For more detailed info, you can refer to this doc: YAML template.

Conditional dependson for multiple stage YAML

I have a YAML pipeline which contains some template files.
Within my pipeline, there are 4 stages that run in parallel to apply DSC. I then have a destroy task which i would like to run, only when all 4 tasks have ran successfully. When i try to add a depends on with a list:
dependsOn:
- Stage_A
- Stage_B
- Stage_C
- Stage_D
The error I get is:
The 'dependsOn' parameter is not a valid String.
My template YAML looks like:
...
stages:
...
- template: Apply-DSC.yml
parameters:
azureSub: '[sub]'
AutoAccountResourceGroup: 'rg'
AutoAccountName: 'aa'
environment: 'b1'
stageDependsOn: 'b1_apply'
- template: Destroy-Pipeline.yml
parameters:
azureSub: '[sub]'
terraformStorageAccountResourceGroup: 'rg'
terraformStorageAccountName: '[]'
terraformStorageContainerName: '[]'
terraformStorageRemoteStateKey: '[].tfstate'
environment: 'b1'
terraformEnvironmentFileName: 'B01'
dependsOn: 'Stage_A'
I have 4 stages within my Apply-DSC.yml
Stage_A
Stage_B
Stage_C
Stage_D
Question is, is this possible for my destroy stage to await a successful deployment of Stages A-D when using these stage templates?
Thanks.
Edit: Adding Destroy-Pipeline.yml
# Run & upload Terraform plan
parameters:
- name: azureSub
type: string
- name: terraformStorageAccountResourceGroup
type: string
- name: terraformStorageAccountName
type: string
- name: terraformStorageContainerName
type: string
- name: terraformStorageRemoteStateKey
type: string
- name: environment
type: string
- name: terraformEnvironmentFileName
type: string
- name: dependsOn
type: string
stages:
- stage: Destroy_${{ parameters.environment }}
dependsOn: ${{ parameters.dependsOn }}
jobs:
- deployment: '${{ parameters.environment }}_Destroy'
displayName: '${{ parameters.environment }} Destroy'
environment: '${{ parameters.environment }} destroy'
pool:
vmImage: windows-latest
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: 'drop'
name: 'Download_Terraform_code'
displayName: 'Download Terraform code'
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller#0
inputs:
terraformVersion: '$(TerraformVersion)'
displayName: 'Install Terraform'
- task: TerraformCLI#0
inputs:
command: 'init'
workingDirectory: '$(Pipeline.Workspace)/Drop'
backendType: 'azurerm'
backendServiceArm: '${{ parameters.azureSub }}'
backendAzureRmResourceGroupName: '${{ parameters.terraformStorageAccountResourceGroup }}'
backendAzureRmStorageAccountName: '${{ parameters.terraformStorageAccountName }}'
backendAzureRmContainerName: '${{ parameters.terraformStorageContainerName }}'
backendAzureRmKey: '${{ parameters.terraformStorageRemoteStateKey }}'
allowTelemetryCollection: false
displayName: 'Terraform Init'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
terraform workspace select $(WorkspaceEnvironment)
workingDirectory: '$(Pipeline.Workspace)/Drop'
displayName: 'Select Workspace'
- task: TerraformCLI#0
inputs:
command: 'plan'
environmentServiceName: '${{ parameters.azureSub }}'
commandOptions: '-destroy -var-file="./environments/${{ parameters.terraformEnvironmentFileName }}.tfvars" -input=false'
allowTelemetryCollection: false
workingDirectory: '$(Pipeline.Workspace)/Drop'
displayName: 'Plan Destroy'
- task: TerraformCLI#0
inputs:
command: 'destroy'
workingDirectory: '$(Pipeline.Workspace)/Drop'
environmentServiceName: '${{ parameters.azureSub }}'
commandOptions: '-var-file="./environments/${{ parameters.terraformEnvironmentFileName }}.tfvars" -input=false '
allowTelemetryCollection: false
displayName: 'Run Destroy'
I changed the type from string to object
parameters:
- name: dependsOn
type: object
default: []
Then within my template block i added the object like:
- template: Destroy-Pipeline.yml
parameters:
...
dependsOn: ['Stage_A', 'Stage_B' ...]

YAML Extends Template - Unrecognized value: 'else'. Located at position 1 within expression: else

Expanding on my first question. I am now experiencing a new error Unrecognized value: 'else'. Located at position 1 within expression: else that I cannot figure out a solution to. The issue happens when I add the 'else' condition within the foreach of the job.steps.
Build YAML
resources:
repositories:
- repository: self
type: git
ref: refs/heads/Development
- repository: AdoRestrictions
type: git
name: utl-yaml-templates
ref: refs/heads/main
trigger: none
pool:
name: PROD
extends:
template: ADO_Stage_Restrictions_Dev.yml#AdoRestrictions
parameters:
stageObjs:
- stage: 'BuildStage'
displayName: 'Build Test'
jobs:
- job: 'BuildJob'
displayName: 'Build'
steps:
- task: PowerShell#2
displayName: 'PS Hello World'
inputs:
targetType: inline
script: |
Write-Host "Hello World"
Working Extends YAML
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
- job: ${{ job.job }}
displayName: ${{ job.displayName }}
steps:
- ${{ each step in job.steps }}:
- ${{ if or(startsWith(step.task, 'PowerShell'),startsWith(step.task, 'CmdLine'),startsWith(step.task, 'Bash'),startsWith(step.task, 'ShellScript'),containsValue(step.task, 'Script'),containsValue(step.task, 'CLI'),containsValue(step.task, 'PowerShell')) }}:
- task: PowerShell#2
displayName: 'Unapproved - ${{ step.displayName }}'
inputs:
targetType: inline
script: Write-Output "Unapproved Task - Scripting and CLI tasks are not approved for use in yaml pipelines"
Broken Extends YAML
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
- job: ${{ job.job }}
displayName: ${{ job.displayName }}
steps:
- ${{ each step in job.steps }}:
- ${{ if or(startsWith(step.task, 'PowerShell'),startsWith(step.task, 'CmdLine'),startsWith(step.task, 'Bash'),startsWith(step.task, 'ShellScript'),containsValue(step.task, 'Script'),containsValue(step.task, 'CLI'),containsValue(step.task, 'PowerShell')) }}:
- task: PowerShell#2
displayName: 'Unapproved - ${{ step.displayName }}'
inputs:
targetType: inline
script: Write-Output "Unapproved Task - Scripting and CLI tasks are not approved for use in yaml pipelines"
- ${{ else }}:
- ${{ step }}
Full Error Message
(Line: 22, Col: 13): Unrecognized value: 'else'. Located at position 1 within expression: else. (Line: 22, Col: 13): Expected at least one key-value pair in the mapping
Update - Working Solution
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
- job: ${{ job.job }}
displayName: ${{ job.displayName }}
steps:
- ${{ each step in job.steps }}:
#Disallow any type of scripting tasks
- ${{ if or(startsWith(step.task, 'PowerShell'),startsWith(step.task, 'CmdLine'),startsWith(step.task, 'Bash'),startsWith(step.task, 'ShellScript'),contains(step.task, 'Script'),contains(step.task, 'CLI'),contains(step.task, 'PowerShell')) }}:
- task: PowerShell#2
displayName: 'Unapproved Task - ${{ step.displayName }}'
inputs:
targetType: inline
script: throw "Unapproved Task - Scripting and CLI tasks are not approved for use in yaml pipelines"
#Not a scripting task perform additional checks
- ${{ if and(not(startsWith(step.task, 'PowerShell')),not(startsWith(step.task, 'CmdLine')),not(startsWith(step.task, 'Bash')),not(startsWith(step.task, 'ShellScript')),not(contains(step.task, 'Script')),not(contains(step.task, 'CLI')),not(contains(step.task, 'PowerShell')) ) }}:
#Validate azure subscription provided for the environment
- ${{ if contains(step.task, 'Azure') }}:
- ${{ each pair in step }}:
${{ if eq(pair.key, 'inputs') }}:
inputs:
${{ each attribute in pair.value }}:
${{ if contains(attribute.key, 'Subscription') }}:
${{ if and(ne(attribute.value, 'sub-name1'), ne(attribute.value, 'sub-name2'), ne(attribute.value, 'sub-name3')) }}:
${{ attribute.value }}: ''
${{ if or(eq(attribute.value, 'sub-name1'), eq(attribute.value, 'sub-name2'), eq(attribute.value, 'sub-anme3')) }}:
${{ pair.key }}: ${{ pair.value }}
${{ if ne(pair.key, 'inputs') }}:
${{ if eq(pair.key, 'displayName') }}:
${{ pair.key }}: 'Invalid Azure Subscription - ${{ pair.value }}'
${{ if ne(pair.key, 'displayName') }}:
${{ pair.key }}: ${{ pair.value }}
#Allow all other tasks
- ${{ if not(contains(step.task, 'Azure')) }}:
- ${{ step }}