Azure Pipeline: Unexpected value 'steps' - azure-devops

Can anybody point me in the right direction here?
When I attempt to kick off the pipeline from the main yaml template the reference template gives me the unexpected 'steps' value.
I attempted to add a stage before the defined job and then receive the unexpected 'stage' message. Looking at some similar previously asked questions I understand a step can't be place directly under a 'stage' but this is not the case here.
parameters:
- name: baseEnvironmentName
type: string
- name: environmentSuffix
type: string
- name: postDeploySteps
type: stepList
default: []
- name: publishedPackageName
type: string
- name: serviceName
type: string
- name: dnsHostName
type: string
jobs:
- deployment: ${{ parameters.environmentSuffix }}Deployment
displayName: '${{ parameters.environmentSuffix }} Deployment'
pool:
vmImage: 'windows-2019'
environment: ${{ parameters.baseEnvironmentName }}-${{ parameters.environmentSuffix }}
variables:
- template: variables/variables.common.yaml
- template: variables/variables.${{ parameters.environmentSuffix }}.yaml
- name: MonitorExceptionAlert_Name
value: '${{ variables.IdPrefix }}-${{ parameters.environmentSuffix }}-${{ parameters.dnsHostName }}-ExceptionAlert'
steps:
- checkout: self
- checkout: common_iac
- task: Random Task Name
displayName: 'Create Environment Resource Group'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
resourceGroupName: '$(ResourceGroup.Name)'
location: '$(ResourceGroup.Location)'
condition: and(succeeded(), ne(variables['CreateResourceGroupInTaskGroup'], 'false'))
- task: Random Task Name 2
displayName: 'Create Application Insights'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
ResourceGroupName: '$(ResourceGroup.Name)'
Location: '$(ResourceGroup.Location)'
instanceName: '$(ApplicationInsights.Name)'

You are using deployment jobs. Try defining a strategy. Try replacing steps: inline with deployment with:
- deployment: ....
strategy:
runOnce:
deploy:
steps:

From the example at the Azure docs templates page, jobs: should contain a - job: which contains steps:. I don't know that jobs: can directly contain steps:.

Related

Dynamic parameters/variables does not work for deployment job in checkout step in template but work directly in pipeline

The dynamic variables with the checkout step do not work with the template.
The same works without a template.
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.tag_name}} ## DOES NOT WORK
Getting below error while running pipeline:
"ERROR: An item with the same key has already been added"
Working:
my_pipeline.yml without template works
parameters:
- name: RELEASE_TAG
displayName: Enter The Master Release Tag Name Example 1.0.0-RELEASE
default: 1.0.0-RELEASE
type: string
trigger:
- none
variables:
db_resource_path: '$(System.DefaultWorkingDirectory)/resources/db'
pipeline_environment_name: 'PROD_ENV'
db_env_name: 'prod'
stages:
- stage: "PROD_DB_DEPLOYMENT"
displayName: "PROD DB Deployment"
pool:
name: $(param.agent.pool.name)
variables:
- group: PROD_VG
jobs:
- job:
steps:
- script: |
echo param release tag: ${{parameters.RELEASE_TAG}}
- deployment: Deploy
environment: PROD_ENV
strategy:
runOnce:
deploy:
steps:
- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.RELEASE_TAG}} ## WORKS
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{variables.tag_name}} ## WORKS WITH VAR ALSO
Not Working:
Gives ERROR: An item with the same key has already been added.
my azure-pipelines.yml is:
parameters:
- name: RELEASE_TAG
displayName: Enter The Master Release Tag Name Example 1.0.0-RELEASE
default: 1.0.0-RELEASE
type: string
resources:
repositories:
- repository: templates
type: git
name: MY_PROJECT/MY_TEMPLATE_REPO
trigger:
- none
variables:
tagName: '${{ parameters.RELEASE_TAG }}'
stages:
- stage: "PROD_DB_DEPLOYMENT"
displayName: "PROD DB Deployment"
variables:
- group: PROD_VG
jobs:
- template: my_template.yml#templates
parameters:
tag_name: $(tagName)
db_env_name: 'prod'
agent_pool_name: $(param.agent.pool.name)
db_resource_path: $(System.DefaultWorkingDirectory)/resources/db
pipeline_environment_name: PROD_ENV
is_release: 'true'
My Template is:
parameters:
- name: 'agent_pool_name'
type: string
- name: 'db_resource_path'
type: string
- name: 'pipeline_environment_name'
type: string
- name: 'db_env_name'
type: string
- name: 'is_release'
type: string
default: 'false'
- name: 'tag_name'
type: string
default: '1.0.0-RELEASE'
jobs:
- job:
pool:
name: ${{parameters.agent_pool_name}}
steps:
- script: |
echo param tag_name: ${{parameters.tag_name}}
echo var tag_name: $(tagName)
- deployment: Deploy
pool:
name: ${{parameters.agent_pool_name}}
environment: ${{parameters.pipeline_environment_name}}
strategy:
runOnce:
deploy:
steps:
- checkout: self
- ${{ if eq(parameters.is_release, true) }}:
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/1.0.0-RELEASE ## WORKS
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/$(tag_name) ## DOES NOT WORK
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.tag_name}} ## DOES NOT WORK
Tried below variable option also but get error:Unexpected value 'variables'
Any suggestion, please.
This is due to that you passed $(var) as parameters of the template.
You can pass ${{variables.tagName}} instead.
checked on my side screenshot
In a pipeline, template expression variables (${{ variables.var }}) get processed at compile time, before runtime starts. Macro syntax variables ($(var)) get processed during runtime before a task runs.
Because templates are expanded before the pipeline execution gets planned, so you cannot pass $(var) as parameter to the template.
Please check official doc for the variable syntax.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#understand-variable-syntax
Edit: for error "Unexpected value 'variables'":
Move variables to stage scope: my yaml here
My template here

Applying tags to a CFN stack using Azure DevOps pipeline - list?

Following on from my question the other day...
Passing an object into a template in Azure DevOps pipeline
I am now facing the problem where I'm trying to use the pipeline to apply tags to the AWS CloudFormation stack that gets created from the Azure DevOps pipeline.
Here's the pipeline (some bits of it) :-
# azure-pipelines.yml
parameters:
- name: stackparams
type: object
default:
- displayname: AWS Test User Account
awscredentials: TESTAWS-BuildUser
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies.yml
templateparametersfile: ./Test/TestApp/demoparams.json
tags:
- tag1
- tag2
- Managed=Cloudformation
stages:
- stage: ProdDeploy # name of the stage (A-Z, a-z, 0-9, and underscore)
displayName: Production Deploy # friendly name to display in the UI
jobs:
- deployment: DeveloperRoles # name of the deployment job (A-Z, a-z, 0-9, and underscore)
displayName: Manage Developer Roles using Cloudformation # friendly name to display in the UI
pool:
vmImage: 'ubuntu-latest'
environment: aws-test-appaccount-structure # target environment name and optionally a resource name to record the deployment history; format: <environment-name>.<resource-name>
strategy:
runOnce: #rolling, canary are the other strategies that are supported
deploy:
steps:
- template : ./Templates/CfnUpdateStack/CfnUpdateStack.yml
parameters:
stackparams: ${{ parameters.stackparams }}
runRolesGroupsPolicies: ${{ parameters.runRolesGroupsPolicies }}
Here's the template :-
# CfnUpdateStack.yml
parameters:
- name: stackparams
type: object
- name: runRolesGroupsPolicies
type: boolean
steps:
- ${{ each stackparam in parameters.stackparams }}:
- task: CloudFormationCreateOrUpdateStack#1
displayName: ${{ stackparam.displayname }}
inputs:
awsCredentials: ${{ stackparam.awscredentials }}
regionName: 'eu-west-1'
stackName: 'ConsoleSwitchRolesGroupsPolicies'
templateSource: 'file'
templateFile: ${{ stackparam.templatefile }}
templateParametersFile: ${{ stackparam.templateparametersfile }}
tags: |
Component=RoleUserAccess
${{ stackparam.tags }}
condition: eq('${{ parameters.runRolesGroupsPolicies }}', true)
When I run this I only get the first tag Component=RoleUserAccess. I have managed to get it working, but using this horrendous ugly method of using a variable (not a param) and putting a blank line between each tag. It does work though!
Versions
variables:
- name: tags
value: "tag1
tag2
Managed=Cloudformation"
We have around 20 tags for each resource, so it gets pretty messy. I have cut it down to make the question simpler to read.
Any suggestions?
Thanks,
It seems you are trying to add an array (tags) to an array (stackparams), I don't think it's correct. You may try the following syntax:
# CfnUpdateStack.yml
parameters:
- name: 'tags'
type: object
default: {}
- name: 'displayname'
type: string
default: ''
- name: 'awscredentials'
type: string
default: ''
- name: 'templatefile'
type: string
default: ''
- name: 'templateparametersfile'
type: string
default: ''
steps:
# azure-pipelines.yml
- template: CfnUpdateStack.yml
parameters:
tags:
- tag1
- tag2
- Managed=Cloudformation
displayname: AWS Test User Account
awscredentials: TESTAWS-BuildUser
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies.yml
templateparametersfile: ./Test/TestApp/demoparams.json
- template: CfnUpdateStack.yml
parameters:
tags:
- tag1
- tag2
- Managed=Cloudformation
displayname: AWS Test User Account2
awscredentials: TESTAWS-BuildUser2
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies2.yml
templateparametersfile: ./Test/TestApp/demoparams2.json

azure devOps yml pipeline: pass Boolean value to task prisma-cloud-compute-scan#2 written in template

I want to send Boolean value to task 'prisma-cloud-compute-scan#2' written in a template file.
It always gives below error.
Error: The 'prisma_continue_on_error' parameter value '$(prismaContinueOnError)' is not a valid Boolean.
Main pipeline abc.yml
resources:
repositories:
- repository: templates
type: git
name: my_projects/my-build-templates
ref: refs/heads/features/add-build-template
variables:
name: prismaContinueOnError
value: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
stages:
- stage: "Build"
displayName: Build
jobs:
- template: my_build_stage/my_template.yml#templates
parameters:
prisma_continue_on_error: $(prismaContinueOnError)
Template my_template.yml
parameters:
- name: prisma_continue_on_error
type: boolean
default: false
- name: pool_name
type: string
default: abc_pool
jobs:
- job: Build
pool:
name: ${{parameters.pool_name}}
steps:
- task: prisma-cloud-compute-scan#2
inputs:
scanType: 'images'
twistlockService: 'SERVICE_CONNECTIONM_NAME'
artifact: ...
continueOnError: ${{parameters.prisma_continue_on_error}}
You mixed syntaxes here
variables:
name: prismaContinueOnError
value: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
it should be:
variables:
prismaContinueOnError: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
But this will not solve the issue, because variables are just string. You can't have variable of type boolean. You need to pass there runtime expression which delay type evaluation:
stages:
- stage: "Build"
displayName: Build
jobs:
- template: my_build_stage/my_template.yml#templates
parameters:
prisma_continue_on_error: ${{ variables.prismaContinueOnError }}

Execute Azure Devops job on a pool based on conditional parameter

I am trying to execute an Azure Devops Job on a specific pool based on a condition.
The goal is to switch between self-hosted agent and microsoft agent.
Here is the configuration:
parameters:
custom_agent: true
jobs:
- job: Test
displayName: Test job
- ${{ if eq(parameters.custom_agent, true) }}:
- pool:
name: mypool
demands:
- agent.os -equals Linux
- ${{ if eq(parameters.custom_agent, false) }}:
- pool:
vmImage: 'ubuntu-latest'
steps:
- task: npmAuthenticate#0
Any ideas ?
The below example solved my requirement
parameters:
- name: 'vmImage'
type: string
default: 'ubuntu-latest'
- name: 'agentPool'
type: string
default: ''
jobs:
- job: 'Example'
pool:
${{ if ne(parameters.agentPool, '') }}:
name: ${{ parameters.agentPool }}
${{ if eq(parameters.agentPool, '') }}:
vmImage: ${{ parameters.vmImage }}
steps:
- script: example
Another apporach to conditionally select pools if you use non-vm pools:
variables:
- ${{ if eq(parameters.custom_agent, true) }}:
- name: testJobPool
value: mypool
- ${{ if eq(parameters.custom_agent, false) }}:
- name: testJobPool
value: mypool_second
jobs:
- job: Test
displayName: Test job
pool:
name: $(testJobPool)
steps:
- task: npmAuthenticate#0
This has proved working.
We can specify conditions under which a step, job, or stage will run. We can configure the jobs in the pipeline with different condition entries, and set demands based on those conditions.
A skeleton version looks like this:
parameters:
- name: custom_agent
displayName: Pool Image
type: boolean
default: True
jobs:
- job: selfhostedagent
condition: eq(${{ parameters.custom_agent }}, True)
displayName: 'self_hosted agent'
pool:
name: Default
demands:
- Agent.Name -equals WS-VITOL-01
steps:
- script: echo self_hosted agent
- job: hostedagent
condition: eq(${{ parameters.custom_agent }}, False)
displayName: 'hosted agent'
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo hosted agent
Update1
In addition, we can configure task template, then use the template in the steps.
Result:
It looks like pool is not a valid property of a job type
Try switching your job to a deployment type:
jobs:
- deployment: Test
- ${{ if eq(parameters.custom_agent, true) }}:
pool:
name: mypool
demands:
agent.os -equals Linux
strategy:
runOnce:
deploy:
steps:

Parameterized variable names as task input in Azure Pipelines

I've been trying to make a YAML template that first uses the AzureKeyVault#1 task to get the value ofsome Azure KeyVault secrets, and then uses these secrets for the sqlUsername and sqlPassword in asqlAzureDacpacDeployment#1 task.
I want to make the names of the KeyVault secrets a parameter, so that this template can be used for many different situations.
I've successfully used this technique before, with an AzureKeyVault#1 task and then an AzurePowerShell#4 task, where the secret gets injected as an environment variable for the PowerShell script.
This is a dressed down version of the working template:
parameters:
- name: subscription
type: string
- name: keyVaultSecretName
type: string
- name: keyVault
type: string
jobs:
- job: Run_PowerShell_With_Secret
pool:
name: Azure Pipelines
vmImage: windows-latest
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.subscription }}
keyVaultName: ${{ parameters.keyVault }}
secretsFilter: ${{ parameters.keyVaultSecretName }}
- task: AzurePowerShell#4
inputs:
azureSubscription: ${{ parameters.subscription }}
ScriptPath: 'some_script.ps1'
azurePowerShellVersion: LatestVersion
env:
SECRETVALUE: $(${{ parameters.keyVaultSecretName }})
And here is the template where I can't get the same technique to work:
parameters:
- name: subscription
type: string
- name: keyVault
type: string
- name: sqlServerName
type: string
- name: sqlDatabaseName
type: string
- name: sqlServerAdminSecretName
type: string
- name: sqlServerPasswordSecretName
type: string
- name: dacpacName
type: string
- name: artifactName
type: string
jobs:
- job: Deploy_SQL_Database
pool:
name: Azure Pipelines
vmImage: windows-latest
steps:
- task: DownloadPipelineArtifact#2
inputs:
artifact: ${{ parameters.artifactName }}_artifacts
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.subscription }}
keyVaultName: ${{ parameters.keyVault }}
secretsFilter: '${{ parameters.sqlServerAdminSecretName }}, ${{ parameters.sqlServerPasswordSecretName }}'
- task: sqlAzureDacpacDeployment#1
inputs:
azureSubscription: ${{ parameters.subscription }}
ServerName: ${{ parameters.sqlServerName }}.database.windows.net
DatabaseName: ${{ parameters.sqlDatabaseName }}
sqlUsername: $(${{ parameters.sqlServerAdminSecretName }})
sqlPassword: $(${{ parameters.sqlServerPasswordSecretName }})
DacpacFile: $(Pipeline.Workspace)\${{ parameters.dacpacName }}.dacpac
I can get the template to work if I hardcode the secret names:
parameters:
- name: subscriptionName
type: string
- name: keyVault
type: string
- name: sqlServerName
type: string
- name: sqlDatabaseName
type: string
- name: dacpacName
type: string
- name: artifactName
type: string
jobs:
- job: Deploy_${{ parameters.sqlDatabaseName }}_Database
pool:
name: Azure Pipelines
vmImage: windows-latest
steps:
- checkout: none
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.subscriptionName }}
keyVaultName: ${{ parameters.keyVault }}
secretsFilter: 'SQLServerAdmin, SQLServerPassword'
- task: DownloadPipelineArtifact#2
inputs:
artifact: ${{ parameters.artifactName }}_artifacts
- task: sqlAzureDacpacDeployment#1
inputs:
azureSubscription: ${{ parameters.subscriptionName }}
ServerName: ${{ parameters.sqlServerName }}.database.windows.net
DatabaseName: ${{ parameters.sqlDatabaseName }}
sqlUsername: $(sqlServerAdmin)
sqlPassword: $(sqlServerPassword)
DacpacFile: $(Pipeline.Workspace)\${{ parameters.dacpacName }}.dacpac
Although this works for us for now, I find this sub-optimal. Is there any way I can make these parameterized variable names work?