Issues while passing variable groups parameter to template from azure pipeline.yml - azure-devops

I have declared a variable group Agile-Connections as below and the group do not have any restriction for any Pipeline:
I am using another template called vars.yml to store some other variables:
variables:
- group: Agile-Connections
- name: extensions_dir
value: /apps/agile/product/agile936/integration/sdk/extensions
- name: properties_dir
value: /apps/agile/product/Properties
- name: build_name
value: RestrictPreliminaryBOMPX.jar
- name: resource_name
value: RestrictPreliminaryBOMPX.properties
My Azure Pipeline looks like below, which is calling a deploy.yml template, and I am passing two parameters (connection, environment) from azure pipeline.yml to deploy.yml.
Below is my azure-pipeline.yml:
trigger:
- None
pool:
name: AgentBuildAgile
stages:
- template: templates/build.yml
- stage: DEV_Deployment
variables:
- template: templates/vars.yml
jobs:
- job:
steps:
- script:
echo $(Dev-mnode1)
- template: templates/deploy.yml
parameters:
connection: $(Dev-mnode1)
environment: 'DEV'
Below is my deploy.yml:
parameters:
- name: connection
- name: environment
jobs:
- deployment:
variables:
- template: vars.yml
environment: ${{ parameters.environment }}
displayName: Deploy to ${{ parameters.environment }}
strategy:
runOnce:
deploy:
steps:
- script:
echo Initiating Deployment ${{ parameters.connection }}
- template: copy-artifact.yml
parameters:
connection: ${{ parameters.connection }}
# - template: copy-resources.yml
# parameters:
# connection: ${{ parameters.connection }}
From my deploy.yml I am passing a parameter connection further to another template called copy-artifact.yml, which is below:
parameters:
- name: connection
jobs:
- job:
variables:
- template: vars.yml
displayName: 'Copy jar'
steps:
# - script:
# echo ${{ parameters.connection }}
- task: SSH#0
displayName: 'Task - Backup Existing jar file'
inputs:
sshEndpoint: ${{ parameters.connection }}
runOptions: inline
inline:
if [[ -f ${{ variables.extensions_dir }}/${{ variables.build_name }} ]]; then mv ${{ variables.extensions_dir }}/${{ variables.build_name }} ${{ variables.extensions_dir }}/${{ variables.build_name }}_"`date +"%d%m%Y%H%M%S"`"; echo "Successfully Backed up the existing jar"; fi
Now when I run my pipeline I am getting error message :
The pipeline is not valid. Job Job3: Step SSH input sshEndpoint references service connection $(Dev-mnode1) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
When I print the value of $(Dev-mnode1) using the commented out lines in copy-artifacts.yml file, it prints fine (Dev11 Connection) but when I try to use that as service connection for my ssh task, it gives me the above error.
Also, there is a service connection Dev11 Connection in my project and all the pipelines are allowed to use that service connection.

The pipeline is not valid. Job Job3: Step SSH input sshEndpoint references service connection $(Dev-mnode1) which could not be found. The service connection does not exist or has not been authorized for use.
From the error message, the parameter not get the variable value when expand the template.
Test the same sample and can reproduce the issue.
When you define the variable template at stage level, the variables in template will expand at runtime and the paramter:connection will be expand at compile time. So the correct value cannot be passed to parameter.
To solve this issue, you can define the variable template at root level.
Refer to the sample:
trigger:
- None
pool:
name: AgentBuildAgile
variables:
- template: templates/vars.yml
stages:
- template: templates/build.yml
- stage: DEV_Deployment
jobs:
- job:
steps:
- script:
echo $(Dev-mnode1)
- template: templates/deploy.yml
parameters:
connection: $(Dev-mnode1)
environment: 'DEV'

Related

Azure DevOps: An error occurred while loading the YAML build pipeline. An item with the same key has already been added

Lately I've been trying to make an Azure DevOps pipeline to deploy to 3 environments, with 2 different data centers and 2 different service connections. I've been trying to achieve this with using as little lines of YAML as possible.
After a lot of trial and error, I'm stuck on this message "An error occurred while loading the YAML build pipeline. An item with the same key has already been added."
deploy-env.yaml:
parameters:
- name: OPENSHIFT_NAMESPACE
displayName: 'OpenShift namespace'
type: object
values: []
- name: DCR
displayName: 'Data Center'
type: object
values: []
- name: OSC
displayName: 'Openshift service connection'
type: object
values: []
stages:
- ${{ each namespace in parameters.OPENSHIFT_NAMESPACE }}:
- ${{ each dcr in parameters.DCR }}:
- ${{ each osc in parameters.OSC }}:
- stage: deploy-${{ convertToJson(namespace) }}
jobs:
- deployment: deploy_to_dcr
environment: ${{ namespace }}
displayName: 'Deploy to DCR1'
strategy:
runOnce:
deploy:
steps:
- template: steps/deploy_to_cluster_with_helm_templating.yml#pipeline_templates
parameters:
DATA_CENTER: ${{ dcr }}
OPENSHIFT_NAMESPACE: ${{ namespace }}
OPENSHIFT_SERVICE_CONNECTION: '${{ osc }}'
HELM_VALUES:
- 'global.namespace=${{ namespace }}'
- 'global.data_center=${{ DCR }}'
azure-pipeline.yaml
resources:
repositories:
- repository: pipeline_templates
type: git
name: pipeline-templates
stages:
- template: deploy-env.yaml
parameters:
OPENSHIFT_NAMESPACE:
- development
- test
- acceptance
DCR:
- dcr1
- dcr2
OSC:
- OCP4DCR1
- OCP4DCR2
Does anyone knows why this error occurs? I've found other articles where stage/job names we're not unique, but that is not the case in this example.
Thanks in advance.
This line is getting repeated twelve times, with only four different values:
- stage: deploy-${{ convertToJson(namespace) }}
Stage names must be unique.

YAML Extend Stage Template

Looking to try and get some help figuring out a solution to what seems like a simple task. I am trying to develop some extend YAML templates for security within our ADO pipelines but first I have to get passed this error I am experiencing.
The error being returned by ADO when I try to validate or run the pipeline is
/ADO_Stage_Restrictions_Dev.yml#AdoRestrictions (Line: 7, Col: 3): Unexpected value job
Working Extend Template YAML
This template validates and executes without issue, which to me means I am successfully passing the stages object into the extends template
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
${{ stage }}
Broken Extend Template YAML
This template does not validate and throws the 'Unexpected value job' exception. based on the stage schema I would assume that I would be able to loop the jobs property within the stage.
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- ${{ each job in stage.jobs }}:
${{ job }}
Build YAML
The main yaml file that extends stages
resources:
repositories:
- repository: self
type: git
ref: refs/heads/Development
- repository: AdoRestrictions
type: git
name: utl-yaml-templates
ref: refs/heads/main
trigger: none
pool:
name: PROD
extends:
template: ADO_Stage_Restrictions_Dev.yml#AdoRestrictions
parameters:
stageObjs:
- stage: 'BuildStage'
displayName: 'Build Test'
jobs:
- job: 'BuildJob'
displayName: 'Build'
steps:
- task: PowerShell#2
displayName: 'Hello World'
inputs:
targetType: inline
script: |
Write-Host "Hello World"
There's nothing in your yaml which defines the start of the stage, or declares the start of the list of jobs; that's why it wasn't expecting to find 'job' there. You can add those parts in, like this:
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
${{ job }}

Azure Devops YML Template Conditions and Dependencies

I have a requirement for a couple of job templates but I can’t get the conditions and dependencies configured to run how I want. I have the basic yaml below:
parameters:
- name: environment
displayName: Environment
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
stages:
- stage: app
displayName: App
jobs:
- deployment: kickoff
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: Write-Host "Run kickoff tasks"
- template: undeploy.yml
parameters:
environment: ${{ parameters.environment }}
action: ${{ parameters.action }}
- template: deploy.yml
parameters:
environment: ${{ parameters.environment }}
action: ${{ parameters.action }}
The requirements I have are the below:
Undeploy.yml: if Action = ‘Deploy’, does not run. If Action = ‘Undeploy’, only run this template. Action = ‘Redeploy’, run both templates but undeploy.yml must run first.
Deploy.yml: If Action = ‘Deploy’, only run this template. If Action = ‘Undeploy’, does not run. If Action = ‘Redeploy’, run both templates but deploy.yml must run second.
Closest I can get to is setting the templates yaml as below:
undeploy.yml
parameters:
- name: environment
default: environmentToDeployTo
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
jobs:
- deployment: undeploy
dependsOn: kickoff
condition: ne ('${{parameters.action}}', 'deploy')
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
#undeploy steps here#
deploy.yml
parameters:
- name: environment
default: environmentToDeployTo
- name: action
values:
- deploy
- undeploy
- redeploy
default: redeploy
jobs:
- deployment: dploy
dependsOn: undeploy
condition: ne ('${{parameters.action}}', 'undeploy')
environment:
name: ${{ parameters.environment }}
resourceType: virtualMachine
tags: app
strategy:
runOnce:
deploy:
steps:
- checkout: none
#dploy steps here#
This results only affects deploy.yml but it is not able to run in isolation - the undeploy.yml job template will have to run beforehand. However, without the "dependsOn", I can see no other way to ensure that undeploy.yml runs first.
Is there anyway to achieve this as per requirements outlined above?
Thanks in advance
dependsOn takes the name of a stage or job as input, not the name of a template file. Seeing more of your templates would be useful, but if you could use the name of the stage or job in dependsOn, you might get the result you're looking for.
Think of the expanded template as your guide here - you can refer to a stage or job that's declared in a different template, so long as it expands to the right place when the pipeline runs.

Can't run a templated yaml Azure Devops pipeline with dynamically created stage

My problem
I try to create templated yaml Azure Devops pipeline:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This is my template:
parameters:
envs: {}
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
jobs:
- job: Deploy
steps:
- script: echo Deploy project
displayName: 'Deploy'
- job: Tests
steps:
- script: echo Unit tests
displayName: Test 1
But I get this error message:
An error occurred while loading the YAML build pipeline. The array
must contain at least one element. Parameter name: stages
OK, I modify my main script like this:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- stage: Build
jobs:
- job: Build
steps:
- script: echo Compilation completed...
displayName: 'Compile'
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This time pipeline runs, but only first Job. The Template is not loaded.
What I need
I was able to make this scenario working with a single file script, but I would like to make it working with a templated script.
Is this scenario supported? How I can do?
thanks
Here I have a sample as reference:
In the template YAML file (here I name it template.yaml), write as this.
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In the pipeline YAML (here I name it pipeline.yaml), write as this.
trigger:
- main
extends:
template: template.yaml
Result
If you do not want to hardcode the value of the parameter 'envs' in template.yaml, you can write like as below.
In template.yaml write as this.
parameters:
- name: envs
type: object
default: []
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In pipeline.yaml write as this.
trigger:
- main
stages:
- template: template.yaml
parameters:
envs:
- QUAL
- PROD
- PREPROD
Result. Same as above.
To view more details, you can see "YAML schema reference".

Azure Devops YAML pipeline - how to repeat a task

In my YAML pipeline I have a deployment job:
- stage: deployment_to_development
jobs:
- deployment: deployment_to_development
displayName: Deploy to Development
environment: Development
variables:
- template: migrate-vars.yml
strategy:
runOnce:
deploy:
steps:
- template: migrate-data.yml
The deployment template is a simple DbUp task:
steps:
- task: UpdateDatabaseWithDbUp#2
displayName: Migrate data
inputs:
ConnectionString: 'Data Source=$(DatabaseServer);Initial Catalog=$(DatabaseName);Integrated Security=SSPI'
ScriptPath: '$(Pipeline.Workspace)\data-migrations'
JournalSchemaName: dbo
JournalTableName: _SchemaVersions
LogScriptOutput: true
IncludeSubfolders: true
Order: FolderStructure
TransactionStrategy: SingleTransaction
The variables template defines the server and db name:
variables:
DatabaseServer: 'server'
DatabaseName: 'dbName'
Instances: '_1,_2'
This all works fine for a single database. However, I wish to repeat the task for each instance defined in the Instances variable, i.e for databases named dbName_1, dbName_2. This seemingly simple feat is proving difficult.
I have tried passing the instances as a parameter array and iterating them using
parameters:
param: []
steps:
- ${{each p in parameters.param}}:
but the 'p' variable isn't evaluated in the task.
There have been many more futile attempts. I must be missing something very obvious here. What is it?
I can't test UpdateDatabaseWithDbUp#2 but I have sth what explain how you can achieve your goal. First define template.yaml
parameters:
- name: 'instances'
type: object
default: {}
- name: 'server'
type: string
default: ''
steps:
- ${{ each instance in parameters.instances }}:
- script: echo ${{ parameters.server }}:${{ instance }}
then reuse this template in your build:
steps:
- template: template.yaml
parameters:
instances:
- test1
- test2
server: someServer
And here is the result: