I want to define a deployment job (via a template) but, running my azure pipeline, the following errors is displayed:
job-deploy.yml#templates: Expected mapping end
Where is my issue ?
Following the template called:
parameters:
- name: clientBaseName
type: string
- name: environment
type: string
- name: aks
type: string
- name: helm
type: string
default: 'helm3'
values:
- 'helm2'
- 'helm3'
jobs:
- deployment: deploy_{{ parameters.environment }}
displayName: 'Deploy a MyPlace client.'
environment: approvals-demo-core
strategy:
runOnce:
preDeploy:
steps:
- template: ../tasks/task-chart-setup.yml
parameters:
helm: ${{ parameters.helm }}
deploy:
steps:
- template: ../tasks/task-chart-deploy.yml
parameters:
type: data
namespace: ${{ parameters.clientBaseName }}-{{ parameters.environment }}
charts: ./charts/data
values: ./output/{{ parameters.environment }}/data.yaml
aks: {{ parameters.aks }}
- template: ../tasks/task-chart-deploy.yml
parameters:
type: services
namespace: ${{ parameters.clientBaseName }}-{{ parameters.environment }}
charts: ./charts/services
values: ./output/{{ parameters.environment }}/services.yaml
aks: {{ parameters.aks }}
- template: ../tasks/task-chart-deploy.yml
parameters:
type: jobs
namespace: ${{ parameters.clientBaseName }}-{{ parameters.environment }}
charts: ./charts/jobs
values: ./output/{{ parameters.environment }}/jobs.yaml
aks: {{ parameters.aks }}
Expected mapping end usually refers to the error in the yaml syntax format. "$" is missing from the reference variable in your yaml file.
You need to change {{ parameters.environment }} to ${{ parameters.environment }}
parameters:
- name: clientBaseName
type: string
- name: environment
type: string
- name: aks
type: string
- name: helm
type: string
default: 'helm3'
values:
- 'helm2'
- 'helm3'
jobs:
- deployment: deploy_${{ parameters.environment }}
displayName: 'Deploy a MyPlace client.'
# environment: approvals-demo-core
strategy:
runOnce:
preDeploy:
steps:
- template: ../tasks/task-chart-setup.yml
parameters:
helm: ${{ parameters.helm }}
deploy:
steps:
- template: ../tasks/task-chart-deploy.yml
parameters:
type: data
namespace: ${{ parameters.clientBaseName }}-${{ parameters.environment }}
charts: ./charts/data
values: ./output/{{ parameters.environment }}/data.yaml
aks: ${{ parameters.aks }}
- template: ../tasks/task-chart-deploy.yml
parameters:
type: services
namespace: ${{ parameters.clientBaseName }}-${{ parameters.environment }}
charts: ./charts/services
values: ./output/${{ parameters.environment }}/services.yaml
aks: ${{ parameters.aks }}
- template: ../tasks/task-chart-deploy.yml
parameters:
type: jobs
namespace: ${{ parameters.clientBaseName }}-${{ parameters.environment }}
charts: ./charts/jobs
values: ./output/${{ parameters.environment }}/jobs.yaml
aks: ${{ parameters.aks }}
Related
I have this problem. When trying to use variable groups with conditional templates pipeline gives me this error:
templates/vars-qa.yml (Line: 9, Col: 1): While parsing a block mapping, did not find expected key.
Here is the azure-pipelines.yml
trigger:
- main
- qa
- development
- staging
#//
resources:
- repo: self
variables:
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- template: templates/vars-qa.yml
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/qa') }}:
- template: templates/vars-qa.yml
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/development') }}:
- template: templates/vars-dev.yml
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/staging') }}:
- template: templates/vars-qa.yml
stages:
- template: templates/transform-settings.yml
- template: templates/build-image.yml
And vars-qa.yml where error is happen:
variables:
imageRepository: 'service-qa'
dockerRegistryServiceConnection: 'dec124f0-814f-4511-94d3-b3396698767508'
containerRegistry: 'test.azurecr.io'
imagePullSecret: 'test1334fe31-auth'
dockerfilePath: '**/Dockerfile'
vmImageName: 'ubuntu-latest'
tag: '$(Build.BuildId)'
- group: dev-secrets
- name: ConnectionStrings.DefaultConnection
value: $(psql-conn-str-dev)
This is happen in -group: dev-secrets.The group is existing and the psql-conn-str-dev already to.I have tried with $(variables.psql-conn-str-dev) but result was the same. Everything works correctly without templates
There is a problem in the indent of your template YAML.
You cannot define a sequence item when in a mapping.
Please try and use an online YAML validator tool to check this:
https://codebeautify.org/yaml-validator
I converted your variables to sequence items.
I think this will work better:
variables:
- name: imageRepository
value: 'service-qa'
- name: dockerRegistryServiceConnection
value: 'dec124f0-814f-4511-94d3-b3396698767508'
- name: containerRegistry
value: 'test.azurecr.io'
- name: imagePullSecret
value: 'test1334fe31-auth'
- name: dockerfilePath
value: '**/Dockerfile'
- name: vmImageName
value: 'ubuntu-latest'
- name: tag
value: '$(Build.BuildId)'
- group: dev-secrets
- name: ConnectionStrings.DefaultConnection
value: $(psql-conn-str-dev)
I want to run a loop in a template to download two artifacts with specific versions.
I have been trying to formulate solutions for this but no luck yet, this is what I've came up until now but i think its not supported.
Can someone point me in the right direction if this is possible?
pipeline.yml
variables:
- template: project.variables.yml
jobs:
- job: 'Deploy'
steps:
- template: instantclient.template.yml
parameters:
artifacts:
oracle-instantclient:
package: 'oracle-instantclient'
packageVersion: ${{ variables.oracle-instantclient }}
oracle-data-access-components:
package: 'oracle-data-access-components'
packageVersion: ${{ variables.oracle-data-access-components }}
instantclient.template.yml
parameters:
- name: artifacts
type: object
- name: feed
default: ahitapplicationteam
- name: downloadDirectory
default: deployment/s
steps:
- ${{ each artifact in parameters.artifacts}}:
- template: artifacts.template.yml
parameters:
packageVersion: ${{ packageVersion }}
feed: ${{ parameters.feed }}
package: ${{ package }}
downloadDirectory: ${{ parameters.downloadDirectory }}
artifacts.template.yml
parameters:
- name: packageVersion
- name: feed
- name: package
- name: downloadDirectory
steps:
- task: UniversalPackages#0
displayName: 'Downloading | Feed: ${{ parameters.feed }} | Package: ${{ parameters.package }}' #| PackageVersion: ${{ parameters.packageVersion }}
inputs:
command: 'download'
downloadDirectory: ${{ parameters.downloadDirectory }}
vstsFeed: ${{ parameters.feed }}
vstsFeedPackage: ${{ parameters.package }}
vstsPackageVersion: ${{ parameters.packageVersion }}
verbosity: 'Debug'
You're on the right track. You need to add the - character to each item in your object to convert it into an array. The object can be an array of simple strings or complex objects. As an object, you can access the properties of your objects in the each loop.
The use of ${{ variables.oracle-data-access-components }} assumes that the oracle-data-access-components variable is available at compile-time when the pipeline is initially processed.
Whether you want to break it into 3 templates is a stylistic decision. I went with 2 templates to simplify readability, but a third template will provide you will some validation for required parameters.
pipeline.yml
variables:
- template: project.variables.yml
jobs:
- job: 'Deploy'
steps:
- template: instantclient.template.yml
parameters:
artifacts:
- name: 'oracle-instantclient'
version: ${{ variables.oracle-instantclient }}
- name: 'oracle-data-access-components'
version: ${{ variables.oracle-data-access-components }}
instantclient.template.yml
parameters:
# list of package to download (name, version)
- name: artifacts
type: object
# azure artifact feed name
- name: feed
type: string
default: 'ahitapplicationteam'
# download path for artifacts
- name: downloadDirectory
type: string
default: 'deployment/s'
steps:
# loop through the artifacts (name, version)
- ${{ each artifact in parameters.artifacts}}:
# download the artifact
- task: UniversalPackages#0
displayName: 'Downloading | Feed: ${{ parameters.feed }} | Package: ${{ artifact.name }}' #| PackageVersion: ${{ artifact.version }}
inputs:
command: 'download'
downloadDirectory: ${{ parameters.downloadDirectory }}
vstsFeed: ${{ parameters.feed }}
vstsFeedPackage: ${{ artifact.name }}
vstsPackageVersion: ${{ artifact.version }}
verbosity: 'Debug'
I am using outputs on each job as a hack to enable Github environments to control if my reusable workflow runs.
My only concern is the "ENV_AWS_ACCESS_KEY_ID" & "ENV_AWS_SECRET_ACCESS_KEY". These secrets are environment specific. How does the reusable workflow know what secret I am passing in?
Is there a risk with the current setup it could get overwritten if two environments got ran at the same time?
name: Used to rollback docker containers
on:
workflow_call:
inputs:
tag_to_identify_containers:
description: The last known containers prior to deployment
type: choice
required: true
options:
- last-known-testing
- last-known-integrate
- last-known-production
new_tag_to_apply_to_containers:
type: choice
required: true
options:
- testing-latest
- integrate-latest
- production-latest
jobs:
rollback_on_testing:
runs-on: ubuntu-latest
name: Rollback on testing
outputs:
signal_deployment: ${{ steps.step_id.outputs.environment }}
environment:
name: test
url: https://test.###/
steps:
- id: step_id
run: echo "::set-output name=environment::test"
retag_and_rollback_test:
needs: rollback_on_testing
if: needs.rollback_on_testing.outputs.signal_deployment == 'test'
uses: ###/###/.github/workflows/container-tagger.yml#main
with:
tag_to_identify_containers: ${{ github.event.inputs.tag_to_identify_containers }}
new_tag_to_apply_to_containers: ${{ github.event.inputs.new_tag_to_apply_to_containers }}
aws-region: eu-west-2
run_cron_and_cycle_containers: true
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.SHARED_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SHARED_AWS_SECRET_ACCESS_KEY }}
ENV_AWS_ACCESS_KEY_ID: ${{ secrets.THIS_AWS_ACCESS_KEY_ID }}
ENV_AWS_SECRET_ACCESS_KEY: ${{ secrets.THIS_AWS_SECRET_ACCESS_KEY }}
rollback_on_integrate:
runs-on: ubuntu-latest
name: Rollback on Integrate
outputs:
signal_deployment: ${{ steps.step_id.outputs.environment }}
environment:
name: integrate
url: https://integrate.###/
steps:
- id: step_id
run: echo "::set-output name=environment::integrate"
retag_and_rollback_integrate:
needs: rollback_on_integrate
if: needs.rollback_on_integrate.outputs.signal_deployment == 'integrate'
uses: ###/###/.github/workflows/container-tagger.yml#main
with:
tag_to_identify_containers: ${{ github.event.inputs.tag_to_identify_containers }}
new_tag_to_apply_to_containers: ${{ github.event.inputs.new_tag_to_apply_to_containers }}
aws-region: eu-west-2
run_cron_and_cycle_containers: true
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.SHARED_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SHARED_AWS_SECRET_ACCESS_KEY }}
ENV_AWS_ACCESS_KEY_ID: ${{ secrets.THIS_AWS_ACCESS_KEY_ID }}
ENV_AWS_SECRET_ACCESS_KEY: ${{ secrets.THIS_AWS_SECRET_ACCESS_KEY }}
rollback_on_production:
runs-on: ubuntu-latest
name: Rollback on Production
outputs:
signal_deployment: ${{ steps.step_id.outputs.environment }}
environment:
name: production
url: https://###/
steps:
- id: step_id
run: echo "::set-output name=environment::production"
retag_and_rollback_production:
needs: rollback_on_integrate
if: needs.rollback_on_integrate.outputs.signal_deployment == 'production'
uses: ###/###/.github/workflows/container-tagger.yml#main
with:
tag_to_identify_containers: ${{ github.event.inputs.tag_to_identify_containers }}
new_tag_to_apply_to_containers: ${{ github.event.inputs.new_tag_to_apply_to_containers }}
aws-region: eu-west-2
run_cron_and_cycle_containers: true
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.SHARED_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.SHARED_AWS_SECRET_ACCESS_KEY }}
ENV_AWS_ACCESS_KEY_ID: ${{ secrets.THIS_AWS_ACCESS_KEY_ID }}
ENV_AWS_SECRET_ACCESS_KEY: ${{ secrets.THIS_AWS_SECRET_ACCESS_KEY }}
An idea would be to use a matrix for your GitHub reusable workflow.
name: Reusable workflow with matrix strategy
on:
push:
jobs:
ReuseableMatrixJobForDeployment:
strategy:
matrix:
stage: [test, integration, production]
uses: octocat/octo-repo/.github/workflows/deployment.yml#main
with:
environment: ${{ matrix.stage }}
tag_to_identify_containers: ${{ github.event.inputs.tag_to_identify_containers }}
new_tag_to_apply_to_containers: ${{ github.event.inputs.new_tag_to_apply_to_containers }}
aws-region: eu-west-2
run_cron_and_cycle_containers: true
secrets: inherit
When GitHub runs the workflow, your reusable workflow should have the environment "name" set to:
jobs:
rollback_on_testing:
runs-on: ubuntu-latest
name: Rollback on testing
outputs:
signal_deployment: ${{ steps.step_id.outputs.environment }}
environment:
name: ${{inputs.environment}}
url: https://test.###/
which should give you access to the environment's secrets inherited... "secrets: inherit".
I want to pass variable group to my yaml template with deployment job.
Is it possible to use one of the values from variable group as environment name and connection name in deployment job?
My case is a little complicated, sorry for that :D
variables:
- group: ServiceVariables
pool: 'My Agent Pool'
extends:
template: /releases/deployment.dev.yml
parameters:
variableGroups:
- Gen2DEVVariables
- PaymentService
deployment.dev.yml
parameters:
- name: variableGroups
type: object
default: []
stages:
- stage: Deployment
variables:
- ${{ each value in parameters.variableGroups }}:
- group: ${{ value }}
jobs:
- template: /templates/jobs/deployToKubernetes.yml
parameters:
environmentName: $(azure_environment_name)
dockerRegistryConnection: $(serviceConnection_dockerRegistry)
kubernetesServiceConnection: $(serviceConnection_kubernetes)
variableGroups: ${{ parameters.variableGroups }}
and deployToKubernetes.yml
parameters:
- name: variableGroups
type: object
default: []
- name: environmentName
type: string
values:
- Develop
- Test
- UAT
- Production
- name: dockerRegistryConnection
- name: kubernetesServiceConnection
jobs:
- deployment: ${{ parameters.environmentName }}
environment: ${{ parameters.environmentName }}
variables:
- ${{ each value in parameters.variableGroups }}:
- group: ${{ value }}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: KubernetesManifest#0
displayName: Create Secret
inputs:
action: 'createSecret'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: 'value'
secretType: 'dockerRegistry'
secretName: 'value'
dockerRegistryEndpoint: ${{ parameters.dockerRegistryConnection }}
- task: KubernetesManifest#0
displayName: Deploy to Kubernetes cluster
inputs:
action: 'deploy'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: 'value'
manifests: 'value'
I have two templates (build-tsmain-project.yml and build-single-project.yml) which I call from another template like this:
steps:
- ${{ each project in parameters.projects }}:
- ${{ if ne(project.disable, 'true') }}:
- ${{ if eq(project.name, 'TSMain') }}:
- template: build-tsmain-project.yml
parameters:
name: ${{ project.name }}
shortName: ${{ project.shortName }}
path: ${{ project.name }}.sln
moreBuildArgs: ${{ parameters.restoreArgs }} ${{ parameters.moreBuildArgs }} ${{ project.moreBuildArgs }}
testFilter: ${{ project.testFilter }}
${{ if parameters.withCoverage }}:
moreTestArgs: ${{ parameters.coverageArgs }}
publishTestResults: false
noTest: ${{ project.noTest }}
- ${{ if ne(project.name, 'TSMain') }}:
- template: build-single-project.yml
parameters:
name: ${{ project.name }}
shortName: ${{ project.shortName }}
path: ${{ project.name }}.sln
moreBuildArgs: ${{ parameters.restoreArgs }} ${{ parameters.moreBuildArgs }} ${{ project.moreBuildArgs }}
testFilter: ${{ project.testFilter }}
${{ if parameters.withCoverage }}:
moreTestArgs: ${{ parameters.coverageArgs }}
publishTestResults: false
noTest: ${{ project.noTest }}
That is a problem, because I duplicate the parameters - not DRY.
The only working solution that is DRY which I could find involves introducing a new template (build-single-project-or-tsmain.yml):
parameters:
- name: template_name
type: string
- name: template_parameters
type: object
steps:
- template: ${{ parameters.template_name }}
parameters: ${{ parameters.template_parameters }}
Which allows me to rewrite the original code like this:
steps:
- ${{ each project in parameters.projects }}:
- ${{ if ne(project.disable, 'true') }}:
- template: build-single-project-or-tsmain.yml
parameters:
${{ if eq(project.name, 'TSMain') }}:
template_name: build-tsmain-project.yml
${{ if ne(project.name, 'TSMain') }}:
template_name: build-single-project.yml
template_parameters:
name: ${{ project.name }}
shortName: ${{ project.shortName }}
path: ${{ project.name }}.sln
moreBuildArgs: ${{ parameters.restoreArgs }} ${{ parameters.moreBuildArgs }} ${{ project.moreBuildArgs }}
testFilter: ${{ project.testFilter }}
${{ if parameters.withCoverage }}:
moreTestArgs: ${{ parameters.coverageArgs }}
publishTestResults: false
noTest: ${{ project.noTest }}
Now I am wondering if there is a way to avoid introducing a new template and still have a DRY solution?
We use Azure DevOps Server 2020 (on-prem).