Short question: how do I do the "enable" part in the pipeline below?
Long question: I have 3 jobs that I need to generate dynamically based on the parameters "params" due to business requirements. The job generated only works if the Job1, Job2 or Job3 is selected as true. Can I access parameters using string like dynamic in c#? ie: parameters["Job1"] or parameters["Job2"] or parameters["Job3"] ?
trigger: none
#Package Parameter
parameters:
- name: Job1
displayName: 'Job1'
type: boolean
default: false
- name: Job2
displayName: 'Job2'
type: boolean
default: false
- name: Job3
displayName: 'All Teams'
type: boolean
default: false
- name: 'params'
type: object
default: ["Job1",
"Job2",
"Job3"]
pool:
vmImage: ubuntu-latest
stages:
- stage: Build_dev
dependsOn:
jobs:
- ${{ each testJob in parameters.params }}:
- job:
enable: **If(parameters[testJob] == true) then enable this job**
displayName: Build ${{ testJob }}
steps:
- script: Build ${{ testJob }}
This is one way you can disable the jobs that are not set as true in the parameters. You can also use conditions for your jobs to skip them
azure-pipeline.yml
trigger: none
#Package Parameter
parameters:
- name: Job1
displayName: 'Job1'
type: boolean
default: false
- name: Job2
displayName: 'Job2'
type: boolean
default: false
- name: Job3
displayName: 'All Teams'
type: boolean
default: false
pool:
vmImage: ubuntu-latest
stages:
- stage: Build_dev
dependsOn: []
jobs:
- ${{ each parameter in parameters }}:
- ${{ if eq(parameter.Value, 'true') }}:
- template: job-template.yml
parameters:
jobName: ${{ parameter.Key }}
job-template.yml
parameters:
jobName: ''
jobs:
- job: ${{ parameters.jobName }}
displayName: Build ${{ parameters.jobName }}
steps:
- script: echo "Build ${{ parameters.jobName }}"
I have a YAML pipeline which contains some template files.
Within my pipeline, there are 4 stages that run in parallel to apply DSC. I then have a destroy task which i would like to run, only when all 4 tasks have ran successfully. When i try to add a depends on with a list:
dependsOn:
- Stage_A
- Stage_B
- Stage_C
- Stage_D
The error I get is:
The 'dependsOn' parameter is not a valid String.
My template YAML looks like:
...
stages:
...
- template: Apply-DSC.yml
parameters:
azureSub: '[sub]'
AutoAccountResourceGroup: 'rg'
AutoAccountName: 'aa'
environment: 'b1'
stageDependsOn: 'b1_apply'
- template: Destroy-Pipeline.yml
parameters:
azureSub: '[sub]'
terraformStorageAccountResourceGroup: 'rg'
terraformStorageAccountName: '[]'
terraformStorageContainerName: '[]'
terraformStorageRemoteStateKey: '[].tfstate'
environment: 'b1'
terraformEnvironmentFileName: 'B01'
dependsOn: 'Stage_A'
I have 4 stages within my Apply-DSC.yml
Stage_A
Stage_B
Stage_C
Stage_D
Question is, is this possible for my destroy stage to await a successful deployment of Stages A-D when using these stage templates?
Thanks.
Edit: Adding Destroy-Pipeline.yml
# Run & upload Terraform plan
parameters:
- name: azureSub
type: string
- name: terraformStorageAccountResourceGroup
type: string
- name: terraformStorageAccountName
type: string
- name: terraformStorageContainerName
type: string
- name: terraformStorageRemoteStateKey
type: string
- name: environment
type: string
- name: terraformEnvironmentFileName
type: string
- name: dependsOn
type: string
stages:
- stage: Destroy_${{ parameters.environment }}
dependsOn: ${{ parameters.dependsOn }}
jobs:
- deployment: '${{ parameters.environment }}_Destroy'
displayName: '${{ parameters.environment }} Destroy'
environment: '${{ parameters.environment }} destroy'
pool:
vmImage: windows-latest
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: 'drop'
name: 'Download_Terraform_code'
displayName: 'Download Terraform code'
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller#0
inputs:
terraformVersion: '$(TerraformVersion)'
displayName: 'Install Terraform'
- task: TerraformCLI#0
inputs:
command: 'init'
workingDirectory: '$(Pipeline.Workspace)/Drop'
backendType: 'azurerm'
backendServiceArm: '${{ parameters.azureSub }}'
backendAzureRmResourceGroupName: '${{ parameters.terraformStorageAccountResourceGroup }}'
backendAzureRmStorageAccountName: '${{ parameters.terraformStorageAccountName }}'
backendAzureRmContainerName: '${{ parameters.terraformStorageContainerName }}'
backendAzureRmKey: '${{ parameters.terraformStorageRemoteStateKey }}'
allowTelemetryCollection: false
displayName: 'Terraform Init'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
terraform workspace select $(WorkspaceEnvironment)
workingDirectory: '$(Pipeline.Workspace)/Drop'
displayName: 'Select Workspace'
- task: TerraformCLI#0
inputs:
command: 'plan'
environmentServiceName: '${{ parameters.azureSub }}'
commandOptions: '-destroy -var-file="./environments/${{ parameters.terraformEnvironmentFileName }}.tfvars" -input=false'
allowTelemetryCollection: false
workingDirectory: '$(Pipeline.Workspace)/Drop'
displayName: 'Plan Destroy'
- task: TerraformCLI#0
inputs:
command: 'destroy'
workingDirectory: '$(Pipeline.Workspace)/Drop'
environmentServiceName: '${{ parameters.azureSub }}'
commandOptions: '-var-file="./environments/${{ parameters.terraformEnvironmentFileName }}.tfvars" -input=false '
allowTelemetryCollection: false
displayName: 'Run Destroy'
I changed the type from string to object
parameters:
- name: dependsOn
type: object
default: []
Then within my template block i added the object like:
- template: Destroy-Pipeline.yml
parameters:
...
dependsOn: ['Stage_A', 'Stage_B' ...]
I have a bicep file as below to create role assignment to resourceGroup scope using azure devops pipeline.
main.bicep
targetScope = 'resourceGroup'
#description('Principal type of the assignee.')
#allowed([
'Device'
'ForeignGroup'
'Group'
'ServicePrincipal'
'User'
])
param principalType string
#description('the id for the role defintion, to define what permission should be assigned')
param RoleDefinitionId string
#description('the id of the principal that would get the permission')
param principalId string
#description('the role deffinition is collected')
resource roleDefinition 'Microsoft.Authorization/roleDefinitions#2018-01-01-preview' existing = {
scope: subscription()
name: RoleDefinitionId
}
resource RoleAssignment 'Microsoft.Authorization/roleAssignments#2020-10-01-preview' = {
name: guid(resourceGroup().id, RoleDefinitionId, principalId)
properties: {
roleDefinitionId: roleDefinition.id
principalId: principalId
principalType: principalType
}
}
This is my pipeline where I want to build the bicep and pass multiple principle iD as array.but its failing
pipeline.yaml.
parameters:
- name: roleList
type: object
stages:
- stage: BuilD_Roles_ARM_Artifact
displayName: 'Build_ARM_Template'
jobs:
- ${{ each role in parameters.roleList }}:
- job: BuilD_ARM_Artifact_${{ role.environment }}_${{ role.rolesname }}
displayName: '${{ role.rolesname }}'
variables:
- name: subscription
${{ if or(eq(role.environment, 'development'), eq(role.environment, 'staging')) }}:
value: 'mynonprod'
${{ if eq(role.environment, 'production')}}:
value: "myprod"
${{ if eq(role.environment, 'dr')}}:
value: "mydr"
workspace:
clean: all
pool:
${{ if eq(role.environment, 'development')}}:
name: devpool
${{ if eq(role.environment, 'staging')}}:
name: stagepool
${{ if eq(role.environment, 'production')}}:
name: az-prod-spoke
${{ if eq(role.environment, 'dr')}}:
name: drpool
steps:
- bash: |
resourceGroup=${{ role.resourceGroup }}
echo "##vso[task.setvariable variable=resourceGroup]$resourceGroup"
principalType=${{ role.principalType }}
echo "##vso[task.setvariable variable=principalType]$principalType"
principalid=${{ role.principalid }}
echo "##vso[task.setvariable variable=principalid]$principalid"
roleDefinitionId=${{ role.roleDefinitionId }}
echo "##vso[task.setvariable variable=roleDefinitionId]$roleDefinitionId"
- bash: az bicep build --file template/main.bicep
displayName: 'Compile Bicep to ARM'
- task: qetza.replacetokens.replacetokens-task.replacetokens#3
inputs:
rootDirectory: '$(System.DefaultWorkingDirectory)/'
targetFiles: '$(System.DefaultWorkingDirectory)/template/parameters.json'
encoding: 'auto'
writeBOM: true
actionOnMissing: 'warn'
keepToken: false
tokenPrefix: '#{'
tokenSuffix: '}#'
useLegacyPattern: false
enableTelemetry: true
- task: AzureCLI#2
displayName: "validate the templates"
inputs:
azureSubscription: ${{ variables.subscription }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az group deployment validate --resource-group $(resourceGroup) --template-file $(System.DefaultWorkingDirectory)/template/main.json --parameters $(System.DefaultWorkingDirectory)/template/parameters.json'
- task: AzureCLI#2
displayName: "verify the change result"
inputs:
azureSubscription: ${{ variables.subscription }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az deployment group what-if --resource-group $(resourceGroup) --template-file $(System.DefaultWorkingDirectory)/template/main.json --parameters $(System.DefaultWorkingDirectory)/template/parameters.json'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(System.DefaultWorkingDirectory)/template/'
ArtifactName: 'arm-source-${{ role.environment }}-${{ role.rolesname }}'
publishLocation: 'Container'
name: "Publish_arm_code"
displayName: "Publish arm code as build artifact"
- stage: Create_RoleAssignment
displayName: 'Create RoleAssignment'
jobs:
- ${{ each role in parameters.roleList }}:
- deployment: deploy_role_${{ role.environment }}_${{ role.rolesname }}
displayName: '${{ role.rolesname }}'
variables:
- name: resourceGroup
value: ${{ role.resourceGroup }}
- name: subscription
${{ if or(eq(role.environment, 'development'), eq(role.environment, 'staging')) }}:
value: 'mynonprod'
${{ if eq(role.environment, 'production')}}:
value: "myprod"
${{ if eq(role.environment, 'dr')}}:
value: "mydr"
${{ if eq(variables.subscription, 'mynonprod') }}:
environment: NON-PROD-RBAC
${{ if eq(variables.subscription, 'myprod') }}:
environment: PROD-RBAC
${{ if eq(variables.subscription, 'mydr') }}:
environment: DR-RBAC
pool:
${{ if eq(variables.subscription, 'mynonprod') }}:
name: devpool
${{ if eq(variables.subscription, 'mytest') }}:
name: stagepool
${{ if eq(variables.subscription, 'myprod') }}:
name: az-prod-spoke
${{ if eq(variables.subscription, 'mydr') }}:
name: drpool
strategy:
runOnce:
deploy:
steps:
- download: none
- task: DownloadBuildArtifacts#0
inputs:
artifactName: 'arm-source-${{ role.environment }}-${{ role.rolesname }}'
downloadPath: $(System.ArtifactsDirectory)
- task: CopyFiles#2
inputs:
sourceFolder: $(System.ArtifactsDirectory)/arm-source-${{ role.environment }}-${{ role.rolesname }}
contents: '**'
targetFolder: $(System.DefaultWorkingDirectory)/arm-source-${{ role.environment }}-${{ role.rolesname }}
cleanTargetFolder: true
- task: AzureCLI#2
displayName: "Create the change result"
inputs:
azureSubscription: ${{ variables.subscription }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az deployment group create --resource-group $(resourceGroup) --template-file $(System.DefaultWorkingDirectory)/arm-source-${{ role.environment }}-${{ role.rolesname }}/main.json --parameters $(System.DefaultWorkingDirectory)/arm-source-${{ role.environment }}-${{ role.rolesname }}/parameters.json'
and this is my pipeline input file
name: $(Build.SourceBranchName)-$(Build.BuildId)
trigger: none
stages:
- template: azure-pipeline.yaml
parameters:
roleList:
- rolesname: rolename1
environment: development
scope: resourcegroup
principalType: Group
principalid: xxxxxxxxxxx,yyyyyyyy, zzzzzzzzz
roleDefinitionId: acdxxxxxxxxxxxxxxxxxxxxx # reader id
resourceGroup: myrg-1
- rolesname: rolename2
environment: development
scope: resourcegroup
principalType: Group
principalid: aaaaaaaa,bbbbbbbbbb,cccccccccc
roleDefinitionId: acdxxxxxxxxxxxxxxxxxxxxx # reader id
resourceGroup: myrg-2
- rolesname: rolename3
environment: development
scope: resourcegroup
principalType: Group
principalid:
roleDefinitionId: acdxxxxxxxxxxxxxxxxxxxxx # reader id
resourceGroup: myrg-3
so here first I am building the bicep to ARM file and replacing the variables with the pipelines variables over a loop, Which is creating mutiple ARM templates together.
So I am looking for 2 things.
I would be able to pass list of PrincipleIDs over the input, for each roleassignment. The above template syntax is failing if I add multiple PrincipleIDs
looking for a way to iterate through all the roles paramaters i9f any changes added and have a single ARM template with the inputs. So that it will produce a single ARM build artifact for only the modification to roleassignment items.
Adding the modified files
New bicep file
targetScope = 'resourceGroup'
#description('Principal type of the assignee.')
#allowed([
'Device'
'ForeignGroup'
'Group'
'ServicePrincipal'
'User'
])
param principalType string
#description('the id for the role defintion, to define what permission should be assigned')
param RoleDefinitionId string
#description('the id of the principal that would get the permission')
param principalId string
#description('the role deffinition is collected')
resource roleDefinition 'Microsoft.Authorization/roleDefinitions#2018-01-01-preview' existing = {
scope: subscription()
name: RoleDefinitionId
}
resource RoleAssignment 'Microsoft.Authorization/roleAssignments#2020-10-01-preview' = [for id in split(principalId, ','): {
name: guid(resourceGroup().id, RoleDefinitionId, principalId)
properties: {
roleDefinitionId: roleDefinition.id
principalId: principalId
principalType: principalType
}
}]
parameters file
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"principalType": {
"value": "#{principalType}#"
},
"RoleDefinitionId": {
"value": "#{RoleDefinitionId}#"
},
"principalId": {
"value": "#{principalId}#"
}
}
}
pipeline file.
- rolesname: readerall
environment: development
scope: resourcegroup
principalType: Group
principalid: aaaaaaaaaaaaaa,bbbbbbbbbbbbbbbbb,ccccccccccccccccccccc,ddddddddddddddddddddd
roleDefinitionId: acddxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resourceGroup: aks-rg
- rolesname: reader_apimrg_all
environment: development
scope: resourcegroup
principalType: Group
principalid: aaaaaaaaaaaaaa,bbbbbbbbbbbbbbbbb,ccccccccccccccccccccc,ddddddddddddddddddddd
roleDefinitionId: acddxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resourceGroup: apim-rg
- rolesname: reader_lawrg_all
environment: development
scope: resourcegroup
principalType: Group
principalid: aaaaaaaaaaaaaa,bbbbbbbbbbbbbbbbb,ccccccccccccccccccccc,ddddddddddddddddddddd
roleDefinitionId: acddxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
resourceGroup: la-rg
Here the principal ids are passed as string:
principalid: xxxxxxxxxxx,yyyyyyyy,zzzzzzzzz
In the bicep file, you could split the string to create multiple role assignments:
resource roleAssignments 'Microsoft.Authorization/roleAssignments#2020-10-01-preview' = [for id in split(principalId, ','): {
name: guid(resourceGroup().id, RoleDefinitionId, id)
properties: {
roleDefinitionId: roleDefinition.id
principalId: id
principalType: principalType
}
}]
Regarding your second question, it sounds quite complex. You probably would need to have a preparation task that will iterate and check which role assignments already exist and then create a complex object to pass to the bicep file. Because the ARM API is idempotent, not sure why are you trying only to deploy the new changes ?
Can anybody point me in the right direction here?
When I attempt to kick off the pipeline from the main yaml template the reference template gives me the unexpected 'steps' value.
I attempted to add a stage before the defined job and then receive the unexpected 'stage' message. Looking at some similar previously asked questions I understand a step can't be place directly under a 'stage' but this is not the case here.
parameters:
- name: baseEnvironmentName
type: string
- name: environmentSuffix
type: string
- name: postDeploySteps
type: stepList
default: []
- name: publishedPackageName
type: string
- name: serviceName
type: string
- name: dnsHostName
type: string
jobs:
- deployment: ${{ parameters.environmentSuffix }}Deployment
displayName: '${{ parameters.environmentSuffix }} Deployment'
pool:
vmImage: 'windows-2019'
environment: ${{ parameters.baseEnvironmentName }}-${{ parameters.environmentSuffix }}
variables:
- template: variables/variables.common.yaml
- template: variables/variables.${{ parameters.environmentSuffix }}.yaml
- name: MonitorExceptionAlert_Name
value: '${{ variables.IdPrefix }}-${{ parameters.environmentSuffix }}-${{ parameters.dnsHostName }}-ExceptionAlert'
steps:
- checkout: self
- checkout: common_iac
- task: Random Task Name
displayName: 'Create Environment Resource Group'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
resourceGroupName: '$(ResourceGroup.Name)'
location: '$(ResourceGroup.Location)'
condition: and(succeeded(), ne(variables['CreateResourceGroupInTaskGroup'], 'false'))
- task: Random Task Name 2
displayName: 'Create Application Insights'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
ResourceGroupName: '$(ResourceGroup.Name)'
Location: '$(ResourceGroup.Location)'
instanceName: '$(ApplicationInsights.Name)'
You are using deployment jobs. Try defining a strategy. Try replacing steps: inline with deployment with:
- deployment: ....
strategy:
runOnce:
deploy:
steps:
From the example at the Azure docs templates page, jobs: should contain a - job: which contains steps:. I don't know that jobs: can directly contain steps:.
I want to pass variable group to my yaml template with deployment job.
Is it possible to use one of the values from variable group as environment name and connection name in deployment job?
My case is a little complicated, sorry for that :D
variables:
- group: ServiceVariables
pool: 'My Agent Pool'
extends:
template: /releases/deployment.dev.yml
parameters:
variableGroups:
- Gen2DEVVariables
- PaymentService
deployment.dev.yml
parameters:
- name: variableGroups
type: object
default: []
stages:
- stage: Deployment
variables:
- ${{ each value in parameters.variableGroups }}:
- group: ${{ value }}
jobs:
- template: /templates/jobs/deployToKubernetes.yml
parameters:
environmentName: $(azure_environment_name)
dockerRegistryConnection: $(serviceConnection_dockerRegistry)
kubernetesServiceConnection: $(serviceConnection_kubernetes)
variableGroups: ${{ parameters.variableGroups }}
and deployToKubernetes.yml
parameters:
- name: variableGroups
type: object
default: []
- name: environmentName
type: string
values:
- Develop
- Test
- UAT
- Production
- name: dockerRegistryConnection
- name: kubernetesServiceConnection
jobs:
- deployment: ${{ parameters.environmentName }}
environment: ${{ parameters.environmentName }}
variables:
- ${{ each value in parameters.variableGroups }}:
- group: ${{ value }}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: KubernetesManifest#0
displayName: Create Secret
inputs:
action: 'createSecret'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: 'value'
secretType: 'dockerRegistry'
secretName: 'value'
dockerRegistryEndpoint: ${{ parameters.dockerRegistryConnection }}
- task: KubernetesManifest#0
displayName: Deploy to Kubernetes cluster
inputs:
action: 'deploy'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: 'value'
manifests: 'value'