I have a yaml. where I need to pass serviceconnections to a script/template based on an another task which retrieve all required subscriptions.
Question is : Can I pass a dynamic value to a service connection? It is giving me compile time error.
My code below:
trigger: none
pr: none
parameters:
- name: AzureSubscription
type: object
default:
xxx:
Sub: xxx
yyy:
Sub: yyy
jobs:
- job: Updating
condition: succeeded()
pool:
vmImage: "windows-latest"
strategy:
maxParallel: 10
matrix: ${{ parameters.AzureSubscription }}
steps:
- task: AzurePowerShell#5
displayName: Tes
inputs:
azureSubscription: 'zzz'
ScriptType: 'InlineScript'
Inline: |
Write-Output "subcriptionList ---- $(Sub)"
FailOnStandardError: true
azurePowerShellVersion: 'LatestVersion'
pwsh: true
- task: AzurePowerShell#4
displayName: Updating
inputs:
**azureSubscription: ${{ sub }}** # here it is giving me error?
ScriptType: 'FilePath'
ScriptPath: '$(System.DefaultWorkingDirectory)/Foundation/xxxxx.ps1'
azurePowerShellVersion: 'LatestVersion'
So in the 2nd task, I am passing subscription from my parameter.
Error is : Unrecognized value: 'sub'.
Can someone help me?
This is possible with some creativity but not native to the task. Rather using variables and dynamically loading the variable file.
I usually declare the Service Connection name as a variable template file in a separate repository. This allows for reuse across all projects in the org, not required but find it easier that way. Part of the template name would be the environment being deployed to. So a template file might be called azure.dev.yml or azure.uat.yml and look like:
variables:
AzureSubscriptionServiceConnectionName: Azure - Dev
Then a variable defined within the scope of the stage/job would load the template file like below, assuming that a parameter or a local variable would be passed in with the given environmentName.
variables:
- template: /azure.${{ parameters.environmentName }}.yml
Then the stage/job can reference this variable via:
${{ variables.AzureSubscriptionServiceConnectionName }}
Here is some more Microsoft Documentation on YAML Pipeline Variable Scope
You cannot set azureSubcription dynamically. It is known limitation.
#JoeGaggler this feature isn't supported today. Usage of service endpoints (Azure Subscription is one of kind) in release/build definition is controlled by some permissions. At the time of saving a release/build definition service validates that the author (whoever is saving the definition) has appropriate permissions on the endpoint. If we support variable replacements for service endpoint input then service can't validate that the author has required permissions or not and it might become a security issue.
Related
I have an Azure DevOps Deployment YAML Pipeline that creates an Azure App Service and deploys code to it. The actual pipeline is more complex, but I am simplifying it for this question.
Currently my pipeline can successfully deploy to a specific Azure Subscription (Service Connector) with resource names defined in variables.
I need to parametrize the pipeline so that it can deploy to several different environments (means Azure Subscriptions) using multiple Service Connectors. Each environment has a different Azure Resource naming convention.
Is there any way to read the value of pipeline variables from an XML or JSON file? This way I can have multiple config files for each environment and store them as part of my repository.
Is this a right approach for multi-environment deployment pipeline configuration?
You can use variable templates. There is another interesting link: Learn more about variable reuse with templates.
Here I have this flat folder structure (for the clarity of the sample):
.
| deploy-app.job.yaml
| deploy-app.pipeline.yaml
| variables.dev.yaml
| variables.prod.yaml
So here we're trying to run the reusable job deploy-app.job.yaml with different variable sets.
I've defined some variables in each variable.{env}.yaml files
# variables.dev.yaml
variables:
vmImage: ubuntu-20.04
serviceConnection: dev-service-connection
# variables.prod.yaml
variables:
vmImage: ubuntu-20.04
serviceConnection: prod-service-connection
The deploy-app.job.yaml file accepts a parameter that allow to inject a variable template:
# deploy-app.job.yaml
parameters:
- name: envVariablesTemplate
type: string
jobs:
- deployment: deploy
variables:
# Inject the verianle template here
- template: ${{ parameters.envVariablesTemplate }}
pool:
# Use the variable from the template
vmImage: ${{ variables.vmImage }}
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI#2
displayName: Hello from azure cli
inputs:
# Use the variable from the template
azureSubscription: ${{ variables.serviceConnection }}
scriptType: pscore
scriptLocation: inlineScript
inlineScript: echo 'Hello from azure cli'
In the main pipeline, I can create different stages and inject the desired vairables:
# deploy-app.pipeline..yaml
stages:
- stage: dev
condition: succeeded()
jobs:
- template: ./deploy-app.job.yaml
parameters:
envVariablesTemplate: ./variables.dev.yaml
- stage: prod
dependsOn: dev
condition: succeeded()
jobs:
- template: ./deploy-app.job.yaml
parameters:
envVariablesTemplate: ./variables.prod.yaml
Based on your needs, you can add multiple variable templates, having a naming convention etc. Really up to you and depends on the complexity of your pipelines.
By using XML transformation, we can perform the operation. Check the below link to get the complete steps.
https://www.dragonspears.com/blog/how-to-handle-continuous-deployment-across-multiple-environments
Create different stages.
Create build transformation using XML transformer
Another option is to utilize ‘XML variable substitution.’ Sensitive information can be stored and secured within Azure Pipelines vs. in plain text transformation files.
Continuous Deployment
The same steps mentioned here are available in the link mentioned above.
I need to set azureSubscription property for AzurePowerShell#5 task.
The value depends on EnvName parameter, which is as follows:
- name: EnvName
displayName: Environment Name
type: string
default: dev
values:
- dev
- test
- uat
- prod
I want to do this in PowerShell in Job #1 in SetVariables task:
if (${{ parameters.EnvName }} -eq "prod") {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Production Subscription"
} else {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Non-Production Subscription"
}
And then assign the value to the task in Job #2:
- job: job2
dependsOn: job1
variables:
AzureSubscription: $[ dependencies.job1.outputs['SetVariables.AzureSubscription'] ]
steps:
- task: AzurePowerShell#5
inputs:
azureSubscription: '$(AzureSubscription)'
Unfortunately, this doesn't work. When executing the pipeline, I immediately get following error:
There was a resource authorization issue: "The pipeline is not valid.
Job job2: Step AzurePowerShell input ConnectedServiceNameARM
references service connection $(AzureSubscription) which could not be
found. The service connection does not exist or has not been
authorized for use. For authorization details, refer to
https://aka.ms/yamlauthz. Job job2: Step AzurePowerShell
input ConnectedServiceNameARM references service connection
$(AzureSubscription) which could not be found. The service connection
does not exist or has not been authorized for use. For authorization
details, refer to https://aka.ms/yamlauthz."
If I create AzureSubscription as a parameter instead of a runtime-computed variable, then everything works fine. But I would prefer not to, as the pipeline user shouldn't need to set its value, and doesn't need to know which value is correct.
It seems that azureSubscription property of AzurePowerShell#5 task is evaluated during compile-time. So I need to evaluate my variable (or parameter, or whatever) during compile-time too.
The only solution I came up with is to determine which job to run based on EnvName parameter, something like this:
- ${{ if eq(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Production Subscription
- ${{ if ne(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Non-Production Subscription
But in that case I have 2 almost identical task declarations, this complicates my YAML quite a bit.
Another solution would be to use PowerShell#2 task instead of AzurePowerShell#5. That I would also rather not do.
Is there a better way to achieve my goal?
I have a DevOps pipeline that gives me this error:
There was a resource authorization issue: "The pipeline is not valid. Job ExecutionTerraform: Step AzureCLI input connectedServiceNameARM references service connection Azure: $(subscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
The configuration I am using is looking up the Subscription name dynamically.
The step I use for that is:
- bash: |
# pull the subscription data
# ... read data into local variables
# set the shared variables
echo "##vso[task.setvariable variable=subscriptionId]${SUBSCRIPTION_ID}"
echo "##vso[task.setvariable variable=subscriptionName]${SUBSCRIPTION_NAME}"
From there I attempt to call the Azure CLI via a template:
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
Inside the template my CLI step uses:
steps:
- task: AzureCLI#2
displayName: Test CLI
inputs:
azureSubscription: "ARMTest ${{ parameters.azureSubscriptionName }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az --version
addSpnToEnvironment: true
useGlobalConfig: true
It looks like Pipelines is trying to preemptively check authorization without noticing that there's a variable in there. What am I doing wrong here that is causing Azure to attempt to resolve that at the wrong time?
I do this in other pipelines without issues and I am not sure what is different in this particular instance.
Update 1: Working Template I have Elsewhere
Full template:
parameters:
- name: environment
type: string
jobs:
- job: AKSCredentials
displayName: "AKS Credentials Pull"
steps:
- task: AzureCLI#2
displayName: AKS Credentials
inputs:
azureSubscription: "Azure: testbed-${{ parameters.environment }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az aks get-credentials -g testbed-${{ parameters.environment }} -n testbed-${{ parameters.environment }}-aks
addSpnToEnvironment: true
useGlobalConfig: true
This is not possible because azure subscription needs to be known at compilation time. You set your variable on run time.
Here an issue with similar case when it is explained:
run time variables aren't supported for service connection OR azure subscription. In your code sample, you are referring to AzureSubscription variable which will get initialized at the run time (but not at save time). Your syntax is correct but you need to set AzureSubscription variable as part of variables.
If you define your variables like:
variables:
subscriptionId: someValue
subscriptionName: someValue
and then you will use it
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
it should work. But since you set up your variables on runtime it causes your issue.
Template:
parameters:
- name: PathPrefix
displayName: 'Path prefix'
type: string
default: ''
steps:
- task: DotNetCoreCLI#2
displayName: 'dotnet restore'
inputs:
command: restore
projects: ${{parameters.PathPrefix}}**/$(Build.DefinitionName).sln
Pipeline:
resources:
repositories:
- repository: devops
name: foo/devops
type: git
ref: master
trigger:
branches:
include:
- refs/heads/*
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: windows-latest
steps:
- checkout: self
- template: azure/pipelines/pipeline.yaml#devops
parameters:
PathPrefix: $(Build.DefinitionName)
Run error during the restore step:
##[error]No files matched the search pattern.
Even setting verbosityRestore: detailed on the restore step doesn't give me any more information.
If I don't set PathPrefix, it seems to use the default empty string and find the solution file (in some cases). However, if I do set the prefix, which is needed in some repos, it can't find the file. I've tried various ways of referencing the parameter within the template (${{}}, $(), $[] and others) and different ways of specifying it within the pipeline, including hard-coding the path (though I want to use the variable instead), but nothing works.
I thought maybe variables would work instead, so I also tried specifying variables in the pipeline and using them in the template, but that results in the same error. Defining the variable in the template gave me a compilation error for the template (unexpected token 'variable' or something similar).
Look at what you're passing and mentally expand the results.
${{parameters.PathPrefix}}**/$(Build.DefinitionName).sln
If Build.DefinitionName is foo, and you pass that in as PathPrefix, then what you get is:
foo**/foo.sln
It looks like you want an extra forward slash in there, so you get foo/**/foo.sln.
Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
environment: ''
jobs:
- job: preDeploy
variables:
EnvironmentName: preDeploy-${{ parameters.environment }}
steps:
- checkout: none
- pwsh: |
echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
dependsOn: preDeploy
variables:
EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
steps:
- checkout: none
- pwsh: |
Write-Host "$(EnvironmentNameCopy)"
name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
- stage: deployQA
jobs:
- template: azure-pipelines-1.yml
parameters:
environment: FromTemplate1
- template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.