set parameter based on another parameter - azure-devops

I need to set azureSubscription property for AzurePowerShell#5 task.
The value depends on EnvName parameter, which is as follows:
- name: EnvName
displayName: Environment Name
type: string
default: dev
values:
- dev
- test
- uat
- prod
I want to do this in PowerShell in Job #1 in SetVariables task:
if (${{ parameters.EnvName }} -eq "prod") {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Production Subscription"
} else {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Non-Production Subscription"
}
And then assign the value to the task in Job #2:
- job: job2
dependsOn: job1
variables:
AzureSubscription: $[ dependencies.job1.outputs['SetVariables.AzureSubscription'] ]
steps:
- task: AzurePowerShell#5
inputs:
azureSubscription: '$(AzureSubscription)'
Unfortunately, this doesn't work. When executing the pipeline, I immediately get following error:
There was a resource authorization issue: "The pipeline is not valid.
Job job2: Step AzurePowerShell input ConnectedServiceNameARM
references service connection $(AzureSubscription) which could not be
found. The service connection does not exist or has not been
authorized for use. For authorization details, refer to
https://aka.ms/yamlauthz. Job job2: Step AzurePowerShell
input ConnectedServiceNameARM references service connection
$(AzureSubscription) which could not be found. The service connection
does not exist or has not been authorized for use. For authorization
details, refer to https://aka.ms/yamlauthz."
If I create AzureSubscription as a parameter instead of a runtime-computed variable, then everything works fine. But I would prefer not to, as the pipeline user shouldn't need to set its value, and doesn't need to know which value is correct.
It seems that azureSubscription property of AzurePowerShell#5 task is evaluated during compile-time. So I need to evaluate my variable (or parameter, or whatever) during compile-time too.
The only solution I came up with is to determine which job to run based on EnvName parameter, something like this:
- ${{ if eq(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Production Subscription
- ${{ if ne(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Non-Production Subscription
But in that case I have 2 almost identical task declarations, this complicates my YAML quite a bit.
Another solution would be to use PowerShell#2 task instead of AzurePowerShell#5. That I would also rather not do.
Is there a better way to achieve my goal?

Related

DevOps Pipeline AzureCLI#2 with dynamic azureSubscription

I have a DevOps pipeline that gives me this error:
There was a resource authorization issue: "The pipeline is not valid. Job ExecutionTerraform: Step AzureCLI input connectedServiceNameARM references service connection Azure: $(subscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
The configuration I am using is looking up the Subscription name dynamically.
The step I use for that is:
- bash: |
# pull the subscription data
# ... read data into local variables
# set the shared variables
echo "##vso[task.setvariable variable=subscriptionId]${SUBSCRIPTION_ID}"
echo "##vso[task.setvariable variable=subscriptionName]${SUBSCRIPTION_NAME}"
From there I attempt to call the Azure CLI via a template:
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
Inside the template my CLI step uses:
steps:
- task: AzureCLI#2
displayName: Test CLI
inputs:
azureSubscription: "ARMTest ${{ parameters.azureSubscriptionName }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az --version
addSpnToEnvironment: true
useGlobalConfig: true
It looks like Pipelines is trying to preemptively check authorization without noticing that there's a variable in there. What am I doing wrong here that is causing Azure to attempt to resolve that at the wrong time?
I do this in other pipelines without issues and I am not sure what is different in this particular instance.
Update 1: Working Template I have Elsewhere
Full template:
parameters:
- name: environment
type: string
jobs:
- job: AKSCredentials
displayName: "AKS Credentials Pull"
steps:
- task: AzureCLI#2
displayName: AKS Credentials
inputs:
azureSubscription: "Azure: testbed-${{ parameters.environment }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az aks get-credentials -g testbed-${{ parameters.environment }} -n testbed-${{ parameters.environment }}-aks
addSpnToEnvironment: true
useGlobalConfig: true
This is not possible because azure subscription needs to be known at compilation time. You set your variable on run time.
Here an issue with similar case when it is explained:
run time variables aren't supported for service connection OR azure subscription. In your code sample, you are referring to AzureSubscription variable which will get initialized at the run time (but not at save time). Your syntax is correct but you need to set AzureSubscription variable as part of variables.
If you define your variables like:
variables:
subscriptionId: someValue
subscriptionName: someValue
and then you will use it
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
it should work. But since you set up your variables on runtime it causes your issue.

AzureDevops: Can we pass dynamic value as service connection in yaml

I have a yaml. where I need to pass serviceconnections to a script/template based on an another task which retrieve all required subscriptions.
Question is : Can I pass a dynamic value to a service connection? It is giving me compile time error.
My code below:
trigger: none
pr: none
parameters:
- name: AzureSubscription
type: object
default:
xxx:
Sub: xxx
yyy:
Sub: yyy
jobs:
- job: Updating
condition: succeeded()
pool:
vmImage: "windows-latest"
strategy:
maxParallel: 10
matrix: ${{ parameters.AzureSubscription }}
steps:
- task: AzurePowerShell#5
displayName: Tes
inputs:
azureSubscription: 'zzz'
ScriptType: 'InlineScript'
Inline: |
Write-Output "subcriptionList ---- $(Sub)"
FailOnStandardError: true
azurePowerShellVersion: 'LatestVersion'
pwsh: true
- task: AzurePowerShell#4
displayName: Updating
inputs:
**azureSubscription: ${{ sub }}** # here it is giving me error?
ScriptType: 'FilePath'
ScriptPath: '$(System.DefaultWorkingDirectory)/Foundation/xxxxx.ps1'
azurePowerShellVersion: 'LatestVersion'
So in the 2nd task, I am passing subscription from my parameter.
Error is : Unrecognized value: 'sub'.
Can someone help me?
This is possible with some creativity but not native to the task. Rather using variables and dynamically loading the variable file.
I usually declare the Service Connection name as a variable template file in a separate repository. This allows for reuse across all projects in the org, not required but find it easier that way. Part of the template name would be the environment being deployed to. So a template file might be called azure.dev.yml or azure.uat.yml and look like:
variables:
AzureSubscriptionServiceConnectionName: Azure - Dev
Then a variable defined within the scope of the stage/job would load the template file like below, assuming that a parameter or a local variable would be passed in with the given environmentName.
variables:
- template: /azure.${{ parameters.environmentName }}.yml
Then the stage/job can reference this variable via:
${{ variables.AzureSubscriptionServiceConnectionName }}
Here is some more Microsoft Documentation on YAML Pipeline Variable Scope
You cannot set azureSubcription dynamically. It is known limitation.
#JoeGaggler this feature isn't supported today. Usage of service endpoints (Azure Subscription is one of kind) in release/build definition is controlled by some permissions. At the time of saving a release/build definition service validates that the author (whoever is saving the definition) has appropriate permissions on the endpoint. If we support variable replacements for service endpoint input then service can't validate that the author has required permissions or not and it might become a security issue.

Dynamic variables not available in other stages of azure pipeline

I am new to azure pipelines and am currently experimenting with the passing variables to the later jobs. Here is the current snippet whereby I am trying to extract the project version from the pom file using maven help: evaluate plugin. The pomVersion variable is populated but is not available in the same step or later steps in the second job via the projectVersionDynamic variable.
stages:
- stage: FirstStage
jobs:
- job: FirstJob
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
pomVersion=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
echo $pomVersion ##Prints semantic version 2.27.0-SNAPSHOT
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion'
echo '##vso[task.setvariable variable=projectVersionStatic;isOutput=true]2.27.0'
echo $(Task1.projectVersionDynamic) ##Error message
echo $projectVersionDynamic ##Is empty
name: Task1
displayName: Task1 in JobOne of FirstStage
- job: SecondJob
dependsOn: FirstJob
variables:
DynVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionDynamic'] ]
StaVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionStatic'] ]
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'SecondJob'
echo $(DynVar) ##Is empty
echo $DynVar ##Is empty
echo $(StaVar) ##Prints 2.27.0
echo $StaVar ##Is empty
displayName: Task in JobTwo of FirstStage
Observation: projectVersionDynamic value does not get populated and is not available in the same task or subsequent tasks within or in later jobs / stages. However, the static variable gets populated in projectVersionStatic without any issues.
Is it possible to set dynamic values for user-defined variables in azure pipelines or is it that I am doing something wrong? I see an example here under the Stages section where it seems to be working.
Variables in Azure Pipelines can be really tricky sometimes. The documentation isn't always crystal-clear on how it's supposed to work. Looking at your example, a cpuple of observations:
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion' - your single quotes need to be double quotes to expand the value of $pomVersion into the echo statement
(and this is where things get fuzzy): The purpose of task.setvariable is to communicate values to other tasks or jobs. From the documentation: "This doesn't update the environment variables, but it does make the new variable available to downstream steps within the same job."
$variable syntax won't work because the task.setVariable doesn't inject into the running environment - rather, it's a signal to the pipeline to capture the output and store it away for later use.
$(variable) syntax won't work because it's expanded just before the job starts, so it's too late to capture here.
If you use my suggestion in point 1) about double-quoting the task.setVariable, you should see the value available in the second job.
For anybody having a use case where variables defined in one azure pipeline stage needs to be used in another stage, this is how it can be done:
Following snipett of azure pipeline that defines a variable and make it available to be used in a job in the same stage and in a job in another stage:
stages:
- stage: stage1
jobs:
- job: job1
steps:
- task: "AzureCLI#2"
name: step1
inputs:
inlineScript: |
my_variable=a-value
echo "##vso[task.setvariable variable=var_name;isOutput=true]$my_variable"
- job: job2
variables:
stage1_var: $[ dependencies.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage1_var)
- stage: stage2
jobs:
- job: job1
variables:
stage2_var: $[ stageDependencies.stage1.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage2_var)

Azure Devops - passing variables between job templates

Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
  environment: ''
jobs:
- job: preDeploy
  variables:
    EnvironmentName: preDeploy-${{ parameters.environment }}
  steps:
  - checkout: none
  - pwsh: |
      echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
    name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
  dependsOn: preDeploy
  variables:
    EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
  steps:
  - checkout: none
  - pwsh: |
      Write-Host "$(EnvironmentNameCopy)"
    name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
  - stage: deployQA
    jobs:
    - template: azure-pipelines-1.yml
      parameters:
        environment: FromTemplate1
    - template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.

Azure YAML Get variable from a job run in a previous stage

I am creating YAML pipeline in Azure DevOps that consists of two stages.
The first stage (Prerequisites) is responsible for reading the git commit and creates a comma separated variable containing the list of services that has been affected by the commit.
The second stage (Build) is responsible for building and unit testing the project. This Stage consists of many templates, one for each Service. In the template script, the job will check if the relevant Service in in the variable created in the previous stage. If the job finds the Service it will continue to build and test the service. However if it cannot find the service, it will skip that job.
Run.yml:
stages:
- stage: Prerequisites
jobs:
- job: SetBuildQueue
steps:
- task: powershell#2
name: SetBuildQueue
displayName: 'Set.Build.Queue'
inputs:
targetType: inline
script: |
## ... PowerShell script to get changes - working as expected
Write-Host "Build Queue Auto: $global:buildQueueVariable"
Write-Host "##vso[task.setvariable variable=buildQueue;isOutput=true]$global:buildQueueVariable"
- stage: Build
jobs:
- job: StageInitialization
- template: Build.yml
parameters:
projectName: Service001
projectLocation: src/Service001
- template: Build.yml
parameters:
projectName: Service002
projectLocation: src/Service002
Build.yml:
parameters:
projectName: ''
projectLocation: ''
jobs:
- job:
displayName: '${{ parameters.projectName }} - Build'
dependsOn: SetBuildQueue
continueOnError: true
condition: and(succeeded(), contains(dependencies.SetBuildQueue.outputs['SetBuildQueue.buildQueue'], '${{ parameters.projectName }}'))
steps:
- task: NuGetToolInstaller#1
displayName: 'Install Nuget'
Issue:
When the first stages runs it will create a variable called buildQueue which is populated as seen in the console output of the PowerShell script task:
Service001 Changed
Build Queue Auto: Service001;
However when it gets to stage two and it tries to run the build template, when it checks the conditions it returns the following output:
Started: Today at 12:05 PM
Duration: 16m 7s
Evaluating: and(succeeded(), contains(dependencies['SetBuildQueue']['outputs']['SetBuildQueue.buildQueue'], 'STARS.API.Customer.Assessment'))
Expanded: and(True, contains(Null, 'service001'))
Result: False
So my question is how do I set the dependsOn and condition to get the information from the previous stage?
It because you want to access the variable in a different stage from where you defined them. currently, it's impossible, each stage it's a new instance of a fresh agent.
In this blog you can find a workaround that involves writing the variable to disk and then passing it as a file, leveraging pipeline artifacts.
To pass the variable FOO from a job to another one in a different stage:
Create a folder that will contain all variables you want to pass; any folder could work, but something like mkdir -p $(Pipeline.Workspace)/variables might be a good idea.
Write the contents of the variable to a file, for example echo "$FOO" > $(Pipeline.Workspace)/variables/FOO. Even though the name could be anything you’d like, giving the file the same name as the variable might be a good idea.
Publish the $(Pipeline.Workspace)/variables folder as a pipeline artifact named variables
In the second stage, download the variables pipeline artifact
Read each file into a variable, for example FOO=$(cat $(Pipeline.Workspace)/variables/FOO)
Expose the variable in the current job, just like we did in the first example: echo "##vso[task.setvariable variable=FOO]$FOO"
You can then access the variable by expanding it within Azure Pipelines ($(FOO)) or use it as an environmental variable inside a bash script ($FOO).