DevOps Pipeline AzureCLI#2 with dynamic azureSubscription - azure-devops

I have a DevOps pipeline that gives me this error:
There was a resource authorization issue: "The pipeline is not valid. Job ExecutionTerraform: Step AzureCLI input connectedServiceNameARM references service connection Azure: $(subscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
The configuration I am using is looking up the Subscription name dynamically.
The step I use for that is:
- bash: |
# pull the subscription data
# ... read data into local variables
# set the shared variables
echo "##vso[task.setvariable variable=subscriptionId]${SUBSCRIPTION_ID}"
echo "##vso[task.setvariable variable=subscriptionName]${SUBSCRIPTION_NAME}"
From there I attempt to call the Azure CLI via a template:
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
Inside the template my CLI step uses:
steps:
- task: AzureCLI#2
displayName: Test CLI
inputs:
azureSubscription: "ARMTest ${{ parameters.azureSubscriptionName }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az --version
addSpnToEnvironment: true
useGlobalConfig: true
It looks like Pipelines is trying to preemptively check authorization without noticing that there's a variable in there. What am I doing wrong here that is causing Azure to attempt to resolve that at the wrong time?
I do this in other pipelines without issues and I am not sure what is different in this particular instance.
Update 1: Working Template I have Elsewhere
Full template:
parameters:
- name: environment
type: string
jobs:
- job: AKSCredentials
displayName: "AKS Credentials Pull"
steps:
- task: AzureCLI#2
displayName: AKS Credentials
inputs:
azureSubscription: "Azure: testbed-${{ parameters.environment }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az aks get-credentials -g testbed-${{ parameters.environment }} -n testbed-${{ parameters.environment }}-aks
addSpnToEnvironment: true
useGlobalConfig: true

This is not possible because azure subscription needs to be known at compilation time. You set your variable on run time.
Here an issue with similar case when it is explained:
run time variables aren't supported for service connection OR azure subscription. In your code sample, you are referring to AzureSubscription variable which will get initialized at the run time (but not at save time). Your syntax is correct but you need to set AzureSubscription variable as part of variables.
If you define your variables like:
variables:
subscriptionId: someValue
subscriptionName: someValue
and then you will use it
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
it should work. But since you set up your variables on runtime it causes your issue.

Related

AzureStaticWebApp#0 not recognizing deployment token from variable

Hi have the following code that deploys an artifact to an Azure Static Web App:
...
variables:
- name: staticWebAppDeploymentToken
...
# This steps reads the deployment token of the static web app and assigns it on a variable
- task: AzureCLI#2
displayName: 'Retrieve static web app deployment token'
inputs:
azureSubscription: xxxx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name xxxx-xxxx-$(environment) | jq .properties.apiKey)
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken;]$output"
- task: AzureStaticWebApp#0
inputs:
output_location: '/'
cwd: '$(Pipeline.Workspace)/artifact'
skip_app_build: true
azure_static_web_apps_api_token: $(staticWebAppDeploymentToken)
I get the error:
I've set the System.Debug variable to true, and I see the value is set in the variable. I've also printed the variable and the value is there.
I can't understand what I'm doing wrong. What is the correct way to set a variable in bash and use it on another non-bash step?
I've tried hardcoding the value and also passing it as a parameter from the library, and that works, but that is not what I want.
Test the same script to get the token and pass it to Azure Static web APP task, I can reproduce the same issue.
The root cause of this issue is that when you set the variable in Azure CLI task with the command, the pipeline variable will contain "". For example: "tokenvalue".
The expected deployment token in Azure Static web APP task, it will not contain double quotes.
To solve this issue, you need to add a step in Azure CLI task to remove the double quotes.
Here is an example:
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: xx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name kevin0824 | jq .properties.apiKey)
var2=`sed -e 's/^"//' -e 's/"$//' <<<"$output"`
echo $var2
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken; isOutput=true]$var2"
- task: AzureStaticWebApp#0
displayName: 'Static Web App: '
inputs:
app_location: /
api_location: api
skip_app_build: false
skip_api_build: false
is_static_export: false
verbose: false
azure_static_web_apps_api_token: '$(staticWebAppDeploymentToken)'

Azure DevOps Deployment Job not finding registered variable during run

I am trying to use an Azure DevOps deployment job to create a ServiceNow Standard Change Request, register the sys_id to the pipeline, then use it in subsequent phases of the Deployment Job. I have a Python utility that creates a Change Request, and filters the sys_id registering it as a variable. I can access said variable in the same "phase" of the Deployment Job, but the next phase is not working as expected, well,the docs don't really cover any use like this other than contrived uses. I think I was following the Set Variables in Scripts See my pipeline below.
- task: Bash#3
name: snow
displayName: Create Standard RFC from Template
# This task, I'm registering the SYS_ID of the RFC being created. I want to use this throughout the rest
# of the Deployment Job.
inputs:
targetType: inline
script: |
export sys_id=$(servicenow standard create template $(std_tmpl_sys_id) --query="result.sys_id.value")
echo "##vso[task.setvariable variable=rfc_sys_id;isoutput=true]$sys_id"
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
- task: Bash#3
displayName: Progress RFC to Scheduled
# This works, for this one task.
inputs:
targetType: inline
script: |
servicenow standard update $(snow.rfc_sys_id) state=Scheduled
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
deploy:
steps:
- task: Bash#3
displayName: Install ServiceNow
inputs:
targetType: inline
script: |
pip install snow --index-url=https://azure:$(System.AccessToken)#pkgs.dev.azure.com/$(ADO_ORG)/$(ADO_PROJ)/_packaging/python-azure-artifacts/pypi/simple/
- task: Bash#3
displayName: Progress RFC to Implement
# Here, I attempt to get the registered variable from the preDeploy "phase" and use it as a Shell Variable
# because otherwise Azure DevOps would try to just execute it as a shell command.
inputs:
targetType: inline
script: |
servicenow standard update ${RFC_SYS_ID} state=Implement
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['preDeploy.rfc_sys_id'] ]
Also found here:
https://gist.github.com/FilBot3/d8184b3c0b1c887e7e99884b051bd73c#file-azure-pipelines-yaml-L89-L131
Is it even possible to do this in Azure DevOps YAML Pipelines using a Deployment Job?
Can you try two of them? I think only the step-name is missing.
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['preDeploy.snow.rfc_sys_id'] ]
or
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['BuildPythonApp.snow.rfc_sys_id'] ]

set parameter based on another parameter

I need to set azureSubscription property for AzurePowerShell#5 task.
The value depends on EnvName parameter, which is as follows:
- name: EnvName
displayName: Environment Name
type: string
default: dev
values:
- dev
- test
- uat
- prod
I want to do this in PowerShell in Job #1 in SetVariables task:
if (${{ parameters.EnvName }} -eq "prod") {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Production Subscription"
} else {
echo "##vso[task.setvariable variable=AzureSubscription;isOutput=true]Non-Production Subscription"
}
And then assign the value to the task in Job #2:
- job: job2
dependsOn: job1
variables:
AzureSubscription: $[ dependencies.job1.outputs['SetVariables.AzureSubscription'] ]
steps:
- task: AzurePowerShell#5
inputs:
azureSubscription: '$(AzureSubscription)'
Unfortunately, this doesn't work. When executing the pipeline, I immediately get following error:
There was a resource authorization issue: "The pipeline is not valid.
Job job2: Step AzurePowerShell input ConnectedServiceNameARM
references service connection $(AzureSubscription) which could not be
found. The service connection does not exist or has not been
authorized for use. For authorization details, refer to
https://aka.ms/yamlauthz. Job job2: Step AzurePowerShell
input ConnectedServiceNameARM references service connection
$(AzureSubscription) which could not be found. The service connection
does not exist or has not been authorized for use. For authorization
details, refer to https://aka.ms/yamlauthz."
If I create AzureSubscription as a parameter instead of a runtime-computed variable, then everything works fine. But I would prefer not to, as the pipeline user shouldn't need to set its value, and doesn't need to know which value is correct.
It seems that azureSubscription property of AzurePowerShell#5 task is evaluated during compile-time. So I need to evaluate my variable (or parameter, or whatever) during compile-time too.
The only solution I came up with is to determine which job to run based on EnvName parameter, something like this:
- ${{ if eq(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Production Subscription
- ${{ if ne(parameters.EnvName, 'prod') }}:
- script: echo run AzurePowerShell#5 with Non-Production Subscription
But in that case I have 2 almost identical task declarations, this complicates my YAML quite a bit.
Another solution would be to use PowerShell#2 task instead of AzurePowerShell#5. That I would also rather not do.
Is there a better way to achieve my goal?

AzureDevops: Can we pass dynamic value as service connection in yaml

I have a yaml. where I need to pass serviceconnections to a script/template based on an another task which retrieve all required subscriptions.
Question is : Can I pass a dynamic value to a service connection? It is giving me compile time error.
My code below:
trigger: none
pr: none
parameters:
- name: AzureSubscription
type: object
default:
xxx:
Sub: xxx
yyy:
Sub: yyy
jobs:
- job: Updating
condition: succeeded()
pool:
vmImage: "windows-latest"
strategy:
maxParallel: 10
matrix: ${{ parameters.AzureSubscription }}
steps:
- task: AzurePowerShell#5
displayName: Tes
inputs:
azureSubscription: 'zzz'
ScriptType: 'InlineScript'
Inline: |
Write-Output "subcriptionList ---- $(Sub)"
FailOnStandardError: true
azurePowerShellVersion: 'LatestVersion'
pwsh: true
- task: AzurePowerShell#4
displayName: Updating
inputs:
**azureSubscription: ${{ sub }}** # here it is giving me error?
ScriptType: 'FilePath'
ScriptPath: '$(System.DefaultWorkingDirectory)/Foundation/xxxxx.ps1'
azurePowerShellVersion: 'LatestVersion'
So in the 2nd task, I am passing subscription from my parameter.
Error is : Unrecognized value: 'sub'.
Can someone help me?
This is possible with some creativity but not native to the task. Rather using variables and dynamically loading the variable file.
I usually declare the Service Connection name as a variable template file in a separate repository. This allows for reuse across all projects in the org, not required but find it easier that way. Part of the template name would be the environment being deployed to. So a template file might be called azure.dev.yml or azure.uat.yml and look like:
variables:
AzureSubscriptionServiceConnectionName: Azure - Dev
Then a variable defined within the scope of the stage/job would load the template file like below, assuming that a parameter or a local variable would be passed in with the given environmentName.
variables:
- template: /azure.${{ parameters.environmentName }}.yml
Then the stage/job can reference this variable via:
${{ variables.AzureSubscriptionServiceConnectionName }}
Here is some more Microsoft Documentation on YAML Pipeline Variable Scope
You cannot set azureSubcription dynamically. It is known limitation.
#JoeGaggler this feature isn't supported today. Usage of service endpoints (Azure Subscription is one of kind) in release/build definition is controlled by some permissions. At the time of saving a release/build definition service validates that the author (whoever is saving the definition) has appropriate permissions on the endpoint. If we support variable replacements for service endpoint input then service can't validate that the author has required permissions or not and it might become a security issue.

Use a variable name that is stored in another variable in Azure Pipelines

I'm using the AzureKeyVault task to retrieve a secret from the Key Vault. The name of the secret is StorageAccountKey. This name is stored in the variable KeyName. I do it like that
- task: AzureKeyVault#1
displayName: 'Get key'
name: GetKey
inputs:
azureSubscription: '${{ parameters.azureSubscription }}'
KeyVaultName: '$(KeyVaultName)'
SecretsFilter: '$(KeyName)'
Now, in a subsequent task, I would like to access the secret. How would I do that, given that the name of the secret is itself stored in a variable? The following seems not to work
- task: Bash#3
displayName: Create container
inputs:
targetType: 'inline'
script: |
az storage container create \
--name raw \
--account-name storageaccountname \
--account-key $($(dataLakeAccountKeyKeyName))
failOnStderr: true
I'm getting the error
/mnt/azp/azp-linux1_5/_temp/6719378a-b3ee-45d8-aad8-4f6a5e8b581e.sh: line 1: StorageAccountKey: command not found
ERROR: az storage container create: error: argument --account-key: expected one argument
So, it does seem to resolve the inner variable but still fails.
I also struggled to get this done and this has worked for me:
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.azureSubscription }}
KeyVaultName: ${{ parameters.azureKeyVaultName }}
SecretsFilter: '*'
RunAsPreJob: true
- bash: |
#I can now use ${GCP_CREDS}
displayName: GCP auth
env:
GCP_CREDS: $(${{ parameters.azureKeyVaultCredentailsKey }})
Try to use --account-key $(StorageAccountKey)
From "Azure Key Vault task" documentation.
"Values are retrieved as strings. For example, if there is a secret named connectionString, a task variable connectionString is created with the latest value of the respective secret fetched from Azure key vault. This variable is then available in subsequent tasks."
So if you access secret named in azure key vault "StorageAccountKey" then Azure DevOps creates from this place variable called "StorageAccountKey".
I have never used Azure Key Vault but hope it will help you : )