I am using a variable group to store some secrets, that I want to deploy into a key vault as part of my pipeline.
I do this by passing the values as secret parameters to a bicep file via the Azure CLI.
My pipeline step looks like this:
- task: AzureCLI#2
displayName: deploy bicep template
inputs:
azureSubscription: ${{ parameters.azure_subscription }}
scriptType: 'ps'
scriptLocation: 'inlineScript'
inlineScript: |
az deployment group create `
--name foo.$(Build.BuildNumber) `
--resource-group my-rg-name `
--template-file $(Pipeline.Workspace)/iac/main.bicep `
--parameters environment='${{ parameters.environment }}' `
kv_secret_one='$(bd_kv_secret_one)' `
kv_secret_two='$(bd_kv_secret_two)'
Now, the issue is that these secrets could contain all sorts of special characters that break my script. So I found some suggestions that I could use double quotes inside the single quotes like this kv_secret_two='"$(bd_kv_secret_two)"'
However, if the value of the secret contains a " then the script breaks.
This leads me to think there must be a better way of doing this, but I cannot find it.
What is the correct way to pass --parameters so that they are escaped properly, no matter what characters the may contain?
You can do this with a bicep parameter file.
First, construct a parameter file including all your parameter values, escaped as necessary. You could do this in a script task for example.
Secondly, name that file when doing the deployment instead of listing individual parameters:
- task: AzureCLI#2
displayName: deploy bicep template
inputs:
azureSubscription: ${{ parameters.azure_subscription }}
scriptType: 'ps'
scriptLocation: 'inlineScript'
inlineScript: |
az deployment group create `
--name foo.$(Build.BuildNumber) `
--resource-group my-rg-name `
--template-file $(Pipeline.Workspace)/iac/main.bicep `
--parameters #main.parameters.json
You could check this document for double quotation in parameter, and edit your value to escape with the equivalent of the string "{\"key\": \"value\"}"
Related
Currently trying to get the Azure Key Reference from Key Vault using a DevOps Task.
I have it working for PowerShell but I was moved over to a different agent and now we are using PSCore for the script type. In doing so the previous way no longer works.
How do I do the following with PSCore as a scriptType?
- task: AzureCLI#2
displayName: 'Get KeyVault Secret URIs'
inputs:
azureSubscription: '$(Sub_Dev)'
scriptType: 'ps'
failOnStandardError: true
scriptLocation: 'inlineScript'
inlineScript: |
$Dev_keyVaultName= "$(AppService_Dev)"
$secretNames=(
"ServiceBusConnectionStringKey")
$outputVars=(
"ServiceBusConnectionStringKey")
$x=0
while ($x -lt $secretNames.count)
{
$uri=$(az keyvault secret show --vault-name $Dev_keyVaultName --name $secretNames.get($x) --query id --output tsv)
$secretname = $outputVars.get($x)
echo "##vso[task.setvariable variable=$secretname;]#Microsoft.KeyVault(SecretUri=$uri)"
$x++
}
This will loop through all the specified Keys that I send it and return the Reference to the Key, so that I can store the Reference in the Config instead of the actual Secret.
Using the Next section it will return the actual Secret, but I need the reference?
- task: AzureCLI#2
displayName: 'Get KeyVault Secret URIs'
inputs:
azureSubscription: '$(Sub_Dev)'
scriptType: 'pscore'
failOnStandardError: true
scriptLocation: 'inlineScript'
inlineScript: |
$Dev_keyVaultName= "$(AppService_Dev)"
$(sbConnstring)=$(az keyvault secret show --name "ServiceBusConnectionStringKey" --vault-name $Dev_keyVaultName)
Hi have the following code that deploys an artifact to an Azure Static Web App:
...
variables:
- name: staticWebAppDeploymentToken
...
# This steps reads the deployment token of the static web app and assigns it on a variable
- task: AzureCLI#2
displayName: 'Retrieve static web app deployment token'
inputs:
azureSubscription: xxxx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name xxxx-xxxx-$(environment) | jq .properties.apiKey)
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken;]$output"
- task: AzureStaticWebApp#0
inputs:
output_location: '/'
cwd: '$(Pipeline.Workspace)/artifact'
skip_app_build: true
azure_static_web_apps_api_token: $(staticWebAppDeploymentToken)
I get the error:
I've set the System.Debug variable to true, and I see the value is set in the variable. I've also printed the variable and the value is there.
I can't understand what I'm doing wrong. What is the correct way to set a variable in bash and use it on another non-bash step?
I've tried hardcoding the value and also passing it as a parameter from the library, and that works, but that is not what I want.
Test the same script to get the token and pass it to Azure Static web APP task, I can reproduce the same issue.
The root cause of this issue is that when you set the variable in Azure CLI task with the command, the pipeline variable will contain "". For example: "tokenvalue".
The expected deployment token in Azure Static web APP task, it will not contain double quotes.
To solve this issue, you need to add a step in Azure CLI task to remove the double quotes.
Here is an example:
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: xx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name kevin0824 | jq .properties.apiKey)
var2=`sed -e 's/^"//' -e 's/"$//' <<<"$output"`
echo $var2
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken; isOutput=true]$var2"
- task: AzureStaticWebApp#0
displayName: 'Static Web App: '
inputs:
app_location: /
api_location: api
skip_app_build: false
skip_api_build: false
is_static_export: false
verbose: false
azure_static_web_apps_api_token: '$(staticWebAppDeploymentToken)'
I have a DevOps pipeline that gives me this error:
There was a resource authorization issue: "The pipeline is not valid. Job ExecutionTerraform: Step AzureCLI input connectedServiceNameARM references service connection Azure: $(subscriptionName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
The configuration I am using is looking up the Subscription name dynamically.
The step I use for that is:
- bash: |
# pull the subscription data
# ... read data into local variables
# set the shared variables
echo "##vso[task.setvariable variable=subscriptionId]${SUBSCRIPTION_ID}"
echo "##vso[task.setvariable variable=subscriptionName]${SUBSCRIPTION_NAME}"
From there I attempt to call the Azure CLI via a template:
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
Inside the template my CLI step uses:
steps:
- task: AzureCLI#2
displayName: Test CLI
inputs:
azureSubscription: "ARMTest ${{ parameters.azureSubscriptionName }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az --version
addSpnToEnvironment: true
useGlobalConfig: true
It looks like Pipelines is trying to preemptively check authorization without noticing that there's a variable in there. What am I doing wrong here that is causing Azure to attempt to resolve that at the wrong time?
I do this in other pipelines without issues and I am not sure what is different in this particular instance.
Update 1: Working Template I have Elsewhere
Full template:
parameters:
- name: environment
type: string
jobs:
- job: AKSCredentials
displayName: "AKS Credentials Pull"
steps:
- task: AzureCLI#2
displayName: AKS Credentials
inputs:
azureSubscription: "Azure: testbed-${{ parameters.environment }}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az aks get-credentials -g testbed-${{ parameters.environment }} -n testbed-${{ parameters.environment }}-aks
addSpnToEnvironment: true
useGlobalConfig: true
This is not possible because azure subscription needs to be known at compilation time. You set your variable on run time.
Here an issue with similar case when it is explained:
run time variables aren't supported for service connection OR azure subscription. In your code sample, you are referring to AzureSubscription variable which will get initialized at the run time (but not at save time). Your syntax is correct but you need to set AzureSubscription variable as part of variables.
If you define your variables like:
variables:
subscriptionId: someValue
subscriptionName: someValue
and then you will use it
- template: execution-cli.yml
parameters:
azureSubscriptionId: $(subscriptionId)
azureSubscriptionName: $(subscriptionName)
it should work. But since you set up your variables on runtime it causes your issue.
Trying to dynamically retrieve all the variables from a variable group via Azure DevOps task in a YAML Pipeline. Originally tried leveraging the AzureCLI#2 task with the following code to retrieve the variableGroupID which would be used to get the variables inside of it:
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
This command works locally but not when executing on a MS hosted agent like this:
parameters:
variableGroupName: ''
steps:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: ${{ parameters.azureSubscriptionName }}
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
az upgrade
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
write-Host $variableGroupId
$variables = $(az pipelines variable-group variable list --group-id $variableGroupId --org $(System.CollectionUri) --project $(System.TeamProject) -o yaml)
write-Host $variables
This fails with the error:
Before you can run Azure DevOps commands, you need to run the login command (az login if using AAD/MSA identity else az devops login if using PAT token) to setup credentials. Please see https://aka.ms/azure-devops-cli-auth for more information
I have opened up an issue
In the meantime, I tried to run the commands to install the necessary pieces via scripts
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#3
inputs:
azureSubscription: Example - Dev
WebAppName: wapp-Example-dev-eus
Package: $(Pipeline.Workspace)/drop/Web.Example.zip
TakeAppOfflineFlag: True
- task: UsePythonVersion#0
inputs:
versionSpec: '3.x'
architecture: 'x64'
- task: CmdLine#2
displayName: 'Upgrade pip'
inputs:
script: python -m pip install --upgrade pip
- task: CmdLine#2
displayName: 'upgrade azure cli'
inputs:
script: pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
- task: CmdLine#2
displayName: 'Show Azure CLI version'
inputs:
script: az --version
- task: CmdLine#2
displayName: 'Install Azure DevOps Extension'
inputs:
script: az extension add -n azure-devops
- task: CmdLine#2
env:
AZURE_DEVOPS_CLI_PAT: $(patCredential)
displayName: 'Login Azure DevOps Extension'
inputs:
script: echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
- task: CmdLine#2
displayName: 'Show List of Variables'
inputs:
script: |
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
write-Host $variableGroupId
$variables = $(az pipelines variable-group variable list --group-id $variableGroupId --org $(System.CollectionUri) --project $(System.TeamProject) -o yaml)
write-Host $variables
However, when using both latest Ubuntu agents and those designated in the doc get an error:
WARNING: Failed to store PAT using keyring; falling back to file storage.
WARNING: You can clear the stored credential by running az devops logout.
WARNING: Refer https://aka.ms/azure-devops-cli-auth to know more on sign in with PAT.
I have opened up an issue with the documentation team as at the very least the provided steps do not work. Any assistance would be appreciated!
I was getting the same error, and was able to get mine working by adding:
echo $(System.AccessToken) | az devops login
to the top of my inline script. Here's what it looks like:
variables:
variableGroupName: 'my-variable-group'
...
- task: AzureCLI#2
displayName: 'Set environment variables'
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
echo $(System.AccessToken) | az devops login
$groupId = (az pipelines variable-group list `
--organization $(System.CollectionUri) `
--project $(System.TeamProject) `
--group-name $(variableGroupName) | ConvertFrom-Json).id
...
You can use the REST API instead of Azure CLI to get the information. It can be used with the standard tools already present on the Microsoft Hosted agents. It requires only vanilla powershell or powershell core, meaning to works on both windows and linux agents. The below example was successfully tested on windows-latest/windows-2019 and ubuntu-latest/ubuntu-20.04
The approach is the same as with Azure CLI.
List all available groups filtered by name to retrieve the variable group in question
Get all variables in the variable group using the variable group id from step
In fact, the pipeline also has an out of the box PAT token available with read access to variable groups. It is stored in the variable System.AccessToken. Using that instead of a manually managed one will further simplify things.
The script below is executed in a pwsh step, which is the built in Powershell task in Powershell core mode
- pwsh: |
# Construct PAT authentication header
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "user",$env:SYSTEM_ACCESSTOKEN)))
$headers = #{Authorization=("Basic {0}" -f $base64AuthInfo)}
# Retrieve variable group id. Filter the result by setting the groupName query parameter
$variableGroupId = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups?groupName=${{ parameters.variableGroupName }}&api-version=6.0-preview.2").value[0].id
# Retrieve variables in variable group with id $variableGroupId
$variables = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups/${variableGroupId}?api-version=6.0-preview.2").variables
#Print variables as json (for demo purpose)
$variables | ConvertTo-Json
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
displayName: 'Retrieve variables'
Testing the above pipeline in a project on a variable group with two variables yields the following output:
{
"Variable 1": {
"value": "Value 1"
},
"Variable 2": {
"value": "Value 2"
}
}
I'm using the AzureKeyVault task to retrieve a secret from the Key Vault. The name of the secret is StorageAccountKey. This name is stored in the variable KeyName. I do it like that
- task: AzureKeyVault#1
displayName: 'Get key'
name: GetKey
inputs:
azureSubscription: '${{ parameters.azureSubscription }}'
KeyVaultName: '$(KeyVaultName)'
SecretsFilter: '$(KeyName)'
Now, in a subsequent task, I would like to access the secret. How would I do that, given that the name of the secret is itself stored in a variable? The following seems not to work
- task: Bash#3
displayName: Create container
inputs:
targetType: 'inline'
script: |
az storage container create \
--name raw \
--account-name storageaccountname \
--account-key $($(dataLakeAccountKeyKeyName))
failOnStderr: true
I'm getting the error
/mnt/azp/azp-linux1_5/_temp/6719378a-b3ee-45d8-aad8-4f6a5e8b581e.sh: line 1: StorageAccountKey: command not found
ERROR: az storage container create: error: argument --account-key: expected one argument
So, it does seem to resolve the inner variable but still fails.
I also struggled to get this done and this has worked for me:
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.azureSubscription }}
KeyVaultName: ${{ parameters.azureKeyVaultName }}
SecretsFilter: '*'
RunAsPreJob: true
- bash: |
#I can now use ${GCP_CREDS}
displayName: GCP auth
env:
GCP_CREDS: $(${{ parameters.azureKeyVaultCredentailsKey }})
Try to use --account-key $(StorageAccountKey)
From "Azure Key Vault task" documentation.
"Values are retrieved as strings. For example, if there is a secret named connectionString, a task variable connectionString is created with the latest value of the respective secret fetched from Azure key vault. This variable is then available in subsequent tasks."
So if you access secret named in azure key vault "StorageAccountKey" then Azure DevOps creates from this place variable called "StorageAccountKey".
I have never used Azure Key Vault but hope it will help you : )