Use Azure pipeline secret variable to set environment variables on build agent - azure-devops

We have certain functional tests that rely on some secrets. Those secrets are obtained from a Azure Key Vault (AKV) and to connect from build agent, I am using environment variables and AzureIdentity.I set those env variables on the build agent machine using powershell. When I use non-secret pipeline variables, then everything works but when I switch to secret pipeline variable for AZURE_CLIENT_SECRET, the authentication starts to fail. I tried the approach of using a script to set the environment variable from secret pipeline variable, but it does not work. I also tried the approach mentioned here but that does not work either. ANy suggestion on how to set an environment variable using secret pipeline variables?

ANy suggestion on how to set an environment variable using secret pipeline variables?
If you set secret variable in below pipeline.
And then use the script's environment or map the variable within the variables block to pass secrets to your pipeline like below script. See: Set secret variables for details.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable
If you use Azure Key vault variable, we create a secret variable(PAT) in below Azure key vault.
So we can link secrets from an Azure key vault in variable group, as below.
Now we can use this variable group in below script. See: Reference secret variables in variable groups for details.
variables:
- group: 'AKVgroup' # variable group
pool:
vmImage: 'ubuntu-latest'
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable
The other way is using Azure Key Vault task like below script. See: Use secrets from Azure Key Vault in Azure Pipelines for details.
- task: AzureKeyVault#1
inputs:
azureSubscription: 'ARM'
KeyVaultName: 'edwardkey'
SecretsFilter: '*'
RunAsPreJob: true
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable

If you explicitly pass the secret to the script as a parameter then the scrip will have access to it.
If you want to then use that to set an environment variable for use in later scripts you'll can use a different environment variable name and have the script publish that you want it available in subsequent scripts. That sort of defeats the purpose of it being secret but if thats what you want.

Related

Use azure keyvault secret as environment variable in Azure DevOps pipeline

As part of build I am using 'envsubst' command to replace all secrets from environment variables to my application configuration file. We are using Azure DevOps pipeline for our build process and now start using Azure KeyVault to store all these Secrets. Current Issue is that I am not able to make these secrets as environment variable in MS based agent runner. I tried to refer multiple documents but nothing helps.
Did anyone able to set the Azure KeyVault secrets as environment variable on the build agent runner. Any clue/guidance will be of help
Tried using setvariable task but it helps to read the secret and use it within job but not help insetting that variable as environment variable
task: Bash#3
inputs:
targetType: 'inline'
script: |
# Write your commands here
echo "##vso[task.setvariable variable=MySecret;issecret=true]$(MY-SECRET)"
Export and set command inside shell task is not allowing to setup environment variable
Also tried env setting, but that too didn't help
env:
MYSECRET: $(MY-SECRET)
Anyone who have implemented the same, do let me know

Azure DevOps Deployment Job not finding registered variable during run

I am trying to use an Azure DevOps deployment job to create a ServiceNow Standard Change Request, register the sys_id to the pipeline, then use it in subsequent phases of the Deployment Job. I have a Python utility that creates a Change Request, and filters the sys_id registering it as a variable. I can access said variable in the same "phase" of the Deployment Job, but the next phase is not working as expected, well,the docs don't really cover any use like this other than contrived uses. I think I was following the Set Variables in Scripts See my pipeline below.
- task: Bash#3
name: snow
displayName: Create Standard RFC from Template
# This task, I'm registering the SYS_ID of the RFC being created. I want to use this throughout the rest
# of the Deployment Job.
inputs:
targetType: inline
script: |
export sys_id=$(servicenow standard create template $(std_tmpl_sys_id) --query="result.sys_id.value")
echo "##vso[task.setvariable variable=rfc_sys_id;isoutput=true]$sys_id"
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
- task: Bash#3
displayName: Progress RFC to Scheduled
# This works, for this one task.
inputs:
targetType: inline
script: |
servicenow standard update $(snow.rfc_sys_id) state=Scheduled
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
deploy:
steps:
- task: Bash#3
displayName: Install ServiceNow
inputs:
targetType: inline
script: |
pip install snow --index-url=https://azure:$(System.AccessToken)#pkgs.dev.azure.com/$(ADO_ORG)/$(ADO_PROJ)/_packaging/python-azure-artifacts/pypi/simple/
- task: Bash#3
displayName: Progress RFC to Implement
# Here, I attempt to get the registered variable from the preDeploy "phase" and use it as a Shell Variable
# because otherwise Azure DevOps would try to just execute it as a shell command.
inputs:
targetType: inline
script: |
servicenow standard update ${RFC_SYS_ID} state=Implement
env:
SNOW_USER: '$(SNOW_USER)'
SNOW_PASS: '$(SNOW_PASS)'
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['preDeploy.rfc_sys_id'] ]
Also found here:
https://gist.github.com/FilBot3/d8184b3c0b1c887e7e99884b051bd73c#file-azure-pipelines-yaml-L89-L131
Is it even possible to do this in Azure DevOps YAML Pipelines using a Deployment Job?
Can you try two of them? I think only the step-name is missing.
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['preDeploy.snow.rfc_sys_id'] ]
or
RFC_SYS_ID: $[ dependencies.BuildPythonApp.outputs['BuildPythonApp.snow.rfc_sys_id'] ]

How Can I make Secret variables (defined in azure release pipeline) be accessible to my Powershell marketplace task?

How Can I make Secret variable (SecretVar) defined in azure release pipeline be accessible to my Powershell used to create marketplace task (vsix)?
How Can I make Secret variable (SecretVar) defined in azure release pipeline be accessible to my Powershell used to create marketplace task (vsix)?
You could not access the secret variable directly from the task. This behavior is by designed for protecting secret variables from being exposed in the task.
This documentation states that secret variables are:
Not decrypted into environment variables. So scripts and programs run by your build steps are not given access by default.
Decrypted for access by your build steps. So you can use them in password arguments and also pass them explicitly into a script or a program from your build step (for example as $(password)).
That the reason why you could not use the secret variables in your task.
To resolve this issue, we need to explicitly map secret variables:
variables:
GLOBAL_MYSECRET: $(mySecret)
GLOBAL_MY_MAPPED_ENV_VAR: foo
steps:
- Youtask: |
env:
MY_MAPPED_ENV_VAR: $(mySecret) # right way to map to an env variable
Or if the secret variable can be set as arguments, we could use it:
variables:
VMS_USER: $(vmsUser)
VMS_PASS: $(vmsAdminPass)
pool:
vmImage: 'ubuntu-latest'
steps:
- task: AzureFileCopy#4
inputs:
SourcePath: 'my/path'
azureSubscription: 'my-subscription'
Destination: 'AzureVMs'
storage: 'my-storage'
resourceGroup: 'my-rg'
vmsAdminUserName: $(VMS_USER)
vmsAdminPassword: $(VMS_PASS)
If your task does not support env: or arguments to pass secret variables explicitly into a script, you could not use it in the task.
You could check this thread for and the document for some more details.
Update:
My custom marketplace task uses Powershell (not yaml) and that is
where I would like to access it. How can I do that within powershell?
If you want to access the secret variables in the powershell script instead of the inline/powershell task, you could try to pass the value of secret variable through PowerShell parameters:
Param(
[String]$pass
)
if ($pass) { Write-Host "variable is NOT null" }
if (!$pass) { Write-Host "variable is null" }
Check this thread for some details.
Hope this helps.

Azure Devops - How to pass environment variables into dotnet test?

I have a standard .NET Core (Ubuntu) pipeline on Azure Devops and within my Test project, I use environment variables. Within my pipeline, I have defined my group variables like so
variables:
- group: MyApiVariables
Whenever I run the tests for my project
- task: DotNetCoreCLI#2
displayName: "Testing Application"
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '--configuration $(buildConfiguration)'
The actual environment variables aren't passed in. They are blank.
What am I missing to get this running? I've even defined variables in the edit pipeline page too with no luck
- task: Bash#3
inputs:
targetType: 'inline'
script: echo $AppConfigEndpoint
env:
AppConfigEndpoint: $(AppConfigEndpoint)
ApiConfigSection: $(ApiConfigSection)
Thanks!
CASING Strikes again! MyVariableName was turned into MYVARIABLENAME on Azure Devops. I changed my variable names in my group to all caps and it worked. I spent way too much time on this.
Mike, I see that you use variable groups. I assume it may cause your issue. Take a look at variable passing example I made:
First I had to create new variable group in Library:
Here is a pipeline code that reference created variables:
# Set variables group reference
variables:
- group: SampleVariableGroup
steps:
- powershell: 'Write-Host "Config variable=$(configuration) Platform variable=$(platform)"'
displayName: 'Display Sample Variable'
I used PowerShell task to verify if variables were properly passed to the job.
As you can see both configuration & platform values were displayed correctly.
In fact you can't go wrong that way, unless you start to mix variable groups with variables defined in a yaml. In such scenario you'll have to use name/value syntax for the individual (non-grouped) variables.
Please see Microsoft Variable Groups documentation. Such example is well explained there. I also suggest to take closer look at general Variables Documentation.
In case of referencing variables in other tasks here is a great example from MS (it should work everywhere in same manner):
# Set variables once
variables:
configuration: debug
platform: x64
steps:
# Use them once
- task: MSBuild#1
inputs:
solution: solution1.sln
configuration: $(configuration) # Use the variable
platform: $(platform)
# Use them again
- task: MSBuild#1
inputs:
solution: solution2.sln
configuration: $(configuration) # Use the variable
platform: $(platform)
Good luck!

Use a variable name that is stored in another variable in Azure Pipelines

I'm using the AzureKeyVault task to retrieve a secret from the Key Vault. The name of the secret is StorageAccountKey. This name is stored in the variable KeyName. I do it like that
- task: AzureKeyVault#1
displayName: 'Get key'
name: GetKey
inputs:
azureSubscription: '${{ parameters.azureSubscription }}'
KeyVaultName: '$(KeyVaultName)'
SecretsFilter: '$(KeyName)'
Now, in a subsequent task, I would like to access the secret. How would I do that, given that the name of the secret is itself stored in a variable? The following seems not to work
- task: Bash#3
displayName: Create container
inputs:
targetType: 'inline'
script: |
az storage container create \
--name raw \
--account-name storageaccountname \
--account-key $($(dataLakeAccountKeyKeyName))
failOnStderr: true
I'm getting the error
/mnt/azp/azp-linux1_5/_temp/6719378a-b3ee-45d8-aad8-4f6a5e8b581e.sh: line 1: StorageAccountKey: command not found
ERROR: az storage container create: error: argument --account-key: expected one argument
So, it does seem to resolve the inner variable but still fails.
I also struggled to get this done and this has worked for me:
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: ${{ parameters.azureSubscription }}
KeyVaultName: ${{ parameters.azureKeyVaultName }}
SecretsFilter: '*'
RunAsPreJob: true
- bash: |
#I can now use ${GCP_CREDS}
displayName: GCP auth
env:
GCP_CREDS: $(${{ parameters.azureKeyVaultCredentailsKey }})
Try to use --account-key $(StorageAccountKey)
From "Azure Key Vault task" documentation.
"Values are retrieved as strings. For example, if there is a secret named connectionString, a task variable connectionString is created with the latest value of the respective secret fetched from Azure key vault. This variable is then available in subsequent tasks."
So if you access secret named in azure key vault "StorageAccountKey" then Azure DevOps creates from this place variable called "StorageAccountKey".
I have never used Azure Key Vault but hope it will help you : )