I have a Azure Key Vault with different keys (e.g. Key1, Key2, Key3). In some setting, which is dynamically read in one pipeline task, I have value which says which key to use (lets KeyName variable be 'Key2'). How can I read the secret value of the Key2 in the pipeline?
I have tried different combinations and none is working for me.
Test pipeline no. 1 - using the group variable connected to the Key Vault (to make it easier, the KeyName is static here, but in reality, it is set through powershell script during the pipeline):
jobs:
- job: JobA
variables:
- group: KeyVaultGroup #Key vault have secret values "Key1,Key2,Key3..."
- name: KeyName
value: Key2
- name: MyValue
value: $[ variables[variables.KeyName] ]
steps:
- powershell: |
Write-Host "Var1 $($env:VARENV1)"
Write-Host "Var2 $($env:VARENV2)"
env:
VarEnv1: $(MyValue)
VarEnv2: $($(KeyName))
Result is:
Var1
Var2 $(Key2)
MyValue is not working, because the variable is evaluated before the key vault variables are loaded. And when the KeyName is set during the pipeline, it will not work because that too (but this could be solved by using separate job and using output variables to set the KeyName - like in test example no. 2).
Expression $($(KeyName)) is not working, because it will not recursively expand the variable (bug?).
Same problem is when the AzureKeyVault task is used to read the Key Vault values, because it is triggered too late.
Test no. 2 - two separate jobs:
I have used 2 jobs - one to read the key vault and Key name (Job A) and second for the rest (Job B). Problem is, that there is no way how to access the key vault secret values loaded on job A from the job B. I can use only output variables from Job A in the Job B through the dependencies.JobA... but the task AzureKeyVault is not exporting the values as output variables. To do so I will need to use e.g. Powershell task, but in this case, I will need to map the secret values as environment variables into the powershell task, but it means I will loose the dynamic part I need, because it will be statically mapped ( I need to be able to add/remove the values in the key vault without need to change the pipeline). This is no go or I do not know the way how to access the secret variables between jobs without using output variables.
Question:
How to read the secure value from key "Key2" when the Key2 is saved as variable value KeyName and is loaded during the pipeline?
In this case the best way is to use Azure CLI task with azure keyvault command:
- task: AzureCLI#2
inputs:
azureSubscription: 'rg-the-code-manual'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
$secretValue = az keyvault secret show --vault-name tcm-kv --name $(keyName) --query value
echo $secretValue
The content of this script is evaluated at runtime so you can set keyName just before this task and all will be fine. And if you need value of secret as variable you can use logging command to create such.
The easiest way to get rid of double quote will be change output to tsv.
$secretValue = az keyvault secret show --vault-name tcm-kv --name $(keyName) --query value -o tsv
Related
I have setup a in my azure pipeline which create a Redis cache instance through Azure CLI.
There is another task which runs afterwards that pick set the values in my application config file from a pipeline variable named "CacheConnectionKey".
I have to set the variable value manually in the pipeline. I want to automate this process by adding a new task in between both described above. The new task should get PrimaryKey from Redis cache instance and set the value in the pipeline variable (i.e. CacheConnectionKey).
There is a command I have tried in the power shell which is giving me the access keys:
Get-AzRedisCacheKey -ResourceGroupName "MyResourceGroup" -Name "MyCacheKey"
PrimaryKey : pJ+jruGKPHDKsEC8kmoybobH3TZx2njBR3ipEsquZFo=
SecondaryKey : sJ+jruGKPHDKsEC8kmoybobH3TZx2njBR3ipEsquZFo=
Now I want to set PrimaryKey resulting from this command to be set in the pipeline variable CacheConnectionKey so that the next process could use the value properly.
The "process" referred in the question could be anything like a run/job/stage/pipeline I suppose. Regardless, in YAML pipelines, you can set variables at the root, stage, and job level. You can also use a variable group to make variables available across multiple pipelines. Some tasks define output variables, which you can consume in downstream steps, jobs, and stages.
In YAML, you can access variables across jobs and stages by using dependencies. By default, each stage in a pipeline depends on the one just before it in the YAML file. So if you need to refer to a stage that isn't immediately prior to the current one, you can override this automatic default by adding a dependsOn section to the stage.
For example, let's assume you have a task called MyTask, which sets an output variable called MyVar.
To use outputs in the same job:
steps:
- task: MyTask#1 # this step generates the output variable
name: ProduceVar # because we're going to depend on it, we need to name the step
- script: echo $(ProduceVar.MyVar) # this step uses the output variable
To use outputs in a different job:
jobs:
- job: A
steps:
# assume that MyTask generates an output variable called "MyVar"
- task: MyTask#1
name: ProduceVar # because we're going to depend on it, we need to name the step
- job: B
dependsOn: A
variables:
# map the output variable from A into this job
varFromA: $[ dependencies.A.outputs['ProduceVar.MyVar'] ]
steps:
- script: echo $(varFromA) # this step uses the mapped-in variable
For more details on the syntax and examples, check the following articles:
Use output variables from tasks
Dependencies
We have certain functional tests that rely on some secrets. Those secrets are obtained from a Azure Key Vault (AKV) and to connect from build agent, I am using environment variables and AzureIdentity.I set those env variables on the build agent machine using powershell. When I use non-secret pipeline variables, then everything works but when I switch to secret pipeline variable for AZURE_CLIENT_SECRET, the authentication starts to fail. I tried the approach of using a script to set the environment variable from secret pipeline variable, but it does not work. I also tried the approach mentioned here but that does not work either. ANy suggestion on how to set an environment variable using secret pipeline variables?
ANy suggestion on how to set an environment variable using secret pipeline variables?
If you set secret variable in below pipeline.
And then use the script's environment or map the variable within the variables block to pass secrets to your pipeline like below script. See: Set secret variables for details.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable
If you use Azure Key vault variable, we create a secret variable(PAT) in below Azure key vault.
So we can link secrets from an Azure key vault in variable group, as below.
Now we can use this variable group in below script. See: Reference secret variables in variable groups for details.
variables:
- group: 'AKVgroup' # variable group
pool:
vmImage: 'ubuntu-latest'
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable
The other way is using Azure Key Vault task like below script. See: Use secrets from Azure Key Vault in Azure Pipelines for details.
- task: AzureKeyVault#1
inputs:
azureSubscription: 'ARM'
KeyVaultName: 'edwardkey'
SecretsFilter: '*'
RunAsPreJob: true
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Using the mapped env var for this task works and is recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(PAT) # the recommended way to map to an env variable
If you explicitly pass the secret to the script as a parameter then the scrip will have access to it.
If you want to then use that to set an environment variable for use in later scripts you'll can use a different environment variable name and have the script publish that you want it available in subsequent scripts. That sort of defeats the purpose of it being secret but if thats what you want.
How Can I make Secret variable (SecretVar) defined in azure release pipeline be accessible to my Powershell used to create marketplace task (vsix)?
How Can I make Secret variable (SecretVar) defined in azure release pipeline be accessible to my Powershell used to create marketplace task (vsix)?
You could not access the secret variable directly from the task. This behavior is by designed for protecting secret variables from being exposed in the task.
This documentation states that secret variables are:
Not decrypted into environment variables. So scripts and programs run by your build steps are not given access by default.
Decrypted for access by your build steps. So you can use them in password arguments and also pass them explicitly into a script or a program from your build step (for example as $(password)).
That the reason why you could not use the secret variables in your task.
To resolve this issue, we need to explicitly map secret variables:
variables:
GLOBAL_MYSECRET: $(mySecret)
GLOBAL_MY_MAPPED_ENV_VAR: foo
steps:
- Youtask: |
env:
MY_MAPPED_ENV_VAR: $(mySecret) # right way to map to an env variable
Or if the secret variable can be set as arguments, we could use it:
variables:
VMS_USER: $(vmsUser)
VMS_PASS: $(vmsAdminPass)
pool:
vmImage: 'ubuntu-latest'
steps:
- task: AzureFileCopy#4
inputs:
SourcePath: 'my/path'
azureSubscription: 'my-subscription'
Destination: 'AzureVMs'
storage: 'my-storage'
resourceGroup: 'my-rg'
vmsAdminUserName: $(VMS_USER)
vmsAdminPassword: $(VMS_PASS)
If your task does not support env: or arguments to pass secret variables explicitly into a script, you could not use it in the task.
You could check this thread for and the document for some more details.
Update:
My custom marketplace task uses Powershell (not yaml) and that is
where I would like to access it. How can I do that within powershell?
If you want to access the secret variables in the powershell script instead of the inline/powershell task, you could try to pass the value of secret variable through PowerShell parameters:
Param(
[String]$pass
)
if ($pass) { Write-Host "variable is NOT null" }
if (!$pass) { Write-Host "variable is null" }
Check this thread for some details.
Hope this helps.
I am trying to access secret variable to pass it to another script.
I expect following code in pipeline to print Value but it prints some text 'xxx' ragardless of the value of a secret variable
echo xxx
Pipeline Snippet
steps:
- bash: echo This script could use $SYSTEM_ACCESSTOKEN
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
If you want to access a secret variable, you could print it to a file. Check the example below:
steps:
- powershell: |
$env:var1 | Out-File C:\Users\xxx\Desktop\Newfolder\debug.txt
displayName: 'PowerShell Script'
env:
var1: $(System.AccessToken)
But System.Accesstoken is a PAT token generated for the service identity “Project Collection Build Service (account)”, it's not needed to verify the value of System.AccessToken. In addition, if you want to print the value of System.AccessToken to a file, you need to check the Allow scripts to access the OAuth token in the agent job:
Azure pipelines will scan the output and mask the secret, you can simply split it up and print it in two parts.
Here is a bash example:
echo "${MY_SECRET:0:10}" # Print the first 10 characters
echo "${MY_SECRET:10:100}" # Print character 11 - 100
You should of course only do it for debugging purposes and not leave it in your pipeline.
Updates:
If I save secret value to a file and publish that file as an artifact secret is visible in cleartext.
After speaking to my colleagues I have realized that all text in logs if it contains a secret value it will be masked.
It will interesting to see if I have 2 variables viz.
OPEN_VAR='something' # No Secret
and
SECRET_VAR='something' # Values same as above but Secret
if I print $OPEN_VAR ; does it mask value because "something" is also a value of "SECRET_VAR"
This is because SYSTEM_ACCESSTOKEN is a secret. If you do the same with variable which is not a secret you will be able to see value.
How can I consume variables set in CI pipeline directly in CD pipeline?
Example:
If there are multiply variables declared in one group. How to access particular variable in CI/CD pipeline?
Also how to use variable groups for multiple keyvault's?
Thank you
Setting up global variables
YAML
https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
CLASSIC
https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=classic
Using Variables
YAML
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
CLASSIC
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=classic%2Cbatch
Examples:
Batch script: %VARIABLE_NAME%
PowerShell script: $env:VARIABLE_NAME
Bash script: $VARIABLE_NAME
Key Vault
https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=classic#link-secrets-from-an-azure-key-vault
Based on Richard's answer, I managed to set the value of a variable in a variable group to a variable coming from a CI pipeline, and then to read that variable in a CD pipeline.
For that, it is necessary:
To have previously created the variable group, and the variable (its name identified by $(variableName). Let's assume its value would be stored in $(variableValue)).
To find the variable group ID (stored in $(variableGroupId)), which can be done by navigating on Azure DevOps to that variable group. The group ID will then be in the URL.
A Personal Access Token (PAT) with Read & Write access to group variables (called $(personalAccessToken) )
CI pipeline
- powershell: |
az pipelines variable-group variable update --group-id $(variableGroupId) --name $(variableName) --value $(variableValue)
displayName: 'Store the variable in a group variable'
env:
AZURE_DEVOPS_EXT_PAT: $(personalAccessToken)
Then all that's necessary, is to declare the variable group in the CD pipeline. If this variable group is called MyVariableGroup, it can be done in the following way:
CD pipeline
variables:
- group: MyVariableGroup
The variable that was previously set in the CI pipeline, will then be available in the CD pipeline.