Storing and retrieving secrets in Azure keyvault with variable Groups in Azure Devops pipeline - azure-devops

Question

[![enter image description here][1]][1]Using Azure devops pipelines and want to store my secr[![enter image description here][2]][2]ets in Azure KeyVault? How to use a variable group with keyvault integration to retrieve my secrets values and use within my DevOps pipeline.
Below is Yaml script which has hardcoded values of service connection and storage account name,key for to store the terraform tfstate files, my question is how we can pass it as secure type so that no one can see my data. I have created AzureKeyVault and it is linked to provided service connection.
trigger: none
#########################
# Declare Build Agents:-
#########################
pool:
vmImage: ubuntu-latest
######################
#DECLARE PARAMETERS:-
######################
parameters:
- name: ResourceGroup
displayName: Please Provide the Resource Group Name:-
type: object
default: <Please provide the required Name>
- name: Region
displayName: Please Provide the Region Name:-
type: object
default: <Please provide the required Name>
- name: sqlserver
displayName: Please Provide the sqlserver Name:-
type: object
default: <Please provide the required Name>
######################
#DECLARE VARIABLES:-
######################
variables:
TF_VAR_ResourceGroup: ${{ parameters.ResourceGroup }}
TF_VAR_REGION: ${{ parameters.Region }}
TF_VAR_SQLSERVER_NAME: ${{ parameters.sqlserver }}
###################
# Declare Stages:-
###################
stages:
- stage: tfvalidate
jobs:
- job: validate
continueOnError: false
steps:
- task: TerraformInstaller#0
inputs:
terraformVersion: 'latest'
- task: TerraformTaskV3#3
displayName: init
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: $(serviceconnection) # it should pick from my azureKeyvault
backendAzureRmResourceGroupName: 'AzureDevops' # it should pick from my azureKeyvault
backendAzureRmStorageAccountName: 'azuredevopsdev' # it should pick from my azureKeyvault
backendAzureRmContainerName: 'tfstatedev' # it should pick from my azureKeyvault
backendAzureRmKey: 'terrafrom.tfstate' # it should pick from my azureKeyvault
- task: TerraformTaskV3#3
displayName: validate
inputs:
provider: 'azurerm'
command: 'validate'
- stage: tfdeploy
condition: succeeded('tfvalidate')
dependsOn: tfvalidate
jobs:
- job: apply
steps:
- task: TerraformInstaller#0
inputs:
terraformVersion: 'latest'
- task: TerraformTaskV3#3
displayName: init
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: 'dev-Automationaccount_OIDC'
backendAzureRmResourceGroupName: 'AzureDevops'
backendAzureRmStorageAccountName: 'azuredevopsdev'
backendAzureRmContainerName: 'tfstatedev'
backendAzureRmKey: 'terrafrom.tfstate'
- task: TerraformTaskV3#3
displayName: validate
inputs:
provider: 'azurerm'
command: 'validate'
- task: TerraformTaskV3#3
displayName: plan
inputs:
provider: 'azurerm'
command: 'plan'
environmentServiceNameAzureRM: 'dev-Automationaccount_OIDC'
- task: TerraformTaskV3#3
displayName: apply
inputs:
provider: 'azurerm'
command: 'apply'
commandOptions: '-auto-approve'
environmentServiceNameAzureRM: 'dev-Automationaccount_OIDC'
backendAzureRmContainerName: 'tfstatedev'
backendAzureRmKey: 'terrafrom.tfstate'
- task: TerraformTaskV3#3
displayName: validate
inputs:
provider: 'azurerm'
command: 'validate'
- stage: tfdeploy
condition: succeeded('tfvalidate')
dependsOn: tfvalidate
jobs:
- job: apply
steps:
- task: TerraformInstaller#0
inputs:
terraformVersion: 'latest'
- task: TerraformTaskV3#3
displayName: init
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: 'dev-Automationaccount_OIDC'
backendAzureRmResourceGroupName: 'AzureDevops'
backendAzureRmStorageAccountName: 'azuredevopsdev'
backendAzureRmContainerName: 'tfstatedev'
backendAzureRmKey: 'terrafrom.tfstate'
- task: TerraformTaskV3#3
displayName: validate
inputs:
provider: 'azurerm'
command: 'validate'
- task: TerraformTaskV3#3
displayName: plan
inputs:
provider: 'azurerm'
command: 'plan'
environmentServiceNameAzureRM: 'dev-Automationaccount_OIDC'
- task: TerraformTaskV3#3
displayName: apply
inputs:
provider: 'azurerm'
command: 'apply'
commandOptions: '-auto-approve'
environmentServiceNameAzureRM: 'dev-Automationaccount_OIDC'
```
[1]: https://i.stack.imgur.com/W0lU5.png
[2]: https://i.stack.imgur.com/cEm6h.png

Answer 1

You have to assign the correct role to the service principal on the KV:
find the linked service connection
project settings > service connections > manage service principal (opens azure portal) > copy the title of the SP
go to your KV and assign permission to this SP
Access Control (IAM) > +Add > Add role assignment > contributor > User, group or SP > select members: paste the copied title of your SP >> select+review&assign
go to Azure Dev/Ops and connect your KV + add secrets you want to use
Test inside YAML:
YAML config:
And inside the logs of the pipeline:
--> The secret is not exposed in the pipeline
note: make sure that access policies for the SP and the specific KV are set correctly, see: https://learn.microsoft.com/bs-latn-ba/azure/key-vault/general/assign-access-policy?tabs=azure-portal
note: when you run the pipeline you might have to give explicit permissions for the pipeline to access the library group, this only needs to be done once.
Alternative method:
Since you have already set up a connection you can also use an AzureCLI task to get credentials and persistent them through your pipeline,
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-cli-v2?view=azure-pipelines
Set addSpnToEnvironment to true, and use something like this to set the value(s): echo "##vso[task.setvariable variable=SECRET;issecret=true;isoutput=true]$Key"
Hope this helps

Answer 2

Here is another approach I have tried to reproduce in my environment as below and got positive results.
Step 1: Create an Azure key vault and add secrets.
Step 2: Create a service principal and note down the app Id, tenant id and password for later use.
Step 3: Add service principle to key vault access policies as shown below.
• Navigate to the Access policies in the key vault and click on create.
• Select the required permissions and click on next.
Enter the object id which we created in step 2 and click Review+create.
Step 4: Create a service connection in the azure devops with the details created in step 2.
Fill these details and click create.
Step 5: Navigate to Pipelines > Library and click on Variable group
Add the required keys, secretes and certificates under variables.
Step 6: Refer the variable group in the Azure pipelines as below.