I have azure key vault service in which we are maintaining secrets.
I have to deploy APIM service using ARM JOB in Azure devops release pipeline so I have added this job and added configured template.json and parameter.json and how to pass key vault as over ride parameter to ARM job in over ride parameters?. I tried with below option
I have added keyvault job/varaible group in azure pipelines then in over ride params i called $(keyvaultname/secretname) then saved it and ran the pipeline but i am geeting below issue
enter image description here
Please go to Pipelines -> Library -> create a variable group which contain the keyvalut.
Link the Variable group in your pipeline, make sure the variable of secret is listed.
In the ARM task, overwrite the parameters with "$(var)" name.
PFA .
I have created Variable group and then came back to release pipeline arm job then in the override parameter .
Arm job over ride parameter
Variable group
Related
I have an ADF pipeline. The task is to productionize the pipeline. I am using azure devops CI/CD (classic). I am following this documentation
https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters
I have to move the pipeline to test and prod. Thereforem, there are many parameters that are parametrized but few parameters like sql user_name, secret_name are not parametrized.
I want to edit the ARM template and add custom parameter so that I do not have to edit the template.json and paramete-template.json and push them again to repo. The edit option in adf allows to create custom params and therefore generate these in ARM templates when exported.
I have the parameter in the template.
The parameter secretName doesnt appear in ARM template in CD flow
Which mode you are using to configure the parameters?
ARM parameter configuration is only enabled in "GIT mode". Currently it is disabled in "live mode" or "Data Factory" mode.
So, as per above official statement from Microsoft, you should be using Git repository.
Also, take note - Creating a custom Resource Manager parameter configuration creates a file named arm-template-parameters-definition.json in the root folder of your git branch. You must use that exact file name.
There are other multiple ways which you can try to pass secrets in ARM template. Refer this article from devkimchi.com.
After lot of tries and understanding the credential structure the ADF follows for different LinkedServices, we have found that to parametrize a custom nested argument, we have to specify the argument in a nested form. The parameter configuration needs to be edited like this:
For example, the secret name for SQL linked service (using password – connected to azurekeyvault) needs to be like this:
"password": {
"secretName": "="
}
But for the secret type (from azure keyvault) for storage linked service, it has to be like this:
"servicePrincipalCredential": {
"secretName": "="
}
And then these args can be passed directly from azure keyvault if variable groups are connected to keyvault. This solves the problem we were facing.
I have created a Docker Compose in my pipeline and Azure created the code. The azureSubscription and the azureContainerRegistry connection are very clear.
I tried to replace them with variable from the Library but when the pipeline starts I immediately get an error.
There was a resource authorization issue: "The pipeline is not valid. Job Build: Step DockerCompose1 input azureSubscriptionEndpoint references service connection $(AzureSubscription) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Build: Step DockerCompose2 input azureSubscriptionEndpoint references service connection $(AzureSubscription) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Basically, Azure DevOps can't replace the variable with the value for those particular parameters. I don't want to send around those configurations for obviuos reasons.
I saw some old posts where Microsoft said this was an issue in DevOps. Is this issue still there? Is there any way to move those values in the Libray or a variables?
This is still an issue. It have to be an literal or variables defined in YAML. It cannot be variable provied via variable group for instance. Please check these topics:
How to parametrize azureSubscription in azure devops template task
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
I have a service connection created for Azure Repos in a separate Azure DevOps org using a PAT.
I would like to be able to access that PAT from a PowerShell script in my pipeline yml since that same PAT has access to a nuget feed in the external org as well and I want to restore a package using that PAT.
Is there any way to do this? Or do I need to create a variable with that same PAT and store the token twice?
Is there any way to do this? Or do I need to create a variable with that same PAT and store the token twice?
We can't access value of specific PAT via reading the service connection which uses the PAT for authorization. As the only way we use service connection is to copy the connection name/ID into pipeline as a value.
What's more, PAT can only be obtained and saved directly when you create it. After that you can only get PAT from where you saved, and you cannot find it from anywhere in Azure DevOps.
If you want to use that PAT in pipeline, you can save it in a variable group as a secret variable.
Here is the detailed steps:
1.Go to Pipelines -> Library -> Create a variable group -> Add a variable and change its type to secret -> Save.
2.Add the variable group to variables in YAML.
variables:
- group: {variable group}
Then you can get the value of PAT by PowerShell scripts.
I want to use azure key vault secrets in azure devops release pipeline.could someone help me with powershell script in which I can define these variables and pass it in pipeline.
Powershell script that can be added to a release pipeline for fetching variables from key vault
You could configure a Variable Group to connect to an Azure Key Vault:
Go to "Pipelines" and then "Library" and "Add variable group".
Link secrets from an Azure key vault
Configure a Pipeline to make use of the new Variable Group
Then, whatever variables you have in your variable group can be accessed from your build pipe as $(VariableNameHere) including the key vault.
You could check the great document for some more details.
Hope this helps.
I am executing an ARM template which create an Azure Kubernetes Service and other resources from an Azure resource group deployment task.
My ARM template have as a parameters servicePrincipalClientId and servicePrincipalClientSecret sensitive data, which are used to create the Azure Kubernetes cluster, just right here. (This link is my complete ARM template)
So, that I am doing is the following:
I've created the servicePrincipalClientId and servicePrincipalClientSecret as pipeline variables
I previously create the service principal and its servicePrincipalClientId and servicePrincipalClientSecret data, I am using them to create a service connection in order to connect to Azure cloud of this way:
I have a PowerShell task to convert the servicePrincipalClientId and servicePrincipalClientSecret variable values as a secure strings of this way:
$env:secretServicePrincipalClientId = ConvertTo-SecureString '$($env:servicePrincipalClientId)' -AsPlainText -Force
$env:secretServicePrincipalClientSecret = ConvertTo-SecureString '$($env:servicePrincipalClientSecret)' -AsPlainText -Force
To Deploy the resources defined in the ARM template I've created an Azure resource group Deployment task with the following options:
I am using the service connection created to interact with my subscription.
Action to be performed: Create or Update resource group
Template, the ARM template path referenced above.
Override template parameters
I included here all the ARM template parameters values and I want to emphasize here the way how I am referencing the servicePrincipalClientId and servicePrincipalClientSecret variable values:
I am referencing here the secretServicePrincipalClientId and secretServicePrincipalClientSecret variables that I used to convert servicePrincipalClientId and servicePrincipalClientSecret variable values as a secure strings above in my first azure devops task
-servicePrincipalClientId $($secretServicePrincipalClientId)
-servicePrincipalClientSecret $($secretServicePrincipalClientSecret)
.
.
-serviceCidr "100.0.0.0/16"
-dnsServiceIP "100.0.0.10"
-dockerBridgeCidr "172.17.0.1/16"
.
.
So, when I execute the release pipeline, I got this error in the Azure resource group deployment task
2019-10-26T20:05:13.3246017Z The detected encoding for file 'd:\a\r1\a\Project\Deployments\ARMTemplates\Infrastructure\AzResourceGroupDeploymentApproach\testing.json' is 'utf-8'
2019-10-26T20:05:13.3410693Z Starting Deployment.
2019-10-26T20:05:13.3412081Z Deployment name is AzureDevOpsDeployment
2019-10-26T20:05:18.1729784Z There were errors in your deployment. Error code: InvalidTemplateDeployment.
2019-10-26T20:05:18.1730624Z ##[error]The template deployment 'AzureDevOpsDeployment' is not valid according to the validation procedure. The tracking id is 'xxxxxxx'. See inner errors for details.
2019-10-26T20:05:18.1731223Z ##[error]Details:
2019-10-26T20:05:18.1732062Z ##[error]ServicePrincipalNotFound: Provisioning of resource(s) for container service KubernetesCluster-aks in resource group testing failed. Message: {
"code": "ServicePrincipalNotFound",
"message": "Service principal clientID: $($secretServicePrincipalClientId) not found in Active Directory tenant ***, Please see https://aka.ms/aks-sp-help for more details."
}. Details:
2019-10-26T20:05:18.1733305Z ##[error]Task failed while creating or updating the template deployment.
2019-10-26T20:05:18.1765718Z ##[section]Finishing: Azure Deployment:Create Or Update Resource Group action on testing
Looks like the service principal that I am using to connect to Azure cloud does not exist, but that's not true. That service principal exist.
If I include directly in plain text in the task the servicePrincipalClientId and servicePrincipalClientSecret values
-servicePrincipalClientId <servicePrincipalClientId-value>
-servicePrincipalClientSecret <servicePrincipalClientSecret-value>
The Azure resource group task works and the resources in the ARM template are deployed in Azure cloud from Azure DevOps.
What the problem is?
According to this link AKS need a service principal to be created.
Also when we create an Azure Kubernetes Service using az cli the service principal is created automatically.
The same case happens when we create an Azure Kubernetes Service from Azure portal.
So I am creating an Azure Kubernetes Service from Azure Devops executing an ARM template via resource group deployment task using an existing Service principal credentials in the task and in the service connection.
Why my service principal is not recognized?
I try this option to troubleshoot and solve the problem, but I afraid that the problem is not the service principal itself, instead of it, I think I would need to reference the
- servicePrincipalClientId $($secretServicePrincipalClientId) and
-servicePrincipalClientSecret $($secretServicePrincipalClientSecret)
of an special way.
How can I do that?
If someone can point me in the right direction I would appreciate
I decided making easier the execution of the ARM template, removing the secure task where I was converting the values to secure-strings.
So, at the end, I have the pipeline variables defined of a normal way, having administratorLogin as a non encrypted variable there:
And in the Azure Resource Group deployment task I am doing:
Template: I select my ARM template
Deployment mode: Complete
Complete mode deletes resources that are not in your template.
[Warning] This action will delete all the existing resources in the resource group that are not specified in the template.
I chosen this mode, in order to have the idea of all the resources created in the platform came from the ARM template, in order to get a log of the resources or changes made to the infrastructure from the ARM template. Trying to apply the Infrastructure as code approach
Override parameters
I decided include all the parameters values that I got from the ARM template, including the servicePrincipalClientId and servicePrincipalClientSecret variables of this way:
-administratorLogin "my-username"
-administratorLoginPassword $(administratorLoginPassword)
-environmentName "dev"
-location "West Europe"
-servicePrincipalClientId $(servicePrincipalClientId)
-servicePrincipalClientSecret $(servicePrincipalClientSecret)
.
.
-serviceCidr "100.0.0.0/16"
-dnsServiceIP "100.0.0.10"
-dockerBridgeCidr "172.17.0.1/16"
.
.
And the final execution result was:
2019-10-27T15:58:18.3523334Z ##[section]Starting: Azure Deployment:Create Or Update Resource Group action on sentia-assessment-testing
2019-10-27T15:58:18.3886841Z ==============================================================================
2019-10-27T15:58:18.3887055Z Task : Azure resource group deployment
2019-10-27T15:58:18.3887210Z Description : Deploy an Azure Resource Manager (ARM) template to a resource group and manage virtual machines
2019-10-27T15:58:18.3887334Z Version : 2.157.4
2019-10-27T15:58:18.3887438Z Author : Microsoft Corporation
2019-10-27T15:58:18.3887559Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment
2019-10-27T15:58:18.3887710Z ==============================================================================
### WE CAN SEE HERE THAT AZ RESOURCE GROUP TASK CREATE THE RESOURCE GROUP
### IF IT DOES NOT EXIST.
2019-10-27T15:58:19.3677672Z Checking if the following resource group exists: resource-group.
2019-10-27T15:58:19.6898000Z Resource group exists: false.
2019-10-27T15:58:19.6900439Z Creating resource Group: resource-group
2019-10-27T15:58:20.1586233Z Resource Group created successfully.
2019-10-27T15:58:20.1589727Z Creating deployment parameters.
############### THIS IS THE ARM TEMPLATE EXECUTED ##################
2019-10-27T15:58:20.1681560Z The detected encoding for file 'd:\a\r1\a\Github\Deployments\ARMTemplates\Infrastructure\AzResourceGroupDeploymentApproach\testing.json' is 'utf-8'
############### THIS IS THE ARM TEMPLATE EXECUTED END ##################
2019-10-27T15:58:20.1864884Z Starting Deployment.
2019-10-27T15:58:20.1866605Z Deployment name is AzureDevOpsDeployment_91
2019-10-27T16:13:20.7707558Z Successfully deployed the template.
2019-10-27T16:13:20.7834983Z ##[section]Finishing: Azure Deployment:Create Or Update Resource Group action on resource-group
Is this a good and simpler approach, but maybe the ideal scenario could be deal the variables mentioned above as a secure-strings and sharing their values across the different tasks in the release pipeline.