I have an ADF pipeline. The task is to productionize the pipeline. I am using azure devops CI/CD (classic). I am following this documentation
https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters
I have to move the pipeline to test and prod. Thereforem, there are many parameters that are parametrized but few parameters like sql user_name, secret_name are not parametrized.
I want to edit the ARM template and add custom parameter so that I do not have to edit the template.json and paramete-template.json and push them again to repo. The edit option in adf allows to create custom params and therefore generate these in ARM templates when exported.
I have the parameter in the template.
The parameter secretName doesnt appear in ARM template in CD flow
Which mode you are using to configure the parameters?
ARM parameter configuration is only enabled in "GIT mode". Currently it is disabled in "live mode" or "Data Factory" mode.
So, as per above official statement from Microsoft, you should be using Git repository.
Also, take note - Creating a custom Resource Manager parameter configuration creates a file named arm-template-parameters-definition.json in the root folder of your git branch. You must use that exact file name.
There are other multiple ways which you can try to pass secrets in ARM template. Refer this article from devkimchi.com.
After lot of tries and understanding the credential structure the ADF follows for different LinkedServices, we have found that to parametrize a custom nested argument, we have to specify the argument in a nested form. The parameter configuration needs to be edited like this:
For example, the secret name for SQL linked service (using password – connected to azurekeyvault) needs to be like this:
"password": {
"secretName": "="
}
But for the secret type (from azure keyvault) for storage linked service, it has to be like this:
"servicePrincipalCredential": {
"secretName": "="
}
And then these args can be passed directly from azure keyvault if variable groups are connected to keyvault. This solves the problem we were facing.
Related
I have azure key vault service in which we are maintaining secrets.
I have to deploy APIM service using ARM JOB in Azure devops release pipeline so I have added this job and added configured template.json and parameter.json and how to pass key vault as over ride parameter to ARM job in over ride parameters?. I tried with below option
I have added keyvault job/varaible group in azure pipelines then in over ride params i called $(keyvaultname/secretname) then saved it and ran the pipeline but i am geeting below issue
enter image description here
Please go to Pipelines -> Library -> create a variable group which contain the keyvalut.
Link the Variable group in your pipeline, make sure the variable of secret is listed.
In the ARM task, overwrite the parameters with "$(var)" name.
PFA .
I have created Variable group and then came back to release pipeline arm job then in the override parameter .
Arm job over ride parameter
Variable group
We use Synapse git Integration to deploy artifacts such as linked services generated by a Data Warehouse automation tool (JSON files)
It is different then deploying ARM template in ADF.
We created one Azure Key Vault (AKV) per environment so we do have an Azure Key Vault LinkedService in each environment and the linked services has the same name. But each AKV as his own URL so we need to change the URL in the deployed linked services during the CI/CD process.
I read this https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-deployment#use-custom-parameters-of-the-workspace-template
I think I need to create a template to change "Microsoft.Synapse/workspaces/linkedServices"
But I didn't find any example on how to modify the KV url parameters.
Here is the linked services I want to modify,https://myKeyVaultDev.vault.azure.net as to be changed when deploying
{
"name": "myKeyVault",
"properties": {
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://myKeyVaultDev.vault.azure.net"
}
}
}
Not much familiar with the ci/cd and azure devOps yet, but still I need to do it...
I have done this using Azure Devops. When you create the Release pipeline within Azure Devops, one of the options is to "override parameters". at this point you can specify the name of the keyvault and the corresponding value. The corresponding value is configured in a pipeline variable set - which itself can come from the same keyvault.
You don't need to create the template. Synapse already does that and stores it in the publish branch (“workspace_publish”). If you look in that branch you will see the template along with the available parameters that you can override.
More info is available here:
https://www.drware.com/how-to-use-ci-cd-integration-to-automate-the-deploy-of-a-synapse-workspace-to-multiple-environments/
https://techcommunity.microsoft.com/t5/data-architecture-blog/ci-cd-in-azure-synapse-analytics-part-1/ba-p/1964172
From the Azure Key Vault side of things, I believe you're right - you have change the Linked Services section within the template to point to the correct Key Vault base URL.
Azure Key Vault linked service
I don't know if you still are looking for the solution.
In order to parametrize linked service property and specially AKV reference, I think you should modify the template-parameters-definition.json, and add the following section:
"Microsoft.Synapse/workspaces/linkedServices":
{ "*":
{ "properties":
{ "typeProperties":
{ "baseUrl": "|:-connectionString:secureString" }
}
}
}
This will create a parameter for each linked service. The next step is to overrideParameters on SynapseWorkspaceDeployment task on Azure Devops.
I hope you are all well!
I need to ask a question about azure devops, I already read the documentation, but I did not find a way to resolve these doubts
I have the X, Y and Z projects and in the X project create in the Pipeline >> Libray a group of variables called general that I would like to be shared with the Y and Z pipelines, when configuring this group I enabled the option "Allow Access to all pipelines" .
In the YAML of the Y and Z pipelines I made the following configuration:
**variables:
group: general**
When running the pipeline he returns an authorization request and even clicking authorize it, according to the print below:
Print Authorization error
There was a resource authorization issue:
"An error occurred while loading the YAML build pipeline.
Variable group was not found or is not authorized for use.
For authorization details, refer to https://aka.ms/yamlauthz."
My question is, is there any way to share variables between pipelines of different purposes, if there is, can you please send me some documentation that can help me to configure this?
Testing in my side and I can reproduce this issue, setting the Allow access to all pipelines option will enable the variable group to be accessible for any pipelines in current project not across projects. It should be a known limitation and this is the feedback ticket. You can vote and follow this ticket. You can also create a new suggestion ticket here. The product group will review these tickets regularly, and consider to take it as roadmap.
In addition, as a workaround, you could try to add these shared variables to Azure Key Vault, and then use Azure Key Vault task to fetch the latest values of all or a subset of secrets from the vault, and set them as variables that can be used in subsequent tasks of a pipeline. See: Use Azure Key Vault secrets in Azure Pipelines for details.
This link gives a good overview of using variables in Azure DevOps pipelines:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables
In the most common case, you set the variables and use them within the
YAML file.
In the YAML file, you can set a variable at various scopes:
At the root level, to make it available to all jobs in the pipeline.
At the stage level, to make it available only to a specific stage.
At the job level, to make it available only to a specific job.
Q: It sounds like you would like to share variables between YAML pipeline roots, correct? You're currently trying to use variable groups to accomplish this, correct?
Another solution would be to have your pipeline read from a "shared file". For example:
Is there a way to read file from Azure DevOps YAML?
... you can use any scripting language you like to parse the file and
"spit out" whatever you need as a build variable and consume it later
on. here's what I've been doing:
- script: echo "##vso[task.setvariable variable=dp]$(cat $(Build.Repository.LocalPath)/deployment/dp)"
- script: az group delete -n $(dp)-k8s -y --no-wait
In other words:
Create a file with the variables you wish to "export". It can be any script format: Powershell, bash, etc., etc.
Modify your pipeline(s) to read the file and "import" the variable definitions at runtime.
There are many ways to do this. The SO link above is just an example.
Am trying to export ARM template for a resourcegroup where in i have Azure data factory, but when using "Export-AzureRmResourceGroup" it throws a warning as below and none of ADF schema is downloaded in that template. is there a way to download ADF template using powershell??
Export-AzureRmResourceGroup -ResourceGroupName ****************
WARNING: ExportTemplateCompletedWithErrors : Export template operation completed with errors. Some resources were not exported. Please see details for more information.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type 'Microsoft.Databricks/workspaces' is not available. Resources of this type will not be exported to the template.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type '**Microsoft.DataFactory/factories**' is not available. Resources of this type will not be exported to the template.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type 'Microsoft.Portal/dashboards' is not available. Resources of this type will not be exported to the template.
As mentioned there, DataFactory template export is not supported yet.
BUT, there is a solution which we've used in our CI/CD.
First, we've created a dummy generic template for the datafactory (see the steps below), and then used either ARM template exported from the datafactory template or adf_publish (see the notes below) branch to update and finalize the datafactory.
To create a dummy datafactory template:
On the portal try to create a dummy datafactory, and click on "Automation options" (instead of "Create")
This will create an empty datafactory ARM template, which then you can use for your pipelines. Now just click on the "Download" button and store the dummy template somewhere.
Afterwards you can use that template to create a new dummy datafactory if it doesn't exist, and then update it with the real ARM templates provided by either the "Export ARM Template" button inside the datafactory, or the adf_publ
Notes:
1. We came to this solution as the ARM templates which were provided* by the azure portal, were not including the Datafactory resource itself, because of which deployment to a new Resource Group was FAILING.
So, we had to create a dummy DataFactory first, and then update it with actual DataFactory template.
2. By saying "provided* by the azure portal" we mean the ARM templates which are provided when you open the DF and click on "ARM Template" > "Export ARM template", or the one published in adf_publish branch
3. For creating CI/CD we used the other steps mentioned there
This is because those "ResourceTypeSchemas" are not available to use in ARM templates. It doesn't matter if you go into the portal and try to get the resource template. It will show the same error there as well.
You get exactly the same thing when you are trying to do via Azure directly:
However there's a template defined by Microsoft for almost any object, which you can access in here, so if you're starting from scratch, this can be a good starting point to create your template.
If you already have something defined in your existing data factory, then you just need to export ARM template directly from your data factory and merge extracted template with the one from Microsoft documentation.
For this to work you need to include two nodes, because the yare not currently included in this export functionality: for generating the data factory itself
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[parameters('location')]",
"identity": {
"type": "SystemAssigned"
}
}
and you will also probably need a template for storage account
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-08-01",
"name": "[parameters('storageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard_LRS"
},
"kind": "StorageV2"
}
If all you need is the template, and if you are trying to download via PowerShell, then I assume you have access to that RG.So navigate to Azure Portal->Resource Group ( that has the ADF or any resource that you need. From the left nav bar , you'll find this Automation Script .
Click on it and the template will appear. From here, you can download the template directly
Context:
I create a storage account via an ARM template with the following output (among others):
"storageAccountKey": {
"type": "securestring",
"value": "[listKeys(parameters('storageAccountName'), '2018-02-01').keys[0].value]"
}
In a next step I convert the deployment output to Azure DevOps environments variables to make them accessible for a subsequent PowerShell task.
Problem:
With an output of type 'string' this works fine, but not with the type 'securestring' as the value is not made available in the deployment output string.
Question:
How can I make an output of type 'securestring' of an ARM Deployment available as secure Azure DevOps environment variable?
Thanks
Edit:
I could of course query the account key directly in the PowerShell task but I would like to understand how to make a securestring output available.
I dont think you can. securestring type is always omitted. What you should do is use powershell to get the key and use it in the script