CI/CD ADF Synapse - Modify URL in Key Vault Linked service - azure-devops

We use Synapse git Integration to deploy artifacts such as linked services generated by a Data Warehouse automation tool (JSON files)
It is different then deploying ARM template in ADF.
We created one Azure Key Vault (AKV) per environment so we do have an Azure Key Vault LinkedService in each environment and the linked services has the same name. But each AKV as his own URL so we need to change the URL in the deployed linked services during the CI/CD process.
I read this https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-deployment#use-custom-parameters-of-the-workspace-template
I think I need to create a template to change "Microsoft.Synapse/workspaces/linkedServices"
But I didn't find any example on how to modify the KV url parameters.
Here is the linked services I want to modify,https://myKeyVaultDev.vault.azure.net as to be changed when deploying
{
"name": "myKeyVault",
"properties": {
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://myKeyVaultDev.vault.azure.net"
}
}
}
Not much familiar with the ci/cd and azure devOps yet, but still I need to do it...

I have done this using Azure Devops. When you create the Release pipeline within Azure Devops, one of the options is to "override parameters". at this point you can specify the name of the keyvault and the corresponding value. The corresponding value is configured in a pipeline variable set - which itself can come from the same keyvault.
You don't need to create the template. Synapse already does that and stores it in the publish branch (“workspace_publish”). If you look in that branch you will see the template along with the available parameters that you can override.
More info is available here:
https://www.drware.com/how-to-use-ci-cd-integration-to-automate-the-deploy-of-a-synapse-workspace-to-multiple-environments/
https://techcommunity.microsoft.com/t5/data-architecture-blog/ci-cd-in-azure-synapse-analytics-part-1/ba-p/1964172

From the Azure Key Vault side of things, I believe you're right - you have change the Linked Services section within the template to point to the correct Key Vault base URL.
Azure Key Vault linked service

I don't know if you still are looking for the solution.
In order to parametrize linked service property and specially AKV reference, I think you should modify the template-parameters-definition.json, and add the following section:
"Microsoft.Synapse/workspaces/linkedServices":
{ "*":
{ "properties":
{ "typeProperties":
{ "baseUrl": "|:-connectionString:secureString" }
}
}
}
This will create a parameter for each linked service. The next step is to overrideParameters on SynapseWorkspaceDeployment task on Azure Devops.

Related

Adding custom parameter to ADF ARM template

I have an ADF pipeline. The task is to productionize the pipeline. I am using azure devops CI/CD (classic). I am following this documentation
https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters
I have to move the pipeline to test and prod. Thereforem, there are many parameters that are parametrized but few parameters like sql user_name, secret_name are not parametrized.
I want to edit the ARM template and add custom parameter so that I do not have to edit the template.json and paramete-template.json and push them again to repo. The edit option in adf allows to create custom params and therefore generate these in ARM templates when exported.
I have the parameter in the template.
The parameter secretName doesnt appear in ARM template in CD flow
Which mode you are using to configure the parameters?
ARM parameter configuration is only enabled in "GIT mode". Currently it is disabled in "live mode" or "Data Factory" mode.
So, as per above official statement from Microsoft, you should be using Git repository.
Also, take note - Creating a custom Resource Manager parameter configuration creates a file named arm-template-parameters-definition.json in the root folder of your git branch. You must use that exact file name.
There are other multiple ways which you can try to pass secrets in ARM template. Refer this article from devkimchi.com.
After lot of tries and understanding the credential structure the ADF follows for different LinkedServices, we have found that to parametrize a custom nested argument, we have to specify the argument in a nested form. The parameter configuration needs to be edited like this:
For example, the secret name for SQL linked service (using password – connected to azurekeyvault) needs to be like this:
"password": {
"secretName": "="
}
But for the secret type (from azure keyvault) for storage linked service, it has to be like this:
"servicePrincipalCredential": {
"secretName": "="
}
And then these args can be passed directly from azure keyvault if variable groups are connected to keyvault. This solves the problem we were facing.

Allow Cloud Service ARM template deployment to a different subscription than the keyVault

Our KeyVault is in subscription 1 and we have multiple Cloud Services for multiple areas that we need deployed in different subscriptions. While working in Azdo I found out that I am unable to deploy CSES to a subscription that is different than the keyVault since the ARM template used for deployment is trying to access secrets from the keyvault.
Then, when I read this document https://learn.microsoft.com/en-us/azure/cloud-services-extended-support/deploy-prerequisite, it states that the "The key vault must be created in the same region and subscription as the cloud service".
Does anyone know of a way around this? It's imperative that we are able to deploy multiple Cloud Services (for different areas) in different Subscriptions and we only have one keyvault that stores all values used by the cloud services.
As mentioned in the Microsoft Documentation that you have shared , its not possible as its a prerequisite to create Key vault in the same subscription as the cloud Services.
In this Github Issue , it is possible to use secrets from one subscription in another subscription but using certificates is an limitation in ARM templates.
It is recommended by Azure to use for different Key vault for different environments for using Certificates .
Secrets can be referenced as parameters in the ARM template to used by Azure Services but certificate can't be referenced from another subscriptions otherwise you will get the below error :
{
"status": "Failed",
"error": {
"code": "InvalidParameter",
"target": "sourceVault.id",
"message": "The SubscriptionId:\"<id>\" of the request must match the SubscriptionId \"<sharedId>\" contained in the Key Vault Id."
}
}

Programmatically configure Azure DevOps project settings

I'm looking for a way to automate/script Azure DevOps project creation. I know that I can use VSTeam PowerShell module to create a project, but it does not provide a way to programmatically set project settings (the ones on the screenshot below). Is there a way to do that with PowerShell or AZ CLI? In particular I'm looking for a way to control/change these two settings:
Limit job authorization scope to current project for non-release pipelines
Limit job authorization scope to referenced Azure DevOps repositories
You can try to use Rest API:
How to use PowerShell with rest API: Modify Azure Devops Test Case Parameters Through REST API
Rest API Methods: General Settings - Get; General Settings - Update
To create a project you will need to send a POST to https://dev.azure.com/<devopsOrgName>/_apis/projects?api-version=5.0-preview.3
with a body like the below. To get the "template type" value used in the body, you will need to set a GET to https://dev.azure.com/<devopsOrgName>/_apis/process/processes?api-version=5.0-preview.1
` {
"description": "Tailspin Toys",
"name": "Tailspin_TOYS",
"capabilities": {
"versioncontrol": {
"sourceControlType": "Git"
},
"processTemplate": {
"templateTypeId": "24268e03-7eed-4ac0-a178-700881565b99"
}
}
}`
To change specific settings, that you can't find in the documentation, I would suggest using the developer tools in your browser (f12). Watch the network tab and you should see the requests going when you change your setting.
In this instance, I can see the below request when I try these settings
URL: https://dev.azure.com/<devopsOrgName>/_apis/Contribution/HierarchyQuery
Type: POST
Body:
{ "contributionIds":["ms.vss-build-web.pipelines-general-settings-data-provider"],"dataProviderContext":{"properties":{"enforceJobAuthScopeForReleases":"false","sourcePage":{"url":"https://dev.azure.com/<devopsOrgName>/<projectName>/_settings/settings","routeId":"ms.vss-admin-web.project-admin-hub-route","routeValues":{"project":"<projectName>","adminPivot":"settings","controller":"ContributedPage","action":"Execute","serviceHost":"495d404e-cdeb-496a-8496-fccf9df3d9fa (<devopsOrgName>)"}}}} }
Azure DevOps REST isn't the most documented, but when used with your browser develop tools, I haven't found something I couldn't set.

azure devops for ADF and databricks

I am trying to create an azure ci/cd pipeline for my azure data factory in which I have used databricks notebook. Pipeline got created successfully with the ARM template for ADF but I am not able to see any override parameter for databricks workspace URL, that's why i got the same databricks URL in my dev and prod environment.
Can anyone help me to set databricks workspace URL for Dev and prod dynamically?
For anyone facing the same challenge, I just added this code to my arm-template-parameters-definition.json:
"AzureDatabricks": {
"properties": {
"typeProperties": {
"domain": "=",
"existingClusterId":"=",
"accessToken": {
"secretName": "="
}
}
}
}
It will make the parameters explicit to be overriden.
If you are using the default template provided by Microsoft, I inserted the code just under the "ODBC" section factories/linkedServices.
you need to create a token in qa and prod and use that in ur ci cd pipeline..tokens are unique all
over and automatically linked to respective service

Unable to export 'Microsoft.DataFactory/factories' schema using powershell

Am trying to export ARM template for a resourcegroup where in i have Azure data factory, but when using "Export-AzureRmResourceGroup" it throws a warning as below and none of ADF schema is downloaded in that template. is there a way to download ADF template using powershell??
Export-AzureRmResourceGroup -ResourceGroupName ****************
WARNING: ExportTemplateCompletedWithErrors : Export template operation completed with errors. Some resources were not exported. Please see details for more information.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type 'Microsoft.Databricks/workspaces' is not available. Resources of this type will not be exported to the template.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type '**Microsoft.DataFactory/factories**' is not available. Resources of this type will not be exported to the template.
WARNING: ResourceTypeSchemaNotFound : The schema of resource type 'Microsoft.Portal/dashboards' is not available. Resources of this type will not be exported to the template.
As mentioned there, DataFactory template export is not supported yet.
BUT, there is a solution which we've used in our CI/CD.
First, we've created a dummy generic template for the datafactory (see the steps below), and then used either ARM template exported from the datafactory template or adf_publish (see the notes below) branch to update and finalize the datafactory.
To create a dummy datafactory template:
On the portal try to create a dummy datafactory, and click on "Automation options" (instead of "Create")
This will create an empty datafactory ARM template, which then you can use for your pipelines. Now just click on the "Download" button and store the dummy template somewhere.
Afterwards you can use that template to create a new dummy datafactory if it doesn't exist, and then update it with the real ARM templates provided by either the "Export ARM Template" button inside the datafactory, or the adf_publ
Notes:
1. We came to this solution as the ARM templates which were provided* by the azure portal, were not including the Datafactory resource itself, because of which deployment to a new Resource Group was FAILING.
So, we had to create a dummy DataFactory first, and then update it with actual DataFactory template.
2. By saying "provided* by the azure portal" we mean the ARM templates which are provided when you open the DF and click on "ARM Template" > "Export ARM template", or the one published in adf_publish branch
3. For creating CI/CD we used the other steps mentioned there
This is because those "ResourceTypeSchemas" are not available to use in ARM templates. It doesn't matter if you go into the portal and try to get the resource template. It will show the same error there as well.
You get exactly the same thing when you are trying to do via Azure directly:
However there's a template defined by Microsoft for almost any object, which you can access in here, so if you're starting from scratch, this can be a good starting point to create your template.
If you already have something defined in your existing data factory, then you just need to export ARM template directly from your data factory and merge extracted template with the one from Microsoft documentation.
For this to work you need to include two nodes, because the yare not currently included in this export functionality: for generating the data factory itself
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[parameters('location')]",
"identity": {
"type": "SystemAssigned"
}
}
and you will also probably need a template for storage account
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-08-01",
"name": "[parameters('storageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard_LRS"
},
"kind": "StorageV2"
}
If all you need is the template, and if you are trying to download via PowerShell, then I assume you have access to that RG.So navigate to Azure Portal->Resource Group ( that has the ADF or any resource that you need. From the left nav bar , you'll find this Automation Script .
Click on it and the template will appear. From here, you can download the template directly