I want to deploy databricks from dev to UAT.
below is the format for azuredatabricks from ARM parameter template.
I am not sure what to parameterize and pass as access token is used to authorize.
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"value": "datafac1"
},
"AzureDatabricks1_accessToken": {
"value": ""
},
"ls_blob_connectionString": {
"value": ""
},
"ls_datalake_accountKey": {
"value": ""
},
"AzureDatabricks1_properties_typeProperties_existingClusterId": {
"value": "1024----cluster id"
},
"ls_datalake_properties_typeProperties_url": {
"value": "https://storgaename.dfs.core.windows.net/"
}
}
Pls guide me on how to deploy ADF_databricks to UAT
Related
Please, forgive the confusing tittle, if it is, but it does describe the problem I am having
So, I have a linked service in my Azure Datafactory. It is used for Azure SQL Database connect.
The Database name and user name are being taken from the parameters set in linked service itself. Here is a snippet of json config
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=myserver.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KeyVaultLink",
"type": "LinkedServiceReference"
},
"secretName": "DBPassword"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
This works fine when in debug in the Azure portal. However, when I get the ARM Template for the whole thing, during ARM Template deployment it asks for input Connection string for the linked service. If I go to the linked service definition, and look up its connection string it will come this way
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=dmsql.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}"
Then when I input it in the ARM Template deployment should I be replacing "#{linkedService().dbName}" and "#{linkedService().dbUserName}" with actual values at the spot when I am entring it ? I am confused because during the ARM Template deployment there are no separate fields for these parameters, and these (parameters specific to linked service itself) are not present as separate parameters in the ARM Template definition.
I created database in my azure portal
and enabled system assigned managed Identity for sql db.
Image for reference:
I created azure keywault and created secret.
Image for reference:
I have created new access policy for Azure data factory.
Image for reference:
I created Azure data factory and enabled system managed identity.
Image for reference:
I have created new parametrized linked service to connect with database with below parameters dbName and userName. I am taking database name and User name dynamically by using above parameters.
Image for reference:
Linked service is created successfully.
json format of my lined service:
{
"name": "SqlServer1",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
}
}
I exported the arm template of data factory.
This is my linked service in my ARM template:
"SqlServer1_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'SqlServer1'",
"defaultValue": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}"
},
"AzureKeyVault1_properties_typeProperties_baseUrl": {
"type": "string",
"defaultValue": "https://keysqlad.vault.azure.net/"
}
Image for reference:
I have got parameters dbName and userName in my ARM template description.
{
"name": "[concat(parameters('factoryName'), '/SqlServer1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "[parameters('SqlServer1_connectionString')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
}
Image for reference:
If you didn't get parameters in ARM template description copy the value of "connectionString" and modified what you needed to and left the parameters in place and added it to the "connectionString" override parameter in my Azure Release Pipeline, and it will work.
I'm trying to assign RBAC role "Reader" to a list of users in subscription level. When I try to do it for one user ARM template works. But for list of users it gives this error.
InvalidRoleAssignmentId: The role assignment ID 'u4ttmsjymtpe21' is not valid. The role assignment ID must be a GUID.
InvalidRoleAssignmentId: The role assignment ID 'u4ttmsjymtpe20' is not valid. The role assignment ID must be a GUID.
Here's the code I used:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"principalId": {
"type": "array"
},
"builtInRoleType": {
"type": "string"
},
"guidValue": {
"type": "string"
}
},
"variables": {
"unique_string":"[uniqueString(parameters('guidValue'))]",
"Reader": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', 'acdd72a7-3385-48ef-bd42-f606fba81ae7')]"
},
"resources": [
{
"type": "Microsoft.Authorization/roleAssignments",
"apiVersion": "2018-09-01-preview",
"name": "[concat(variables('unique_string'),copyIndex())]",
"copy": {
"name": "useridLoop",
"count": "[length(parameters('principalId'))]"
},
"properties": {
"roleDefinitionId": "[variables(parameters('builtInRoleType'))]",
"principalId": "[parameters('principalId')[copyIndex()]]"
}
}
]
}
This is the parameter file:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"principalId": {
"value": [
"b5*****c-****-****-****-c*****0*****",
"e******d-****-****-****-b*****b*****"
]
},
"builtInRoleType": {
"value": "Reader"
},
"guidValue": {
"value": "[newGuid()]"
}
}
}
InvalidRoleAssignmentId: The role assignment ID 'u4ttmsjymtpe21' is
not valid. The role assignment ID must be a GUID.
InvalidRoleAssignmentId: The role assignment ID 'u4ttmsjymtpe20' is
not valid. The role assignment ID must be a GUID.
Instead of passing the [newGuid()] as value to the parameter, you need to pass it as a default value to the parameter. Because of this, you were landed up with the above error message.
We have made some changes to the above-shared template & tried
deploying the modified template, we are able to assign the users as 'Reader' to the subscription.
Here is the Modified ARM template:
{
"$schema":"https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion":"1.0.0.0",
"parameters":{
"principalId":{
"type":"array"
},
"name":{
"type":"string",
"defaultValue":"[newGuid()]"
}
},
"variables":{
"Reader":"[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/', 'acdd72a7-3385-48ef-bd42-f606fba81ae7')]"
},
"resources":[
{
"type":"Microsoft.Authorization/roleAssignments",
"apiVersion":"2018-09-01-preview",
"name":"[guid(concat(parameters('name'),copyIndex()))]",
"copy":{
"name":"useridLoop",
"count":"[length(parameters('principalId'))]"
},
"properties":{
"roleDefinitionId":"[variables('Reader')]",
"principalId":"[parameters('principalId')[copyIndex()]]"
}
}
]
}
Here is the parameter.json file:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"principalId": {
"value": [
<object-id of the users>
]
}
}
}
Here is the sample output for reference:
Hey I am doing a CI/CD deployment for a logic app, I have a table storage where I store some data, I have two table storage for test and prod environment. I created a parameter called *table_storage_name" in ARM template :
"parameters": {
// ....
"connections_azuretables_1_externalid": {
"defaultValue": "/subscriptions/e5..../resourceGroups/myrg.../providers/Microsoft.Web/connections/azuretables-1",
"type": "String"
},
"table_storage_name": {
"defaultValue": "testdevops",
"type": "String"
}
}
The error comes from when I reference the parameter here in template.json file:
// ...
"Insert_Entity": {
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "ApiConnection",
"inputs": {
"body": {
"PartitionKey": "#body('Parse_JSON')?['name']",
"RowKey": "#body('Parse_JSON')?['last']"
},
"host": {
"connection": {
"name": "#parameters('$connections')['azuretables_1']['connectionId']"
}
},
"method": "post",
// problem occur after this line
"path": "/Tables/#{encodeURIComponent('[parameters('table_storage_name')]')}/entities"
}
}
but get this error:
InvalidTemplate: The template validation failed: 'The template action 'Insert_Entity' at line '1' and column '582' is not valid: "Unable to parse template language expression 'encodeURIComponent([parameters('table_storage_name')])': expected token 'Identifier' and actual 'LeftSquareBracket'.".'.
I tried escaping the quote with a backslash like: encodeURIComponent(\'[parameters('table_storage_name')]\') or encodeURIComponent('[parameters(''table_storage_name'')]') but all of them raise an error. How can I reference a paramter inside encodeURIComponent in an ARM template ?
As discussed in the comments. credits: #marone
"path": "/Tables/#{encodeURIComponent(parameters('table_storage_name'))}/entities"
Found the solution from this link https://platform.deloitte.com.au/articles/preparing-azure-logic-apps-for-cicd
but here are the steps to reference a parameter logic app:
create an ARM parameter table_storage_name_armparam in template.json, in order to use it's value to reference the value of the ARM parameter (yes it's confusing but follow along you'll understand):
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"table_storage_name_armparam": {
"type": "String"
}
},
"variables": {},
"resources": [
{
......
}
Now in the logic app parameter value (in the bottom of json file) create the logic app parameter table_storage_name and the value of this parameter will be the ARM parameter created in step 1:
.......
"parameters": {
"$connections": {
"value": {
"azuretables": {
"connectionId": "[parameters('connections_azuretables_externalid')]",
"connectionName": "azuretables",
"id": "/subscriptions/xxxxx-xxxx-xxxx-xxxxxxxx/providers/Microsoft.Web/locations/francecentral/managedApis/azuretables"
}
}
},
"table_storage_name": {
"value": "[parameters('table_storage_name_armparam')]"
}
}
}
}
]
}
finally, reference the logic app parameter value as follow:
"path": "/Tables/#{encodeURIComponent(parameters('table_storage_name'))}/entities"
Every time I try create an Application Insights resource with
"Application_Type" = "General"
or
"Application_Type" = "Other"
using Azure Template, it is creating as "ASP.NET" type. It seems that the default value for "Application_Type" field is "ASP.NET" or "Web".
How do I create an Application Insights resource with "Application_Type" = "General" using ARM template? I specifically need an Application Insight instance of General type to collect logs from Azure AD B2C so that we can diagnose problems with our custom policies.
The following template will work for you.
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "shuitest4"
},
"type": {
"type": "string",
"defaultValue": "other"
},
"hockeyAppToken": {
"type": "string",
"defaultValue": ""
},
"hockeyAppId": {
"type": "string",
"defaultValue": ""
},
"regionId": {
"type": "string",
"defaultValue": "southcentralus"
},
"requestSource": {
"type": "string",
"defaultValue": "IbizaAIExtension"
}
},
"resources": [
{
"name": "[parameters('name')]",
"type": "microsoft.insights/components",
"location": "[parameters('regionId')]",
"apiVersion": "2014-08-01",
"properties": {
"ApplicationId": "[parameters('name')]",
"Application_Type": "[parameters('type')]",
"HockeyAppToken": "[parameters('hockeyAppToken')]",
"HockeyAppId": "[parameters('hockeyAppId')]",
"Flow_Type": "Redfield",
"Request_Source": "[parameters('requestSource')]"
}
}
]
}
There is a easy way for you to get the template. You could create the resource on Azure Portal, when you click Automation options, you will get the template.
I am using following azure rest api to create a virtual machine in Azure Resource Manager mode
PUT
https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Compute/virtualMachines/{vm-name}?validating={true|false}&api-version={api-version}
The virtual machine gets created and it remains in Creating state as i can see the status or new portal of azure
I am able to RDP into the machine.
But after a login it fails.
What could be the reason?
Any thing i am missing?
Note : I am creating a virtual machine using a image.
Request Json :
{
"properties": {
"hardwareProfile": {
"vmSize": "Standard_A0"
},
"storageProfile": {
"osDisk": {
"osType": "Windows",
"name": "goldentemplate-osDisk",
"createOption": "FromImage",
"image": {
"uri": "https://storagename.blob.core.windows.net/system/Microsoft.Compute/Images/mytemplates/goldentemplate-osDisk.vhd"
},
"vhd": {
"uri": "https://storagename.blob.core.windows.net/vmcontainersagar/sagargoden.vhd"
},
"caching": "None"
}
,
"dataDisks": []
},
"osProfile": {
"computerName": "sagarHostVM",
"adminUsername": "itadmin",
"adminPassword": "Micr0s0ft12!#"
},
"networkProfile": {
"networkInterfaces": [
{
"properties": {},
"id": "/subscriptions/subscritpionid/resourceGroups/harigroup/providers/Microsoft.Network/networkInterfaces/sagarhostnic"
}
]
}
},
"name": "sagarHostVM",
"type": "Microsoft.Compute/virtualMachines",
"location": "WestUs",
"tags": {}
}