I receive the following errors when attempting to deploy an exported Azure Resource Template:
"The template is invalid. Error: 'Required property '$schema' not found in
JSON.
"Domain name is already taken"
Error Content:
The error content is as follows:
{"code":"InvalidTemplateDeployment","details":[{"code":"DeploymentPreflightValidationFailed","target":"/subscriptions/<SOME_GUID_VALUE>/resourceGroups/<SOME_RESOURCE_GROUP_NAME>-Resource/providers/Microsoft.Resources/templateSpecs/<SOME_ARM_TEMPLATE_NAME>/versions/0.0","message":"The
template is invalid. Error: 'Required property '$schema' not found in
JSON. Path '', line 1, position
2.'"},{"code":"BadArgument","message":"Domain name is already taken"}],"message":"The template deployment
'Microsoft.Template-20220817090827' is not valid according to the
validation procedure. The tracking id is
'c27e8ad4-6d96-46e9-92ed-ac8d5f462bf6'. See inner errors for
details."}
Observations:
I have observed the following:
The latest schema path appears to match what I have listed.
Template:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vulnerabilityAssessments_Default_storageContainerPath": {
"type": "SecureString"
},
...
}
The error regarding the $schema property is not referring to the $schema of the template itself, but is instead coming back from preflight (validation) of one of the Microsoft.Resources/templateSpecs/versions resources in your template.
I can repro this myself with the following file I exported (note that Export is stripping out the contents of mainTemplate, which I believe is the root cause of the issue, as it doesn't contain the expected $schema property).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"templateSpecs_asdgasdg_name": {
"defaultValue": "asdgasdg",
"type": "String"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Resources/templateSpecs",
"apiVersion": "2022-02-01",
"name": "[parameters('templateSpecs_asdgasdg_name')]",
"location": "centralus",
"properties": {}
},
{
"type": "Microsoft.Resources/templateSpecs/versions",
"apiVersion": "2022-02-01",
"name": "[concat(parameters('templateSpecs_asdgasdg_name'), '/v1')]",
"location": "centralus",
"dependsOn": [
"[resourceId('Microsoft.Resources/templateSpecs', parameters('templateSpecs_asdgasdg_name'))]"
],
"properties": {
"mainTemplate": {}
}
}
]
}
My team at Microsoft owns the Export Template API - I will follow up internally on this.
Related
Please, forgive the confusing tittle, if it is, but it does describe the problem I am having
So, I have a linked service in my Azure Datafactory. It is used for Azure SQL Database connect.
The Database name and user name are being taken from the parameters set in linked service itself. Here is a snippet of json config
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=myserver.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KeyVaultLink",
"type": "LinkedServiceReference"
},
"secretName": "DBPassword"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
This works fine when in debug in the Azure portal. However, when I get the ARM Template for the whole thing, during ARM Template deployment it asks for input Connection string for the linked service. If I go to the linked service definition, and look up its connection string it will come this way
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=dmsql.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}"
Then when I input it in the ARM Template deployment should I be replacing "#{linkedService().dbName}" and "#{linkedService().dbUserName}" with actual values at the spot when I am entring it ? I am confused because during the ARM Template deployment there are no separate fields for these parameters, and these (parameters specific to linked service itself) are not present as separate parameters in the ARM Template definition.
I created database in my azure portal
and enabled system assigned managed Identity for sql db.
Image for reference:
I created azure keywault and created secret.
Image for reference:
I have created new access policy for Azure data factory.
Image for reference:
I created Azure data factory and enabled system managed identity.
Image for reference:
I have created new parametrized linked service to connect with database with below parameters dbName and userName. I am taking database name and User name dynamically by using above parameters.
Image for reference:
Linked service is created successfully.
json format of my lined service:
{
"name": "SqlServer1",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
}
}
I exported the arm template of data factory.
This is my linked service in my ARM template:
"SqlServer1_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'SqlServer1'",
"defaultValue": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}"
},
"AzureKeyVault1_properties_typeProperties_baseUrl": {
"type": "string",
"defaultValue": "https://keysqlad.vault.azure.net/"
}
Image for reference:
I have got parameters dbName and userName in my ARM template description.
{
"name": "[concat(parameters('factoryName'), '/SqlServer1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "[parameters('SqlServer1_connectionString')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
}
Image for reference:
If you didn't get parameters in ARM template description copy the value of "connectionString" and modified what you needed to and left the parameters in place and added it to the "connectionString" override parameter in my Azure Release Pipeline, and it will work.
I'm trying to incorporate some additional security features on my resources such as encryption using customer managed keys. For the Service Bus, this requires the Service Bus to have a managed identity created and granted access to the Key Vault. However, the managed identity is not known until after the Service Bus exists.
In my ARM template, I need to initialize the Service Bus without encryption in order to get a managed identity, grant that identity access to key vault, then update the Service Bus with encryption. However, this deployment process is not repeatable. On subsequent re-deployments, it will fail because encryption cannot be removed from a Service Bus once it is granted. Even if it did work, I would be performing unnecessary steps to remove encryption and add it back on every deployment. It seems that an IAC template that describes the expected final state of the resource cannot be used to both maintain and bootstrap a new environment.
The same problem exists for API Management, I want to add custom domains but they require Key Vault access. It means I can't re-deploy my ARM template without removing the custom domains when the init step gets repeated (or keep a separate set of templates for 'initialization' vs the 'real deployment'.
Is there a better solution for this? I looked into user assigned identities which seems like it could solve the problem, but they aren't supported in ARM templates. I checked if there is a way to make the 'init' step conditional by checking if the resource already exists, and this is also not supported via ARM template.
If you want to maintain a single ARM template, you can use nested deployments to define a resource then reference it again to update it.
In the following example, I create the Service Bus with a system-assigned managed identity, a Key Vault, and an RSA key. In a nested deployment within the same ARM template that depends on that key being generated, I then update the Service Bus to enable encryption. An advantage of being all in the same template is that resourceId and reference can be used with abbreviated syntax (i.e. just the resource name).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "[resourceGroup().name]"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"tenantId": {
"type": "string",
"defaultValue": "[subscription().tenantId]"
}
},
"variables": {
"kv_name": "[concat(parameters('name'), 'kv')]",
"kv_version": "2019-09-01",
"sb_name": "[concat(parameters('name'), 'sb')]",
"sb_version": "2018-01-01-preview",
"sb_keyname": "sbkey"
},
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false
}
},
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "[variables('kv_version')]",
"name": "[variables('kv_name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('sb_name')]"
],
"properties": {
"sku": {
"family": "A",
"name": "Standard"
},
"tenantId": "[parameters('tenantId')]",
"accessPolicies": [
{
"tenantId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.tenantId]",
"objectId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.principalId]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey"
]
}
}
],
// Both must be enabled to encrypt Service Bus at rest.
"enableSoftDelete": true,
"enablePurgeProtection": true
}
},
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "[variables('kv_version')]",
"name": "[concat(variables('kv_name'), '/', variables('sb_keyname'))]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('kv_name')]"
],
"properties": {
"kty": "RSA",
"keySize": 2048,
"keyOps": [
"wrapKey",
"unwrapKey"
],
"attributes": {
"enabled": true
}
}
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "sb_deployment",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/keys', variables('kv_name'), variables('sb_keyname'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false,
"encryption": {
"keySource": "Microsoft.KeyVault",
"keyVaultProperties": [
{
// Ideally should specify a specific version, but no ARM template function to get this currently.
"keyVaultUri": "[reference(variables('kv_name')).vaultUri]",
"keyName": "[variables('sb_keyname')]"
}
]
}
}
}
]
}
}
}
]
}
To deploy this using the Azure CLI:
az group create -g rg-example -l westus2
az deployment group create -g rg-example -f template.json --parameters name=example
Hey I am doing a CI/CD deployment for a logic app, I have a table storage where I store some data, I have two table storage for test and prod environment. I created a parameter called *table_storage_name" in ARM template :
"parameters": {
// ....
"connections_azuretables_1_externalid": {
"defaultValue": "/subscriptions/e5..../resourceGroups/myrg.../providers/Microsoft.Web/connections/azuretables-1",
"type": "String"
},
"table_storage_name": {
"defaultValue": "testdevops",
"type": "String"
}
}
The error comes from when I reference the parameter here in template.json file:
// ...
"Insert_Entity": {
"runAfter": {
"Initialize_variable": [
"Succeeded"
]
},
"type": "ApiConnection",
"inputs": {
"body": {
"PartitionKey": "#body('Parse_JSON')?['name']",
"RowKey": "#body('Parse_JSON')?['last']"
},
"host": {
"connection": {
"name": "#parameters('$connections')['azuretables_1']['connectionId']"
}
},
"method": "post",
// problem occur after this line
"path": "/Tables/#{encodeURIComponent('[parameters('table_storage_name')]')}/entities"
}
}
but get this error:
InvalidTemplate: The template validation failed: 'The template action 'Insert_Entity' at line '1' and column '582' is not valid: "Unable to parse template language expression 'encodeURIComponent([parameters('table_storage_name')])': expected token 'Identifier' and actual 'LeftSquareBracket'.".'.
I tried escaping the quote with a backslash like: encodeURIComponent(\'[parameters('table_storage_name')]\') or encodeURIComponent('[parameters(''table_storage_name'')]') but all of them raise an error. How can I reference a paramter inside encodeURIComponent in an ARM template ?
As discussed in the comments. credits: #marone
"path": "/Tables/#{encodeURIComponent(parameters('table_storage_name'))}/entities"
Found the solution from this link https://platform.deloitte.com.au/articles/preparing-azure-logic-apps-for-cicd
but here are the steps to reference a parameter logic app:
create an ARM parameter table_storage_name_armparam in template.json, in order to use it's value to reference the value of the ARM parameter (yes it's confusing but follow along you'll understand):
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"table_storage_name_armparam": {
"type": "String"
}
},
"variables": {},
"resources": [
{
......
}
Now in the logic app parameter value (in the bottom of json file) create the logic app parameter table_storage_name and the value of this parameter will be the ARM parameter created in step 1:
.......
"parameters": {
"$connections": {
"value": {
"azuretables": {
"connectionId": "[parameters('connections_azuretables_externalid')]",
"connectionName": "azuretables",
"id": "/subscriptions/xxxxx-xxxx-xxxx-xxxxxxxx/providers/Microsoft.Web/locations/francecentral/managedApis/azuretables"
}
}
},
"table_storage_name": {
"value": "[parameters('table_storage_name_armparam')]"
}
}
}
}
]
}
finally, reference the logic app parameter value as follow:
"path": "/Tables/#{encodeURIComponent(parameters('table_storage_name'))}/entities"
I use arm template to deploy a storage account. However, I got an error saying: StorageAccountAlreadyExists: The storage account named xxx already exists.
My release pipeline is set to incremental, so shouldn't really show this error.
I changed storage account name to a new one, not only it worked the first time, but I can keep on deploying the same pipeline and no error ever thrown out.
Looks like it is something specific to this account, however, I can't see anything special. The arm template we use is also quite normal (something we got from official examples before).
{
"$schema": "http://schema.management.azure.com/schemas/2019-06-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"StorageDescriptor": {
"type": "string",
"defaultValue": "StorageAccount",
"metadata": {}
},
"StorageAccountName": {
"type": "string",
"defaultValue": "[toLower(concat(parameters('StorageDescriptor'), resourceGroup().name))]",
"metadata": { "Description": "Override name for the storage account" }
},
"StorageType": {
"type": "string",
"defaultValue": "Standard_LRS",
"allowedValues": [
"Standard_LRS",
"Standard_ZRS",
"Standard_GRS",
"Standard_RAGRS",
"Premium_LRS"
]
},
"Environment": {
"type": "string",
"defaultValue": "PreProd",
"metadata": { "description": "PreProd or Prod" }
}
},
"variables": {
},
"resources": [
{
"name": "[parameters('StorageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"location": "[resourceGroup().location]",
"apiVersion": "2019-06-01",
"dependsOn": [],
"tags": {
"displayName": "Web Job Storage Account"
},
"properties": {
"accountType": "[parameters('StorageType')]"
}
}
],
"outputs": {
}
}
Even though your release pipeline is set to incremental, the storage account name must be unique every time you deploy. Refer to: here.
Arm template deployment fail with 409 error for one specific storage account
You need to check if the storage account attributes have been changed through the Azure/PowerShell portal by somebody else, and are different than the ones specified on the ARM template.
To resolve this issue, please try to export the template and update it in the Azure devops repo:
Then, we could update this new exported template file as you want and deploy with it.
As test, I could keep on deploying the same pipeline and no error ever thrown out.
Context:
I deploy a storage account as well as one or more containers with the following ARM template with Azure DevOps respectively a Resource Deployment Task:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the Azure Storage account."
}
},
"containerNames": {
"type": "array",
"metadata": {
"description": "The names of the blob containers."
}
},
"location": {
"type": "string",
"metadata": {
"description": "The location in which the Azure Storage resources should be deployed."
}
}
},
"resources": [
{
"name": "[parameters('storageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2018-07-01",
"location": "[parameters('location')]",
"kind": "StorageV2",
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"properties": {
"accessTier": "Hot"
}
},
{
"name": "[concat(parameters('storageAccountName'), '/default/', parameters('containerNames')[copyIndex()])]",
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2018-03-01-preview",
"dependsOn": [
"[parameters('storageAccountName')]"
],
"copy": {
"name": "containercopy",
"count": "[length(parameters('containerNames'))]"
}
}
],
"outputs": {
"storageAccountName": {
"type": "string",
"value": "[parameters('storageAccountName')]"
},
"storageAccountKey": {
"type": "string",
"value": "[listKeys(parameters('storageAccountName'), '2018-02-01').keys[0].value]"
},
"storageContainerNames": {
"type": "array",
"value": "[parameters('containerNames')]"
}
}
}
Input can be e.g.
-storageAccountName 'stor1' -containerNames [ 'con1', 'con2' ] -location 'westeurope'
In an next step I create Stored Access Policies for the containers deployed.
Problem:
The first time I do that everything works fine. But if I execute the pipeline a second time the Stored Access Policies gets deleted by the deployment of the template. The storage account itself with its containers and blobs are not deleted (as it should be). This is unfortunate because I want to keep the Stored Access Policy with its starttime and expirytime as deployed the first time, furthermore I expect that the SAS also become invalid (not tested so far).
Questions:
Why is this happening?
How can I avoid this problem respectively keep the Stored Access Policies?
Thanks
After doing some investigation this seems to be by design. When deploying ARM templates for storage accounts the PUT operation is used, i.e. elements that are not specified within the template are removed. As it is not possible to specify Shared Access Policies for containers within an ARM template for Storage Accounts existing ones get deleted when the template is redeployed...