Azure Blob Storage Deployment: Stored Access Policy gets deleted - azure-devops

Context:
I deploy a storage account as well as one or more containers with the following ARM template with Azure DevOps respectively a Resource Deployment Task:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the Azure Storage account."
}
},
"containerNames": {
"type": "array",
"metadata": {
"description": "The names of the blob containers."
}
},
"location": {
"type": "string",
"metadata": {
"description": "The location in which the Azure Storage resources should be deployed."
}
}
},
"resources": [
{
"name": "[parameters('storageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2018-07-01",
"location": "[parameters('location')]",
"kind": "StorageV2",
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"properties": {
"accessTier": "Hot"
}
},
{
"name": "[concat(parameters('storageAccountName'), '/default/', parameters('containerNames')[copyIndex()])]",
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2018-03-01-preview",
"dependsOn": [
"[parameters('storageAccountName')]"
],
"copy": {
"name": "containercopy",
"count": "[length(parameters('containerNames'))]"
}
}
],
"outputs": {
"storageAccountName": {
"type": "string",
"value": "[parameters('storageAccountName')]"
},
"storageAccountKey": {
"type": "string",
"value": "[listKeys(parameters('storageAccountName'), '2018-02-01').keys[0].value]"
},
"storageContainerNames": {
"type": "array",
"value": "[parameters('containerNames')]"
}
}
}
Input can be e.g.
-storageAccountName 'stor1' -containerNames [ 'con1', 'con2' ] -location 'westeurope'
In an next step I create Stored Access Policies for the containers deployed.
Problem:
The first time I do that everything works fine. But if I execute the pipeline a second time the Stored Access Policies gets deleted by the deployment of the template. The storage account itself with its containers and blobs are not deleted (as it should be). This is unfortunate because I want to keep the Stored Access Policy with its starttime and expirytime as deployed the first time, furthermore I expect that the SAS also become invalid (not tested so far).
Questions:
Why is this happening?
How can I avoid this problem respectively keep the Stored Access Policies?
Thanks

After doing some investigation this seems to be by design. When deploying ARM templates for storage accounts the PUT operation is used, i.e. elements that are not specified within the template are removed. As it is not possible to specify Shared Access Policies for containers within an ARM template for Storage Accounts existing ones get deleted when the template is redeployed...

Related

Azure ARM Template parameters for parametrized linked service

Please, forgive the confusing tittle, if it is, but it does describe the problem I am having
So, I have a linked service in my Azure Datafactory. It is used for Azure SQL Database connect.
The Database name and user name are being taken from the parameters set in linked service itself. Here is a snippet of json config
"typeProperties": {
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=myserver.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KeyVaultLink",
"type": "LinkedServiceReference"
},
"secretName": "DBPassword"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
This works fine when in debug in the Azure portal. However, when I get the ARM Template for the whole thing, during ARM Template deployment it asks for input Connection string for the linked service. If I go to the linked service definition, and look up its connection string it will come this way
"connectionString": "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=dmsql.database.windows.net;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().dbUserName}"
Then when I input it in the ARM Template deployment should I be replacing "#{linkedService().dbName}" and "#{linkedService().dbUserName}" with actual values at the spot when I am entring it ? I am confused because during the ARM Template deployment there are no separate fields for these parameters, and these (parameters specific to linked service itself) are not present as separate parameters in the ARM Template definition.
I created database in my azure portal
and enabled system assigned managed Identity for sql db.
Image for reference:
I created azure keywault and created secret.
Image for reference:
I have created new access policy for Azure data factory.
Image for reference:
I created Azure data factory and enabled system managed identity.
Image for reference:
I have created new parametrized linked service to connect with database with below parameters dbName and userName. I am taking database name and User name dynamically by using above parameters.
Image for reference:
Linked service is created successfully.
json format of my lined service:
{
"name": "SqlServer1",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
}
}
I exported the arm template of data factory.
This is my linked service in my ARM template:
"SqlServer1_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'SqlServer1'",
"defaultValue": "Integrated Security=False;Data Source=dbservere;Initial Catalog=#{linkedService().dbName};User ID=#{linkedService().userName}"
},
"AzureKeyVault1_properties_typeProperties_baseUrl": {
"type": "string",
"defaultValue": "https://keysqlad.vault.azure.net/"
}
Image for reference:
I have got parameters dbName and userName in my ARM template description.
{
"name": "[concat(parameters('factoryName'), '/SqlServer1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"parameters": {
"dbName": {
"type": "String"
},
"userName": {
"type": "String"
}
},
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": "[parameters('SqlServer1_connectionString')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "DBPASSWORD"
},
"alwaysEncryptedSettings": {
"alwaysEncryptedAkvAuthType": "ManagedIdentity"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
}
Image for reference:
If you didn't get parameters in ARM template description copy the value of "connectionString" and modified what you needed to and left the parameters in place and added it to the "connectionString" override parameter in my Azure Release Pipeline, and it will work.

How to create idempotent, re-deployable ARM templates that utilize Key Vault? Circular dependencies present issues

I'm trying to incorporate some additional security features on my resources such as encryption using customer managed keys. For the Service Bus, this requires the Service Bus to have a managed identity created and granted access to the Key Vault. However, the managed identity is not known until after the Service Bus exists.
In my ARM template, I need to initialize the Service Bus without encryption in order to get a managed identity, grant that identity access to key vault, then update the Service Bus with encryption. However, this deployment process is not repeatable. On subsequent re-deployments, it will fail because encryption cannot be removed from a Service Bus once it is granted. Even if it did work, I would be performing unnecessary steps to remove encryption and add it back on every deployment. It seems that an IAC template that describes the expected final state of the resource cannot be used to both maintain and bootstrap a new environment.
The same problem exists for API Management, I want to add custom domains but they require Key Vault access. It means I can't re-deploy my ARM template without removing the custom domains when the init step gets repeated (or keep a separate set of templates for 'initialization' vs the 'real deployment'.
Is there a better solution for this? I looked into user assigned identities which seems like it could solve the problem, but they aren't supported in ARM templates. I checked if there is a way to make the 'init' step conditional by checking if the resource already exists, and this is also not supported via ARM template.
If you want to maintain a single ARM template, you can use nested deployments to define a resource then reference it again to update it.
In the following example, I create the Service Bus with a system-assigned managed identity, a Key Vault, and an RSA key. In a nested deployment within the same ARM template that depends on that key being generated, I then update the Service Bus to enable encryption. An advantage of being all in the same template is that resourceId and reference can be used with abbreviated syntax (i.e. just the resource name).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "[resourceGroup().name]"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"tenantId": {
"type": "string",
"defaultValue": "[subscription().tenantId]"
}
},
"variables": {
"kv_name": "[concat(parameters('name'), 'kv')]",
"kv_version": "2019-09-01",
"sb_name": "[concat(parameters('name'), 'sb')]",
"sb_version": "2018-01-01-preview",
"sb_keyname": "sbkey"
},
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false
}
},
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "[variables('kv_version')]",
"name": "[variables('kv_name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('sb_name')]"
],
"properties": {
"sku": {
"family": "A",
"name": "Standard"
},
"tenantId": "[parameters('tenantId')]",
"accessPolicies": [
{
"tenantId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.tenantId]",
"objectId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.principalId]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey"
]
}
}
],
// Both must be enabled to encrypt Service Bus at rest.
"enableSoftDelete": true,
"enablePurgeProtection": true
}
},
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "[variables('kv_version')]",
"name": "[concat(variables('kv_name'), '/', variables('sb_keyname'))]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('kv_name')]"
],
"properties": {
"kty": "RSA",
"keySize": 2048,
"keyOps": [
"wrapKey",
"unwrapKey"
],
"attributes": {
"enabled": true
}
}
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "sb_deployment",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/keys', variables('kv_name'), variables('sb_keyname'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false,
"encryption": {
"keySource": "Microsoft.KeyVault",
"keyVaultProperties": [
{
// Ideally should specify a specific version, but no ARM template function to get this currently.
"keyVaultUri": "[reference(variables('kv_name')).vaultUri]",
"keyName": "[variables('sb_keyname')]"
}
]
}
}
}
]
}
}
}
]
}
To deploy this using the Azure CLI:
az group create -g rg-example -l westus2
az deployment group create -g rg-example -f template.json --parameters name=example

Arm template deployment fail with 409 error for one specific storage account

I use arm template to deploy a storage account. However, I got an error saying: StorageAccountAlreadyExists: The storage account named xxx already exists.
My release pipeline is set to incremental, so shouldn't really show this error.
I changed storage account name to a new one, not only it worked the first time, but I can keep on deploying the same pipeline and no error ever thrown out.
Looks like it is something specific to this account, however, I can't see anything special. The arm template we use is also quite normal (something we got from official examples before).
{
"$schema": "http://schema.management.azure.com/schemas/2019-06-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"StorageDescriptor": {
"type": "string",
"defaultValue": "StorageAccount",
"metadata": {}
},
"StorageAccountName": {
"type": "string",
"defaultValue": "[toLower(concat(parameters('StorageDescriptor'), resourceGroup().name))]",
"metadata": { "Description": "Override name for the storage account" }
},
"StorageType": {
"type": "string",
"defaultValue": "Standard_LRS",
"allowedValues": [
"Standard_LRS",
"Standard_ZRS",
"Standard_GRS",
"Standard_RAGRS",
"Premium_LRS"
]
},
"Environment": {
"type": "string",
"defaultValue": "PreProd",
"metadata": { "description": "PreProd or Prod" }
}
},
"variables": {
},
"resources": [
{
"name": "[parameters('StorageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"location": "[resourceGroup().location]",
"apiVersion": "2019-06-01",
"dependsOn": [],
"tags": {
"displayName": "Web Job Storage Account"
},
"properties": {
"accountType": "[parameters('StorageType')]"
}
}
],
"outputs": {
}
}
Even though your release pipeline is set to incremental, the storage account name must be unique every time you deploy. Refer to: here.
Arm template deployment fail with 409 error for one specific storage account
You need to check if the storage account attributes have been changed through the Azure/PowerShell portal by somebody else, and are different than the ones specified on the ARM template.
To resolve this issue, please try to export the template and update it in the Azure devops repo:
Then, we could update this new exported template file as you want and deploy with it.
As test, I could keep on deploying the same pipeline and no error ever thrown out.

How to add Roles in AppInsight Access control using ARM template

I am trying to add Role Assignment in AppInsight Access control using ARM template.
I am able to create AppInsight with ARM template but unable to move ahead with addition of Role Assignment in App Insight Access control . Following is my code for creation of App Insight using ARM template
"resources": [
{
"type": "Microsoft.Insights/components",
"kind": "web",
"name": "[parameters('components_AppInsightPoc_name')]",
"apiVersion": "2015-05-01",
"location": "westus2",
"scale": null,
"properties": {
"Application_Type": "web",
"Flow_Type": "Redfield",
"Request_Source": "IbizaAIExtension",
"HockeyAppId": null,
"SamplingPercentage": null
}
}
]
You can use this snippet to add RBAC roles to the resource:
{
"type": "Microsoft.Insights/components/providers/roleAssignments",
"apiVersion": "2017-05-01",
"name": "[concat(parameters('components_AppInsightPoc_name'), '/Microsoft.Authorization/', guid('something'))]",
"properties": {
"roleDefinitionId": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/', 'role_guid')]",
"principalId": "user_guid",
"scope": "[resourceId('Microsoft.Insights/components', parameters('components_AppInsightPoc_name'))"
}
}
you can get role guids with powershell:
Get-AzRoleDefinition
Able to add RBAC for App Insights with the following code
"resources": [
{
"type": "Microsoft.Insights/components/providers/roleAssignments",
"apiVersion": "2017-05-01",
"name": "[concat(parameters('AppInsightName'),'/Microsoft.Authorization/',guid('AppInsightName'))]",
"properties": {
"roleDefinitionId": "[variables(parameters('builtInRoleType'))]",
"principalId": "[parameters('principalId')]"
}
}
]
I posted an answer on this question: Apply Azure RBAC to resource using ARM
I have an array of Role IDs that I wanted to add as Owners at the App Insight resource, without making the users Owners at the Resource Group level. I didn't want to use the nested resource approach as I wanted to iterate over an array of objects to dynamically create the roles, so after tweaking the type, name, and scope attributes, the following resource block is what ended up working for me:
{
"comments": "Add the Application Insights resource",
"apiVersion": "2014-04-01",
"name": "[variables('appInsightsName')]",
"type": "Microsoft.Insights/components",
"location": "[resourceGroup().location]",
"properties": {
"ApplicationId": "[variables('appInsightsName')]"
}
},
{
"comments": "Add the IAM roles to the App Insights resource",
"condition": "[parameters('isProduction')]",
"type": "Microsoft.Insights/components/providers/roleAssignments",
"name": "[concat(variables('appInsightsName'),'/Microsoft.Authorization/',guid(parameters('roleAssignments')[copyIndex()].principalId))]",
"apiVersion": "2017-05-01",
"location": "[resourceGroup().location]",
"properties": {
"roleDefinitionId": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635')]", // Owner Role
"principalId": "[parameters('roleAssignments')[copyIndex()].principalId]",
"scope": "[resourceId('Microsoft.Insights/components', variables('appInsightsName'))]"
},
"copy": {
"name": "appInsightsRoleAssignments",
"count": "[length(parameters('roleAssignments'))]"
},
"dependsOn": [
"[resourceId('Microsoft.Insights/components', variables('appInsightsName'))]"
]
}
Pay close attention to the segments in the type, name, and scope properties of the Microsoft.Insights/components/providers/roleAssignments resource.

PowerShell script to create an Application Insights resource with type: General

Every time I try create an Application Insights resource with
"Application_Type" = "General"
or
"Application_Type" = "Other"
using Azure Template, it is creating as "ASP.NET" type. It seems that the default value for "Application_Type" field is "ASP.NET" or "Web".
How do I create an Application Insights resource with "Application_Type" = "General" using ARM template? I specifically need an Application Insight instance of General type to collect logs from Azure AD B2C so that we can diagnose problems with our custom policies.
The following template will work for you.
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "shuitest4"
},
"type": {
"type": "string",
"defaultValue": "other"
},
"hockeyAppToken": {
"type": "string",
"defaultValue": ""
},
"hockeyAppId": {
"type": "string",
"defaultValue": ""
},
"regionId": {
"type": "string",
"defaultValue": "southcentralus"
},
"requestSource": {
"type": "string",
"defaultValue": "IbizaAIExtension"
}
},
"resources": [
{
"name": "[parameters('name')]",
"type": "microsoft.insights/components",
"location": "[parameters('regionId')]",
"apiVersion": "2014-08-01",
"properties": {
"ApplicationId": "[parameters('name')]",
"Application_Type": "[parameters('type')]",
"HockeyAppToken": "[parameters('hockeyAppToken')]",
"HockeyAppId": "[parameters('hockeyAppId')]",
"Flow_Type": "Redfield",
"Request_Source": "[parameters('requestSource')]"
}
}
]
}
There is a easy way for you to get the template. You could create the resource on Azure Portal, when you click Automation options, you will get the template.