How to encrypt Azure Data Factory with CMK (customer managed key) - azure-data-factory

Parameters.json
"parameters": {
"dataFactoryName": { "type": "string", "metadata": { "description": "Name of the data factory. Name must be globally unique" } },
"resourceTags": { "type": "object" },
"diagnosticSettingsStorageAccount": { "type": "string", "metadata": { "description": "Resource ID of the storage account used to store diagnostic logs" } },
"cmkIdentity": {
"type": "string"
},
"vaultBaseUrl": {
"type": "string"
},
"keyName": {
"type": "string"
},
"keyVersion": {
"type": "string"
}
},
Template.json
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[resourceGroup().location]",
"tags": "[parameters('resourceTags')]",
"identity": {
"type": "SystemAssigned,UserAssigned",
"userAssignedIdentities": {"[parameters('cmkIdentity')]": {}}},
"properties": {
"publicNetworkAccess": "Disabled",
"encryption": {
"identity": {
"userAssignedIdentity": "[parameters('cmkIdentity')]"
},
"vaultBaseUrl": "[parameters('vaultBaseUrl')]",
"keyName": "[parameters('keyName')]",
"keyVersion": "[parameters('keyVersion')]"
}
},
"dependsOn": ["[resourceId('Microsoft.OperationalInsights/workspaces',variables('workspaceName'))]"]
},
I am passing these values in:
cmkIdentity: "/subscriptions/xxxxx/resourcegroups/xxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-identity"
vaultBaseUrl: https://testkeyvault123.vault.azure.net/
keyName: test-key
keyVersion: t5dca2a5xxxxx399we5
The Validation passes and the Data Factory is deployed. I can see the test-identity in the Managed Identity section. But when I open the Data Factory's UI and navigate to Manage and in Customer managed key I don't see anything. All fields are empty, see picture below:
Provided access policy to test-identity to the test key vault. Can't figure out what the problem is.
UPDATED parameters and template
parameters
"parameters": {
"dataFactoryName": { "type": "string", "metadata": { "description": "Name of the data factory. Name must be globally unique" } },
"resourceTags": { "type": "object" },
"diagnosticSettingsStorageAccount": { "type": "string", "metadata": { "description": "Resource ID of the storage account used to store diagnoistic logs" } },
"cmkIdentity": {
"type": "object",
"defaultValue": {
"/subscriptions/xxxxx/resourcegroups/xxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-identity": {
}
}
},
"vaultBaseUrl": {
"type": "string"
},
"keyName": {
"type": "string"
},
"keyVersion": {
"type": "string"
}
},
template.json
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[resourceGroup().location]",
"tags": "[parameters('resourceTags')]",
"identity": {
"type": "SystemAssigned,UserAssigned",
"principalId": "",
"tenantId": "",
"userAssignedIdentities": "[parameters('cmkIdentity')]"
},
"properties": {
"publicNetworkAccess": "Disabled",
"encryption": {
"identity": {
"userAssignedIdentity": "[parameters('cmkIdentity')]"
},
"vaultBaseUrl": "[parameters('vaultBaseUrl')]",
"keyName": "[parameters('keyName')]",
"keyVersion": "[parameters('keyVersion')]"
}
},
"dependsOn": ["[resourceId('Microsoft.OperationalInsights/workspaces',variables('workspaceName'))]"]
},

I have tried doing it both the ways via json template and even through portal works fine, just to have another parameter which is object type as below:
Updated Info : Please add a new parameter as an cmkidentity_obj which is type: object and have the other parameter cmkidentity as type: string
and pass the string in :
"encryption": {
"identity": {
"userAssignedIdentity": "[parameters('cmkidentity')]"
},
"VaultBaseUrl": "[parameters('dataFactory_properties_encryption_VaultBaseUrl')]",
"KeyName": "[parameters('dataFactory_properties_encryption_KeyName')]",
"KeyVersion": "[parameters('dataFactory_properties_encryption_KeyVersion')]"
}
Parameter- cmkidentity_obj abd pass it in below template.json :
"cmkidentity_obj": {
"type": "object",
"defaultValue": {
"/subscriptions/xxxxx/resourcegroups/xxxxx/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-identity": {
}
}
}
and then passed this object in my template.json :
"identity": {
"type": "[parameters('dataFactory_identity_type')]",
"principalId": "",
"tenantId": "",
"userAssignedIdentities": "[parameters('cmkidentity_obj')]"
}
This deployed successfully with about any error and was able to view my Customer Managed Key in Azure Data Factory(UI), kindly try the same and see.
My Template.json:
"resources": [
{
"name": "[parameters('factoryName')]",
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"properties": {
"encryption": {
"identity": {
"userAssignedIdentity": "[parameters('cmkidentity')]"
},
"VaultBaseUrl": "[parameters('dataFactory_properties_encryption_VaultBaseUrl')]",
"KeyName": "[parameters('dataFactory_properties_encryption_KeyName')]",
"KeyVersion": "[parameters('dataFactory_properties_encryption_KeyVersion')]"
}
},
"dependsOn": [],
"location": "[parameters('dataFactory_location')]",
"identity": {
"type": "[parameters('dataFactory_identity_type')]",
"principalId": "",
"tenantId": "",
"userAssignedIdentities": "[parameters('cmkIdentity_obj')]"
}
}
]

Related

How to make a FunctionApp setting slotSpecific using ARM templates

We are using ARM templates to deploy function apps but the slotSetting: true property is not respected and I cannot find any modern documentation as to how to make app settings slot specific.
THis is my app settings snippet in my ARM template
{
"name": "AzureWebJobs.HandleFiscalFrResponse.Disabled",
"value": "1",
"slotSetting": true
}
the setting and the value works but the slotSettings attribute is ignored silently, no error is shown its just ignored.
What is the correct way to make a function app setting slot specific?
I have reproduced the issue and able to resolve, please follow the below steps
Open VS code and create a file using .json extension and se the below code
Thanks #patelchandni for the ARM Template code.
My Filename.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"functionAppName": {
"type": "string",
"defaultValue": "[format('tar-{0}', uniqueString(resourceGroup().id))]"
},
"storageAccountType": {
"type": "string",
"defaultValue": "Standard_LRS",
"allowedValues": [
"Standard_LRS",
"Standard_GRS",
"Standard_RAGRS"
]
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"appInsightsLocation": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"functionWorkerRuntime": {
"type": "string",
"defaultValue": "node",
"allowedValues": [
"dotnet",
"node",
"python",
"java"
]
},
"functionPlanOS": {
"type": "string",
"defaultValue": "Windows",
"allowedValues": [
"Windows",
"Linux"
]
},
"functionAppPlanSku": {
"type": "string",
"defaultValue": "EP1",
"allowedValues": [
"EP1",
"EP2",
"EP3"
]
},
"linuxFxVersion": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Only required for Linux app to represent runtime stack in the format of 'runtime|runtimeVersion'. For example: 'python|3.9'"
}
}
},
"variables": {
"hostingPlanName": "[parameters('functionAppName')]",
"applicationInsightsName": "[parameters('functionAppName')]",
"storageAccountName": "[concat(uniquestring(resourceGroup().id), 'azfunctions')]",
"isReserved": "[if(equals(parameters('functionPlanOS'), 'Linux'), true(), false())]",
"slotContentShareName": "[concat(parameters('functionAppName'), '-deployment')]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-02-01",
"name": "[variables('storageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "[parameters('storageAccountType')]"
},
"kind": "Storage"
},
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2021-02-01",
"name": "[variables('hostingPlanName')]",
"location": "[parameters('location')]",
"sku": {
"tier": "ElasticPremium",
"name": "[parameters('functionAppPlanSku')]",
"family": "EP"
},
"properties": {
"maximumElasticWorkerCount": 20,
"reserved": "[variables('isReserved')]"
},
"kind": "elastic"
},
{
"type": "microsoft.insights/components",
"apiVersion": "2020-02-02",
"name": "[variables('applicationInsightsName')]",
"location": "[parameters('appInsightsLocation')]",
"tags": {
"[concat('hidden-link:', resourceId('Microsoft.Web/sites', variables('applicationInsightsName')))]": "Resource"
},
"properties": {
"Application_Type": "web"
},
"kind": "web"
},
{
"type": "Microsoft.Web/sites",
"apiVersion": "2021-02-01",
"name": "[parameters('functionAppName')]",
"location": "[parameters('location')]",
"kind": "[if(variables('isReserved'), 'functionapp,linux', 'functionapp')]",
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]",
"[resourceId('Microsoft.Insights/components', variables('applicationInsightsName'))]"
],
"properties": {
"reserved": "[variables('isReserved')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
"siteConfig": {
"linuxFxVersion": "[if(variables('isReserved'), parameters('linuxFxVersion'), json('null'))]",
"appSettings": [
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference(resourceId('microsoft.insights/components', variables('applicationInsightsName')), '2015-05-01').InstrumentationKey]"
},
{
"name": "AzureWebJobsStorage",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';EndpointSuffix=', environment().suffixes.storage, ';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';EndpointSuffix=', environment().suffixes.storage, ';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "[toLower(parameters('functionAppName'))]"
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~4"
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "[parameters('functionWorkerRuntime')]"
},
{
"name": "WEBSITE_NODE_DEFAULT_VERSION",
"value": "~14"
}
]
}
}
},
{
"type": "Microsoft.Web/sites/slots",
"apiVersion": "2021-02-01",
"name": "[concat(parameters('functionAppName'), '/deployment')]",
"kind": "[if(variables('isReserved'), 'functionapp,linux', 'functionapp')]",
"location": "[parameters('location')]",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', parameters('functionAppName'))]"
],
"properties": {
"reserved": "[variables('isReserved')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
"siteConfig": {
"linuxFxVersion": "[if(variables('isReserved'), parameters('linuxFxVersion'), json('null'))]",
"appSettings": [
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference(resourceId('microsoft.insights/components', variables('applicationInsightsName')), '2015-05-01').InstrumentationKey]"
},
{
"name": "AzureWebJobsStorage",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';EndpointSuffix=', environment().suffixes.storage, ';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'), ';EndpointSuffix=', environment().suffixes.storage, ';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value)]"
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "[variables('slotContentShareName')]"
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~4"
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "[parameters('functionWorkerRuntime')]"
},
{
"name": "WEBSITE_NODE_DEFAULT_VERSION",
"value": "~14"
}
]
}
}
}
]
}
Click on the below marked one to create parameter file
Click new as shown in below picture
Use the below code in FileName.parameters.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"functionAppName": {
"value": "zapper01"
},
"storageAccountType": {
"value": "Standard_LRS"
},
"location": {
"value": "EastUS"
},
"appInsightsLocation": {
"value": "EastUS"
},
"functionWorkerRuntime": {
"value": "node"
},
"functionPlanOS": {
"value": "Windows"
},
"functionAppPlanSku": {
"value": "EP1"
},
"linuxFxVersion": {
"value": "3.9"
}
}
}
To login to azure portal, run the PowerShell command
az login
Set subscription by using
az account set --subscription "Subscription ID xxxxxx-xxxxxxx-xxxxxxx-xxxxx"
Deploy to azure portal
New-AzResourceGroupDeployment -ResourceGroupName "ResourceGroupName" -TemplateFile "FileName.json" -TemplateParameterFile "Filename.parameters.json"
After execution you will get below one as result in power shall
After deploying into azure portal open the function app and select deployment slot
Update
I have deployed the below code in Custom deployment In Azure portal
Thanks, #seligj95 for the ARM Template code.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"baseResourceName": {
"type": "string",
"metadata": {
"description": "Name of the resource"
},
"maxLength": 15
},
"appSettingName": {
"type": "string",
"metadata": {
"description": "Name of the app setting"
},
"maxLength": 24
},
"environments": {
"defaultValue": [
"Dev",
"QA",
"UAT",
"Preview"
],
"type": "array",
"metadata": {
"description": "Array with the names for the environment slots"
},
"maxLength": 19
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"variables": {
"standardPlanMaxAdditionalSlots": 4,
"webAppPortalName": "[concat(parameters('baseResourceName'), 'Portal')]",
"appServicePlanName": "[concat('AppServicePlan-', parameters('baseResourceName'))]",
"stickyAppSettingName": "[concat(parameters('appSettingName'), '-sticky')]"
},
"resources": [
{
"apiVersion": "2020-06-01",
"type": "Microsoft.Web/serverfarms",
"kind": "app",
"name": "[variables('appServicePlanName')]",
"location": "[parameters('location')]",
"comments": "This app service plan is used for the web app and slots.",
"tags": {
"displayName": "AppServicePlan"
},
"properties": { },
"sku": {
"name": "[if(lessOrEquals(length(parameters('environments')), variables('standardPlanMaxAdditionalSlots')), 'S1', 'P1')]"
}
},
{
"apiVersion": "2020-06-01",
"type": "Microsoft.Web/sites",
"kind": "app",
"name": "[variables('webAppPortalName')]",
"location": "[parameters('location')]",
"comments": "This is the web app, also the default 'nameless' slot.",
"tags": {
"displayName": "WebApp"
},
"properties": {
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]",
"siteConfig": {
"appSettings": [
{
"name": "[parameters('appSettingName')]",
"value": "value"
},
{
"name": "[variables('stickyAppSettingName')]",
"value": "value"
}
]
}
},
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
]
},
{
"apiVersion": "2020-06-01",
"type": "Microsoft.Web/sites/slots",
"name": "[concat(variables('webAppPortalName'), '/', parameters('environments')[copyIndex()])]",
"kind": "app",
"location": "[parameters('location')]",
"comments": "This specifies the web app slots.",
"tags": {
"displayName": "WebAppSlots"
},
"properties": {
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
},
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', variables('webAppPortalName'))]"
],
"copy": {
"name": "webPortalSlot",
"count": "[length(parameters('environments'))]"
}
},
{
"apiVersion": "2020-06-01",
"name": "[concat(variables('webAppPortalName'), '/slotconfignames')]",
"type": "Microsoft.Web/sites/config",
"comments": "This specifies the sticky (slot setting) application settings.",
"dependsOn": [
"[resourceId('Microsoft.Web/Sites', variables('webAppPortalName'))]"
],
"properties": {
"appSettingNames": [
"[variables('stickyAppSettingName')]"
]
}
}
]
}
I have selected the second slot as specific sticky slot in arm template, So the second slot is deployed

I'm trying to deploy an Azure Data Factory service using Azure Resource Manager Templates but getting an error The request content was invalid

I'm trying to deploy an Azure Data Factory service using Azure Resource Manager Templates but getting an error The request content was invalid and could not be desterilized:
'Could not find member 'name' on object of type 'Template'. Path 'properties.template.name', line 1, position 34.'.
You can use the below ARM template for creating a Azure storage account and Azure data factory and linking both of them via linked service:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"metadata": {
"_generator": {
"name": "bicep",
"version": "0.4.1.14562",
"templateHash": "8367564219536411224"
}
},
"parameters": {
"dataFactoryName": {
"type": "string",
"defaultValue": "[format('datafactory{0}', uniqueString(resourceGroup().id))]",
"metadata": {
"description": "Data Factory Name"
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location of the data factory."
}
},
"storageAccountName": {
"type": "string",
"defaultValue": "[format('storage{0}', uniqueString(resourceGroup().id))]",
"metadata": {
"description": "Name of the Azure storage account that contains the input/output data."
}
},
"blobContainerName": {
"type": "string",
"defaultValue": "[format('blob{0}', uniqueString(resourceGroup().id))]",
"metadata": {
"description": "Name of the blob container in the Azure Storage account."
}
}
},
"functions": [],
"variables": {
"dataFactoryLinkedServiceName": "ArmtemplateStorageLinkedService",
"dataFactoryDataSetInName": "ArmtemplateTestDatasetIn",
"dataFactoryDataSetOutName": "ArmtemplateTestDatasetOut",
"pipelineName": "ArmtemplateSampleCopyPipeline"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-04-01",
"name": "[parameters('storageAccountName')]",
"location": "[parameters('location')]",
"sku": {
"name": "Standard_LRS"
},
"kind": "StorageV2"
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2021-04-01",
"name": "[format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName'))]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
]
},
{
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[parameters('location')]",
"identity": {
"type": "SystemAssigned"
}
},
{
"type": "Microsoft.DataFactory/factories/linkedservices",
"apiVersion": "2018-06-01",
"name": "[format('{0}/{1}', parameters('dataFactoryName'), variables('dataFactoryLinkedServiceName'))]",
"properties": {
"type": "AzureBlobStorage",
"typeProperties": {
"connectionString": "[format('DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}', parameters('storageAccountName'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2021-04-01').keys[0].value)]"
}
},
"dependsOn": [
"[resourceId('Microsoft.DataFactory/factories', parameters('dataFactoryName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
]
},
{
"type": "Microsoft.DataFactory/factories/datasets",
"apiVersion": "2018-06-01",
"name": "[format('{0}/{1}', parameters('dataFactoryName'), variables('dataFactoryDataSetInName'))]",
"properties": {
"linkedServiceName": {
"referenceName": "[variables('dataFactoryLinkedServiceName')]",
"type": "LinkedServiceReference"
},
"type": "Binary",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "[format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName'))]",
"folderPath": "input",
"fileName": "emp.txt"
}
}
},
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices/containers', split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[0], split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[1], split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[2])]",
"[resourceId('Microsoft.DataFactory/factories', parameters('dataFactoryName'))]",
"[resourceId('Microsoft.DataFactory/factories/linkedservices', parameters('dataFactoryName'), variables('dataFactoryLinkedServiceName'))]"
]
},
{
"type": "Microsoft.DataFactory/factories/datasets",
"apiVersion": "2018-06-01",
"name": "[format('{0}/{1}', parameters('dataFactoryName'), variables('dataFactoryDataSetOutName'))]",
"properties": {
"linkedServiceName": {
"referenceName": "[variables('dataFactoryLinkedServiceName')]",
"type": "LinkedServiceReference"
},
"type": "Binary",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "[format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName'))]",
"folderPath": "output"
}
}
},
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices/containers', split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[0], split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[1], split(format('{0}/default/{1}', parameters('storageAccountName'), parameters('blobContainerName')), '/')[2])]",
"[resourceId('Microsoft.DataFactory/factories', parameters('dataFactoryName'))]",
"[resourceId('Microsoft.DataFactory/factories/linkedservices', parameters('dataFactoryName'), variables('dataFactoryLinkedServiceName'))]"
]
},
{
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"name": "[format('{0}/{1}', parameters('dataFactoryName'), variables('pipelineName'))]",
"properties": {
"activities": [
{
"name": "MyCopyActivity",
"type": "Copy",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"typeProperties": {
"source": {
"type": "BinarySource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true
}
},
"sink": {
"type": "BinarySink",
"storeSettings": {
"type": "AzureBlobStorageWriterSettings"
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "[variables('dataFactoryDataSetInName')]",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "[variables('dataFactoryDataSetOutName')]",
"type": "DatasetReference"
}
]
}
]
},
"dependsOn": [
"[resourceId('Microsoft.DataFactory/factories', parameters('dataFactoryName'))]",
"[resourceId('Microsoft.DataFactory/factories/datasets', parameters('dataFactoryName'), variables('dataFactoryDataSetInName'))]",
"[resourceId('Microsoft.DataFactory/factories/datasets', parameters('dataFactoryName'), variables('dataFactoryDataSetOutName'))]"
]
}
]
}
Reference:
Create an Azure Data Factory using an Azure Resource Manager template (ARM template) - Azure Data Factory | Microsoft Docs

How to reference already existing Azure Portal created storage accounts in an ARM template

I want to create a function app with ARM template such that the ARM template references already existing storage accounts (inputstgdev and outputstgdev) which I have already parameterized. I would like the ARM template to use the inputstgdev storage account as its attached storage account such that it does not have to create a new storage account. The source control of the function app is referenced to a Gitrepo which I have also parameterized in the ARM template. Each time I run the ARM template I get the following error message
##[error]Deployment template validation failed: 'The resource '/subscriptions/bea8ac84-24a4-4e53-9198-e3b0107547d4/resourceGroups/dev-rgp/providers/Microsoft.Web/sites/functionapp/sourcecontrols/web' at line '1' and column '3069' doesn't depend on parent resource '/subscriptions/bea8ac84-24a4-4e53-9198-e3b0107547d4/resourceGroups/dev-rgp/providers/Microsoft.Web/sites/functionapp'. Please add dependency explicitly using the 'dependsOn' syntax. Please see aka.ms/arm-template/#resources for
Any suggestions what the issue might be or possible solutions.
I look forward to your response
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"InputstorageAccount": {
"defaultValue": "inputstgdev",
"type": "String"
},
"GitrepoBranch": {
"type": "string",
"defaultValue": "master",
"metadata": {
"description": "Name of the branch to use when deploying (Default = master)."
}
},
"GitrepoURL": {
"type": "string",
"defaultValue": "https://github.com/FBoucher/AzUnzipEverything.git",
"metadata": {
"description": "URL to repo (Default = master)."
}
},
"InputcontainerName": {
"type": "string",
"defaultValue": "inputcontainer",
"metadata": {
"description": "Specifies the name of the blob container."
}
},
"OutputstorageAccount": {
"defaultValue": "outputstgdev",
"type": "String"
},
"OutputcontainerName": {
"type": "string",
"defaultValue": "outputcontainer",
"metadata": {
"description": "Specifies the name of the blob container."
}
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(parameters('InputstorageAccount'), '/default/', parameters('InputcontainerName'))]",
"properties": {
"publicAccess": "None"
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2019-06-01",
"name": "[concat(parameters('OutputstorageAccount'), '/default/', parameters('OutputcontainerName'))]",
"properties": {
"publicAccess": "None"
}
},
{
"name": "serviceplan",
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2018-02-01",
"location": "[resourceGroup().location]",
"sku": {
"name": "F1",
"capacity": 1
},
"tags": {
"displayName": "serviceplan"
},
"properties": {
"name": "serviceplan"
}
},
{
"name": "functionapp",
"type": "Microsoft.Web/sites",
"apiVersion": "2018-11-01",
"location": "[resourceGroup().location]",
"kind": "functionapp",
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', 'serviceplan')]",
"[resourceId('Microsoft.Storage/storageAccounts', parameters('InputstorageAccount'))]"
],
"properties": {
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', 'serviceplan')]",
"siteConfig": {
"appSettings": [
{
"name": "AzureWebJobsDashboard",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('InputstorageAccount'), ';AccountKey=', listKeys(parameters('InputcontainerName'),'2015-05-01-preview').key1)]"
},
{
"name": "AzureWebJobsStorage",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('InputstorageAccount'), ';AccountKey=', listKeys(parameters('InputcontainerName'),'2015-05-01-preview').key1)]"
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('InputstorageAccount'), ';AccountKey=', listKeys(parameters('InputcontainerName'),'2015-05-01-preview').key1)]"
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "[toLower('functionapp')]"
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~2"
},
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference(resourceId('microsoft.insights/components/', 'applicationInsightsName'), '2015-05-01').InstrumentationKey]"
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "dotnet"
}
]
}
},
"resources":[
{
"apiVersion": "2015-08-01",
"name": "web",
"type": "sourcecontrols",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/', parameters('InputstorageAccount'))]"
],
"properties": {
"RepoUrl": "[parameters('GitrepoURL')]",
"branch": "[parameters('GitrepoBranch')]",
"publishRunbook": true,
"IsManualIntegration": true
}
}
]
}
]
}
You encountered this error message due to the wrong dependency.
You should use "[resourceId('Microsoft.Web/sites/', 'functionapp')]" instead of "[resourceId('Microsoft.Web/sites/', parameters('InputstorageAccount'))]".
And you should delete the storage dependency.
"[resourceId('Microsoft.Storage/storageAccounts', parameters('InputstorageAccount'))]"
By the way, as you already have a storage account, you just need to paste your connection string in the value of AzureWebJobsStorage and AzureWebJobsDashboard.
Just like
{
"name": "AzureWebJobsDashboard",
"value": "{connectionstring}"
}

Getting error on null and empty string while copying a csv file from blob container to Azure SQL DB

I tried all combination on the datatype of my data but each time my data factory pipeline is giving me this error:
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorColumnNameNotAllowNull,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Empty or Null string found in Column Name 2. Please make sure column name not null and try again.,Source=Microsoft.DataTransfer.Common,'",
"failureType": "UserError",
"target": "xxx",
"details": []
}
My Copy data source code is something like this:{
"name": "xxx",
"description": "uuu",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"wildcardFileName": "*"
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "AzureSqlSink"
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"name": "populationId",
"type": "Guid"
},
"sink": {
"name": "PopulationID",
"type": "String"
}
},
{
"source": {
"name": "inputTime",
"type": "DateTime"
},
"sink": {
"name": "inputTime",
"type": "DateTime"
}
},
{
"source": {
"name": "inputCount",
"type": "Decimal"
},
"sink": {
"name": "inputCount",
"type": "Decimal"
}
},
{
"source": {
"name": "inputBiomass",
"type": "Decimal"
},
"sink": {
"name": "inputBiomass",
"type": "Decimal"
}
},
{
"source": {
"name": "inputNumber",
"type": "Decimal"
},
"sink": {
"name": "inputNumber",
"type": "Decimal"
}
},
{
"source": {
"name": "utcOffset",
"type": "String"
},
"sink": {
"name": "utcOffset",
"type": "Int32"
}
},
{
"source": {
"name": "fishGroupName",
"type": "String"
},
"sink": {
"name": "fishgroupname",
"type": "String"
}
},
{
"source": {
"name": "yearClass",
"type": "String"
},
"sink": {
"name": "yearclass",
"type": "String"
}
}
]
}
},
"inputs": [
{
"referenceName": "DelimitedTextFTDimensions",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "AzureSqlTable1",
"type": "DatasetReference"
}
]
}
Can anyone please help me understand the issue. I see in some blogs they ask me use treatnullasempty but I am not allowed to modify the JSON. is there a way to do that??
I suggest to using Data Flow DerivedColumn, DerivedColumn can help you build expression to replace the null column.
For example:
Derived Column, if Column_2 is null =true, return 'dd' :
iifNull(Column_2,'dd')
Mapping the column
Reference: Data transformation expressions in mapping data flow
Hope this helps.
fixed it.it was a easy fix as one of my column in destination was marked as not null, i changed it as null and it worked.

Authoratative Source Dcoumentation for Azure ARM Template Schema

I am looking for the official documentation for Azure Desired State Configuration JSON schemas. Documentation and examples I find online go back quite a while and I feel the schemas I am using are not current. I am currently seeking the most current schema and a description of the fields for Microsoft.Powershell.DSC. This is the schema Visual Studio provides but I would like to have a complete understanding of each field and values they expect.
{
"name": "Microsoft.Powershell.DSC",
"type": "extensions",
"location": "[parameters('location')]",
"apiVersion": "2015-06-15",
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', parameters('vm-SP1-Name'))]"
],
"tags": {
"displayName": "test"
},
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.9",
"autoUpgradeMinorVersion": true,
"forceUpdateTag": "[parameters('testUpdateTagVersion')]",
"settings": {
"configuration": {
"url": "[concat(parameters('_artifactsLocation'), '/', variables('testArchiveFolder'), '/', variables('testArchiveFileName'))]",
"script": "test.ps1",
"function": "Main"
},
"configurationArguments": {
"nodeName": "[parameters('vm-SP1-Name')]"
}
},
"protectedSettings": {
"configurationUrlSasToken": "[parameters('_artifactsLocationSasToken')]"
}
}
}
The official schema definition for the ARM PowerShell Desired State Configuration (DSC) JSON Schema can be found at the link below:
Schema Json
"dscExtension": {
"type": "object",
"properties": {
"publisher": {
"enum": ["Microsoft.Powershell"]
},
"type": {
"enum": ["DSC"]
},
"typeHandlerVersion": {
"type": "string",
"minLength": 1
},
"autoUpgradeMinorVersion": {
"type": "boolean"
},
"settings": {
"type": "object",
"properties": {
"modulesUrl": {
"type": "string"
},
"configurationFunction": {
"type": "string"
},
"properties": {
"type": "string"
},
"wmfVersion": {
"type": "string"
},
"privacy": {
"type": "object",
"properties": {
"dataCollection": {
"type": "string"
}
}
}
},
"required": ["modulesUrl", "configurationFunction"]
},
"protectedSettings": {
"type": "object",
"properties": {
"dataBlobUri": {
"type": "string"
}
}
}
},
"required": ["publisher", "type", "typeHandlerVersion", "autoUpgradeMinorVersion", "settings", "protectedSettings"]
},
The schema was documented in this blog. Feel free to ping me on the PowerShell slack (preferred) or twitter if you have any questions, #TravisPlunk. If I don't know, I can direct them to the right place.