Linked Service parameterization not working for Linked Service of type Azure Data Explorer (Kusto) - azure-data-factory

I initially successfully created the following linked service in ADFv2 of type AzureDataExplorer for accessing my database in ADX called CustomerDB:-
{
"name": "ls_AzureDataExplorer",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "CustomerDB"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
This worked smoothly. Some values I had to mask for obvious reasons but just wanted to say that there is no issue with this connection. Now inspired from this Microsoft documentation I am trying to create a generic version of this linked service, which makes sense because otherwise if I have 10 databases in the cluster, I will have to create 10 different linked services.
So I tried to create the parameterized version in the following manner:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
But while publishing the changes I keep getting the following error:-
Is there any solution to this?
The article clearly says that:-
For all other data stores, you can parameterize the linked service by selecting the Code icon on the Connections tab and using the JSON editor
So as per that my changes should have been published successfully. But I keep getting the error.

It appears I need to specify the parameter elsewhere in the same JSON. The followed worked:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"parameters": {
"DBName": {
"type": "string"
}
},
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}

Related

Enable .NET Core collection in AppService with ARM or PowerShell

I would like to enable the .NET Core collection level with applicationInsight in Azure AppService as shown in below picture. From Azure Portal it works well. Keep in mind that by default value is set to disabled:
Now I would like to automate this using either ARM template or powershell.
I did a export template to see how it looks on ARM but there is no settings related to .NET Core collection
I check documentation on MS website but nothing about ARM template with enabling collection too
In PowerShell same problem
Is there anyone in the community who know how to enable the collection using ARM or PowerShell ?
Thanks a lot !
It's simply setting an app setting called XDT_MicrosoftApplicationInsights_Mode with value recommended (to enable) or default (to disable) as described here. You can do that both in ARM or PowerShell as below.
ARM (check appSettings part):
{
"resources": [
{
"name": "[parameters('name')]",
"type": "Microsoft.Web/sites",
"properties": {
"siteConfig": {
"appSettings": [
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference('microsoft.insights/components/AppMonitoredSite', '2015-05-01').InstrumentationKey]"
},
{
"name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
"value": "[reference('microsoft.insights/components/AppMonitoredSite', '2015-05-01').ConnectionString]"
},
{
"name": "ApplicationInsightsAgent_EXTENSION_VERSION",
"value": "~2"
},
{
"name": "XDT_MicrosoftApplicationInsights_Mode ",
"value": "recommended"
}
]
},
"name": "[parameters('name')]",
"serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"hostingEnvironment": "[parameters('hostingEnvironment')]"
},
"dependsOn": [
"[concat('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"microsoft.insights/components/AppMonitoredSite"
],
"apiVersion": "2016-03-01",
"location": "[parameters('location')]"
},
{
"apiVersion": "2016-09-01",
"name": "[parameters('hostingPlanName')]",
"type": "Microsoft.Web/serverfarms",
"location": "[parameters('location')]",
"properties": {
"name": "[parameters('hostingPlanName')]",
"workerSizeId": "[parameters('workerSize')]",
"numberOfWorkers": "1",
"hostingEnvironment": "[parameters('hostingEnvironment')]"
},
"sku": {
"Tier": "[parameters('sku')]",
"Name": "[parameters('skuCode')]"
}
},
{
"apiVersion": "2015-05-01",
"name": "AppMonitoredSite",
"type": "microsoft.insights/components",
"location": "West US 2",
"properties": {
"ApplicationId": "[parameters('name')]",
"Request_Source": "IbizaWebAppExtensionCreate"
}
}
],
"parameters": {
"name": {
"type": "string"
},
"hostingPlanName": {
"type": "string"
},
"hostingEnvironment": {
"type": "string"
},
"location": {
"type": "string"
},
"sku": {
"type": "string"
},
"skuCode": {
"type": "string"
},
"workerSize": {
"type": "string"
},
"serverFarmResourceGroup": {
"type": "string"
},
"subscriptionId": {
"type": "string"
}
},
"$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0"
}
Powershell:
$app = Get-AzWebApp -ResourceGroupName "AppMonitoredRG" -Name "AppMonitoredSite" -ErrorAction Stop
$newAppSettings = #{} # case-insensitive hash map
$app.SiteConfig.AppSettings | %{$newAppSettings[$_.Name] = $_.Value} # preserve non Application Insights application settings.
$newAppSettings["APPINSIGHTS_INSTRUMENTATIONKEY"] = "012345678-abcd-ef01-2345-6789abcd"; # set the Application Insights instrumentation key
$newAppSettings["APPLICATIONINSIGHTS_CONNECTION_STRING"] = "InstrumentationKey=012345678-abcd-ef01-2345-6789abcd"; # set the Application Insights connection string
$newAppSettings["ApplicationInsightsAgent_EXTENSION_VERSION"] = "~2"; # enable the ApplicationInsightsAgent
$newAppSettings["XDT_MicrosoftApplicationInsights_Mode"] = "recommended"; # set the APM collection to recommended
$app = Set-AzWebApp -AppSettings $newAppSettings -ResourceGroupNamrecommendede $app.ResourceGroup -Name $app.Name -ErrorAction Stop

Error in running Azure Data Factory Pipeline. Linked Service Reference not found

I am facing the below issue in creating an Azure Machine Learning Batch Execution activity to execute a scoring ML experiment. Please help:
Please let me know if any other relevant information is needed. I am new to this so, please help
Created an AzureML Linked Service as below:
{
"name": "PredictionAzureML",
"properties": {
"typeProperties": {
"mlEndpoint": "https://ussouthcentral.services.azureml.net/workspaces/xxxxx/jobs",
"apiKey": "xxxxxxxx=="
},
"type": "AzureML"
}
}
Created Pipeline as below:
{
"name": "pipeline1",
"properties": {
"description": "use AzureML model",
"activities": [
{
"name": "MLActivity",
"description": "description",
"type": "AzureMLBatchExecution",
"policy": {
"timeout": "02:00:00",
"retry": 1,
"retryIntervalInSeconds": 30
},
"typeProperties": {
"webServiceInput": "PredictionInputDataset",
"webServiceOutputs": {
"output1": "PredictionOutputDataset"
}
},
"inputs": [
{
"name": "PredictionInputDataset"
}
],
"outputs": [
{
"name": "PredictionOutputDataset"
}
],
"linkedServiceName": "PredictionAzureML"
}
]
}
}
Getting the below error:
{
"errorCode": "2109",
"message": "'linkedservicereference' with reference name 'PredictionAzureML' can't be found.",
"failureType": "UserError",
"target": "MLActivity"
}
I got this working in Data Factory v2, so apologies if you are using v1.
Try putting the linkedServiceName as an object in the JSON outside of the typeProperties and use the following structure:
"linkedServiceName": {
"referenceName": "PredictionAzureML",
"type": "LinkedServiceReference"
}
Hope that helps!
Please use "Trigger" instead of "Debug" in the UX. You need publish your pipeline first before click "Trigger" Button.
Please follow this doc to update your payload. It should look like the following.
{
"name": "AzureMLExecutionActivityTemplate",
"description": "description",
"type": "AzureMLBatchExecution",
"linkedServiceName": {
"referenceName": "AzureMLLinkedService",
"type": "LinkedServiceReference"
},
"typeProperties": {
"webServiceInputs": {
"<web service input name 1>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService1",
"type": "LinkedServiceReference"
},
"FilePath":"path1"
},
"<web service input name 2>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService1",
"type": "LinkedServiceReference"
},
"FilePath":"path2"
}
},
"webServiceOutputs": {
"<web service output name 1>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService2",
"type": "LinkedServiceReference"
},
"FilePath":"path3"
},
"<web service output name 2>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService2",
"type": "LinkedServiceReference"
},
"FilePath":"path4"
}
},
"globalParameters": {
"<Parameter 1 Name>": "<parameter value>",
"<parameter 2 name>": "<parameter 2 value>"
}
}
}

PowerShell script to create an Application Insights resource with type: General

Every time I try create an Application Insights resource with
"Application_Type" = "General"
or
"Application_Type" = "Other"
using Azure Template, it is creating as "ASP.NET" type. It seems that the default value for "Application_Type" field is "ASP.NET" or "Web".
How do I create an Application Insights resource with "Application_Type" = "General" using ARM template? I specifically need an Application Insight instance of General type to collect logs from Azure AD B2C so that we can diagnose problems with our custom policies.
The following template will work for you.
{
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "shuitest4"
},
"type": {
"type": "string",
"defaultValue": "other"
},
"hockeyAppToken": {
"type": "string",
"defaultValue": ""
},
"hockeyAppId": {
"type": "string",
"defaultValue": ""
},
"regionId": {
"type": "string",
"defaultValue": "southcentralus"
},
"requestSource": {
"type": "string",
"defaultValue": "IbizaAIExtension"
}
},
"resources": [
{
"name": "[parameters('name')]",
"type": "microsoft.insights/components",
"location": "[parameters('regionId')]",
"apiVersion": "2014-08-01",
"properties": {
"ApplicationId": "[parameters('name')]",
"Application_Type": "[parameters('type')]",
"HockeyAppToken": "[parameters('hockeyAppToken')]",
"HockeyAppId": "[parameters('hockeyAppId')]",
"Flow_Type": "Redfield",
"Request_Source": "[parameters('requestSource')]"
}
}
]
}
There is a easy way for you to get the template. You could create the resource on Azure Portal, when you click Automation options, you will get the template.

Multiple VMExtensions per handler not supported for OS type 'Linux

I have created Azure VM using ARM template. Now I want to install Java and Mongodb on Azure VM.
When I try to use Multiple CustomScript, I get the following error.
Multiple VMExtensions per handler not supported for OS type 'Linux
Below are my configuration:-
parameters:-
"javaPackageName": {
"type": "string",
"defaultValue": "openjdk-7-jdk",
"allowedValues": [
"openjdk-6-jdk",
"openjdk-7-jdk",
"openjdk-8-jdk"
]
},
"tomcatPackageName": {
"type": "string",
"defaultValue": "tomcat7",
"allowedValues": [
"tomcat6",
"tomcat7",
"tomcat8"
]
}
variables:-
"mongoInstallCentos": "https:/..install-mongo.json"
{
"type": "Microsoft.Compute/virtualMachines/extensions",
"name": "[concat(parameters('virtualMachineName'),'/javainstall')]",
"apiVersion": "2015-05-01-preview",
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachineName'))]"
],
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": ["https://..java-tomcat-install.sh"],
"commandToExecute": "[concat('sh java-tomcat-install.sh',' ',parameters('javaPackageName'),' ',parameters('tomcatPackageName'))]"
}
}
},
{
"type": "Microsoft.Compute/virtualMachines/extensions",
"name": "[concat(parameters('virtualMachineName'),'/mongoinstall')]",
"apiVersion": "2015-05-01-preview",
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', parameters('virtualMachineName'))]"
],
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": ["https://..mongo-install.sh"],
"commandToExecute": "sh mongo-install.sh"
}
}
},
Is there any solution use Multiple CustomScript in ARM template? Kindly help me to solve this.
Multiple VMExtensions per handler not supported for OS type 'Linux
Currently , it is not possible to run Multiple CustomScript Extensions at the deployment time.
According to your scenario, you can author an entry point script that calls the dependent scripts, then upload the entry point script, dependent scripts and any other dependent binaries to the script location(Azure storage blob or GitHub). More information please refer to this link.
Also, you could refer to this similar question.

Swagger UI doesn't show embedded json properties model

I am using the swagger tool for documenting my Jersey based REST API (the swaggerui I am using was downloaded on June 2014 don't know if this issue has been fixed in later versions but as I made a lot of customization to its code so I don't have the option to download the latest without investing lot of time to customize it again).
So far and until now, all my transfer objects have one level deep properties (no embedded pojos). But now that I added some rest paths that are returning more complex objects (two levels of depth) I found that SwaggerUI is not expanding the JSON model schema when having embedded objects.
Here is the important part of the swagger doc:
...
{
"path": "/user/combo",
"operations": [{
"method": "POST",
"summary": "Inserts a combo (user, address)",
"notes": "Will insert a new user and a address definition in a single step",
"type": "UserAndAddressWithIdSwaggerDto",
"nickname": "insertCombo",
"consumes": ["application/json"],
"parameters": [{
"name": "body",
"description": "New user and address combo",
"required": true,
"type": "UserAndAddressWithIdSwaggerDto",
"paramType": "body",
"allowMultiple": false
}],
"responseMessages": [{
"code": 200,
"message": "OK",
"responseModel": "UserAndAddressWithIdSwaggerDto"
}]
}]
}
...
"models": {
"UserAndAddressWithIdSwaggerDto": {
"id": "UserAndAddressWithIdSwaggerDto",
"description": "",
"required": ["user",
"address"],
"properties": {
"user": {
"$ref": "UserDto",
"description": "User"
},
"address": {
"$ref": "AddressDto",
"description": "Address"
}
}
},
"UserDto": {
"id": "UserDto",
"properties": {
"userId": {
"type": "integer",
"format": "int64"
},
"name": {
"type": "string"
},...
},
"AddressDto": {
"id": "AddressDto",
"properties": {
"addressId": {
"type": "integer",
"format": "int64"
},
"street": {
"type": "string"
},...
}
}
...
The embedded objects are User and Address, their models are being created correctly as shown in the json response.
But when opening the SwaggerUI I can only see:
{
"user": "UserDto",
"address": "AddressDto"
}
But I should see something like:
{
"user": {
"userId": "integer",
"name": "string",...
},
"address": {
"addressId": "integer",
"street": "string",...
}
}
Something may be wrong in the code that expands the internal properties, the javascript console doesn't show any error so I assume this is a bug.
I found the solution, there is a a line of code that needs to be modified to make it work properly:
In the swagger.js file there is a getSampleValue function with a conditional checking for undefined:
SwaggerModelProperty.prototype.getSampleValue = function(modelsToIgnore) {
var result;
if ((this.refModel != null) && (modelsToIgnore[this.refModel.name] === 'undefined'))
...
I updated the equality check to (removing quotes):
modelsToIgnore[this.refModel.name] === undefined
After that, SwaggerUI is able to show the embedded models.