Azure Data Factory V2: Custom Activity inside a If Condition activity - azure-data-factory

I'm working on an Azure Data Factory V2 Pipeline but I having a problem when I try to execute a "Custom activity" inside an "If Condition Activity".
If I try to test my pipeline with "Test Run" button on the ADF's Web interface, this error appeare:
{"code":"BadRequest","message":"Activity PPL_ANYFBRF01 failed: Invalid linked service reference. Name: LNK_BATCH_AZURE","target"...}
I'm sure that there is no error in the linked service reference's name. If I create a "Custom Activity" directly in my pipeline, it's working.
I think it can be a syntax error on my activity but I can't find it.
Here is my "If Condition Activity"'s Json template (the expression "#equal(0,0)" is just for testing purpose):
{
"name": "IfPointComptageNotExist",
"type": "IfCondition",
"dependsOn": [
{
"activity": "PointComptage",
"dependencyConditions": [
"Succeeded"
]
},
{
"activity": "SousPointComptage",
"dependencyConditions": [
"Succeeded"
]
}
],
"typeProperties": {
"expression": {
"value": "#equal(0,0)",
"type": "Expression"
},
"ifTrueActivities": [
{
"type": "Custom",
"name": "CustomActivityTest",
"linkedServiceName": {
"referenceName": "LNK_BATCH_AZURE",
"type": "LinkedServiceReference"
},
"typeProperties": {
"command": "Batch.exe",
"resourceLinkedService": {
"referenceName": "LNK_BLOB_STORAGE",
"type": "LinkedServiceReference"
},
"folderPath": "/test/app/"
}
}
]
}
},
Thank you in advance for your help.

The problem is now solved. I have recreate the pipeline and it's working now.
Regards,
Julien.

Related

How to check multiple conditions in ADF

enter image description hereI have a scenario where I have to check two conditions and if both are true then execute set of activities in ADF.
I tried if condition activity inside a if condition but ADF is not allowing it.
so basically my design is two lookups to read data and then if condition to check condition 1, if that is true then go inside again two lookups to read data and if condition to check condition 2. But it is not working.
is there any other work around for this?
I tried AND condition inside IF condition activity but it is not working. Please suggest.
Since we cannot use IF within IF activity, we can leverage multiple conditions within an IF activity via expressions which includes if,and ,or etc functions.
The below JSON pipeline is somewhat an example :
{
"name": "pipeline7",
"properties": {
"activities": [
{
"name": "If Condition1",
"type": "IfCondition",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"expression": {
"value": "#and(greater(pipeline().parameters.Test2, pipeline().parameters.Test1),greater(pipeline().parameters.Test4, pipeline().parameters.Test3))",
"type": "Expression"
},
"ifFalseActivities": [
{
"name": "Wait2",
"type": "Wait",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
}
],
"ifTrueActivities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": 1
}
}
]
}
}
],
"parameters": {
"Test1": {
"type": "int",
"defaultValue": 1
},
"Test2": {
"type": "int",
"defaultValue": 2
},
"Test3": {
"type": "int",
"defaultValue": 3
},
"Test4": {
"type": "int",
"defaultValue": 4
}
},
"annotations": []
}
}
Your approach is correct in case if you dont want to create another pipeline and use execute activity within the IF activity for another comparision

Extensibility point on release results page

I am part of VS performance team, I need to create a new tab for the Release result page.Release result page
Previously it was possible with the target ms.vss-releaseManagement-web.release-details-view but that no longer works on the new experience, do you know how the contribution configuration should look?, this is what I have so far:
"id": "pit-tab",
"type": "ms.vss-releaseManagement-web.release-summary-tab",
"targets": [
"ms.vss-releaseManagement-web.release-details-view"
],
"properties": {
"name": "Performance",
"uri": "dist/index.html",
"order": "99",
"icon": "asset://img/icon.png",
"supportsMobile": true
}
Try with below script:
{
"id": "release-environments-tab",
"type": "ms.vss-web.tab",
"description": "Adds a 'Merlin Insert' tab to the release environment.",
"targets": [
"ms.vss-releaseManagement-web.release-environment-editor-tab"
],
"properties": {
"name": "Merlin Insert",
"order": "30",
"uri": "index.html"
}
},

Linked Service parameterization not working for Linked Service of type Azure Data Explorer (Kusto)

I initially successfully created the following linked service in ADFv2 of type AzureDataExplorer for accessing my database in ADX called CustomerDB:-
{
"name": "ls_AzureDataExplorer",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "CustomerDB"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
This worked smoothly. Some values I had to mask for obvious reasons but just wanted to say that there is no issue with this connection. Now inspired from this Microsoft documentation I am trying to create a generic version of this linked service, which makes sense because otherwise if I have 10 databases in the cluster, I will have to create 10 different linked services.
So I tried to create the parameterized version in the following manner:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
But while publishing the changes I keep getting the following error:-
Is there any solution to this?
The article clearly says that:-
For all other data stores, you can parameterize the linked service by selecting the Code icon on the Connections tab and using the JSON editor
So as per that my changes should have been published successfully. But I keep getting the error.
It appears I need to specify the parameter elsewhere in the same JSON. The followed worked:-
{
"name": "ls_AzureDataExplorer_Generic",
"properties": {
"parameters": {
"DBName": {
"type": "string"
}
},
"type": "AzureDataExplorer",
"annotations": [],
"typeProperties": {
"endpoint": "https://mycluster.xxxxmaskingregionxxxx.kusto.windows.net",
"tenant": "xxxxmaskingtenantidxxxx",
"servicePrincipalId": "xxxxmaskingspxxxx",
"servicePrincipalKey": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "MySecret"
},
"database": "#{linkedService().DBName}"
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}

Azure DevOps Role based menu item

Currently I have build an Azure DevOps extension with a menu item who is visible for everyone. This was not a problem, but now we want to hide some of the pages for specific user groups.
I can only find limited information about setting up custom menu items and I didn't find any solution how to make the menu item role or team based.
Below you can see my current extension file. Can anyone tell me how to solve this problem?
{
// ...
"contributions": [
{
"id": "xxx-hub-group",
"type": "ms.vss-web.hub-group",
"description": "XXX",
"targets": [
"ms.vss-web.project-hub-groups-collection"
],
"properties": {
"name": "XXX",
"icon": "img/xxx.png"
}
},
{
"id": "sub-xxx",
"type": "ms.vss-web.hub",
"description": "sub-xxx",
"targets": [
".xxx-hub-group"
],
"properties": {
"name": "sub-xxx",
"title": "sub-xxx",
"icon": "img/sub-xxx.png",
"uri": "pages/sub-xxx.html",
"supportedScopes": [
"project_team"
]
}
}
]
}

Error in running Azure Data Factory Pipeline. Linked Service Reference not found

I am facing the below issue in creating an Azure Machine Learning Batch Execution activity to execute a scoring ML experiment. Please help:
Please let me know if any other relevant information is needed. I am new to this so, please help
Created an AzureML Linked Service as below:
{
"name": "PredictionAzureML",
"properties": {
"typeProperties": {
"mlEndpoint": "https://ussouthcentral.services.azureml.net/workspaces/xxxxx/jobs",
"apiKey": "xxxxxxxx=="
},
"type": "AzureML"
}
}
Created Pipeline as below:
{
"name": "pipeline1",
"properties": {
"description": "use AzureML model",
"activities": [
{
"name": "MLActivity",
"description": "description",
"type": "AzureMLBatchExecution",
"policy": {
"timeout": "02:00:00",
"retry": 1,
"retryIntervalInSeconds": 30
},
"typeProperties": {
"webServiceInput": "PredictionInputDataset",
"webServiceOutputs": {
"output1": "PredictionOutputDataset"
}
},
"inputs": [
{
"name": "PredictionInputDataset"
}
],
"outputs": [
{
"name": "PredictionOutputDataset"
}
],
"linkedServiceName": "PredictionAzureML"
}
]
}
}
Getting the below error:
{
"errorCode": "2109",
"message": "'linkedservicereference' with reference name 'PredictionAzureML' can't be found.",
"failureType": "UserError",
"target": "MLActivity"
}
I got this working in Data Factory v2, so apologies if you are using v1.
Try putting the linkedServiceName as an object in the JSON outside of the typeProperties and use the following structure:
"linkedServiceName": {
"referenceName": "PredictionAzureML",
"type": "LinkedServiceReference"
}
Hope that helps!
Please use "Trigger" instead of "Debug" in the UX. You need publish your pipeline first before click "Trigger" Button.
Please follow this doc to update your payload. It should look like the following.
{
"name": "AzureMLExecutionActivityTemplate",
"description": "description",
"type": "AzureMLBatchExecution",
"linkedServiceName": {
"referenceName": "AzureMLLinkedService",
"type": "LinkedServiceReference"
},
"typeProperties": {
"webServiceInputs": {
"<web service input name 1>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService1",
"type": "LinkedServiceReference"
},
"FilePath":"path1"
},
"<web service input name 2>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService1",
"type": "LinkedServiceReference"
},
"FilePath":"path2"
}
},
"webServiceOutputs": {
"<web service output name 1>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService2",
"type": "LinkedServiceReference"
},
"FilePath":"path3"
},
"<web service output name 2>": {
"LinkedServiceName":{
"referenceName": "AzureStorageLinkedService2",
"type": "LinkedServiceReference"
},
"FilePath":"path4"
}
},
"globalParameters": {
"<Parameter 1 Name>": "<parameter value>",
"<parameter 2 name>": "<parameter 2 value>"
}
}
}