Use output from Web Activity call as variable - azure-data-factory

I'm using ADFv2 to transfer some data. As a part of this operation I need some configuration values to pass into the pipeline.
The config values must be pulled at runtime from a REST service - not as parameters.
I can successfully query the REST service with Web Activity and I can see the output in the debug view.
Now the problem :)
How do I use this output in other activities further in the pipeline?
My Web Activity configuration is like this:
{
"name": "Web1",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"url": "https://myazurefunction.azurewebsites.net/api/MyFunction",
"method": "GET",
"headers": {
"Content-Type": "application/json"
}
}
I have tried to access the output after is has executed, but it seems empty:
#activity('Web1').Output
#activity('Web1').output
#string(activity('Web1').Output)
they are all empty. Any suggestions?
Thanks!

I set up an ADF2 and try to get a response.
This works for me:
#string(activity('Post').output)
Have you checked the output in the debugging?
Here is my output:
{
"test": {
"value": 123,
"text": abc
},
"concat": 123abc
}
I use the stored procedure to insert the values into the destination table on a Logical Server.

In ADFv2, you access the output of previous activities using #activity('ActivityName').output.
For the web activity defined, the response from your function should be in JSON format, so you would reference specific JSON values using their attribute names in the response. For example, your defined web activity, named Web1, calls a function that returns a response of:
{
"foo": "bar",
"some": "value"
}
To use the value of foo in a subsequent ADF activity, you would reference #activity('Web1').output.foo. ADFv2 provides multiple type conversion functions, should you need the returned value converted to another type.
If your function is returning an empty JSON response back, you may want to inspect the response from your function using Postman or another tool to ensure you are returning a properly formatted response, and that your function isn't failing for another reason.
Inside your Azure function code, you should be returning a JSON object, along with a success code, similar to return req.CreateResponse(HttpStatusCode.OK, json);.
Also note that if you reference a property of the response and it does not exist, ADF will fail at that point, so you can use an If Condition activity to check for the required values to better handle failures in ADFv2.

Related

how to get a particular field from web activity output in azure data factory

I am trying to create a pipeline where I want to store a particular value from a web activity in azure data factory, in a variable, so that I can pass it to other activities.
I want to get the export ID but I keep running into errors.
The response of the web activity looks like this:
{
"requestId": "----",
"result": [
{
"exportId": "---",
"format": "CSV",
"status": "Created",
"createdAt": "2020-12-15T16:03:01Z"
}
],
"success": true
}
I have tried the following methods but it fails: #string(activity('Web1').output.result.exportId
#string(activity('Web1').output.result[0].exportId
#string(activity('Web1').output.result.[0]
first(#string(activity('Web1').output.result)
enter image description here
enter image description here
I have tried this. Your second expression should work #string(activity('Web1').output.result[0].exportId)
My test
Output of Web activity
These expressions also work fine on my side, you can have a try:
#string(activity('Web1').output['result'][0]['exportId'])
#string(activity('Web1').output.result[0].exportId)
#string(first(activity('Web1').output['result']).exportId)
#string(json(activity('Web1').output.response)[0]['Id'])

Actions Builder webhookResponse Unexpected internal error at List Response

I tried to add a List Response from my webhook and always receiving an error such as:
Unexpected internal error id=c57c97b2-0b6f-492b-88a3-3867cf2e7203
(The id changes each time.)
After comparing the expected JSON webhookResponse from the Docs with the generated Response from the Actions SDK I found a difference at the typeOverrides object:
JSON from Docs
"typeOverrides": [
{
"name": "prompt_option",
"synonym": {
"entries": []
},
"typeOverrideMode": "TYPE_REPLACE"
}
]
Generated JSON Response from Actions SDK
"typeOverrides": [
{
"name": "prompt_option",
"synonym": {
"entries": []
},
"mode": "TYPE_REPLACE"
}
]
There seems to be an error in the example documentation, but the reference docs say that it should be mode. I've tested it both ways, and that isn't causing the error.
The likely problem is that if you're replying with a List, you must do two things:
You need a Slot in the Scene that will accept the Type that you specify in the typeOverride.name. (And remember - you're updating the Type, not the name of the Slot.)
In the prompt for this slot, you must call the webhook that generates the list. (It has to be that slots prompt. You can't request it in On Enter, for example.)

How to increment a parameter in an Azure Data Factory Until Activity?

I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.
I have a Pipeline Variable called SkipIndex that defaults to 0. Inside the Until loop I have a Copy Data Activity that works (HTTP source to BLOB sink), then a Set Variable Activity that I am trying to get to increment this Variable.
{
"name": "Add 50 to SkipIndex",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Copy next to temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "SkipIndex",
"value": {
"value": "50++",
"type": "Expression"
}
}
}
Everything I have tried results in errors such as "The expression contains self referencing variable. A variable cannot reference itself in the expression." and the one above with 50++ causes a sink error during debug.
How can I get the Until loop to increment this variable after it retrieves data?
Agree that REST Connector does supports pagination but does not for Client Certificates Authentication type.
For the idea of your Until activity scenario,i am tripped by the can't self-reference a variable in an expression limitation also. Maybe you could make a little trick on that: Add one more variable to persist the index number.
For example,i got 2 variables: count and indexValue
Until Activity:
Inside Until Activity:
V1:
V2:
BTW, no usage of 50++ in ADF.

Retrieve UserName from ServiceNow

I am able to retrieve records for a particular Incident ID using Invoke-RestMethod. However, while retrieving the data, values like Resolved To, Updated By, etc. get populated by a sysid.
Resolved By comes in this format:
https<!>://devinstance.servicenow.com/api/sysid, value= sysid
I would like to view the username instead of the sysid.
The 'User ID' (user_name) isn't on the Incident, it's on the sys_user table, so you'll have to dot-walk to it.
If you're using the table API, you'll need to specify a dot-walked field to return, using the sysparm_fields query parameter.
This is no problem, just specify your endpoint like this:
$uri = "https://YOUR_INSTANCE.service-now.com/api/now/table/incident?sysparm_query=number%3DINC0000001&sysparm_fields=resolved_by.user_name"
I've specified a query for a specific incident number is requested, but you can replace that with whatever your query is.The important part is sysparm_fields=resolved_by.user_name. You'll want to specify any other fields you need here, as well.
The JSON I get as a result of running this API call, is the following:
{
"result": [
{
"resolved_by.user_name": "admin"
}
]
}
Note the element name: "resolved_by.user_name".
Another option for doing this, would be to tell the API to return both display, and actual values by specifying the sysparm_display_value parameter and setting it to all to return both sys_id and display value, or just true to return only display values.
Your URI would then look like this:
https://dev12567.service-now.com/api/now/table/incident?sysparm_query=resolved_byISNOTEMPTY%5Enumber%3DINC0000001&sysparm_display_value=all
And your JSON would contain the following:
"number": {
"display_value": "INC0000001",
"value": "INC0000001"
},
"resolved_by": {
"display_value": "System Administrator",
"link": "https://YOUR_INSTANCE.service-now.com/api/now/table/sys_user/6816f79cc0a8016401c5a33be04be441",
"value": "6816f79cc0a8016401c5a33be04be441"
},
"sys_updated_by": {
"display_value": "admin",
"value": "admin"
},
This would be accessed by:
answer.result[n].resolved_by.display_value

How to modify ajax response before jsTree creation?

How can I modify ajax response before jsTree creation? I would like to access each node ID and add prefix to it. On jsTree page the only clue is this: the function will receive two arguments - the node being loaded & a function". I need to do that before the tree is actually created, to avoid duplicate ID in the document.
"json_data" : {
"ajax" : {
"type": "GET",
"url" : "tree.json"
},
"data" : function(node, func){
//how to use that?
}}
I have expected to get JSON data here, modify it and return? But this will explode.
I have successfully manipulated data using the success callback in the instantiation of the jsTree. In my case, I am parsing XML data returned as JSON from a .NET webmethod. It should work for your case in a similar manner.
"ajax": {
"type": "GET"
"url": "tree.json",
"success": function (x) {
//manipulate string x to change id's here
return x;
}, ...
Another method is to use the "complete" callback function to manipulate the jsTree in its final form. I don't recommend it in your case of duplicate id's, however.