How to increment a parameter in an Azure Data Factory Until Activity? - azure-data-factory

I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.
I have a Pipeline Variable called SkipIndex that defaults to 0. Inside the Until loop I have a Copy Data Activity that works (HTTP source to BLOB sink), then a Set Variable Activity that I am trying to get to increment this Variable.
{
"name": "Add 50 to SkipIndex",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Copy next to temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "SkipIndex",
"value": {
"value": "50++",
"type": "Expression"
}
}
}
Everything I have tried results in errors such as "The expression contains self referencing variable. A variable cannot reference itself in the expression." and the one above with 50++ causes a sink error during debug.
How can I get the Until loop to increment this variable after it retrieves data?

Agree that REST Connector does supports pagination but does not for Client Certificates Authentication type.
For the idea of your Until activity scenario,i am tripped by the can't self-reference a variable in an expression limitation also. Maybe you could make a little trick on that: Add one more variable to persist the index number.
For example,i got 2 variables: count and indexValue
Until Activity:
Inside Until Activity:
V1:
V2:
BTW, no usage of 50++ in ADF.

Related

how to get a particular field from web activity output in azure data factory

I am trying to create a pipeline where I want to store a particular value from a web activity in azure data factory, in a variable, so that I can pass it to other activities.
I want to get the export ID but I keep running into errors.
The response of the web activity looks like this:
{
"requestId": "----",
"result": [
{
"exportId": "---",
"format": "CSV",
"status": "Created",
"createdAt": "2020-12-15T16:03:01Z"
}
],
"success": true
}
I have tried the following methods but it fails: #string(activity('Web1').output.result.exportId
#string(activity('Web1').output.result[0].exportId
#string(activity('Web1').output.result.[0]
first(#string(activity('Web1').output.result)
enter image description here
enter image description here
I have tried this. Your second expression should work #string(activity('Web1').output.result[0].exportId)
My test
Output of Web activity
These expressions also work fine on my side, you can have a try:
#string(activity('Web1').output['result'][0]['exportId'])
#string(activity('Web1').output.result[0].exportId)
#string(first(activity('Web1').output['result']).exportId)
#string(json(activity('Web1').output.response)[0]['Id'])

ADF CDM Source Transformation not reading data

I am having an issue with the inline data set for Common Data Model in Azure Data Factory.
Simply, everything in ADF appears to connect and read from my manifest file and entity definition - but when I click the "Data preview" button, I always get "No output data" - which I find bizarre, as the data can be read perfectly when using the CDM connector to the same files in PowerBI. What am I doing wrong to mean that the data is not read into the data preview and subsequent transformations in the mapping data flow?
My Manifest file looks as below (referring to an example entity):
{
"$schema": "CdmManifest.cdm.json",
"jsonSchemaSemanticVersion": "1.0.0",
"imports": [
{
"corpusPath": "cdm:/foundations.cdm.json"
}
],
"manifestName": "manifestExample",
"explanation": "example",
"entities": [
{
"type": "LocalEntity",
"entityName": "Entityname",
"entityPath": "folder/EntityName.cdm.json/Entityname",
"dataPartitions": [
{
"location": "folder/data/Entityname/Entityname.csv",
"exhibitsTraits": [
{
"traitReference": "is.partition.format.CSV",
"arguments": [
{
"name": "columnHeaders",
"value": "true"
},
{
"name": "delimiter",
"value": ","
}
]
}
]
}
]
},
...
I am having exactly same output message "No output data". I am using json not manifest. If i sink the source it moves no data but without error. My CDM originates from PowerBI dataflow. The PowerApps works fine but historization and privileges make it useless.
Edit:
On Microsofts info on preview feature we can find this
screen. I will make a guess that CDM the ADS sources is not the same which orignates from Power BI.

Actions Builder webhookResponse Unexpected internal error at List Response

I tried to add a List Response from my webhook and always receiving an error such as:
Unexpected internal error id=c57c97b2-0b6f-492b-88a3-3867cf2e7203
(The id changes each time.)
After comparing the expected JSON webhookResponse from the Docs with the generated Response from the Actions SDK I found a difference at the typeOverrides object:
JSON from Docs
"typeOverrides": [
{
"name": "prompt_option",
"synonym": {
"entries": []
},
"typeOverrideMode": "TYPE_REPLACE"
}
]
Generated JSON Response from Actions SDK
"typeOverrides": [
{
"name": "prompt_option",
"synonym": {
"entries": []
},
"mode": "TYPE_REPLACE"
}
]
There seems to be an error in the example documentation, but the reference docs say that it should be mode. I've tested it both ways, and that isn't causing the error.
The likely problem is that if you're replying with a List, you must do two things:
You need a Slot in the Scene that will accept the Type that you specify in the typeOverride.name. (And remember - you're updating the Type, not the name of the Slot.)
In the prompt for this slot, you must call the webhook that generates the list. (It has to be that slots prompt. You can't request it in On Enter, for example.)

Implemented a Resource Type: How does Concourse use the output of the check, in, and out scripts?

Reading the Concourse documentation about Implementing a Resource Type, in regards to what the check, in, and out scripts must emit, it is not clear why this output is needed or how Concourse uses it. My questions are:
1) How does Concourse use the output of the check script, the in script, and the out script?
2) And, why is it required that the in and out script emit the version? What happens if you don't?
For context, here is the relevant parts of the documentation:
1) For the check script:
...[it] must print the array of new versions, in chronological order,
to stdout, including the requested version if it's still valid.
For example:
[
{ "ref": "61cbef" },
{ "ref": "d74e01" },
{ "ref": "7154fe" }
]
2) For the in script:
The script must emit the fetched version, and may emit metadata as a list of key-value pairs. This data is intended for public consumption and will make it upstream, intended to be shown on the build's page.
For example:
{
"version": { "ref": "61cebf" },
"metadata": [
{ "name": "commit", "value": "61cebf" },
{ "name": "author", "value": "Hulk Hogan" }
]
}
3) Similar to the in script, the out script:
The script must emit the resulting version of the resource. For
example, the git resource emits the sha of the commit that it just
pushed.
For example:
{
"version": { "ref": "61cebf" },
"metadata": [
{ "name": "commit", "value": "61cebf" },
{ "name": "author", "value": "Mick Foley" }
]
}
Concourse uses the check result to verify if there is any new resource available. According to your pipeline definition, the presence of a new resource would trigger a job. The in is therefore used to read the specific resource using parameters provided by the pipeline whilst the out would take care of writing them.
As your in is going to use the information provided by the check you may want to use a similar structure, but you're not obliged to. It is useful to echo the same version information in your check/in/out in order to be able to log it and understand each resource in your pipeline is belonging to which version.

Use output from Web Activity call as variable

I'm using ADFv2 to transfer some data. As a part of this operation I need some configuration values to pass into the pipeline.
The config values must be pulled at runtime from a REST service - not as parameters.
I can successfully query the REST service with Web Activity and I can see the output in the debug view.
Now the problem :)
How do I use this output in other activities further in the pipeline?
My Web Activity configuration is like this:
{
"name": "Web1",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"url": "https://myazurefunction.azurewebsites.net/api/MyFunction",
"method": "GET",
"headers": {
"Content-Type": "application/json"
}
}
I have tried to access the output after is has executed, but it seems empty:
#activity('Web1').Output
#activity('Web1').output
#string(activity('Web1').Output)
they are all empty. Any suggestions?
Thanks!
I set up an ADF2 and try to get a response.
This works for me:
#string(activity('Post').output)
Have you checked the output in the debugging?
Here is my output:
{
"test": {
"value": 123,
"text": abc
},
"concat": 123abc
}
I use the stored procedure to insert the values into the destination table on a Logical Server.
In ADFv2, you access the output of previous activities using #activity('ActivityName').output.
For the web activity defined, the response from your function should be in JSON format, so you would reference specific JSON values using their attribute names in the response. For example, your defined web activity, named Web1, calls a function that returns a response of:
{
"foo": "bar",
"some": "value"
}
To use the value of foo in a subsequent ADF activity, you would reference #activity('Web1').output.foo. ADFv2 provides multiple type conversion functions, should you need the returned value converted to another type.
If your function is returning an empty JSON response back, you may want to inspect the response from your function using Postman or another tool to ensure you are returning a properly formatted response, and that your function isn't failing for another reason.
Inside your Azure function code, you should be returning a JSON object, along with a success code, similar to return req.CreateResponse(HttpStatusCode.OK, json);.
Also note that if you reference a property of the response and it does not exist, ADF will fail at that point, so you can use an If Condition activity to check for the required values to better handle failures in ADFv2.