I am trying to create a pipeline where I want to store a particular value from a web activity in azure data factory, in a variable, so that I can pass it to other activities.
I want to get the export ID but I keep running into errors.
The response of the web activity looks like this:
{
"requestId": "----",
"result": [
{
"exportId": "---",
"format": "CSV",
"status": "Created",
"createdAt": "2020-12-15T16:03:01Z"
}
],
"success": true
}
I have tried the following methods but it fails: #string(activity('Web1').output.result.exportId
#string(activity('Web1').output.result[0].exportId
#string(activity('Web1').output.result.[0]
first(#string(activity('Web1').output.result)
enter image description here
enter image description here
I have tried this. Your second expression should work #string(activity('Web1').output.result[0].exportId)
My test
Output of Web activity
These expressions also work fine on my side, you can have a try:
#string(activity('Web1').output['result'][0]['exportId'])
#string(activity('Web1').output.result[0].exportId)
#string(first(activity('Web1').output['result']).exportId)
#string(json(activity('Web1').output.response)[0]['Id'])
Related
I am going to send a custom event to EventGrid Custom topic , through ADF using web activity .
in the header I use
aeg-sas-key , and then the value is the key of the eventGrid
body is something like this
[{
"id": "1807",
"eventType": "recordInserted",
"subject": "myapp/vehicles/motorcycles",
"eventTime": "2017-08-10T21:03:07+00:00",
"data": {
"make": "test",
"model": "test"
},
"dataVersion": "1.0"
}]
however , I got the error
and one has experience or idea how to do that?
is there anyway to secure the Key ?
I would advise you to use managed identity authentication. Please do the following to achieve this:
Create a Data Factory with System Managed Identity
Create an Event Grid Topic and copy the topic endpoint into your notepad.
Grant the Data Factory access to the Event Grid Topic, by assigning the EventGrid Data Sender role.
Go to your Data Factory and create a pipeline with the Web Activity
Configure the web activity the following way:
Web Activity Config:
URL: <your-event-grid-topic-endpoint> (similar to https://<your-topic-name>.northeurope-1.eventgrid.azure.net/api/events)
Method: POST
Body: Sample body:
[{
"id": "#{pipeline().RunId}",
"eventType": "PipelineFinished",
"subject": "myapp/pipelines/mydataset",
"eventTime": "#{formatDateTime(utcnow(), 's')}",
"data": {
"DataFactory": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"PipelineGroupId": "#{pipeline().GroupId}",
"PipelineTriggerId": "#{pipeline().GroupId}",
"PipelineTriggerName": "#{pipeline().TriggerName}",
"PipelineTriggerType": "#{pipeline().TriggerType}",
"PipelineTriggeredByPipelineName": "#{pipeline()?.TriggeredByPipelineName}",
"PipelineTriggeredByPipelineRunId": "#{pipeline()?.TriggeredByPipelineRunId}"
},
"dataVersion": "1.0"
}]
Integration Runtime: Select any runtime
Authentication: System Assigned Managed Identity
Resource: https://eventgrid.azure.net/
Here is an image of the activity and pipeline:
Data Factory Pipeline Image
We are working on implementing a custom logging solution. Most of the information what we need is already present in log analytics from data factory analytics solution but for getting log info on data flows, there is a challenge. When querying, we get this error in output. "Too large to parse".
Since data flows are complex and critical piece in a pipeline, we are in desperate need to get data like rows copied, skipped, read etc of each activities with in data flow. can you pls help how to get those info?
You can get the same information shown in the ADF portal UI by making a POST request to the below REST endpoint. You can find more information and read about authentication on the following link https://learn.microsoft.com/en-us/rest/api/datafactory/pipelineruns/querybyfactory
You can choose to query by factory or for a specific pipeline run id depending on your needs.
https://management.azure.com/subscriptions/<subscription id>/resourcegroups/<resource group name>/providers/Microsoft.DataFactory/factories/<ADF resource Name>/pipelineruns/<pipeline run id>/queryactivityruns?api-version=2018-06-01
Below is an example of the data you can get from one stage:
{
"stage": 7,
"partitionTimes": [
950
],
"lastUpdateTime": "2020-07-28 18:24:55.604",
"bytesWritten": 0,
"bytesRead": 544785954,
"streams": {
"CleanData": {
"type": "select",
"count": 241231,
"partitionCounts": [
950
],
"cached": false
},
"ProductData": {
"type": "source",
"count": 241231,
"partitionCounts": [
950
],
"cached": false
}
},
"target": "MergeWithDeltaLakeTable",
"time": 67589,
"progressState": "Completed"
}
I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.
I have a Pipeline Variable called SkipIndex that defaults to 0. Inside the Until loop I have a Copy Data Activity that works (HTTP source to BLOB sink), then a Set Variable Activity that I am trying to get to increment this Variable.
{
"name": "Add 50 to SkipIndex",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Copy next to temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "SkipIndex",
"value": {
"value": "50++",
"type": "Expression"
}
}
}
Everything I have tried results in errors such as "The expression contains self referencing variable. A variable cannot reference itself in the expression." and the one above with 50++ causes a sink error during debug.
How can I get the Until loop to increment this variable after it retrieves data?
Agree that REST Connector does supports pagination but does not for Client Certificates Authentication type.
For the idea of your Until activity scenario,i am tripped by the can't self-reference a variable in an expression limitation also. Maybe you could make a little trick on that: Add one more variable to persist the index number.
For example,i got 2 variables: count and indexValue
Until Activity:
Inside Until Activity:
V1:
V2:
BTW, no usage of 50++ in ADF.
I'm using ADFv2 to transfer some data. As a part of this operation I need some configuration values to pass into the pipeline.
The config values must be pulled at runtime from a REST service - not as parameters.
I can successfully query the REST service with Web Activity and I can see the output in the debug view.
Now the problem :)
How do I use this output in other activities further in the pipeline?
My Web Activity configuration is like this:
{
"name": "Web1",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"url": "https://myazurefunction.azurewebsites.net/api/MyFunction",
"method": "GET",
"headers": {
"Content-Type": "application/json"
}
}
I have tried to access the output after is has executed, but it seems empty:
#activity('Web1').Output
#activity('Web1').output
#string(activity('Web1').Output)
they are all empty. Any suggestions?
Thanks!
I set up an ADF2 and try to get a response.
This works for me:
#string(activity('Post').output)
Have you checked the output in the debugging?
Here is my output:
{
"test": {
"value": 123,
"text": abc
},
"concat": 123abc
}
I use the stored procedure to insert the values into the destination table on a Logical Server.
In ADFv2, you access the output of previous activities using #activity('ActivityName').output.
For the web activity defined, the response from your function should be in JSON format, so you would reference specific JSON values using their attribute names in the response. For example, your defined web activity, named Web1, calls a function that returns a response of:
{
"foo": "bar",
"some": "value"
}
To use the value of foo in a subsequent ADF activity, you would reference #activity('Web1').output.foo. ADFv2 provides multiple type conversion functions, should you need the returned value converted to another type.
If your function is returning an empty JSON response back, you may want to inspect the response from your function using Postman or another tool to ensure you are returning a properly formatted response, and that your function isn't failing for another reason.
Inside your Azure function code, you should be returning a JSON object, along with a success code, similar to return req.CreateResponse(HttpStatusCode.OK, json);.
Also note that if you reference a property of the response and it does not exist, ADF will fail at that point, so you can use an If Condition activity to check for the required values to better handle failures in ADFv2.
I am able to retrieve records for a particular Incident ID using Invoke-RestMethod. However, while retrieving the data, values like Resolved To, Updated By, etc. get populated by a sysid.
Resolved By comes in this format:
https<!>://devinstance.servicenow.com/api/sysid, value= sysid
I would like to view the username instead of the sysid.
The 'User ID' (user_name) isn't on the Incident, it's on the sys_user table, so you'll have to dot-walk to it.
If you're using the table API, you'll need to specify a dot-walked field to return, using the sysparm_fields query parameter.
This is no problem, just specify your endpoint like this:
$uri = "https://YOUR_INSTANCE.service-now.com/api/now/table/incident?sysparm_query=number%3DINC0000001&sysparm_fields=resolved_by.user_name"
I've specified a query for a specific incident number is requested, but you can replace that with whatever your query is.The important part is sysparm_fields=resolved_by.user_name. You'll want to specify any other fields you need here, as well.
The JSON I get as a result of running this API call, is the following:
{
"result": [
{
"resolved_by.user_name": "admin"
}
]
}
Note the element name: "resolved_by.user_name".
Another option for doing this, would be to tell the API to return both display, and actual values by specifying the sysparm_display_value parameter and setting it to all to return both sys_id and display value, or just true to return only display values.
Your URI would then look like this:
https://dev12567.service-now.com/api/now/table/incident?sysparm_query=resolved_byISNOTEMPTY%5Enumber%3DINC0000001&sysparm_display_value=all
And your JSON would contain the following:
"number": {
"display_value": "INC0000001",
"value": "INC0000001"
},
"resolved_by": {
"display_value": "System Administrator",
"link": "https://YOUR_INSTANCE.service-now.com/api/now/table/sys_user/6816f79cc0a8016401c5a33be04be441",
"value": "6816f79cc0a8016401c5a33be04be441"
},
"sys_updated_by": {
"display_value": "admin",
"value": "admin"
},
This would be accessed by:
answer.result[n].resolved_by.display_value