Azure data factory - custom mapping for Rest service - azure-data-factory

So, I am creating a Copy activity that reads from SQL Server table and have to send the data to an API end point with the PATCH request.
API provider specified that the body must be in the form of
"updates":[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
However, my sql table maps to json this way (without the wrapper 'updates:')
[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
I use the copy activity with the sink data set being of type Rest ..
How can we modify the mapping, so that schema gets wrapped by "updates" object ?

Using copy data activity, there might not be any possibility to wrap the data (array of objects) to an updates key.
To do this, I have used a lookup activity to get the data, set variable activity to wrap the data with an updates object key and finally, use Web activity with PATCH method and above variable value as body to complete the activity.
The following is the sample data I have taken for my SQL server table.
Use look up activity to select the data from this table using table or query option (I used query option). The debug output would be as follows:
NOTE: If your data is not same as in sample table I have taken, try using the query option so the output would be something as shown below
In the set variable activity, I have used an array variable and used the following dynamic content to wrap the above array of objects with updates key.
#array(json(concat('{"updates":',string(activity('Lookup1').output.value),'}')))
Now in the Web activity, choose all the necessary settings (PATCH method, authorizations, headers, URL, etc.,) and give the body as follows (I used a fake REST api as a demo):
#variables('tp')[0]
Since I am using the Fake REST API, the activity succeeds, but checking the Web activity debug input shows what is the body that is being passed to the Rest API. The following is an image for reference:

Related

Translating a GUID to a text value, from an API response in a Power Automate Flow

I'm using MS Automate to solve an integration challenge between two systems we use in our Project Management lifecycle. I have a custom connector written by the vendor of System A which allows me to create a Flow in MS Automate which is triggered when a record is Created or Updated.
So far, so good. However, the method in the connector provided by System A returns the new or updated record containing a number of fields which contain value GUIDs as the fields are 'choice' type fields e.g. Department, Status etc. What I end up with is a record where Status = "XXXXXX-000000-00000-00000" etc. The vendor also provides a restful API endpoint which I can query, which returns a JSON collection of fields, which include a 'Choices' section for each field of this type which is a standard JSON which looks like:
{
"Id": "156e6c29-24b3-4413-af91-80a62a04d443",
"Order": 110,
"InternalName": "PrjStatus",
"DisplayName": "Status",
"ColumnType": 5,
"ColumnAggregate": 0,
"Choices": {
"69659014-be4d-eb11-bf94-00155df8457c": "(0) Not Set",
"c30c50e5-51cf-ea11-bfd3-00155de84703": "(1) On Track",
"c40c50e5-51cf-ea11-bfd3-00155de84703": "(2) At Risk",
"c50c50e5-51cf-ea11-bfd3-00155de84703": "(3) Off Track",
"6a659014-be4d-eb11-bf94-00155df8457c": "(4) Not Tracked"
},
Technical problem:
What I have is the GUID of the choice (not the field). I need to take the GUID, in this case "6a659014-be4d-eb11-bf94-00155df8457c" and translate it into "(4) Not Tracked" and store this in a variable to write to a SharePoint list. I need to do this for about 30 fields which are similar in the record.
I've created the flow and the connector has given me the record with a list of fields, some of which contain value GUIDs. I know which fields these are and I have the Display Names of these fields.
I have added a HTTP call to the provided API endpoint (lets call it GetFields), which has returned a 200 response, the body of the response containing a JSON collection of the 50 or so fields in System A.
I can't work out how to parse the body of the response for the GUID I have for each field value and ensure I have the right corresponding text value, so I can then write it to a field variable, and then create a SharePoint record, all wrapped up in an MS Automate flow.
I hope I've understood you correctly but from what I can work out, you want to dynamically select the value of the choice from the GUID you've been provided (by whatever means).
I created a small flow to prove the concept. Firstly, these two steps setup the scenario, the first being the GUID you want to extract the choice value for and the second being the JSON object itself ...
The third step will take the value from the first variable and use it dynamically in an expression to extract that key from the JSON and return the value.
This is the expression ...
variables('JSON')?['Choices'][variables('Choice ID')]
You an see I'm just using the variable in the path expression over the top of the JSON object to extract the key I want.
This is the end result ...

How to send SQL table data with Web Activity of Azure Data Factory?

I have Lookup with query "Select CustomerName, CustomerSales From Customers"
I have Web Activity for Notification purpose and I would like to send list of CustomerNames and Customer Sales. (For example Tesla, 1234000. There are 1-50 rows)
I can successfully send static text with Web Activity, but don't know how to build body for sending all data.
How to send data from table with Web Activity?
I assume you have unselected "firs row" only option inside Lookup activity. If yes, then your lookup data will actually come as array in output json of lookup activity.
So, you should consider writing below expression to get that array.
#activity('Fetch').output.value

Syntax to retrieve the specific object (Ex: server, group etc) value in below json data in azure data factory

I have a row and in that we have a column called data. The below json is present in the data column. So I want to retrieve single object from json like name present inside the survey node.
{"build":{"version":"8.3.4.23"},"survey":{"database":"abc","id":952,"name":"ravi","server":"ABCD0045","group":"JKLMN"}}
I will provide survey and name as the input parameters and my output has to be ravi. is there any syntax to get this in azure data factory dataflow activity.
You can use Parse transformation in Data Flow to parse your data column.
Screenshot:
Then you can use this expression json.survey.name to get the value ravi.

How to use Moodle REST API Call mod_data_add_entry (and parameters)?

I'm trying to set up a webpage that communicates with a Moodle page. I need different data from a database activity and want to create new entries. Note that I am not talking about the SQL database in BG, it is the activity database in courses.
The information should be retrieved/transferred via the REST API, an HTML POST Request. My problem is that I don't know how to add a new record to the database activity because I cannot transfer the data array. Only the first parameter given appears in my database.
E.g. i tried ...&wsfunction=mod_data_add_entry&databaseid=10&data[0][fieldid]=66&data[0][value]=12&data[1][fieldid]=67&data[1][value]=test
And many other combinations. Always only the first parameter is shown in the database.
The docs tell me this (Pseudocode):
//The fields data to be created
list of (
object {
fieldid int //The field id.
subfield string Default to "" //The subfield name (if required).
value string //The contents for the field always JSON encoded.
}
)
Alternatively:
REST (POST parameters)
data[0][fieldid]= int
data[0][subfield]= string
data[0][value]= string
I cannot find anywhere else something called a "subfield".
Any ideas?
Okay, found it. You have to put your values in "", unless they are not a number. Seems like there is a connection with this special activity because you don't have to do it elsewhere.

Referencing JSON payload value in Azure Data Factory for If condition

I have a Json file like so as a payload returned from an API call - which is an http dataset type in data factory.
{
"count": 2,
"name": "DatasetABC",
"columnNames": [
"Column_1",
"Column_2"
],
"rows": ["1234",
"5678"
]
}
I would like to be able to use the count records returned in an If condition. Im wondering what I need to use to get the value of "count" which is 2.
Any help appreciated.
Based on your description, i suppose you could use LookUp Activity in Azure Data Factory.
Lookup activity can retrieve a dataset from any of the Azure Data Factory-supported data sources. Use it in the following scenario:
Dynamically determine which objects to operate on in a subsequent
activity, instead of hard coding the object name. Some object examples
are files and tables. Lookup activity reads and returns the content of
a configuration file or table. It also returns the result of executing
a query or stored procedure. The output from Lookup activity can be
used in a subsequent copy or transformation activity if it's a
singleton value. The output can be used in a ForEach activity if it's
an array of attributes.
For example,maybe you could access the count value by using #{activity('MyLookupActivity').output.firstRow.count} in the IF activity.