Azure Data Factory Rest API - With AccountCode, APIKey & Token - azure-data-factory

Question

I have a use case wherein, I need to connect to an API for data request.
The API requires a valid token to process the requests.
To generate the token, I have a accountCode & secret key
Assume BaseURL as
BaseURL - http://api.xxxxx.com/{accountCode}/data (Value of account needs to be passed)
**Below script in Python/Java needs to be run to fetch the dateToken & token
If we use Python 3.6 or above. Below is the code -**
--START-- {
import time
import requests
from hashlib import md5
account_code = "<account_name>"
key = "<api_key>"
actual_unix_time = int(time.time_ns() / 1000) # in milliseconds
TTL = 31536000000 # for 1 year
expiration_time = actual_unix_time + TTL
base_url = "https://api.xxxxx.com"
url = f"/{account_code}/data?fromDate=last6Hours&granularity=minute&type=ALL%2CVOD%2CLIVE&operation=reduceJoin&metrics=bufferratio"
pre_url = f"{url}&dateToken={expiration_time}"
token_generated = md5(f"{pre_url}{key}".encode('utf-8'))
token_value = token_generated.hexdigest()
request_url = f"{base_url}{pre_url}&token={token_value}"
response = requests.get(request_url)
print(response)
print(response.text)
} --END--
- If we use Java. Below is the code -
--START-- {
var key = pm.environment.get("NPAW-API-KEY");
var base_url = "https://api.xxxxx.com";
var url = pm.request.url.toString();
var path = url.replace(base_url, '');
var pre_url = pm.variables.replaceIn(path);
var moment = require('moment');
var actual_unix_time = moment().unix()*1000;
var TTL = 31536000000
var expiration_time = (actual_unix_time + TTL);
var pre_url = pre_url+"&dateToken="+expiration_time;
var token_generated = CryptoJS.MD5(pre_url + key).toString();
var token_value = token_generated;
var request_url = (base_url+pre_url+'&token='+token_value).toString();
}--END--
Example of how the final URL - https://api.xxxxx.com/kb-vivek/data?fromDate=today&granularity=hour&type=ALL,VOD,LIVE&operation=reduceJoin&metrics=views,playtime&dateToken=1699016056000&token=7a9c97a4d4f108d1d32be2f7f8d00731
I tried to use Postman, wherein, I could pass the above script in the Pre-Request script and set environment variables for accountCode & Secret Key and I was able to achieve the result as desired.
Question: How can I achieve this in Azure Data Factory?

Answer 1

To achieve the requirement, we need to use a combination of set variables and dataflows (to generate md5 hex string and store final url in a file).
First, I have created 4 parameters with values as shown below:
base_url: https://api.xxxxx.com
account_code: <account_name>
key: <api_key>
TTL: 31536000000
First, I have created a variable to build url. I used the following dynamic content:
/#{pipeline().parameters.account_code}/data?fromDate=last6Hours&granularity=minute&type=ALL%2CVOD%2CLIVE&operation=reduceJoin&metrics=bufferratio
Next, I have built the pre_url with the following dynamic content:
#{variables('url')}&dateToken=#{add(div(sub(ticks(utcNow()), ticks('1970-01-01')),10),31536000000)}
Now, to encode the string and convert it to md5 hex string, I have used dataflow. I have passed base_url, pre_url and key to the dataflow from the pipeline.
I have taken a sample csv file with only one row from blob storage (the data in this file does not matter but make sure it has 1 row only).
I have created a derived column to create final URL by concatenating base_url, pre_url and encoded md5 hex string. Use the following content:
$base_url+$pre_url+'&token='+md5(encode(concat($pre_url,$key)))
Now I am writing this data to a file by using output to single file option in the sink settings.
When I debug the pipeline, the file will be written to my storage account. The contents of the file will be as shown below:
NOTE:
Now since you want to generate date token once and use it for a year, I have written the data to a file.
You run this pipeline once and generate a file with required URL (as above). Anytime you want to access this URL, you can use look up activity to access the URL anywhere required.
I have used utcNow() to generate dateToken. But if you have any specific date in mind, you can simply use that in the correct format (in place of utcNow()).