I have difficulties creating an ADF pipeline that pulls data from a REST API using oAuth and writes it to azure blob storage in JSON format. I am a python developer and relatively new to adf. I will explain in more detail below.
I need to obtain an access token by posting a request to the token endpoint. This step works fine.
I need to pull the data from the API using a GET request in a copy activity. Also I need to include pagination. Is there something wrong on the way I define the pagination rules? "#odata.nextLink" is full description of the json key; is this correctly spelled in the value section?
I need to copy the data to a blob storage in json format. I created the sink dataset.
Upon running the pipeline it asks for 2 pipeline parameters, what am I supposed to write here?
I get the following error, what am I doing wrong?
Related
I want to perform some validation checks in ADF on my input data and any validation failures want to capture into Azure log analytics.
Can someone guide me how to capture the custom logs into log analytics through Azure Data Factory please.
Any example dataflow/pipeline would be very helpful.
Thanks,
Kumar
If I understand correctly you want to be able to get the Azure Monitor logs for ADF and query/store these logs?
Well the good news is most of the information you would want to see is already collected through Azure Monitor.
One of the simple methods to pull the information is to use the Azure Monitor REST API. You can then store the response into a file or table, or you can just query the api for specific pipelines or triggers etc.
Here is a link with example of Authorization and using the Azure Monitor API:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#authenticating-azure-monitor-requests
This is an example of the HTTP URL using the Azure Monitor REST API to get Activity Run data (Dynamic content syntax):
#{concat('https://management.azure.com/subscriptions/', linkedService().SubscriptionID, '/resourceGroups/', linkedService().ResourceGroupName, '/providers/Microsoft.DataFactory/factories/', linkedService().DataFactoryName, '/pipelineruns/', linkedService().RunID, '/queryActivityruns?api-version=2018-06-01')}
Here are all the different ADF Metrics that can be pulled from Azure Monitor:
https://learn.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor#data-factory-metrics
Here is an example of a REST Linked service in ADF that can be used to call the rest api:
You could then create a dataset that will pass all the values to the linked service so that you can call the API, and copy the response to a DB, like here:
This particular example is getting the status of a specific Pipeline RunID, but it can be much more broad than that.
Here is what the request body looks like so that I could filter to just failed pipelines in the last day:
I'm looking into this myself, as far as I can tell you would have to use a REST or HTTP connector to send a POST request to the HTTP Data Collector API in Log Analytics. More details here: https://learn.microsoft.com/en-gb/azure/azure-monitor/logs/data-collector-api
Need some guidance or suggestion
Scenario:
We are trying fetch a file using RestAPI and save it to Azure Data Lake.
Journey:
Login, GetToken, Invoke GetFile API, and save to AzureDataLake
We are trying 2 options
using LogicApp-- this work fine, but apparently this is not approved PaaS service.
Using Data Factory-- here we are facing some issue. We are able to invoke the RestEndpoint, but unable to parse the response into a file (pdf, img etc...). How can I achieve this?
Best,
Ashwin
Please change to use HTTP connector :
HTTP connector is generic to retrieve data from any HTTP endpoint, for example, to download files.
REST connector is used to copy the REST JSON response.
You can refer this documentation
I am trying to connect SalesForce via ConnectedAPP. I am successful to authorize and get the access and refresh tokens for the REST API operations.
I can able to post one lead into SF via API end points. I found that in the below URL's
I found bulk API documentation also.
But in that, Soap API and csv are being used to to the operations of bulk inserts.But i am trying to do using bearer token and JSON as the payload to push data.
Please provide me some samples or a approach to achieve this via REST API bulk inserts.
The Bulk API is not a SOAP API. Its endpoints accept payloads in XML and JSON format, and accepts batch data files in XML, JSON, and CSV formats.
You can use the access token supplied from your OAuth flow to authenticate to the Bulk API. You can choose to submit job details in either JSON or XML. See Prepare XML and JSON Files for details of the data file format when using JSON input.
There are multiple example implementations of the Bulk API available in open source projects, such as salesforce-bulk.
I am trying to get data from a 3rd party API into an Azure SQL DB using Azure Data Factory with out using SSIS.
This lead me down a rabbit hole and I have been searching for 3 days now and cannot find a solution.I must be missing something.
I have tried using Azure Data Factory and the copy data controls.
I then tried using power apps and cant find anything that helps.
I then tried custom connector, from scrach, from postman and from OpenAPI
I cannot get any of it to work!
I really thought this would be easier than this?
I have read almost all of the standard MicroSoft documentation and none of it helps with my specific scenario.
I have a third party web site that I get an authorisation token from using a username and password with grant-type=password
Using this token I then get JSON data from the site.
I want to get this data into my SQL DB in Azure.
Thats it!
Any help would be greatly appreciated.
Thanks...
PS:Next step is the same thing but API returns XML, rabbit hole 2...
Here is a official guide that import data from Azure Storage to Azure SQL Database using pipeline.
I think this guide will be helpful for you. Before you follow this guide to meet your requirement, you should import your JSON data from 3rd party API first. So you can use Azure logic App to finish this process quickly.
I have finished this process , it is easy and just few steps :
1. Getting access token from 3rd party identity provider(here I use Azure AD as identity provider, just for demo) using http request action:
2. Generally , we can get an access token from step1 , use this auth info to call your 3rd API to fetch json data using http request action directly:
The API I am using is a sample demo one, just reply a json as :
{
"name": "stan",
"id": "123"
}
Using create blob action to create a blob file with the json content from previous step (you should create a storage account with a blob container first , so that you can store files, here I name the json file with guid)
Run this logic App you can see that a json file has been created in Azure storage account with the json data from API :
So the steps left , you can just refer to the official guide to import data to your SQL, just selecting json as resource format in Configure source section.
If you have anything unclear , pls feel free to let me know. Hope it helps : )
I want to do a one time pull of everything in Wisconsin from this Web API Endpoint. https://developers.google.com/civic-information/docs/v2/representatives/representativeInfoByDivision
Then I want to load the data into an Azure DB that I created.
I've reviewed several articles and videos and I haven't made much progress. I currently have the following; an Azure subscription, a storage account, an Azure SQL Server and DB. I've tried using the Copy Data activity in my Data Factory. For the source Base URL I'm using https://www.googleapis.com/civicinfo/v2/representatives. The "Test Connection" is successful. For the next screen "Data Set" for the entry "Relative URL" I'm using the same thing and it's giving the following error. "Relative Url Fail to read from REST resource"
I expect to be able to map the data from the Google API to columns in tables that I create in my Azure DB.
Try https://www.googleapis.com/civicinfo/v2/ as base URL and representatives as relative URL.
And the end, relative URL will be concat with base URL. And try rest connector of ADF.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest