I'm trying to pull data from Hubspot to my SQL Server Database through an Azure Data Factory pipeline with the usage of a REST dataset. I have problems setting up the right pagination rules. I've already spent a day on Google and MS guides, but I find it hard to get it working properly.
This is the source API. I am able to connect and pull the first set of 20 rows. It gives an offset which is usable with vidoffset= which is returned in the body.
I need to return the result of vid-offset from the body to the HTTP request. Also the process needs to stop when has-more results in 'false'.
I tried to reproduce the same in my environment and I got the below results:
First I create a linked service with this URL: https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo&vidOffset
Then after I created the pagination end condition rule with $.has-more and absolute URL.
For demo purpose, I took sink as a storage account.
The pipeline run success full look at the below image for reference.
For more information refer this Ms Document
Related
The API call:
https://dev.azure.com/{organization}/{project}/_apis/test/Plans/{planId}/Suites/{suiteId}/points?api-version=7.0
Returns JSON data with objects "LastTestRun" and "LastTestResult" that contain their Ids. I am trying to find a proper way of getting older (not last) data of TestResults of the given TestPoint using API calls.
I tried the following API call:
https://dev.azure.com/{organization}/{project}/_apis/test/runs?planId=123
That returns me a list of all TestRuns in the given TestPlan. I can loop through the list and make API call for every single TestRun:
https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results
And, then retrieve the TestPointId. The problem is I'm working with large amounts of data, so I'm looping through 7000 API calls and it takes too much time (parallel of course).
I tried calling the last API call with $select so I can retrieve only the TestPointId but it is not supported.
A version of the AzureDevops I am working with: Version Azure DevOps Server 2020 Update 1.2.
Based on your expectation, I have tested as well to get test results with only test point value and tested get the runs of a specific test case in a test suite. Neither succeeded.
Currently there seemed no official API support the test point filter. For this, you may access this URL: https://aka.ms/AzDevOpsIdeas to submit any comments and proposals for future releases and implementations.
Timeout issue for http connector and web activity
Web activity and http connector on adf
We have tried loading data through Copy Activity using REST API with Json data some columns are getting skipped which is having no data at its first row. We have also tried REST API with cv data but it's throwing error. We have tried using Web Activity but its payload size is 4MB, so it is getting failed with timeout issue. We have tried using HTTP endpoint but its payload size is 0.5 MB, so it is also getting failed with timeout issue
In Mapping settings, Toggle on the advanced editor and give the respective value in collection reference to cross apply the value for nested Json data. Below is the approach.
Rest connector is used in source dataset. Source Json API is taken as in below image.
Then Sink dataset is created for Azure SQL database. Once the pipeline is run, few columns are not copied to database.
Therefore, In Mapping settings of copy actvity,
1. Schema is imported
2. Advanced editor is turned on
3. Collection reference given.
When pipeline is run after the above changes, all columns are copied in SQL database.
I am trying to incrementally load data from a ServiceNow data source into an Azure SQL table as per the guide from Microsoft https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-portal. This uses 2 lookup activities to find 2 dates, I then filter the source data in the copy activity to only return and insert rows where the sys_updated_on date is between the 2 lookup values.
I would now like to lookup a value from a REST API dataset. However, I do not get the option to choose my REST dataset in the lookup activity. It just does not appear as an option. The REST URL is setup to return me one date value which I need to pass into the WHERE clause of my source in the copy data. If I cannot retrieve this value in the lookup, how else can I pass it to my WHERE clause?
Currently I use activity('LookupOldWaterMarkActivity').output.firstRow.watermarkvalue and
convertTimeZone(activity('LookupNewWaterMarkActivity').output.firstRow.watermarkvalue
Thanks
As per the Microsoft official document, the Rest dataset is supported in lookup activity.
You can post feedback from an Azure Data factory or raise a support request for fixing the issue.
As a workaround, you can create an HTTP dataset with JSON format and use the output value in later activities.
I'm attempting to pull data from the Square Connect v1 API using ADF. I'm utilizing a Copy Activity with a REST source. I am successfully pulling back data, however, the results are unexpected.
The endpoint is /v1/{location_id}/payments. I have three parameters, shown below.
I can successfully pull this data via Postman.
The results are stored in a Blob and are as if I did not specify any parameters whatsoever.
Only when I hardcode the parameters into the relative path
do I get correct results.
I feel I must be missing a setting somewhere, but which one?
You can try setting the values you want into a setVariable activity, and then have your copyActivity reference those variables. This will tell you whether it is an issue with the dynamic content or not. I have run into some unexpected behavior myself. The benefit of the intermediate setVariable activity is twofold. Firstly it coerces the datatype, secondly, it lets you see what the value is.
My apologies for not using comments. I do not yet have enough points to comment.
I've set up an Azure Data Factory pipeline to transfer the data from one table in our SQL Server Database to our new Azure Search service. The transfer job continuously fails giving the following error:
Copy activity encountered a user error at Sink side:
GatewayNodeName=SQLMAIN01,ErrorCode=UserErrorAzuerSearchOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error
happened when writing data to Azure Search Index
'001'.,Source=Microsoft.DataTransfer.ClientLibrary.AzureSearch,''Type=Microsoft.Rest.Azure.CloudException,Message=Operation
returned an invalid status code
'RequestEntityTooLarge',Source=Microsoft.Azure.Search,'.
From what I've read thus far, Request Entity Too Large error is a standard HTTP error 413 found inside REST API. Of all the research I've done though, nothing helps me understand how I can truly diagnose and resolve this error.
Has anyone dealt with this with specific context to Azure? I would like to find out how to get all of our database data into our Azure Search service. If there are adjustments that can be made on the Azure side to increase the allowed request size, the process for doing so certainly is not readily-available anywhere I've seen on the internet nor in the Azure documentation.
This error means that the batch size written by Azure Search sink into Azure Search is too large. The default batch size is 1000 documents (rows). You can decrease it to a value that balances size and performance by using writeBatchSize property of the Azure Search sink. See Copy Activity Properties in Push data to an Azure Search index by using Azure Data Factory article.
For example, writeBatchSize can be configured on the sink as follows:
"sink": { "type": "AzureSearchIndexSink", "writeBatchSize": 200 }