We're trying to put together a proof-of-concept where we read data from an API and store it in a blob. We have a For Each activity that loops through a file that has parameters that are used in an API call. We are trying to do this in parallel. The first API call works fine, but the second call returns a 429 error. Are we asking the impossible?
Usually error code 429 meaning is too many requests. Inside ForEach activity, use Sequence execution option and see if that helps.
Related
I am using input as rest api url .And I am trying to save the response to a sql table.When I run the pipeline the pipeline run successfully,But it is showing zero rows copied.
I tested the api in postman.I am able to see the reponse data (9 mb)
Anybody else got this same issue,Please help me
I tried to reproduce and faced similar problem Its not inserting any records.
The problem is causing due to API returns response in Json and pipeline doesn't know which object value should store in which column.
To resolve this use Mapping. import the scma and amp the paricular columns as below:
Output:
I think the intend here is copy the response json to SQL and if thats the case then we cannot do that with copy activity .
One way is you can use a web activity to call the API and after that you can call a Stored proc activity and pass the response as a input paramter to the SP . The SP will insert the record in the table . But 9MB of response is too big , i doubt if the web activity can handle that .
The API call:
https://dev.azure.com/{organization}/{project}/_apis/test/Plans/{planId}/Suites/{suiteId}/points?api-version=7.0
Returns JSON data with objects "LastTestRun" and "LastTestResult" that contain their Ids. I am trying to find a proper way of getting older (not last) data of TestResults of the given TestPoint using API calls.
I tried the following API call:
https://dev.azure.com/{organization}/{project}/_apis/test/runs?planId=123
That returns me a list of all TestRuns in the given TestPlan. I can loop through the list and make API call for every single TestRun:
https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results
And, then retrieve the TestPointId. The problem is I'm working with large amounts of data, so I'm looping through 7000 API calls and it takes too much time (parallel of course).
I tried calling the last API call with $select so I can retrieve only the TestPointId but it is not supported.
A version of the AzureDevops I am working with: Version Azure DevOps Server 2020 Update 1.2.
Based on your expectation, I have tested as well to get test results with only test point value and tested get the runs of a specific test case in a test suite. Neither succeeded.
Currently there seemed no official API support the test point filter. For this, you may access this URL: https://aka.ms/AzDevOpsIdeas to submit any comments and proposals for future releases and implementations.
Hi i am processing a set of ~50K records from a pipe delimeted flat kn azure data factory and need to invoke a rest API call for each input record. So, I am using a foreach loop to access each record and inside the loop, I am using a copy activity to invoke a rest API call.
My question is, can I invoke the rest API call in bulk for all the records at once, as the foreach loop is slowing the pipeline execution. I want to remove the foreach loop and also process the API json response and store it in azure sql database.
Thanks
You will have to check the Pagination properties so that you can decide how much payload you need to return from source API:
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support
Also, if you need to store the API JSON response in Azure SQL, then you can do so with many built in functions like JSON_PATH
More details can be found in this link:
https://learn.microsoft.com/en-us/azure/azure-sql/database/json-features
Using the Amplitude Dashboard REST API, I am attempting to get a count of all unique times a custom event was triggered with a filter of a custom event property. However I am unable to even get the simplest event segmentation to run, though all other endpoint hits (except Funnels which also use the e event parameter) are working as expected. In other words, my auth is working and I'm able to successfully get data from all the endpoints that don't require the e event parameter.
Here is an example of a constructed endpoint using event segmentation that is as simple as I believe it could possible be and which is
failing with a 400 error.
https://amplitude.com/api/2/events/segmentation?e=\{"event_type":"_active"\}&start=20170401&end=20170402
While the call I want to do is ultimately more complex and involves a filter, I'm unable to just get this call which is one of the simplest event segmentation calls possible, considering the e, start, and end parameters are all required.
Try properly percent escaping the e parameter.
https://amplitude.com/api/2/events/segmentation?e=%7B%22event_type%22%3A%22_active%22%7D&start=20170401&end=20170402
I want to read logs from server until a certain time and return via an api endpoint (e.g., like tail). Since the file can be long, I want to return some output continuously
Is it possible to return some output continuously or does one api call only returns discretely in one step?
FWIW, I am using scalatra.