All,
I am trying to pull all data from successfactors using oData API but it give only 1000 rows at a time
How to pull all the data at a time?
Thanks
The OData API can return a maximum number of 1000 records in a single page.
To get the next page of your dataset, you have to call the __next API (which carries a skip token) from previous response. Repeat this action until you get all data via API.
Related
I have a API source in an ADF DataFlow task. The API source gives me the current page and the toatl number of pages in the body of the response. I want to use that information to paginate through my API source. I'm able to paginate through it just fine outside of a DataFlow activity using the range function. The issue is that the Rest transformation in a DataFlow activity does not support the range function. I've been trying to use the AbsoluteUrl function plus an expression to do add one to the current page returned by the body but either pagination does not accept expressions or I cannot figure out the syntax
I have a url like this:
BaseURL/fabricationcodes?facets=relatedArticles:Not%20Empty&page={PageNumber}.
In this example my rest linked service URL has everything I need minus the &page=pageNumber. So I'm trying to add that part with the key/value pair function of AbsoluteUrl. The Key being &page= and the value should be currentPage +1. My desire is for it to get the first page, page 0 and then add +1 to that to formulate the next pages url. the end condition being when body.totalPages == body.currentPage
I've tried a bunch of different expression formaulations but none seem to work and debugging in a Data flow is tough b/c the logging and error messaging is poor
What I have right now.
As data flow don't support Range option or you cannot use dynamic expression to get page from API response.
To work around the issue, you can use Data Flow activity within ForEach loop using range function in dynamic expression.
First take a web activity and pass the URL of the Rest API as below Ito get the total no of Pages from API response
then take a for each activity to iterate on API like pagination give the Dynamic expression as #range(1,activity('Web1').output.total_pages)
I will iterate the API till the respective range in sequential manner.
create parameter with type string in source DataSource.
give that parameter as dynamic value in relative URL.
after this gave parameter value as ?page=#{item()} to give the no coming from range to the page.
OUTPUT:
I have a project scenario to get all values from an endpoint URL. I am using ADF Pipeline but I'm having some issues with pagination.
To get the following values, I need to make requests with the PaginationCursor value in the current body response in the following request header.
I have read that ADF supports the following case, which would be mine.
Next request’s header = property value in current response body ADF - Pagination support
I don't know how to use the following attributes in order to use the paginationCursor value from the current response body in the header of the next request.
Attributes for pagination in ADF
I tried to reproduce above but not successful. Instead, if you want to do it without pagination, you can try this approach.
First create a web activity with any page URL of your API to get the total number of pages count.
In ForEach create an array for page numbers using the count from web activity as
#range(1,activity('Web1').output.total_pages)
Inside ForEach use the copy activity and give the source REST dataset parameter for the page number like ?page=#{item()}.
In the sink dataset also, create a dataset for each page with the dataset parameter value like APIdataset#{item()}.csv. This generates the sink dataset names like APIdataset1.csv, APIdataset2.csv,...
Now, you can copy from your REST API without pagination.
My repro for your reference:
Copy activity:
I could solve this problem with the following attributes.
Solution
In the Headers I had to put the name of the header of the next call. In my case the name is PaginationCursor and I got the value of this header from the actual body response called paginationCursor.
I need to fetch items from JAMA API, and I do not get the Tag value with the data.
I tried to call GET/{item}/{id} API and then I also need to execute GET/{tags}/{id}/{item} to discover item tag, which takes too much time for multiple items.
Is there any more efficient way to get all data in one API call?
There is the /items/{id}/tags endpoint: https://rest.jamasoftware.com/#operation_getTagsOnItem
For the Marketing API, I know that I'm able to make one call to retrieve all of the adsets from a certain account along with their insights, but am I able to specify the date_preset for the insights edge in that same call?
For example, the following gives me lifetime insights stats:
/v2.4/{accountID}/adcampaigns?fields=insights
To be clear - I know this is possible to retrieve by making separate calls for each adset id (where I know I can specify the date_preset); instead, I'd like to do this via the call where I get a long list of the ad sets plus their insights details in one go.
Yes this is possible using query expansion, however you probably should not do it in this anyway.
Using query expansion results in multiple requests being executed in one HTTP call, in this case one to get all the adcampaigns, and then N requests where N is the number of adcampaigns returned. This will in turn affect your rate limiting.
The most efficient way to request all insights for all adcampaigns (ad sets) is instead to request them at the account level, specifying aggregation level:
/v2.4/act_{ADACCOUNT_ID}/insights?date_preset=last_7_days&level=campaign
This requires just 1 request, or the number of requests to retrieve the total number of pages.
If you really want to achieve this with query expansion, you can do the following for example:
/v2.4/act_{ADACCOUNT_ID}/adcampaigns?fields=insights.date_preset(last_30_days).time_increment(all_days)
You can see the parameters to insights that would normally be query parameters of the form param_name=param_value are now in the form of param_name(param_value).
To specify the date_preset , here is the correct format . Its important to use insights as edge to get the date_preset filtering .
/v2.10/act_{ADACCOUNT_ID}/insights?fields=impressions,clicks,ctr,unique_clicks,unique_ctr,spend,cpc&date_preset=last_3d
The above one is tested with latest Graph Api version(2.10) as of now . FOr more info related to the date_preset values refer to there api docs .
https://developers.facebook.com/docs/marketing-api/insights/parameters
I would like to record the various GET requests to my API in a table and use that table as part of the calculation of what to return for future GET requests.
Perhaps the easiest test example would be a GET function that returns the number of GET requests in the last hour.
The REST protocol says that GET requests should only have data returns.
Do I need to POST the request and then GET the results of the same request?
You can easily achieve that with nodejs
You should have the requests saved in a json file or database for example and have another service that returns this saved data.
Take a look at expressjs
Best luck