I am trying to load all paginated data from API (as shown below) to Qlik Sense server but it only gives first page data whereas my requirement is to load all the paginated data to the server.
I have to use `\XML/JSON script in the request body to achieve the goal.
However, tried with Offset pagination type but looks like Total records path is not specified by the API data provider.
What about not relying on Qlik to perform the pagination?
Instead manually loop through all possible pages and when there is no data - break the loop. You will have define the total pages (set to some silly number)
Pseudo code:
// set the total possible pages
let customTotalPages = 100000;
// loop through each page
for a = 1 to $(customTotalPages)
// set the pagenumber in the body
// to be the current iteration number
let body = "<body>
<method>GET_N08_LIST</method>
<params>
<fil></fil>
<pagenumber>$(a)</pagenumber>
</params>
</body>
";
// load the data
TemTable:
Load * From ApiEndpoint WITH CONNECTION (
BODY "$(body)"
);
// check if the api response have any rows
let tempTableRows = NoOfRows('TempTable');
// break the loop if there are no more pages to process
if $(tempTableRows) = 0 then
Exit
end if
// if the api response returned data
// then concatenate this data to main table (which is outside the loop somewhere)
Concatenate (MainTable) Load * Resident TempTable;
// drop the temp table in each iteration
Drop TempTable;
next
Related
So, I am creating a Copy activity that reads from SQL Server table and have to send the data to an API end point with the PATCH request.
API provider specified that the body must be in the form of
"updates":[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
However, my sql table maps to json this way (without the wrapper 'updates:')
[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
I use the copy activity with the sink data set being of type Rest ..
How can we modify the mapping, so that schema gets wrapped by "updates" object ?
Using copy data activity, there might not be any possibility to wrap the data (array of objects) to an updates key.
To do this, I have used a lookup activity to get the data, set variable activity to wrap the data with an updates object key and finally, use Web activity with PATCH method and above variable value as body to complete the activity.
The following is the sample data I have taken for my SQL server table.
Use look up activity to select the data from this table using table or query option (I used query option). The debug output would be as follows:
NOTE: If your data is not same as in sample table I have taken, try using the query option so the output would be something as shown below
In the set variable activity, I have used an array variable and used the following dynamic content to wrap the above array of objects with updates key.
#array(json(concat('{"updates":',string(activity('Lookup1').output.value),'}')))
Now in the Web activity, choose all the necessary settings (PATCH method, authorizations, headers, URL, etc.,) and give the body as follows (I used a fake REST api as a demo):
#variables('tp')[0]
Since I am using the Fake REST API, the activity succeeds, but checking the Web activity debug input shows what is the body that is being passed to the Rest API. The following is an image for reference:
I am following a sample for Azure DocumentDB below. In the sample, C# code queries for documents in the DocumentDB.
https://github.com/Azure/azure-documentdb-dotnet/blob/master/samples/rest-from-.net/Program.cs
Line 182:
var qry = new SqlQuerySpec { query = "SELECT * FROM root" };
var r = client.PostWithNoCharSetAsync(new Uri(baseUri, resourceLink), qry).Result;
The problem is the result 'r' only contains the first 100 documents. If I use the Client SDK, I can get more than 100. I tried using stream, but had no luck so far. Any help would be appreciated!
For a SQL query the results are returned in segments if the result set is too large. The results are returned in chunks of 100 items or 1 MB (whichever limit is hit first) by default.
You can either use continuation tokens to get each segment after another. Or you set the x-ms-max-item-count custom header in a request to increase the limit to an appropriate value.
You can have a look the the REST API for further details.
For the sample program you have to add the line
client.DefaultRequestHeaders.Add("x-ms-max-item-count", "1000");
in order to get 1000 documents instead of 100.
I'm just guessing here, but it might be worth a shot. Here's the documentation from MSDN that describes the List action:
https://learn.microsoft.com/en-us/rest/api/documentdb/list-documents
In the "Headers" section under "Response" it is mentioned that you might get an optional token in the header "x-ms-continuation". Based on the description you have to issue another GET request with this token specified to get the other elements of the result set.
Can you check whether you get a header like this in the response? If so, you can issue another get request with this token specified (see the same documentation page under "Request").
I am attempting to use the rest client to query a webservice for data. The flow is as follows:
POST request with the query that returns a cursor Id (Get Initial Cursor)
GET request with the cursor ID to retrieve the first batch of 5000 rows
Along with the data, the response contains the next and previous cursor id (Extract Cursor Id, Get data with cursor)
Use the next cursor from the request in step 2 to get more data and possibly another next cursor.
If next cursor is missing from the response, no more data is available.
What is the best way to implement a looping construct to keep retrieving data till 'next cursor' is unavailable?
Below is the diagram of my transformation:
I am using Azure Mobile Web Services for my backend data exposed via REST/JSON. I have not been able to locate documentation that states how many results are posted per page, and how to page through them, as I need to incorporate server side paging for my Angular app.
GITs API has something as follows:
Requests that return multiple items will be paginated to 30 items by default. You can specify further pages with the ?page parameter. For some resources, you can also set a custom page size up to 100 with the ?per_page parameter.
Is there anything similar in Azure's Mobile Web Service API/Does anyone know the results per page and how to page thru them? Ex. https://myrestcall.net/tables/articles?page=2
If you are using the Javascript client you can check out this page
How to: Return data in pages
By default, Mobile Services only returns 50 rows in a given request, unless the client explicitly asks for more data in the response. The following code shows how to implement paging in returned data by using the take and skip clauses in the query. The following query, when executed, returns the top three items in the table.
var query = todoItemTable.take(3).read().done(function (results) {
alert(JSON.stringify(results));
}, function (err) {
alert("Error: " + err);
});
Notice that the take(3) method was translated into the query option $top=3 in the query URI.
The following revised query skips the first three results and returns the next three after that. This is effectively the second "page" of data, where the page size is three items.
var query = todoItemTable.skip(3).take(3).read().done(function (results) {
alert(JSON.stringify(results));
}, function (err) {
alert("Error: " + err);
});
Again, you can view the URI of the request sent to the mobile service. Notice that the skip(3) method was translated into the query option $skip=3 in the query URI.
I have a TSQL stored procedure that can run for a few minutes and return a few million records, I need to display that data in an ASP.NET Grid (Infragistics WebDataGrid to be precise). Obviously I don't want return all data at once and need to setup some kind of paging options - every time user selects another page - another portion of data is loaded from the DB. But I can't run the SP every time new page is requested - it would take too much time.
What would be the best way to persist data from the SP, so when user selects a new page - new data portion would be loaded by a simple SELECT... WHERE from that temp data storage?
A few options
One:
If the user only pages forward then you could just hold the connection open and use a DataReader. Just .Read() as needed.
Two:
Create a #temp table using the userID as part of the name to store the results. I don't like this as if user aborts sometimes tables are left over. About 1/2 second hit to create and drop the #temp. Store the entire results or just the PK and create the page detail on demand.
Three:
Use a DataReader to read the the PK into a List<>. It is faster than you would guess. That List is only going to IIS (not to the browser). List can be referenced by ordinal [] and preserves the sort. Get the detail for a page as required. The problem here is where PK in (3,9,2,6) will not return them in that order. I use TVP to pass the order, PK so the page is sorted by order. I do exactly this and get pages loads for objects with 20 properties 40 rows at a time and it takes less than 1/2 second. Do one query per table (NOT one per row) then assemble assign properties in .NET. Use DataReader (not DataTable). And you can even run the reader on a backgroundworker and pass back the first page of PKs using progresschanged.
Have you look at Server Side Paging (article is 2005, but will work with 2008 and CTEs). Also - just wondering, is there any reason you are returning that many rows? I can't see a very good use of a human paging through a million records even if the page size was 1000.