I am attempting to use the rest client to query a webservice for data. The flow is as follows:
POST request with the query that returns a cursor Id (Get Initial Cursor)
GET request with the cursor ID to retrieve the first batch of 5000 rows
Along with the data, the response contains the next and previous cursor id (Extract Cursor Id, Get data with cursor)
Use the next cursor from the request in step 2 to get more data and possibly another next cursor.
If next cursor is missing from the response, no more data is available.
What is the best way to implement a looping construct to keep retrieving data till 'next cursor' is unavailable?
Below is the diagram of my transformation:
Related
So, I am creating a Copy activity that reads from SQL Server table and have to send the data to an API end point with the PATCH request.
API provider specified that the body must be in the form of
"updates":[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
However, my sql table maps to json this way (without the wrapper 'updates:')
[{"key1":"value1","key2":"value2","key3":"value3" },
{"key1":"value1","key2":"value2","key3":"value3" }, ...
.... {"key1":"value1","key2":"value2","key3":"value3" }]
I use the copy activity with the sink data set being of type Rest ..
How can we modify the mapping, so that schema gets wrapped by "updates" object ?
Using copy data activity, there might not be any possibility to wrap the data (array of objects) to an updates key.
To do this, I have used a lookup activity to get the data, set variable activity to wrap the data with an updates object key and finally, use Web activity with PATCH method and above variable value as body to complete the activity.
The following is the sample data I have taken for my SQL server table.
Use look up activity to select the data from this table using table or query option (I used query option). The debug output would be as follows:
NOTE: If your data is not same as in sample table I have taken, try using the query option so the output would be something as shown below
In the set variable activity, I have used an array variable and used the following dynamic content to wrap the above array of objects with updates key.
#array(json(concat('{"updates":',string(activity('Lookup1').output.value),'}')))
Now in the Web activity, choose all the necessary settings (PATCH method, authorizations, headers, URL, etc.,) and give the body as follows (I used a fake REST api as a demo):
#variables('tp')[0]
Since I am using the Fake REST API, the activity succeeds, but checking the Web activity debug input shows what is the body that is being passed to the Rest API. The following is an image for reference:
I'm new to getStream.io and i'm retrieving feed from flatfeed don't know how to paginate response in flutter.I know we pass offset and limit but how could I know to change offset value dynamically.
Paginated methods such as flatFeed.getPaginatedEnrichedActivities return a json response with a field "next". This field is something of this order:
"/api/v1.0/enrich/feed/user/sacha/?api_key=$keyField&id_lt=$idLtField&limit=$limitField"
Essentially you need to parse the URL parameters limit and id_lt in order to query the next page and so on until "next" is null.
We have simplified this process in the core package with loadMoreEnrichedActivities. In the tutorial there is an Activities Pagination section where you can read more about it
I am using smartsheet as a database and I want to query the smartsheet by column name equals value like in sql db for example: To get the particular row of Employee sheet where salary equal to 10000. But documentation describes only how to get list of rows and how to update and delete rows by row id.
https://smartsheet-platform.github.io/api-docs/?java
What i want is to achieve without knowing id of the row. But I can do by search function by searching the salary of the employee
https://api.smartsheet.com/2.0/search?query=10000
and the response of above call will have row id and again i should make a call with rowid to get that row by below call which I don't want.
GET /sheets/{sheetId}/rows/{rowId}
Can anyone help me out?
Although some folks do try to use Smartsheet like a database, it's not really designed to be queried via API in the same way that you'd use transact SQL to query a SQL database.
I don't believe the scenario that you've described (i.e., using the Smartsheet API to retrieve row(s) in a sheet where [column value] = [specified value]) is possible. Instead, you'll have to use the API to get the data in the sheet (Get Sheet) and then in your code, query the data that you received in the API response (to identify the row(s) that match your query criteria).
I am following a sample for Azure DocumentDB below. In the sample, C# code queries for documents in the DocumentDB.
https://github.com/Azure/azure-documentdb-dotnet/blob/master/samples/rest-from-.net/Program.cs
Line 182:
var qry = new SqlQuerySpec { query = "SELECT * FROM root" };
var r = client.PostWithNoCharSetAsync(new Uri(baseUri, resourceLink), qry).Result;
The problem is the result 'r' only contains the first 100 documents. If I use the Client SDK, I can get more than 100. I tried using stream, but had no luck so far. Any help would be appreciated!
For a SQL query the results are returned in segments if the result set is too large. The results are returned in chunks of 100 items or 1 MB (whichever limit is hit first) by default.
You can either use continuation tokens to get each segment after another. Or you set the x-ms-max-item-count custom header in a request to increase the limit to an appropriate value.
You can have a look the the REST API for further details.
For the sample program you have to add the line
client.DefaultRequestHeaders.Add("x-ms-max-item-count", "1000");
in order to get 1000 documents instead of 100.
I'm just guessing here, but it might be worth a shot. Here's the documentation from MSDN that describes the List action:
https://learn.microsoft.com/en-us/rest/api/documentdb/list-documents
In the "Headers" section under "Response" it is mentioned that you might get an optional token in the header "x-ms-continuation". Based on the description you have to issue another GET request with this token specified to get the other elements of the result set.
Can you check whether you get a header like this in the response? If so, you can issue another get request with this token specified (see the same documentation page under "Request").
From a MySQL query, I am trying to iterate on every rows to:
Test the existence of an object using a webservice
Then if the object doesn't exist yet, I will add it using another webservice, otherwise I do nothing.
Update in any case a MySQL DB table.
Here is what I've done so far:
First I create the connection
I get the columns I need from my MySQL query
I start iterating
I call my webservice which give me a String holding XML (column name: ResultXML)
I extract from this XML the values I need (column name: ResultExtract)
Then I want to test if one of the values extracted is equal to a specific value (let's say 10).
So from my my tExtractXMLField, I get something like :
150
10
So this would mean that I won't have called the 2nd webservice.
I can't find how to look for my value in the result (the result can also be empty and then no row would come from the tExtractXMLField).
I tried to use a tJava after the tExtractXMLField component to split my result as a List but the tJava component is running before the 1st webservice has finished so I get NULL.
Maybe there is a better way to do what I've already done, and if so, I am also pleased to receive your recommandation.