I'm syncing a REST API where we don't have a next page, but we do have the total pagecount as a viable option. This loops over the REST API 8 times just fine, but the problem is, it's not incrementing the page query parameter by 1 each time, which results in it fetching the first page 8 times.
I've tried these variations of the baseurl & relativeurl:
baseurl: https://api.****.com/****/export/****.json?lastModifiedStart=****&lastModifiedEnd=****&page=1
baseurl: https://api.****.com/****/export/****.json?lastModifiedStart=****&lastModifiedEnd=****
baseurl: https://api.****.com/****
relativeurl: export/****.json?lastModifiedStart=****&lastModifiedEnd=****&page=1
relativeurl: export/****.json?lastModifiedStart=****&lastModifiedEnd=****
with this as the pagination rule:
"paginationRules": {
"QueryParameters.page": "RANGE:1:$.metadata.pagecount:",
"supportRFC5988": "true" }
Here's what the $.metadata returns from the api:
"metadata": {
"path": "export/****.json",
"receivedat": "2021-12-21T13:01:57-0500",
"endrow": 10000,
"resourcename": "export",
"isprivate": 0,
"page": 1,
"startrow": 1,
"timezone": "America/New_York",
"x-format-date": false,
"pagecount": 8,
"recordcount": 74081 }
What am I doing wrong?
EDIT:
Image Link of Setup
I found out that the paginationRule for QueryParameters was turning page into essentially a variable. This was one of the page requests it made:
baseurl: ...?lastModifiedStart=2021-01-01&lastModifiedEnd=2021-12-31&7Size=1000&7=1
This tells me that page was being treated as a literal parameter for the url and was incrementing 'page' to be 1-8 instead of the value of page, so to fix this, I used the following setup:
baseurl: ...?lastModifiedStart=2021-01-01&lastModifiedEnd=2021-12-31&pageSize=1000&page=pagenum
and this as the pagination rule:
"paginationRules": {
"QueryParameters.pagenum": "RANGE:1:$.metadata.pagecount:",
"supportRFC5988": "true" }
After doing this, the requests worked like a charm!
Related
working with a MEAN Stack and I have three GET requests for the same URL/Route. One is to get a generalised summary of long-term emotions, the other is to get a summary of emotions by dates entered, and lastly, a summary of emotions related to a user-entered tag associated with individual emotion entries.
My first GET request is throwing no issues but the second GET request throws an error: Cannot read property 'length' of undefined
The error points to the following line:
48| each emotion in dateEmotions
Below is the relative code associated with the error:
Jade
each emotion in dateEmotions
.side-emotions-group
.side-emotions-label
p.emotion-left= emotion.emotionName
p.pull-right(class= emotion.emotionLevel) (#{emotion.emotionLevel}%)
.side-emotions-emotion.emotion-left
GET Request
module.exports.emotionsListByDates = function (req, res) {
Emo.aggregate([
{ $match :
{ "date" : { $gte: ISODate("2018-04-09T00:00:00.000Z"), $lt: ISODate("2018-04-13T00:00:00.000Z") } }
}, { "$group": {
"_id": null,
"averageHappiness": {"$avg": "$happiness"},
"averageSadness": {"$avg": "$sadness"},
"averageAnger": {"$avg": "$anger"},
"averageSurprise": {"$avg": "$surprise"},
"averageContempt": {"$avg": "$contempt"},
"averageDisgust": {"$avg": "$disgust"},
"averageFear": {"$avg": "$fear"},
}}
], function (e, docs) {
if (e) {
res.send(e);
} else {
res.render('dashboard', {
title: "ReacTrack - User Dashboard",
pageHeader: {
title: "User Dashboard",
strapline: "View your emotional data here."
},
dateEmotions: docs
})
}
});
};
This question is already getting pretty long, but I have another GET Request pointed to that URL and it is not throwing any errors, and the only difference is that I am not matching the db records by date in that query. I can post the working code if need be.
Edit
After some experimenting, I am able to get each of the three routes working individually if I comment out the other two. It's when multiple routes pull in the multiple requests that causes issues. For example, here are the routes at present where the ctrlDashboard.emotionsListByDates is working:
// Dashboard Routes
//router.get(/dashboard', ctrlDashboard.emotionsListGeneralised);
router.get('/dashboard', ctrlDashboard.emotionsListByDates);
//router.get('/dashboard', ctrlDashboard.emotionsListByTag);
If I comment out two routes and leave one running, and comment out the respective each emotion in emotions each emotion in dateEmotions and each emotion in tagEmotions blocks in the Jade file and leave the correct one uncommented, then that route will work, it seems to be when I am firing multiple routes. Is this bad practice, or incorrect? Should all queries be in the one GET request if on the same URL?
Thanks.
Apologies, new to routing and RESTful APIs but after some researching into the topic, I now understand the fault.
I assumed that the URL used in routing was the URL you wanted the data to populate...which it still kinda is, but I thought if I wanted to populate the dashboard page, I had to use that exact route and I did not realise I could post the data to different URL routes and take the data from those URLs to populate the one page.
Fixed by adding /date and /tag to those routes and using AJAX to perform those requests and populate the main page.
Thanks all.
I have the same problem but I'm using React+Redux+Fetch. So is it not a good practice dispatch more the one request in the same time and from the same page to a specific url?
I would know what causes that problem. I've found some discussions about it could be a mongoose issue.
My code:
MymongooObject.find(query_specifiers, function(err, data) {
for (let i = 0; i < data.length; ++i) {
...
}
}
Error:
TypeError: Cannot read property 'length' of undefined
I'm using ADFv2 to transfer some data. As a part of this operation I need some configuration values to pass into the pipeline.
The config values must be pulled at runtime from a REST service - not as parameters.
I can successfully query the REST service with Web Activity and I can see the output in the debug view.
Now the problem :)
How do I use this output in other activities further in the pipeline?
My Web Activity configuration is like this:
{
"name": "Web1",
"type": "WebActivity",
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false
},
"typeProperties": {
"url": "https://myazurefunction.azurewebsites.net/api/MyFunction",
"method": "GET",
"headers": {
"Content-Type": "application/json"
}
}
I have tried to access the output after is has executed, but it seems empty:
#activity('Web1').Output
#activity('Web1').output
#string(activity('Web1').Output)
they are all empty. Any suggestions?
Thanks!
I set up an ADF2 and try to get a response.
This works for me:
#string(activity('Post').output)
Have you checked the output in the debugging?
Here is my output:
{
"test": {
"value": 123,
"text": abc
},
"concat": 123abc
}
I use the stored procedure to insert the values into the destination table on a Logical Server.
In ADFv2, you access the output of previous activities using #activity('ActivityName').output.
For the web activity defined, the response from your function should be in JSON format, so you would reference specific JSON values using their attribute names in the response. For example, your defined web activity, named Web1, calls a function that returns a response of:
{
"foo": "bar",
"some": "value"
}
To use the value of foo in a subsequent ADF activity, you would reference #activity('Web1').output.foo. ADFv2 provides multiple type conversion functions, should you need the returned value converted to another type.
If your function is returning an empty JSON response back, you may want to inspect the response from your function using Postman or another tool to ensure you are returning a properly formatted response, and that your function isn't failing for another reason.
Inside your Azure function code, you should be returning a JSON object, along with a success code, similar to return req.CreateResponse(HttpStatusCode.OK, json);.
Also note that if you reference a property of the response and it does not exist, ADF will fail at that point, so you can use an If Condition activity to check for the required values to better handle failures in ADFv2.
Greeting everyone, I have a datatable in my html page that I populated using REST API. I can create new row and also update or delete by selecting a row and clicking the edit or delete button.
But currently I am unable to delete update or delete multiple row at once due to url error,
e.g : PUT http://127.0.0.1:8000/dashboard/content_detail/5,7,9/ 404 (Not Found)
how can I split this this into several separate url with respective id when I update or delete.
e.g :
/dashboard/content_detail/5
/dashboard/content_detail/7
/dashboard/content_detail/9
Below is my code, any help is much appreciated thank you.
idSrc: 'id',
ajax: {
create: {
type: 'POST',
url: content_path,
data: function (content_data) {
var create_data = {};
$.each(content_data.data, function (id, value) {
create_data['name'] = value['name'];
create_data['description'] = value['description'];
create_data['category'] = value['category'];
});
return create_data;
},
success: function () {
content_table.api().ajax.reload();
}
},
edit: {
type: 'PUT',
url: '/dashboard/content_detail/_id_/',
data: function (content_data) {
var updated_data = {};
$.each(content_data.data, function (id, value) {
updated_data['description'] = value['description'];
updated_data['category'] = value['category'];
updated_data['name'] = value['name'];
});
return updated_data;
},
success: function () {
content_table.api().ajax.reload();
}
},
remove: {
type: 'DELETE',
url: '/dashboard/content_detail/_id_/',
data: function (content_data) {
var deleted_data = {};
$.each(content_data.data, function (id, value) {
deleted_data['id'] = id;
});
return deleted_data;
},
success: function () {
content_table.api().ajax.reload();
}
}
},
If you're going to allow the update of a large number of items at once, then PATCH might be your friend:
Looking at the RFC 6902 (which defines the Patch standard), from the client's perspective the API could be called like
PATCH /authors/{authorId}/book
[
{ "op": "replace", "path": "/dashboard/content_detail/5", "value": "test"},
{ "op": "remove", "path": "/dashboard/content_detail", "value": [ "7", "9" ]}
]
From a design perspective you don't want several ids in your url.
I would prefer single calls for each change, thinking in resources you only manipulate one at a time.
In case this is a perfomance issue, I recommend a special url marked with action or something simliar, to make clear this ist not REST.
In HTTP it is not required for information to only exist on a single resource. It is possible to have multiple resources that represent the same underlying data.
It's therefore not out of the question to create a resource that 'represents' a set of other resources that you wish to DELETE or PUT to.
I do agree that it might not be the most desirable. I think we tend to prefer having information only exist in a single part of tree, and I think we like to avoid situations where updating a resource effects a secondary resource's state. However, if you are looking for a strictly RESTful solution to solve this problem, I think it's the right way.
Therefore a url design such as:
/dashboard/content_detail/5,7,9/
Is not necessarily non-RESTful or goes against the HTTP protocol. The fact that you're getting a 404 on that URL currently has to do with your application framework, not the protocol (HTTP) or architecture (REST) of your API.
However, for cases such as these I feel I would personally be inclined to sometimes create a separate POST endpoint that, acting outside of REST like an RPC endpoint. Specifically for these types of batch requests.
I have a textbox bound to KendoUI autocomplete widget. The JS code looks like this:
var dataSourceImeSearch = {
type: "json",
transport: {
read: {
url: "#Url.Action("ImeSearch")",
contentType: "application/json; charset=utf-8",
type: "POST"
},
parameterMap: function (data, type) {
if (type == "read") {
if (data.filter) {
data = $.extend({ sort: null, filter: data.filter.filters[0] }, data);
} else {
data = $.extend({ sort: null, filter: null }, data);
}
return JSON.stringify(data);
} else {
return JSON.stringify({ model: data });
}
}
},
batch: false,
pageSize: 10,
serverPaging: true,
serverFiltering: true,
serverSorting: true,
schema: {
errors: "Errors",
data: "Data",
total: "TotalRecordCount",
model: myModel
},
error: function (e) {
if (e.errors) {
alert(e.errors);
}
}
};
$("#Ime").kendoAutoComplete({
dataTextField: "PunoIme",
filter: "contains",
minLength: 3,
dataSource: dataSourceImeSearch
});
I am experiencing a weird thing here. Autocomplete is working in terms that when I type the third character it runs to the server and gets JSON data back from there and shows first ten results. The thing is that this textbox is searching large datasets, so for some queries with say 4 characters result set can be more than 1000 items. For some reason the widget is not figuring out that there are more than 10 results and when I scroll down in the autocomplete dropdown which gets shown, it will not fire any search for a second page and so on. You can see that the serverPaging for data source is set to true, but this does not help.
Any help is appreciated. Thank you.
I found out after posting this question that Autocomplete widget does not allow paging by design. This was explained in the KendoUI forums by some Kendo employee as an example of poor UX (if you have autocomplete that needs paging). I would argue that, since in my opinion, the first use case of the autocomplete would be in case of a search of a person, and here I am doing exactly that. The only problem is that if you search by person's second name you can end up with hundreds of results after first 3 or 4 characters and you really need paging for that. If the Kendo people think that this is a case of a bad UX, this should be clearly mentioned in the Autocomplete documentation, and I could really not find any mentioning of it at a single place, and one would think that it might be a good idea to mention something like this to the people so that they don't have to waste all day trying to figure out what went wrong.
In my opinion one of the worst use case examples at all demos at KendoUI web demo pages is the Shared DataSource example, where if you type in 'ch' in the autocomplete textbox in the top, you will end up with 10 results in autocomplete, but 14 in the datagrid bellow. It really strikes me as stupid that nobody in Kendo sees this behavior as odd.
So my answer to my own question would be the following: DO NOT use autocomplete, except in some really really simple use case (I can't really think of a single one that would make sense). I ended up making a whole search form with 5 textboxes and search button in case where I hoped that I was going to be able to use 2 textboxes (one with autocomplete) and a search button.
You have set pageSize:10, which means that only 10 records are returned to the AutoComplete and its dataSource contains only 10 elements, I am afraid that automatic paging is not implemented by default
I want to support pagination in my RESTful API.
My API method should return a JSON list of product via /products/index. However, there are potentially thousands of products, and I want to page through them, so my request should look something like this:
/products/index?page_number=5&page_size=20
But what does my JSON response need to look like? Would API consumers typically expect pagination meta data in the response? Or is only an array of products necessary? Why?
It looks like Twitter's API includes meta data: https://dev.twitter.com/docs/api/1/get/lists/members (see Example Request).
With meta data:
{
"page_number": 5,
"page_size": 20,
"total_record_count": 521,
"records": [
{
"id": 1,
"name": "Widget #1"
},
{
"id": 2,
"name": "Widget #2"
},
{
"id": 3,
"name": "Widget #3"
}
]
}
Just an array of products (no meta data):
[
{
"id": 1,
"name": "Widget #1"
},
{
"id": 2,
"name": "Widget #2"
},
{
"id": 3,
"name": "Widget #3"
}
]
ReSTful APIs are consumed primarily by other systems, which is why I put paging data in the response headers. However, some API consumers may not have direct access to the response headers, or may be building a UX over your API, so providing a way to retrieve (on demand) the metadata in the JSON response is a plus.
I believe your implementation should include machine-readable metadata as a default, and human-readable metadata when requested. The human-readable metadata could be returned with every request if you like or, preferably, on-demand via a query parameter, such as include=metadata or include_metadata=true.
In your particular scenario, I would include the URI for each product with the record. This makes it easy for the API consumer to create links to the individual products. I would also set some reasonable expectations as per the limits of my paging requests. Implementing and documenting default settings for page size is an acceptable practice. For example, GitHub's API sets the default page size to 30 records with a maximum of 100, plus sets a rate limit on the number of times you can query the API. If your API has a default page size, then the query string can just specify the page index.
In the human-readable scenario, when navigating to /products?page=5&per_page=20&include=metadata, the response could be:
{
"_metadata":
{
"page": 5,
"per_page": 20,
"page_count": 20,
"total_count": 521,
"Links": [
{"self": "/products?page=5&per_page=20"},
{"first": "/products?page=0&per_page=20"},
{"previous": "/products?page=4&per_page=20"},
{"next": "/products?page=6&per_page=20"},
{"last": "/products?page=26&per_page=20"},
]
},
"records": [
{
"id": 1,
"name": "Widget #1",
"uri": "/products/1"
},
{
"id": 2,
"name": "Widget #2",
"uri": "/products/2"
},
{
"id": 3,
"name": "Widget #3",
"uri": "/products/3"
}
]
}
For machine-readable metadata, I would add Link headers to the response:
Link: </products?page=5&perPage=20>;rel=self,</products?page=0&perPage=20>;rel=first,</products?page=4&perPage=20>;rel=previous,</products?page=6&perPage=20>;rel=next,</products?page=26&perPage=20>;rel=last
(the Link header value should be urlencoded)
...and possibly a custom total-count response header, if you so choose:
total-count: 521
The other paging data revealed in the human-centric metadata might be superfluous for machine-centric metadata, as the link headers let me know which page I am on and the number per page, and I can quickly retrieve the number of records in the array. Therefore, I would probably only create a header for the total count. You can always change your mind later and add more metadata.
As an aside, you may notice I removed /index from your URI. A generally accepted convention is to have your ReST endpoint expose collections. Having /index at the end muddies that up slightly.
These are just a few things I like to have when consuming/creating an API.
I would recommend adding headers for the same. Moving metadata to headers helps in getting rid of envelops like result , data or records and response body only contains the data we need. You can use Link header if you generate pagination links too.
HTTP/1.1 200
Pagination-Count: 100
Pagination-Page: 5
Pagination-Limit: 20
Content-Type: application/json
[
{
"id": 10,
"name": "shirt",
"color": "red",
"price": "$23"
},
{
"id": 11,
"name": "shirt",
"color": "blue",
"price": "$25"
}
]
For details refer to:
https://github.com/adnan-kamili/rest-api-response-format
For swagger file:
https://github.com/adnan-kamili/swagger-response-template
As someone who has written several libraries for consuming REST services, let me give you the client perspective on why I think wrapping the result in metadata is the way to go:
Without the total count, how can the client know that it has not yet received everything there is and should continue paging through the result set? In a UI that didn't perform look ahead to the next page, in the worst case this might be represented as a Next/More link that didn't actually fetch any more data.
Including metadata in the response allows the client to track less state. Now I don't have to match up my REST request with the response, as the response contains the metadata necessary to reconstruct the request state (in this case the cursor into the dataset).
If the state is part of the response, I can perform multiple requests into the same dataset simultaneously, and I can handle the requests in any order they happen to arrive in which is not necessarily the order I made the requests in.
And a suggestion: Like the Twitter API, you should replace the page_number with a straight index/cursor. The reason is, the API allows the client to set the page size per-request. Is the returned page_number the number of pages the client has requested so far, or the number of the page given the last used page_size (almost certainly the later, but why not avoid such ambiguity altogether)?
just add in your backend API new property's into response body.
from example .net core:
[Authorize]
[HttpGet]
public async Task<IActionResult> GetUsers([FromQuery]UserParams userParams)
{
var users = await _repo.GetUsers(userParams);
var usersToReturn = _mapper.Map<IEnumerable<UserForListDto>>(users);
// create new object and add into it total count param etc
var UsersListResult = new
{
usersToReturn,
currentPage = users.CurrentPage,
pageSize = users.PageSize,
totalCount = users.TotalCount,
totalPages = users.TotalPages
};
return Ok(UsersListResult);
}
In body response it look like this
{
"usersToReturn": [
{
"userId": 1,
"username": "nancycaldwell#conjurica.com",
"firstName": "Joann",
"lastName": "Wilson",
"city": "Armstrong",
"phoneNumber": "+1 (893) 515-2172"
},
{
"userId": 2,
"username": "zelmasheppard#conjurica.com",
"firstName": "Booth",
"lastName": "Drake",
"city": "Franks",
"phoneNumber": "+1 (800) 493-2168"
}
],
// metadata to pars in client side
"currentPage": 1,
"pageSize": 2,
"totalCount": 87,
"totalPages": 44
}
This is an interessting question and may be perceived with different arguments. As per the general standard meta related data should be communicated in the response headers e.g. MIME type and HTTP codes. However, the tendency I seem to have observed is that information related to counts and pagination typically are communicated at the top of the response body. Just to provide an example of this The New York Times REST API communicate the count at the top of the response body (https://developer.nytimes.com/apis).
The question for me is wheter or not to comply with the general norms or adopt and do a response message construction that "fits the purpose" so to speak. You can argue for both and providers do this differently, so I believe it comes down to what makes sense in your particular context.
As a general recommendation ALL meta data should be communicated in the headers. For the same reason I have upvoted the suggested answer from #adnan kamili.
However, it is not "wrong" to included some sort of meta related information such as counts or pagination in the body.
generally, I make by simple way, whatever, I create a restAPI endpoint for example "localhost/api/method/:lastIdObtained/:countDateToReturn"
with theses parameters, you can do it a simple request.
in the service, eg. .net
jsonData function(lastIdObtained,countDatetoReturn){
'... write your code as you wish..'
and into select query make a filter
select top countDatetoreturn tt.id,tt.desc
from tbANyThing tt
where id > lastIdObtained
order by id
}
In Ionic, when I scroll from bottom to top, I pass the zero value, when I get the answer, I set the value of the last id obtained, and when I slide from top to bottom, I pass the last registration id I got