Consider a backend system that stores logs of this form:
{"id": 541, "timestamp": 123, "status": "ok"}
{"id": 681, "timestamp": 124, "status": "waiting"}
...
Assuming that there are MANY logs, a client (e.g. an Android app) wants to sync the log data stored at a server to the client's device for presentation. Since the most recent logs are of more interest to a user, the GET request should be paged and start with the most recent logs and walk its way towards the oldest ones.
What is a proper design for this situation? What about the following design?
Let the server response in reverse order, add parameters lastReceivedIdand size to the request and add a field more=true/false in the response that indicates whether there are more old logs available before the oldest log send in the current request. On the first request set lastRecivedId=-1 indicating that the server should answer with the most recent logs.
Ship 'em all, let the server sort them out. The endpoint simply doesn't care what order they show up in, the server will handle that detail for presentation.
On the client, the client can choose to send the latest logs first, but that's simply coincidence. There's no requirement one way or the other.
There's also no need to send them in any particular order. If the client has a thousand log entries (in chronological order), it can send back batches of 100 starting at 900-1000, then 800-899, etc. The server will figure it out in the end.
Related
Let's say there are two (or more) RESTful microservices serving JSON. Service (A) stores user information (name, login, password, etc) and service (B) stores messages to/from that user (e.g. sender_id, subject, body, rcpt_ids).
Service (A) on /profile/{user_id} may respond with:
{id: 1, name:'Bob'}
{id: 2, name:'Alice'}
{id: 3, name:'Sue'}
and so on
Service (B) responding at /user/{user_id}/messages returns a list of messages destined for that {user_id} like so:
{id: 1, subj:'Hey', body:'Lorem ipsum', sender_id: 2, rcpt_ids: [1,3]},
{id: 2, subj:'Test', body:'blah blah', sender_id: 3, rcpt_ids: [1]}
How does the client application consuming these services handle putting the message listing together such that names are shown instead of sender/rcpt ids?
Method 1: Pull the list of messages, then start pulling profile info for each id listed in sender_id and rcpt_ids? That may require 100's of requests and could take a while. Rather naive and inefficient and may not scale with complex apps???
Method 2: Pull the list of messages, extract all user ids and make bulk request for all relevant users separately... this assumes such service endpoint exists. There is still delay between getting message listing, extracting user ids, sending request for bulk user info, and then awaiting for bulk user info response.
Ideally I want to serve out a complete response set in one go (messages and user info). My research brings me to merging of responses at service layer... a.k.a. Method 3: API Gateway technique.
But how does one even implement this?
I can obtain list of messages, extract user ids, make a call behind the scenes and obtain users data, merge result sets, then serve this final result up... This works ok with 2 services behind the scenes... But what if the message listing depends on more services... What if I needed to query multiple services behind the scenes, further parse responses of these, query more services based on secondary (tertiary?) results, and then finally merge... where does this madness stop? How does this affect response times?
And I've now effectively created another "client" that combines all microservice responses into one mega-response... which is no different that Method 1 above... except at server level.
Is that how it's done in the "real world"? Any insights? Are there any open source projects that are built on such API Gateway architecture I could examine?
The solution which we used for such problem was denormalization of data and events for updating.
Basically, a microservice has a subset of data it requires from other microservices beforehand so that it doesn't have to call them at run time. This data is managed through events. Other microservices when updated, fire an event with id as a context which can be consumed by any microservice which have any interest in it. This way the data remain in sync (of course it requires some form of failure mechanism for events). This seems lots of work but helps us with any future decisions regarding consolidation of data from different microservices. Our microservice will always have all data available locally for it process any request without synchronous dependency on other services
In your case i.e. for showing names with a message, you can keep an extra property for names in Service(B). So whenever a name update in Service(A) it will fire an update event with id for the updated name. The Service(B) then gets consumes the event, fetches relevant data from Service(A) and updates its database. This way even if Service(A) is down Service(B) will function, albeit with some stale data which will eventually be consistent when Service(A) comes up and you will always have some name to be shown on UI.
https://enterprisecraftsmanship.com/2017/07/05/how-to-request-information-from-multiple-microservices/
You might want to perform response aggregation strategies on your API gateway. I've written an article on how to perform this on ASP.net Core and Ocelot, but there should be a counter-part for other API gateway technologies:
https://www.pogsdotnet.com/2018/09/api-gateway-response-aggregation-with.html
You need to write another service called Aggregator which will internally call both services and get the response and merge/filter them and return the desired result. This can be easily achieved in non-blocking using Mono/Flux in Spring Reactive.
An API Gateway often does API composition.
But this is typical engineering problem where you have microservices which is implementing databases per service pattern.
The API Composition and Command Query Responsibility Segregation (CQRS) pattern are useful ways to implement queries .
Ideally I want to serve out a complete response set in one go
(messages and user info).
The problem you've described is what Facebook realized years ago in which they decided to tackle that by creating an open source specification called GraphQL.
But how does one even implement this?
It is already implemented in various popular programming languages and maybe you can give it a try in the programming language of your choice.
Hi I'm building a chat messaging system and am reading/writing to the DB for my first time. I'm creating the method calls to retrieve relevant data I need for this chat. Like with most chat systems, I want to have a general list of message with the name, most recent message, and time/date associated with the most recent message. Once I click on that thread, the corresponding messages will show up. I'm trying to call the DB but am having trouble using the correct command to get the most recent message.
This is what one message contains:
{
"_id": "134a8cba-2195-4ada-bae2-bc1b47d9925a" ,
"clinic_id": 1 ,
"created": 1531157560 ,
"direction": "o" ,
"number": "14383411234" ,
"read": true ,
"text": "hey hows it going"
}
Every single message that is sent and received gets POSTed like this. I'm having trouble coming up with the correct commands to get the most recent message of all the distinct "number" so that for number x, I get its corresponding recent message and with number y, I get its corresponding recent message. "created" is the time when the message was created in UNIX time.
This is what I have started with:
Retrieve all thread numbers:
r.db('d2').table('sms_msg').between([1, r.minval], [1, r.maxval], {index:'clinic_number'}).pluck(['number']).distinct()
Retrieve all messages in specific thread:
r.db('d2').table('sms_msg').getAll([1, "14383411234"], {index:'clinic_number'})
Retrieve recent message for all distinct threads:
r.db('d2').table('sms_msg').filter()....???
Some help would really be appreciated!
That's a very tricky query for any database, and usually involves a multitude of sub-queries. You might want to consider denormalizing it, keeping a reference to last entry in another table, for each number.
But basically, with your current approach, this might work (untested) but might be highly inefficient:
r.table('sms_msg').orderBy(r.desc('created')).group('number').nth(0)
It's usually fast to get the lowest value of the property of a document, but when you want the whole document of a sorted list like this, it is very inefficient in my experience.
Imagine a request that starts a long running process whose output is a large set of records.
We could start the process with a POST request:
POST /api/v1/long-computation
The output consists of a large sequence of numbered records, that must be sent to the client. Since the output is large, the server does not store everything, and so maintains a window of records with a upper limit on the size of the window. Let's say that it stores upto 1000 records (and pauses computation whenever this many records are available). When the client fetches records, the server may subsequently delete those records and so continue with generating more records (as more slots in the 1000-length window are free).
Let's say we fetch records with:
GET /api/v1/long-computation?ack=213
We can take this to mean that the server should return records starting from index 214. When the server receives this request, it can assume that the (well-behaved) client is acknowledging that records up to number 213 are received by the client and so it deletes them, and then returns records starting from number 214 to whatever is available at that time.
Next if the client requests:
GET /api/v1/long-computation?ack=214
the server would delete record 214 and return records starting from 215.
This seems like a reasonable design until it is noticed that GET requests need to be safe and idempotent (see section 9.1 in the HTTP RFC).
Questions:
Is there a better way to design this API?
Is it OK to keep it as GET even though it appears to violate the standard?
Would it be reasonable to make it a POST request such as:
POST /api/v1/long-computation/truncate-and-fetch?ack=213
One question I always feel like that needs to be asked is, are you sure that REST is the right approach for this problem? I'm a big fan and proponent REST, but try to only apply to to situations where it's applicable.
That being said, I don't think there's anything necessarily wrong with expiring resources after they have been used, but I think it's bad design to re-use the same url over and over again.
Instead, when I call the first set of results (maybe with):
GET /api/v1/long-computation
I'd expect that resource to give me a next link with the next set of results.
Although that particular url design does sort of tell me there's only 1 long-computation on the entire system going on at the same time. If this is not the case, I would also expect a bit more uniqueness in the url design.
The best solution here is to buy a bigger hard drive. I'm assuming you've pushed back and that's not in the cards.
I would consider your operation to be "unsafe" as defined by RFC 7231, so I would suggest not using GET. I would also strongly advise you to not delete records from the server without the client explicitly requesting it. One of the principles REST is built around is that the web is unreliable. Under your design, what happens if a response doesn't make it to the client for whatever reason? If they make another request, any records from the lost response will be destroyed.
I'm going to second #Evert's suggestion that you absolutely must keep this design, you instead pick a technology that's build around reliable delivery of information, such as a messaging queue. If you're going to stick with REST, you need to allow clients to tell you when it's safe to delete records.
For instance, is it possible to page records? You could do something like:
POST /long-running-operations?recordsPerPage=10
202 Accepted
Location: "/long-running-operations/12"
{
"status": "building next page",
"retry-after-seconds": 120
}
GET /long-running-operations/12
200 OK
{
"status": "next page available",
"current-page": "/pages/123"
}
-- or --
GET /long-running-operations/12
200 OK
{
"status": "building next page",
"retry-after-seconds": 120
}
-- or --
GET /long-running-operations/12
200 OK
{
"status": "complete"
}
GET /pages/123
{
// a page of records
}
DELETE /pages/123
// remove this page so new records can be made
You'll need to cap out page size at the number of records you support. If the client request is smaller than that limit, you can background more records while they process the first page.
That's just spitballing, but maybe you can start there. No promises on quality - this is totally off the top of my head. This approach is a little chatty, but it saves you from returning a 404 if the new page isn't ready yet.
There is a change to our business logic, where earlier with one of the APIs we use to return a list, for eg. list of employees. Recently we introduced authorization checks, to see if a particular user has permission to view a specific employee.
If say there are 10 employees that should be returned through method GET, due to the missing permission only 5 are returned. The request itself in this case is successful. I am currently not sure how to pass on the information back to the client that there were 5 employees that are filtered out due to missing permission.
Should this be mapped to HTTP status codes? If yes, which status code fits this? Or this is not an error at all?
What would be the best approach in this case?
A status code by itself wouldn't be sufficient to indicate the partial response. The status code 206 sounds close by name but is used when a client specifically requests a partial set of data based on headers.
Use 200. The request was fulfilled successfully after all, and the reason for the smaller set of data is proprietary to your API so extra metadata in the response to indicate a message might be sufficient.
Assuming JSON response:
{
"data": [ ... ],
"messages": [
"Only some data was returned due to permissions."
]
}
If you have many consumers and are worried about backward compatibility you may also want to provide a vendor specific versioned JSON media type:
"Content-Type": "application/vnd.myorg-v2+json"
I wonder how to establish transaction safety in RESTful APIs, where everything is built around single entities.
Example
Database model:
Invoice
Item
User-performed steps in the browser:
Change order number.
Add an item.
Remove an item.
Edit an item.
Requests made:
PATCH/PUT invoice data/order number.
POST item.
DELETE item.
PATCH/PUT item.
Issue
If after any of the requests above an error happens, further calls might mess up the data integrity. Additionally previous requests have to be made undone. E.g. if deleting the item fails, then steps 1 and 2 have to be rewound in order for the overall invoice to be how it was before.
Another problem that might arise is a browser crash, dying internet connection, server failure or whatever.
How can one make sure that certain actions are executed in some kind of transaction to maintain data integrity and safety?
So the thing to remember with REST is the "state transfer" bit. You are not telling the server the steps needed to update a resource, you are telling the server the state the resource should be in after the update, because you have already updated it on the client and are now simply transferring this new state over to the server.
So say you have an Invoice item on the server that looks like this JSON on the server
{
invoice_id: 123,
invoice_description: "Some invoice",
invoice_items: [
{item_id: 10, item_desc: "Large item", order_amount: 34},
{item_id: 11, item_desc: "Small item", order_amount: 400}
]
}
And a user wants to edit that invoice as a single atomic transaction. Firstly they GET the invoice from the server. This essentially says "Give me the current state of the invoice"
GET /invoices/123
The user then edits that invoice anyway they want. They decide the number of large items should be 40 not 34. They decide to delete the small items completely. And finally they decide to add another item of "Extra small" items to the invoice. After the user has edited the invoice the client has the following invoice
{
invoice_id: 123,
invoice_description: "Some invoice",
invoice_items: [
{item_id: 10, item_desc: "Large item", order_amount: 40},
{item_id: 30, item_desc: "Extra small item", order_amount: 5}
]
}
So the client has the invoice in a different state to the server. The user now wants to send this back to the server to be stored, so it PUTs the new state of the invoice to the server.
PUT /invoices/123
which essentially says "Here is the new state of this resource."
Now depending on how fancy you want your validation to be the server can simply accept this new state the invoice is in as it is, or it can do a whole load of validation for each change. How much you want to do is up to you.
You would at the very least want to check that no other client has PUT an updated invoice onto the server while the user was editing this copy of the invoice on their client. You can do this by checking the various headers of the HTTP requests (such as the etag header http://en.wikipedia.org/wiki/HTTP_ETag)
If for any reason the server decides that this update is not valid it simply fails the entire PUT request. This is what gives you transactions in HTTP. A require should either work or fail. If it fails it is the servers responsibility to make sure the resource has not been effected by the failed request. From an implantation point of view on the server you would probably do some validation of the new JSON and then attempt to save the new data to a database within a DB transaction. If anything fails then the database is kept in original state and the user is told that the PUT didn't work.
If the request fails the user should be returned a HTTP status code and response that explains why the PUT require failed. This might be because someone else has edited the invoice while the user was thinking about his changes. It might be because the user is trying to put the invoice into an invalid state (say the user tried to PUT an invoice with no items and this breaks the business logic of the company).
You can of course develop a URI scheme that allows editing of individual items in an invoice, for example
GET /invoices/123/items/10
Would give you the item id 10 from invoice id 123. But if you do this you have to allow the editing of these resources independently of each other. If I delete item 10 by sending the delete command
DELETE /invoice/123/items/10
that action must be an independent transaction. If other requests depend on this you must instead do it as detailed above, by updating the invoice itself in a single request. You should never be able to put the resource into an invalid state through a single HTTP request, or to put it another way it should never require multiple HTTP requests to get the resource into a valid state (and thus never require a string of HTTP requests to work in order to be valid)
Hope that helps
Great answer, thanks a lot. There's just one bit I'm not sure about:
What if the user sends back an updated invoice with a new item that
hasn't got an ID yet, because that is always generated at the server?
AFAIK at least PUT wouldn't be correct here, but POST instead.
However, is that how it's done?
Yes PUT would be wrong here. PUT should be idempotent, which means that you should be able to make multiple PUT requests to a resource and if they are the same request the end result should be the same after all of them.
This makes sense if you think again of state transfer again, doing a PUT of the same state multiple times should still end up with the resource in that state. If I upload a PNG file to a resource 20 times the PNG file should be still the same PNG file as if I just uploaded it once.
So you should not have anything unambiguous in the state that you PUT to the server. If you left out the ID of the item you are essentially saying to the server "As part of this state update create a item". And of course if you run that 10 times you will create 10 new items and the invoice will not be in the same state.
So POST would be better here, and you might want to do that to an "items" end point for clarity if you are just updating the items.
POST /invoices/123/items
[
{item_id: 10, item_desc: "Large item", order_amount: 40},
{item_desc: "Extra small item", order_amount: 5}
]
The server can then return you the state of the invoice's items with the newly created IDs in the body of the response
[
{item_id: 10, item_desc: "Large item", order_amount: 40},
{item_id: 30, item_desc: "Extra small item", order_amount: 5}
]