We have an akka/scala app that has some naively written error handling that I need to fix.
The REST endpoint communicates with an internal actor that makes a remote call to create an order.
It does this using ask, and when the ask times out e.g. because of a network or comms error, we send the client a message over the REST endpoint that the request has failed.
The problem is that the internal actor has its own queuing/retry logic and it will continue to call the remote interface until the request succeeds.
So we have the situation where we've told the client that the request has failed but it's really just queued (and will often eventually succeed). The client resubmits the request and we end up with 100's of duplicate orders.
My question is: does akka support a generic way of rolling back or poisoning an ask message when the ask request times out?
There are a couple of approaches one can take. Neither is generic, as (especially once network communication is involved) any choice made in this area has some cases where it's exactly the wrong thing for the business logic:
If there's something about the orders that can be used to determine that two submitted orders are actually the same (e.g. a client-supplied correlation ID), that can be used in the actor to piggyback on the queuing/retry logic handling the earlier order. This requires some client-visible API changes.
It's also possible to include a stop retrying and ignore if pulled from a queue after this time field in the message; you can set this time based on the ask timeout duration.
Related
Is it proper programming practice/ software design to have a REST API call another REST API? If not what would be the recommended way of handling this scenario?
If I understand your question correctly, then YES, it is extremely common.
You are describing the following, I presume:
Client makes API call to Server-1, which in the process of servicing
this request, makes another request to API Server-2, takes the
response from Server-2, does some reformatting or data extraction, and
packages that up to respond back the the Client?
This sort of thing happens all the time. The downside to it, is that unless the connection between Server-1 and Server-2 is very low latency (e.g. they are on the same network), and the bandwidth used is small, then the Client will have to wait quite a while for the response. Obviously there can be caching between the two back-end servers to help mitigate this.
It is pretty much the same as Server-1 making a SQL query to a database in order to answer the request.
An alternative interpretation of your question might be that the Client is asking Server-1 to queue an operation that Server-2 would pick up and execute asynchronously. This also is very common (it's how Google crawls your website, for instance). This scenario would have Server-1 respond to Client immediately without needing to wait for the results of the operation undertaken by Server-2. A message queue or database table is usually used as an intermediary between servers in this case.
Another approach to that is make your REST API(1) store the request details to a queue table. Make a backend that will check that queue table every let's say 100milliseconds. That backend will be the one who will call the other REST API(2).
In your REST API(1) just create a loop that will check if the transaction on queue has been processed. If yes, get the process details and return it to client, if no, just keep on looping until process is done
Say you are designing a REST API over HTTP for a server "room" where subscribing clients want to monitor public events happening to the room (e.g. a new participant joins the room, another one leaves the room, and so on...) by making long poll requests.
What is the best way to implement this from a server side point of view so that the client will not miss any events between consecutive polls? For example, should the server implement a queue of events which need to exist in the queue until all the subscribers have got them?
Are there any tutorials, examples, some theory on internet about designing such an API and all the things that should be taken into account from the server perspective?
Very short answer - why not just use EventStore?
Short answer - why not just use Event Store as a reference implementation, and adapt their solution to match your implementation constraints?
What is the best way to implement this from a server side point of view so that the client will not miss any events between consecutive polls? For example, should the server implement a queue of events which need to exist in the queue until all the subscribers have got them?
REST by itself offers a few guidelines. There should be no application state stored on the server; the message sent by the client should include any client side state (like current position in the event stream) that the server will need to fulfill the request. The resource identified in the request is an abstraction - so the client can send messages to, for example "the event that comes after event 7", which makes sense even if that next event doesn't exist yet. The uniform interface should be respected, to allow for scaling via caches and the like that are outside of the control of the server. The representation of the state of the resource should be hypermedia, with controls that allow the client to advance after it has consumed the currently available messages.
HTTP throws in a few more specifics. Since there is no tracking of client state on the server, reading from the queue is a safe operation. Therefore, one of the safe HTTP methods (GET, to be precise) should be used for the read. Since GET doesn't actually support content body in the request, the information that the server will need should all be packed into the header of the request.
In other words, the URI is used to specify the current position of the client in the event stream.
Atom Syndication provides a good hypermedia format for event processing - the event stream maps to a feed, events map to entries.
By itself, those pieces give you a big head start on an event processor that conforms to the REST architectural constraints. You just need to bolt long polling onto it.
To get a rough idea at how you might implement long polling on your own, you can take a look at the ticketing demo, written by Michael Barker (maintainer of LMAX Disruptor).
The basic plot in Michael's demo is that a single writer thread is tracking (a) all of the clients currently waiting for an update and (b) the local cache of events. That thread reads a batch of events, identifies which requests need to be notified, responds to each of those requests in turn, and then advances to process the next batch of events.
I tend to think of the local cache of events as a ring buffer (like the disruptor itself, but private to the writer thread). The writer thread knows (from the information in the HTTP request) the position of each client in the event stream. Comparing that position to the current pointer in the ring buffer, each pending request can be classified has
Far Past The position that the client is seeking has already been evicted from the cache. Redirect the client to a "cold" persistent copy of that location in the stream, where it can follow the hypermedia controls to catch up to the present.
Recent Past The position that the client is seeking is currently available in the cache, so immediately generate a response to the client with the events that are available, and dispatch that response.
Near future The position that the client is seeking is not available in the cache, but the writer anticipates being able to satisfy that request before the SLA expires. So we park the client until more events arrive.
Far future The position that the client is seeking is not available in the cache, and we don't anticipate that we will be able to satisfy the request in the allotted time. So we just respond now, and let the client decide what to do.
(If you get enough polling clients that you need to start scaling out the long polling server, you need to consider the case where those servers get out of sync, and a client gets directed from a fast server to one that has fallen behind. So you'll want to have instrumentation in place that lets you track how often this is happening, so that you can apply the appropriate remedy).
There are also some edge cases to consider -- if a very large batch comes in, then you may need to evict the events your clients are waiting on before you get a chance to send them.
Simple, have the client pass in the timestamp (or id, or index) of the last message they received.
Requesting GET /rooms/5/messages returns all the messages the server knows about, like
[
{
"message": "hello",
"timestamp": "2016-07-18T18:44:34Z"
},
{
"message": "world",
"timestamp": "2016-07-18T18:47:16Z"
}
]
The client then long polls the server with GET /rooms/5/messages?since=2016-07-18T18:47:16Z which returns either all the messages since that time (if there are any) or blocks until the room has a new message.
Send reference number with all the events.
Cleint will call with reference number of the latest event received. You will block long poll request if no event is available and respond once event is available again with new reference number.
In Case events are already available it will return all events generated after the request reference number event.
I strongly recommend using WebSockets. Check out socket.io. Long polling is a hack that isn't necessarily desirable and isn't really "supported".
Long polling is not a good idea. Specifically when one wants to live monitor the changes those happen at server side.There are mechanisms where server send the notifications to clients for the changes. This can be achieved by using, as gcoreb already mentioned, Socket.io (Nodejs stack) or SignalR (.net stack).
I understand the main principles behind both. I have however a thought which I can't answer.
Benchmarks show that WebSockets can serve more messages as this website shows: http://blog.arungupta.me/rest-vs-websocket-comparison-benchmarks/
This makes sense as it states the connections do not have to be closed and reopened, also the http headers etc.
My question is, what if the connections are always from different clients all the time (and perhaps maybe some from the same client). The benchmark suggests it's the same clients connecting from what I understand, which would make sense keeping a constant connection.
If a user only does a request every minute or so, would it not be beneficial for the communication to run over REST instead of WebSockets as the server frees up sockets and can handle a larger crowd as to speak?
To fix the issue of REST you would go by vertical scaling, and WebSockets would be horizontal?
Doe this make sense or am I out of it?
This is my experience so far, I am happy to discuss my conclusions about using WebSockets in big applications approached with CQRS:
Real Time Apps
Are you creating a financial application, game, chat or whatever kind of application that needs low latency, frequent, bidirectional communication? Go with WebSockets:
Well supported.
Standard.
You can use either publisher/subscriber model or request/response model (by creating a correlationId with each request and subscribing once to it).
Small size apps
Do you need push communication and/or pub/sub in your client and your application is not too big? Go with WebSockets. Probably there is no point in complicating things further.
Regular Apps with some degree of high load expected
If you do not need to send commands very fast, and you expect to do far more reads than writes, you should expose a REST API to perform CRUD (create, read, update, delete), specially C_UD.
Not all devices prefer WebSockets. For example, mobile devices may prefer to use REST, since maintaining a WebSocket connection may prevent the device from saving battery.
You expect an outcome, even if it is a time out. Even when you can do request/response in WebSockets using a correlationId, still the response is not guaranteed. When you send a command to the system, you need to know if the system has accepted it. Yes you can implement your own logic and achieve the same effect, but what I mean, is that an HTTP request has the semantics you need to send a command.
Does your application send commands very often? You should strive for chunky communication rather than chatty, so you should probably batch those change request.
You should then expose a WebSocket endpoint to subscribe to specific topics, and to perform low latency query-response, like filling autocomplete boxes, checking for unique items (eg: usernames) or any kind of search in your read model. Also to get notification on when a change request (write) was actually processed and completed.
What I am doing in a pet project, is to place the WebSocket endpoint in the read model, then on connection the server gives a connectionID to the client via WebSocket. When the client performs an operation via REST, includes an optional parameter that indicates "when done, notify me through this connectionID". The REST server returns saying if the command was sent correctly to a service bus. A queue consumer processes the command, and when done (well or wrong), if the command had notification request, another message is placed in a "web notification queue" indicating the outcome of the command and the connectionID to be notified. The read model is subscribed to this queue, gets messessages and forward them to the appropriate WebSocket connection.
However, if your REST API is going to be consumed by non-browser clients, you may want to offer a way to check of the completion of a command using the async REST approach: https://www.adayinthelifeof.nl/2011/06/02/asynchronous-operations-in-rest/
I know, that is quite appealing to have an low latency UP channel available to send commands, but if you do, your overall architecture gets messed up. For example, if you are using a CQRS architecture, where is your WebSocket endpoint? in the read model or in the write model?
If you place it on the read model, then you can easy access to your read DB to answer fast search queries, but then you have to couple somehow the logic to process commands, being the read model the responsible of send the commands to the write model and notify if it is unable to do so.
If you place it on the write model, then you have it easy to place commands, but then you need access to your read model and read DB if you want to answer search queries through the WebSocket.
By considering WebSockets part of your read model and leaving command processing to the REST interface, you keep your loose coupling between your read model and your write model.
I have 15 worker clients and one master connected through internet. Job & data are been passed through REST api in json format.
Jobs are not restricted to any particular client. Any worker can query for the available job in regular interval(say 30 seconds), process it and will update the status.
In this scenario, how can I prevent same records been sent to different clients while GET request.
Followings are my solution approach to overcome this issue:
Take top 5 unprocessed records from the database and make it as SENT and expose via REST GET.
But the problem is, it creates inconsistency. Some times, the client doesn't got data due to network connectivity issue. But in server, it will be marked as SENT. So, no other clients can get that data. It will remain as SENT forever.
Get the list from server, and reply back the list of job IDs to Server as received. But in-between this time gap, some other clients also getting same set of Jobs.
You've stumbled upon a fundamental problem in distributed systems: there is no way to know if the other side received your message. You can certainly improve the situation with TCP and ack messages. But if you never get the ACK did the message never arrive, did it arrive but the recipient die before processesing, or did the recipient send he ACK and the ACK get dropped?
That means you need to design your system to handle receiving data more than once.
You offer two partial solutions; if you combine them, your solution starts to look like how SQS works. Mark the item as pending_ack with a timestamp. After client replies, it is marked sent. Any pending_ackss past a certain time period are eligible to be resent.
Pick your time period to allow for slow network and slow clients and it boils down to only sending duplicates when you really don't know if the client died or not.
Maybe you should reconsider the approach to blocking resources. REST architecture - by definition is not obliged to save information about client. Instead, you may want to consider optimistic concurrency control (http://en.wikipedia.org/wiki/Optimistic_concurrency_control).
Consider a queue of items on server. The client then reads 10 queued items at a time using a REST web service. Naturally, when the client has consumed these items the server should remove them server-side.
Q: What is the best approach if we consider both robustness, network load and restfulness?
I can think of three possible solutions:
The client asks for new items. The server then...
sends item 1..10 (GET) and removes them immediately. Hopefully the items arrived at the client.
sends item 1..10 (GET), client sends ACK for 1..10 (DELETE), and the server removes the items.
sends item 1..10 (GET). Next time the client asks for 11..20 (GET), the previous items are removed on the server.
I believe both #1 and #3 violate the restful principle. E.g. Only the DELETE method may delete objects. However, they both avoid the data traffic for the ACK command.
Not sure what's best here. Perhaps there is an even better solution?
Here's the answer in votable format. I hope it helps clarify your options a bit more.
It's important in a REST-style architecture that the implementation of an API not change the implementation of any underlying protocols -- in this case it means that GET requests should be idempotent. While idempotent does not mean that the underlying resources can't change, or go away forever (AKA be deleted), having that happen as a direct or indirect result of a GET seems to not be in accordance with the spirit of the protocol.
Any system that guarantees delivery of a message requires some kind of handshake to single that the intended receiver successfully received the message -- if HTTP is the protocol in question, then that implies two requests. Even in the case where the behavior of GET were modified to lazily delete the resources, the handshake is still present -- it has just been shifted in time. Again, if HTTP is the protocol in questions, then using the existing methods of GET and then DELETE for the retrieval and deletion for that handshake seems best. The result shouldn't tax the network any more than any equivalent approach.