I'm implementing a CQRS/ES based system with a RESTful interface which is used by a webapp.
When performing certain actions e.g. creating a new profile I need to be able to check certain conditions, such as uniqueness of the profile ID, or that the person has the right to create a resource under a group. Which means I have a couple of options:
Context: POST/profiles { "email": "unique#example.com" }
From my REST API return 202 from my service with a location of the new resource where my client can poll for it. In this case, however, how do I handle errors as in effect the view will not exist or ever exist.
Create a saga on the initial request then dispatch the event. Once my service creates the view or finds the error then the result is written to the saga. When the saga has been completed return the result to the user.
From these two options - the second seems more reasonable to me, if not more complex. Is this a viable option for building RESTful request/response models on a CQRS/ES event sourced backend?
Yes, the second solution seems to better fit the business.
From what I understand from your case, from the DDD point of view, the creation of a user profile is a business process, with more than one steps (verifying the uniqueness of the profile, creating the profile and recovering from a duplicate profile situation). This process acts like an entity, it starts, runs and ends with a result (success or error). Being an entity it has an ID and it can be viewed as a REST resource. A Saga will be responsible for executing it.
So, in response to the client's request you send the URI of the process resource where the client can poll for the status. In case of error, it reads the error message. In case of success, it gets the URI of its profile.
The first solution can still be used if the use-case is simpler, if the command can be executed synchronously and the client gets the final result (error or success) as an immediate response.
From my REST API return 202 from my service with a location of the new resource where my client can poll for it. In this case, however, how do I handle errors as in effect the view will not exist or ever exist.
The usual answer here is that, as part of the 202 Accepted response, you include monitoring information
The representation sent with this response ought to describe the request's current status and point to (or embed) a status monitor that can provide the user with an estimate of when the request will be fulfilled.
In other words, a link to a resource that will change when the accepted request is finally run.
So in describing the protocol, in addition to the resource that you create, you'll also need to document the representation used when you defer the work for later, and the representation used by the monitor.
When the saga has been completed return the result to the user.
Depending on the work, that may be overkill.
Which is to say, you are raising two different questions here; one of those is whether the request should be handled synchronously (don't respond until the work is done) or asynchronously (return right away, but give the client the means to monitor progress).
The other question is how the work looks from the business layer. If you are going to need multiple transactions to make the change, and if you may need to "revert" previously committed transactions in some variants of the process, then a saga (or a process manager) makes sense.
Set Validation -- the broader term for enforcing an invariant like "uniqueness" -- is awkward. Make sure you study, and ensure that you and the business understand the impact of having a failure.
Related
I would like to clarify what'd be the best way of organizing architecture.
I've got rest api and microservices architecture. I have applied the Database per Service pattern.
So let's imagine that the user wants to create an order(an e-commerce system). But users can have a credit limit. So the flow will be as below:
OrderService creates a pending order. Then push an event about it.
UserService processes that event and publishes either a credit limit exceeded event or a credit reserved event.
OrderService receives the event and changes the state of the order to either approved or canceled.
All look good. But the question is what the user will do during this simple flow?
I mean: the user makes a POST request /orders and ...
Web-service awaits until the order gets approved or canceled(includes surely timeout)?
Web-service returns 200 ok and then the user needs to check the order state with some interval?
use web sockets?
something else?
Unfortunately, Any option from above has its advantages and its disadvantages.
The challenge is that I described the simplest case. In reality, tens services(even third party) could be involved. And of course, I am expecting a high load. So the queue may be filled.
Please propose the solution to discuss. I'm greatly appreciating for an answer as well as links to production ready system documentation.
Kudos for a good question. If we look at Saga Pattern, it offers to do ACID like transactions in a distributed system but with some trade-off. One of the trade-offs is Ensuring Rollback if any of the service or entity fails to do what it was supposed to do. This can go even complex if you have more than 5 services completing a Saga. Though it will be a Highly Scalable option if you can orchestrate.
Here I will propose Following,
User Does a POST request for order
OrderService first checks with UserService regarding credit Limit (It can be a REST API call as it might be a simple DB call for UserService)
Then OrderService can act upon based on the return response from UserService
Advantages of this approach
User can see an immediate response
Less Complex to code thus testable and maintainable Code
Disadvantages of this approach
This solution will not be effective if there are too many external (third party) rest api calls
It can introduce single point of failure
The main thing is, all the options will have trade-offs. It is on you to decide which one suits you the best.
According to me
Follow the my rule - Validate First then action
Validate the order before its goes to order service on UI. You can call the userservice api on UI which gives info user can make a order or not.
Step 1. On UI call the userservice API , which validate the user credit limit
Step 2. If validation success or fail
Step 3. Take the action according to step2 result . If success call the order service if not do what you want to do (Call other service etc..)
Advantage
A good user exprience - User get the validation message "you can not make a order your limit is low" or something like that when trying to make the order. Not at the end get a message "You limit is low/or anything"
Less Complexity of code
A more microservice Arch. -
Since your are follow the microservices then if you call the userservice inside the orderservice means orderservice is dependent on it or you can say tightly bound - here your are lossing the benifit of mircroservice arch.
What happen if the userservice is down. The customer cannot make the order or if you change anything in userservice response then you have to change the orderservice also.
What you do if some more validation (to call other service) need in future before making the order.
Redirection Flexibility - Suppose if on fail of validation from userservice then you have to call other service( other independent microservice ) and if you are using the other service inside the orderservice the no of service increases as project grow
Less Request on OrderService - Request goes orderservice only when the validation is passed from the other service or other thing.
So from my point of view
Try to make microservice independent as much possible.
Validate First and action.
A good user exprience
Easy to handle load (if you think)
With all that , now you less dependent on saga.
Its only my opinion - choose what you think best according to your domain
The question is what the user will do during this
simple flow? I mean: the user makes a POST request /orders and ...
You can reply to user the created orderId with status 'pending' while your saga is running. The request should be ended in 200.
How does user get the status?
Push notification, websocket, email, sms. it is up to you to notify the user about the status of event, success, failure, waiting_for_approval.
1. Web-service awaits until the order gets approved or canceled?
2. Web-service returns 200 ok and the user check the order state with some interval?
3. use web sockets?
4. something else?
Not recommended. Saga should be asynchronous. Waiting for reply is synchronous (request/reply style).
This will work, but if your saga is long running, it might be problematic, eg: anything with validation/approval by a human.
This will work assuming you are replying 200 after order creation in pending state, one thing is that you have to make sure your websocket is active, or else, you still have to notify them through email, sms.
Unfortunately, Any option from above has its advantages and its
disadvantages.
You have to choose your poison.
The challenge is that I described the simplest case. In reality, tens
services(even third party) could be involved. And of course, I am
expecting a high load. So the queue may be filled.
Use orchestrated saga (sort like) Netflix conductor, Uber Cadence, Temporal (Similar to Cadence). This will allow you to do something as shown below. At least, easier.
A flow diagram from google shopping api and how they handle it.
Link for more: https://developers.google.com/shopping-content/guides/about-orders
There is little difference in cancelling an order because there was a problem with the credit limit vs. cancelling it because a forklift ran over the package in the logistics facility.
So a relatively simple solution is to reply with an order accepted and status 200 when the order is placed put that in a queue and let the saga run asynchronously. You should provide both an API (and UI) for the user to check the order status and proactively send status updates like billing going through (which is your fist saga) and also other sagas in the order lifecycle
Regarding how to send notifications that depends on the usage scenario (e.g. you can do a callback for a system-to-system integration, but maybe you want to send an email/sms to an end-user etc)
I have a resource with conditional operations:
/foos/{id}/authorize
/foos/{id}/cancel
The idea is that authorize will change the status of the resource from saved (the default) to authorized (by a third party application). The authorize could return a error from the remote part or could be authorized. Once authorized the resource could not be authorized again so this is not an action that could be called again and again.
The cancel occurs when an authorized resource is revoked. Once cancelled the resource will stay as cancelled forever.
What's the correct verb in a RESTful world for this kind of operation considering that this operation is not safe and could not be considered idempotent as a second call will return a error like "resource already cancelled" and at the same time I'm not creating a new resource, just making a status change in a known resource?
I would use
DELETE /authorization/1234
There's a whole debate around DELETE's idempotence on previously deleted resources. See https://evertpot.com/idempotence-in-http/ and https://leedavis81.github.io/is-a-http-delete-requests-idempotent/
The bottomline here is that idempotence makes sense in a mathematical world where there's always one result, but in HTTP you get two different outcomes -- the server's response and the resource's new state. It becomes difficult identifying what is idempotent and what is not.
In such areas where the HTTP specification is not clear, I recommend pargmatism over dogmatism.
If you really want the client to know if they deleted the resource themselves or if someone else did, then I see no problem responding 404 on a previously deleted resource.
If you don't care, or think that it will never happen (either because there's not enough concurrent access or because all clients always do a GET moments before sending the DELETE), you can happily stick to 204 in all cases.
I've been searching for best practices for preventing the accidental creation of duplicate resources when using POST to create a new resource, for the case where the resource is to be named by the server and hence PUT can't be used. The API I'm building will be used by mobile clients, and the situation I'm concerned about is when the client gets disconnected after submitting the POST request but before getting the response. I found this question, but there was no mention of using a conditional POST, hence my question.
Is doing a conditional POST to the parent resource, analogous to using a conditional PUT to modify a resource, a reasonable solution to this problem? If not, why not?
The client/server interaction would be just like with a conditional PUT:
Client GETs the parent resource, including the ETag reflecting its current state (which would include its subordinate resources),
Client does a conditional POST to the parent resource (includes the parent's ETag value in an If-Match header) to create a new resource,
Client gets disconnected before getting the server response, so doesn't know if it succeeded,
Later, when reconnected, the client resubmits the same conditional POST request,
Either the earlier request didn't reach the server, so the server creates the resource and replies with a 201, or the earlier request did reach the server, so the server replies with a 412 and the duplicate resource isn't created.
Your solution is clever, but less than ideal. Your client may never get his 201 confirmation, and will have to interpret the 412 error as success.
REST afficianados often suggest you create the resource with an empty POST, then, once the client has the id of the newly created resource, he can do an "idempotent" update to fill it. This is nice, but you will likely need to make DB columns nullable that wouldn't otherwise be, and your updates are only idempotent if no-one else is trying to update at the same time.
According to ME, HTTP is flaky. Requests timeout, browser windows get closed, connections get reset, trains go into tunnels with mobile users aboard. There's a simple, robust pattern for dealing with this. Unsafe actions should always be uniquely identified, and servers should store, and be able to repeat if necessary, the response to any unsafe request. This is not HTTP caching, where a request may be served from cache but the cache may be flushed for whatever reason. This is a guarantee by the server application that if an "action" request is seen a second time, the stored response will be repeated without anything else happening. If the action identity is to be generated by the server, then a request-response should be dedicated just to sending the id. If you implement this for one unsafe request, you might as well do it for all of them, and in so doing you will escape numerous thorny problems: successive update requests wiping out other users' changes, or hitting incompatible states ("order already submitted"), successive delete requests generating 404 errors.
I have a little google doc exploring the pattern more fully if you're interested.
I think this scheme would work. If you want to ensure POST does not result in duplicates, you need the client to send something unique in the POST. The server can then verify uniqueness.
You might as well have the client generate a GUID for each request, rather than obtaining this from the server via a GET.
Your steps then become:-
Client generates a GUID
Client does a POST to the resource, which includes the GUID
Client gets disconnected and doesn't know if it succeeded
Client connects again and does another POST with the same GUID
Server checks the GUID, and either creates the resource (if it never received the first POST) or indicates that this was a duplicate
It might be more restful to use PUT, and have the client decide the resource name. If you did not like the choosen name, you could indicate that you had created the resource but that it's canonical location was somewhere of the server's choosing.
Why not simply do duplicate detection on the server based on the actual resource, using whatever internal mechanism the server chooses to use.
It's just safer that way.
Then you return the URL to the appropriate resource (whether it was freshly created or not).
If the parents ETag is based on the state of sub resources, then it's not a reliable mechanism to check for "duplicate resources". All you know is that the parent has "changed", somehow, since last time. How do you even know it's because your old POST was processed after disconnect? Could be anything changed that ETag.
This is basically a optimistic locking scenario being played out, and it comes down to another question. If the resource is already created, what then? Is that an error? Or a feature? Do you care? Is it bad to send a creation request that's silently ignored by the server when the resource already exists?
And if it already exists, but is "different" enough (i.e. say the name matches but the address is different), is that a duplicate? is that an update? is that a error for trying to change an existing resource?
Another solution is to make two trips. One to stage the request, another to commit it. You can query the status of the request when you come back if it's interrupted. If the commit didn't got through, you can commit it again. If it did, you're happy and can move on.
Just depends on how unstable your comms are and how important this particular operation is whether you want to jump through the hoops to do it safely.
I would like to record user actions on my website, not only on POST requests, but on GET requests as well. For example, suppose the user tries to search for a city with the following GET request:
/search_city?name=greenville
This request would return a list of cities with the name "greenville". I'd also like to save this keyword to the server, as the "search history" for a user. I'm planning to just do the save this information during the processing of the GET call.
Is this a violation to RESTful principles? If yes, how do I do this the right way?
I see this kind of audit logging as an invisible side-effect. If the next person to call
/search_city?name=greenville
still gets the same answer then your GET is valid. A similar case would be some kind of cache building, the caller of GET doesn't (need to) know that you're doing some extra work.
Focus on the formal API - send this request get this response.
If there's some resource available in the API where the user search history is available, then it's not OK to do that, since your GET request has a visible side-effect. For instance, a client caching responses is under no obligation to know that any resource changed because he did a GET request to anything else. I think the only way to do this and stay compliant is to explicitly mark the side-effected resource as uncacheable.
In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.
Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
If that's kept only for internal usage, I guess it's fine to do it that way, but I still recommend against it.
Is it appropriate to perform actions with REST, other than simple create (POST), read (GET), update (PUT), and delete (DELETE)? I'm kind of new to the whole RESTful theology, so bear with me, but how should I accomplish the following:
I have a web service that needs to talk to another web service. Web service A needs to "reserve" an object on Web service B. This object has a timeout of validity, but can be deleted immediately if need be. It's essentially a glorified permissions system which requires web services to reserve a space on web service B before taking any actions.
My initial thought was to 1. enable authentication of some sort, 2. in the serverside response to a GET call, reserve the space and return the result, and 3. provide immediate "unreservation" of the object via a DELETE call. Is this still being RESTful?
Yes, it's OK to perform actions with rest. What matters is that these actions should be guided by the representations you exchange.
If you think about the way the web works (via a browser), you do this all the time: you get an HTML form that lets you choose a number of actions you can perform. Then, you submit the form (typically via POST) and the action is performed.
It's good to be able to use DELETE via a programmatic client (which is something that non-AJAX requests in browsers wouldn't support), but the overall approach of a RESTful system should be very similar to what you find for websites (i.e. the focus should be on the representations: the equivalent of web pages in your system).
GET shouldn't have side effects, so don't use GET to make the reservation itself, use something like POST instead.
No - unlikely to be restful
From your description ...
2. in the serverside response to a GET call, reserve the space and return the result
GETs should be idempotent. For this reason alone, your service is unlikely to be restful because the state of the system after the first GET is different.
You really need to consider that a Reservation is a resource and should be created with a POST to a reservations container which will return the URI of the new resource in the Location header of the HTTP response. This UrI can be used by Get to return the resource and updated with a PUT
Post should be used to extend an existing resource and Put for replacing the state of a resource. In your case, consider the Post to be updating a list of Reservations and returning the URI of the new resource (not just the I'd). Put can be used for changing the state associated with the resource identified by the UR
You're on the right track, but your reservation of the object should be with a PUT; you're PUTting a reservation on the object, i.e. changing the underlying object.
PUT is the right verb here, since you know what resource you're modifying, and it should be idempotent for multiple requests.