I would like to clarify what'd be the best way of organizing architecture.
I've got rest api and microservices architecture. I have applied the Database per Service pattern.
So let's imagine that the user wants to create an order(an e-commerce system). But users can have a credit limit. So the flow will be as below:
OrderService creates a pending order. Then push an event about it.
UserService processes that event and publishes either a credit limit exceeded event or a credit reserved event.
OrderService receives the event and changes the state of the order to either approved or canceled.
All look good. But the question is what the user will do during this simple flow?
I mean: the user makes a POST request /orders and ...
Web-service awaits until the order gets approved or canceled(includes surely timeout)?
Web-service returns 200 ok and then the user needs to check the order state with some interval?
use web sockets?
something else?
Unfortunately, Any option from above has its advantages and its disadvantages.
The challenge is that I described the simplest case. In reality, tens services(even third party) could be involved. And of course, I am expecting a high load. So the queue may be filled.
Please propose the solution to discuss. I'm greatly appreciating for an answer as well as links to production ready system documentation.
Kudos for a good question. If we look at Saga Pattern, it offers to do ACID like transactions in a distributed system but with some trade-off. One of the trade-offs is Ensuring Rollback if any of the service or entity fails to do what it was supposed to do. This can go even complex if you have more than 5 services completing a Saga. Though it will be a Highly Scalable option if you can orchestrate.
Here I will propose Following,
User Does a POST request for order
OrderService first checks with UserService regarding credit Limit (It can be a REST API call as it might be a simple DB call for UserService)
Then OrderService can act upon based on the return response from UserService
Advantages of this approach
User can see an immediate response
Less Complex to code thus testable and maintainable Code
Disadvantages of this approach
This solution will not be effective if there are too many external (third party) rest api calls
It can introduce single point of failure
The main thing is, all the options will have trade-offs. It is on you to decide which one suits you the best.
According to me
Follow the my rule - Validate First then action
Validate the order before its goes to order service on UI. You can call the userservice api on UI which gives info user can make a order or not.
Step 1. On UI call the userservice API , which validate the user credit limit
Step 2. If validation success or fail
Step 3. Take the action according to step2 result . If success call the order service if not do what you want to do (Call other service etc..)
Advantage
A good user exprience - User get the validation message "you can not make a order your limit is low" or something like that when trying to make the order. Not at the end get a message "You limit is low/or anything"
Less Complexity of code
A more microservice Arch. -
Since your are follow the microservices then if you call the userservice inside the orderservice means orderservice is dependent on it or you can say tightly bound - here your are lossing the benifit of mircroservice arch.
What happen if the userservice is down. The customer cannot make the order or if you change anything in userservice response then you have to change the orderservice also.
What you do if some more validation (to call other service) need in future before making the order.
Redirection Flexibility - Suppose if on fail of validation from userservice then you have to call other service( other independent microservice ) and if you are using the other service inside the orderservice the no of service increases as project grow
Less Request on OrderService - Request goes orderservice only when the validation is passed from the other service or other thing.
So from my point of view
Try to make microservice independent as much possible.
Validate First and action.
A good user exprience
Easy to handle load (if you think)
With all that , now you less dependent on saga.
Its only my opinion - choose what you think best according to your domain
The question is what the user will do during this
simple flow? I mean: the user makes a POST request /orders and ...
You can reply to user the created orderId with status 'pending' while your saga is running. The request should be ended in 200.
How does user get the status?
Push notification, websocket, email, sms. it is up to you to notify the user about the status of event, success, failure, waiting_for_approval.
1. Web-service awaits until the order gets approved or canceled?
2. Web-service returns 200 ok and the user check the order state with some interval?
3. use web sockets?
4. something else?
Not recommended. Saga should be asynchronous. Waiting for reply is synchronous (request/reply style).
This will work, but if your saga is long running, it might be problematic, eg: anything with validation/approval by a human.
This will work assuming you are replying 200 after order creation in pending state, one thing is that you have to make sure your websocket is active, or else, you still have to notify them through email, sms.
Unfortunately, Any option from above has its advantages and its
disadvantages.
You have to choose your poison.
The challenge is that I described the simplest case. In reality, tens
services(even third party) could be involved. And of course, I am
expecting a high load. So the queue may be filled.
Use orchestrated saga (sort like) Netflix conductor, Uber Cadence, Temporal (Similar to Cadence). This will allow you to do something as shown below. At least, easier.
A flow diagram from google shopping api and how they handle it.
Link for more: https://developers.google.com/shopping-content/guides/about-orders
There is little difference in cancelling an order because there was a problem with the credit limit vs. cancelling it because a forklift ran over the package in the logistics facility.
So a relatively simple solution is to reply with an order accepted and status 200 when the order is placed put that in a queue and let the saga run asynchronously. You should provide both an API (and UI) for the user to check the order status and proactively send status updates like billing going through (which is your fist saga) and also other sagas in the order lifecycle
Regarding how to send notifications that depends on the usage scenario (e.g. you can do a callback for a system-to-system integration, but maybe you want to send an email/sms to an end-user etc)
Related
I am designing a microservice architecture, using a database per service pattern.
Following the example of Order Service and Shipping Service, when a user makes an HTTP REST request to the Order Service, this one fires an event to notify shipping service. All this happens asynchronously. So, what happens with the user experience? I mean, the user needs an immediate response from the HTTP request. How can I handle this scenario?
All this happens asynchronously. So, What happen with the user experience? I mean, the user needs an immediately response from the HTTP request. How can I handle this scenario?
Respond as soon as you have stored the request.
Part of the point of microservices is that you have a system composed of independently deployable elements that do not require coordination.
If you want a system that is reliable even though the services don't have 100% uptime, then you need to have some form of durable message storage so that the sender and the receiver don't need to be running at the same time.
Therefore, your basic pattern for data from the outside is that the information from the incoming HTTP request is copied, not directly into a running service, but instead into the message store, to be processed by the service at some later time.
In other words, your REST API is a facade in front of your storage, not in front of the service itself.
The actor model may be a useful analogy; information moves around by copying messages into different inboxes, and are later consumed by the subscribing actor.
From the perspective of the client, the HTTP response is an acknowledgement that the request has been received and recognized as valid. Think "thank you for your order, we'll send you an email when your purchase is ready for pick up."
On the web, we would include in the response links to other useful resources; click here to see the status of your order, click there to see your history of recent orders, and so on.
For ex. I need to design an API with two features: payment validation and payment creation.
I need both because the end user should know that everything is ok before actually confirm the payment creation.
For creation I have
POST .../payment
with an input body. It returns HTTP 400 in case something went wrong.
The validation (or simulation) does the exact same process than the creation, stopping the process just before persisting the data. For this validation, is it better to have something like
Solution 1
GET .../is-payment-ok
also with an input body. It returns HTTP 200 including a boolean answer and some details.
Here the resource is not the payment, it's the information about the validity of the payment, which seems REST-compliant to me. The drawback being that the user of the API could be confused because if the payment is not valid, then the simulation would return HTTP 200 (with the boolean in the body set to false) while the creation would return HTTP 400.
, or
Solution 2
POST .../payment?simulation=true , or
POST .../payment-simulation
Here the HTTP response code would be exactly the same as for a payment creation. The drawback being that we use a POST, without actually 'posting' any resource.
How would you do it ? Is there a REST rule or common practice for this case ?
While not strictly forbidden, GET requests usually have no request bodies: HTTP GET with request body, so I would not go with Solution 1.
In our APIs (Payment APIs too) we use a query param with the same URL to flag a dry-run (or forecast in our domain terms), just like your Solution 2, and as described here: Dry run strategy for REST API. It gives back the same error codes and messages that non-forecast requests would yield in case of an error.
We needed this due to we use a reactive payment processor, but it only starts if the payment-forecast succeeded. The process give back this forecast result as a response to the user immediately after the process starts, increasing the responsivity of the UI. The actual payment will take a second to be sent for real processing, as some checks are done reactively with other services. It may even fail and the end-user will get a notification about the final status, but this will be very rare.
If you check the HTTP spec, how POST processes a resource is pretty arbitrary, there is no need to make a persistent change in anything: https://www.rfc-editor.org/rfc/rfc7231#section-4.3.3
The POST method requests that the target resource process the
representation enclosed in the request according to the resource's
own specific semantics
[...]
If one or more resources has been created on the origin server as a
result of successfully processing a POST request, the origin server
SHOULD send a 201 (Created) response containing a Location header
field that provides an identifier for the primary resource created
(Section 7.1.2) and a representation that describes the status of the
request while referring to the new resource(s).
Just make sure to document the query param. EDIT: Kubernetes does it: https://kubernetes.io/docs/reference/using-api/api-concepts/#dry-run
The Stripe API is a bit different since if you send a request it will be booked, there is no dry-run (the /capture call is called by the money requester, not the payer, so it is another use-case).
I would suggest you to create 2 endpoints, this is commonly done among payment service providers like UsaEPay,Stripe,etc.
The first endpoint will be something like: POST .../authorize and will receive all the information needed to make the payment such as the amount, payer info and payment method info, then it will create a paymentAuthorization and return the status of it among with an authorizationToken if the authorization succeeded.
The second endpoint will be something like: POST .../capture, in this endpoint you will only look for the paymentAuthorization related to the authorizationToken you generated before and proceed to make the payment
Also you could take a look a the Stripes documentation https://stripe.com/docs/api/payment_intents to understand more about this kind of system structure
I agree with the other answers for using your option 2 with two endpoints to reduce ambiguity.
You use the term validation in your question, and I believe it relays what is going on better than simulation. I would expect a simulation call return the same response as the payment, but not allow the payment to go through.
A validation request would more intuitively let the user know that the API would evaluate the request and at the very least tell you if it was acceptable or not.
Option 1 would go against standard design principles, as a GET request typically has the body ignored if it exists.
Follow your domain's uri path design, or use some of the resources mentioned in the other answers.
POST .../Payment
POST .../Validate/Payment
Depending what type of payments you're working with, you may be interested in a two-phase commit approach. This also aligns more with the REST interface than the RPC-like "is this ok" call.
The idea here is that you create one resource (let's call it the payment authorization, POST /payment-authorizations) which ensures all resources are available, in this case putting a hold on the user's account for the amount of the payment, and returning the id of the new payment.
Now the user can review the payment (GET /payment-authorizations/{id}).
They can cancel it if they want (DELETE /payment-authorizations/{id}), releasing the funds.
Or they can finalize it (PUT /payments/{id}).
The main advantages of this approach are that
It's RESTfull
Payments are idempotent (you can try to finalize a payment as many times as you want, but it will only ever send the money once)
The payment validation step is ACTUALLY USEFUL
Elaborating more on that last item: these API calls you're making are asynchronous, and race conditions are fully possible. You could have a scenario like
Alice Alice's Account Chuck
$10
"can I send $5 to Bob?" ->
<- "yes"
<- "can I withdraw $6 for rent?"
"yes" ->
<- "withdraw $6 for rent"
"ok" ->
$4
"send $5 to Bob" ->
<- "insufficient funds"
Now your client is confused, the system said it could make a payment, but it can't. With a two-phase commit you'd have
Alice Alice's Account Chuck
$10
"authorize $5 to Bob" -> "ok, authz #5"
$5(+5)
<- "authorize $6 for rent"
"insufficient funds" ->
"send payment #5" -> "ok"
$5
Now Alice is still broke, but all the responses we've given are accurate.
As i can see, you are interested in URI Path Design.
Usually:
Singular Nouns are used as to refer a single document resourse into your URL.
Plural Nouns are used to refer to a collections or stores.
Verrbs represent actions, which can be considered as a controller, like Validate or Store.
In REST, a request of type POST can be used to create a new resource into a collection or execute a controller. So, using Post seems to be a correct choice for your API.
Now, in your case, you need to perform two actions Create and Validate:
In your case, i would create two possible URL, one for create a Payment and another one to validate it, sending an Id of the payment.
POST .../Create/Payment
POST .../Simulate/Payment/{IdPayment}
I considered Payment as a single document and Create and Simulate as a controller. In the case of Simulate, i adding am Id to the Payment document to identify it.
There is a book, from O'Reilly, which is easy to read and has all the information that you are looking for. This book is REST API Design Rulebook
I hope this information could be helpful for you.
I'm implementing a CQRS/ES based system with a RESTful interface which is used by a webapp.
When performing certain actions e.g. creating a new profile I need to be able to check certain conditions, such as uniqueness of the profile ID, or that the person has the right to create a resource under a group. Which means I have a couple of options:
Context: POST/profiles { "email": "unique#example.com" }
From my REST API return 202 from my service with a location of the new resource where my client can poll for it. In this case, however, how do I handle errors as in effect the view will not exist or ever exist.
Create a saga on the initial request then dispatch the event. Once my service creates the view or finds the error then the result is written to the saga. When the saga has been completed return the result to the user.
From these two options - the second seems more reasonable to me, if not more complex. Is this a viable option for building RESTful request/response models on a CQRS/ES event sourced backend?
Yes, the second solution seems to better fit the business.
From what I understand from your case, from the DDD point of view, the creation of a user profile is a business process, with more than one steps (verifying the uniqueness of the profile, creating the profile and recovering from a duplicate profile situation). This process acts like an entity, it starts, runs and ends with a result (success or error). Being an entity it has an ID and it can be viewed as a REST resource. A Saga will be responsible for executing it.
So, in response to the client's request you send the URI of the process resource where the client can poll for the status. In case of error, it reads the error message. In case of success, it gets the URI of its profile.
The first solution can still be used if the use-case is simpler, if the command can be executed synchronously and the client gets the final result (error or success) as an immediate response.
From my REST API return 202 from my service with a location of the new resource where my client can poll for it. In this case, however, how do I handle errors as in effect the view will not exist or ever exist.
The usual answer here is that, as part of the 202 Accepted response, you include monitoring information
The representation sent with this response ought to describe the request's current status and point to (or embed) a status monitor that can provide the user with an estimate of when the request will be fulfilled.
In other words, a link to a resource that will change when the accepted request is finally run.
So in describing the protocol, in addition to the resource that you create, you'll also need to document the representation used when you defer the work for later, and the representation used by the monitor.
When the saga has been completed return the result to the user.
Depending on the work, that may be overkill.
Which is to say, you are raising two different questions here; one of those is whether the request should be handled synchronously (don't respond until the work is done) or asynchronously (return right away, but give the client the means to monitor progress).
The other question is how the work looks from the business layer. If you are going to need multiple transactions to make the change, and if you may need to "revert" previously committed transactions in some variants of the process, then a saga (or a process manager) makes sense.
Set Validation -- the broader term for enforcing an invariant like "uniqueness" -- is awkward. Make sure you study, and ensure that you and the business understand the impact of having a failure.
I know that to make a stateless application, we need to transfer the user state back and forth instead of server holds the user state.
However, there must be some states stored in the server, I read this article that the state stored in server is called resource state.So if I am right, client state which we often call should be the same as application state.
So, how do I distinguish these two, since it will determine that whether they should stored in server or transferred.
Take a shopping cart as an example.
If there is 5 steps before a user to complete his purchase, the user's phase where he is(#3,#4) in seems to be an application state, but does this mean if they close the browser and click on pay again, he will have to start from the step1?
What about the items in his chart? If we treat it as application state, we need to put all of the items in the request. But if we do this, when user close the browser and login again, he will not able to find his items again, since the browser can not remember all the items. So it seems we should treat it as a resource state. But if so, when a user click on pay , they will have a different page: go to pay or say "your cart is empty" based on whether his shopping cart is empty or not. So , the same requests with exactly the same param input, comes out the different result, can we still say it is stateless?
Maybe I understand something wrong, can any body please answer how to distinguish different kinds of state and how to treat them differently?
Resource state is a state that needs to be persistent and survivable even after client disconnect/restart/session end/whatever.
Application state should live on the client and should be supplied with each client request(if we are talking about REST architecture and planning to scale our application well).
How to distinguish application state and resource state?
It depends on the task your are working on. E.g. if you are trying to figure out where to save index of the picture that is currently viewed in your gallery, probably, you could do it in your Application state, because you, likely, don't need this state to survive for the next session of this client. Of course, you can save it in your Resource state(database), but it would be overhead(a lot of effort for a very small gain).
But if your are working on multistep purchasing process, probably, it's better to save state of this process in your Resource state(database), because you want this state to be saved permanently. Otherwise, your clients need to refill a lot of information after disconnect/restart/whatever.
Of course you could do it in cookies for example(and it would be Application state), and this state can live after browser restart. But it has the two downsides: 1)This state unavailable on other user's devices, 2)If you are creating genuine REST service, cookies would complicate client's life, because not all clients operate cookies well(except browsers).
Let me just quote one paragraph of the book RESTful Web Services :
The Flickr
web service lets you upload pictures to your account, and those pictures are stored on
the server. It would be crazy to make the client send every one of its pictures along with
every request to flickr.com, just to keep the server from having to store any state
The application state is related to the path that the client can follow to make some actions.
For example, when consulting an article, a link "add to cart" appears. When consulting his cart, a link "pay your order" is provided if you have one article in your cart otherwise this link does not appears. Feel free to the user to make its own application state based on the link that he follows. Basically, application state is a matter of context.
One other quote from the same book mentionned earlier before I go back to you example:
Resource state stays on the server and is
only sent to the client in the form of representations. Application state stays on the
client until it can be used to create, modify, or delete a resource. Then it’s sent to the
server as part of a POST, PUT, or DELETE request, and becomes resource state.
So let's say that you have some authentification mechanism (based on token). And to one user account is associated one cart.
When you are adding items in your cart, you are modifying your cart resource. As resource state are server-side, it's on server.
Suppose that you disconnect, and reconnect like described in your first point. The cart is still here.
As long as the client send the different authentification credential at each request, your application remains stateless.
A good discussion on SO about how to manage it: Do sessions really violate RESTfulness?
Now, what about the fact that: consulting your cart can leads you to 2 different action depending on whether it has items or not.
Pretty simple. What is served by the server to the client depends on the resource state maintains by the server.
A very simple example on this good website. You can see that depending on the amount of money on the account, the server provides a link to the client to make a withdraw or not.
Feel free to the client to make its own application (again) and to follow the link or not.
I recommand you to take a look at HATEOAS and the Richardson Maturity Model that explains it.
By the way, the quotes from the 2 paragraphs are from the same author that this model.
I'm having some trouble with RPC style verbs starting to appear in my API.
My question is: Is it okay to have these verbs, "Submit", "SendForApproval", "Accept" and "Reject" in a Restful API? Is that going against the Restful principles? What's best practice for naming these Actions?
Our business case:
I have an order. It starts out at status "pre-ordered". I want to then make a call to validate the order against our business logic. Depending on the validation results, I then either submit it or send it for approval. The approval process happens externally to our solution and then when it has been through the approval process the Api is called to mark the order as either Accepted or Rejected.
There is a bit more going on than simply updating statuses - we send things to print queues, spin off tracking documents etc.
So the URI's that we have defined are
https://api.site.com/orders // A list of orders
https://api.site.com/orders/{orderId} // An order
https://api.site.com/orders/{orderId}/ValidationErrors // Get a list of validation results
https://api.site.com/orders/{orderId}/Submit // Submit the order for processing
https://api.site.com/orders/{orderId}/SendForApproval // Send to supervisor
https://api.site.com/orders/{orderId}/Accept // Supervisor Accepts the order for processing
https://api.site.com/orders/{orderId}/Reject // Supervisor can Reject the order
You must avoid inserting verbs in your URIs as much as you possibly can. Furthermore, I disagree with user1438038, because the QueryString filters should be used for search purposes instead of specifying actions.
I believe that there're better approaches to solve this problem. First off, take into account that you shouldn't model your RESTful API in a way that it's highly tied to your business objects (supposing that you're using a OO approach). Moreover, remember that in a REST perspective, everything you expose is a resource rather than objects with its state and behavior.
I'm going to suggest you a different approach but I'm not sure if it's going to fulfill your system requirements but maybe it can be useful to give you some ideas. In your case, you can have multiple resources like PreOrders, Orders, Supervisors...
https://api.site.com/preorders
GET - Display all pre-orders.
POST - Create a new pre-order.
Creating a new order:
https://api.site.com/preorders/1
POST - approves the pre-order. This action's gonna create a new Order.
Displaying all orders that are waiting for approval (by the supervisor with ID=1).
https://api.site.com/supervisors/1/orders
Searching for approved and canceled orders:
https://api.site.com/supervisors/1/orders?status=canceled
https://api.site.com/supervisors/1/orders?status=approved
To cancel or approve an order:
https://api.site.com/supervisors/1/orders/1 (DELETE cancels the order)
https://api.site.com/supervisors/1/orders/1 (POST approves the order)
I'm unsure if it's the best approach to solve your problem but I'm pretty sure it's better than using verbs in your URI.
I'd be fine with that, we are using a very similar setup for order management.
However, another possibility would be passing an action parameter, such as:
https://api.site.com/orders/{orderId}/?action=accept
https://api.site.com/orders/{orderId}/?action=reject
...
The general idea of REST is to have resources (such as order or customer). What you want is more of a RPC semantics, but there is no right or wrong. Your approach is reasonable.
Not is a good practice have verbs in REST uris.
So, if you want change a status in any order, you may send a PUT/POST to https://api.site.com/orders/{orderId} updating only the values you want for your bussines logic.