I get the general concept of CQRS, but I've got a few questions when it comes to moving beyond the example code and slide decks that are out there to dealing with real world problems.
Validation
When you need to do validation of a command that involves checking values from the database, what do you do?
Take registration for a service, I must enter a unique email address. One argument i've heard is that its very unlikely that the user will enter a duplicate email address, so just handle it when processing the command and send them an email saying "sorry", or perhaps suggesting they reset their password. This process therefore avoids having a readmodel for the sake of the validation. But how do you deal with the duplicate case in the command handler? how do you know THEN that its a duplicate? check a readmodel? you might as well have used that in the first place for better usability.
Changes to functionality/fixing bugs
What happens when you need to change the way a command works, or fix a bug? In the append-only philosophy, what do I do with all the old commands and command handlers? I can't rename them to _legacy and hide them away otherwise my event deserialization won't work. What elegant solutions are there to dealing with this?
Thanks
See http://codebetter.com/blogs/gregyoung/archive/2010/08/12/eventual-consistency-and-set-validation.aspx and the cqrs mailing list for the various discussions on this very topic.
Event versioning (there's no need for command versioning in the same sense as event versioning) is also discussed on the cqrs mailing list. Events are replayed to get to the current state of an aggregate, not commands. This way your functionality can evolve. There's no way to change the past, but there are ways to change the present/future. In the odd case you have to start tracking state in a different way, provide an explicit command. For new properties on events, just provide defaults as you would for a new database column. If those new values are based on state already present, then model an explicit command to precalculate them. The advantage is that you can do this asynchronously, whereas a database upgrade forces you to change the code as well. How that would work is better asked on the cqrs mailing list.
One warning! Don't take on event sourcing too lightly. You can get by with just CQRS. I see people switching over to CQRS+ES when most of the time CQRS will do.
So come on over to groups.google.com/group/dddcqrs and get help. Another useful resource is cqrsinfo.com
Related
I have been recently looking into event sourcing and have some questions about the interactions with clients.
So event-sourcing sounds great. decoupling all your microservices, keeping your information in immutable events and formulating a stored states off of that to fit your needs is really handy. Having event propagate through your system/services and reacting to events in their own way is all fine.
The issue i am having lies with understanding the client interaction.
So you want clients to interact with the system, but they need to do this now by events. They can not longer submit a state to mutate your existing one.
So the question is how do clients fire off specific event and interact with (not only an event based system) but a system based on event sourcing.
My understanding is that you no longer use the rest api as resources (which you can get, update, delete, etc.. handling them as a resource), but you instead post to an endpoint as an event.
So how do these endpoint work?
my second question is how does the user get responses back?
for instance lets say we have an event to place an order.
your going to fire off an event an its going to do its thing. Again my understanding is that you dont now validate the request, e.g. checking if the user ordering the order has enough money, but instead fire it to be place and it will be handled in the system.
e.g. it will not be
- order placed
- this will be picked up by the pricing service and it will either fire an reserved money or money exceeded event based on if the user can afford it.
- The order service will then listen for those and then mark the order as denied or not enough credit.
So because this is a async process and the user has fired and forgotten, how do you then show the user it has either failed or succeeded? do you show them an order confirmation page with the order status as it is (even if its pending)
or do you poll it until it changes (web sockets or something).
I'm sorry if a lot of this is all nonsense, I am still learning about this architecture and am very much in the mindset of a monolith with REST responses.
Any help would be appreciated.
The issue i am having lies with understanding the client interaction.
Some of the issue may be understanding, but I promise you a fair share of the issue is that the literature sucks.
In particular, the word "Event" gets re-used a lot of different ways. If you aren't paying very careful attention to which meaning is being used, you are going to get knotted.
Event Sourcing is really about persistence - how does a micro-server store its private copy of state for later re-use? Instead of destructively overwriting our previous state, we write new information that links back to the previous state. If you imagine each microservice storing each change of state as a commit in its own git repository, you are in the right ballpark.
That's a different animal from using Event Messages to communicate information between one microservice and another.
There's some obvious overlap, of course, because the one message that you are likely to share with other microservices is "I just changed state".
So how do these endpoint work?
The same way that web forms do. I send you a representation of a form, the client displays the form to you. You fill in your data and submit the form, the client processes the contents of the form, and sends back to me an HTTP request with a "FormSubmitted" event in the message body.
You can achieve similar results by sending new representations of the state, but its a bit error prone to strip away the semantic intent and then try to guess it again on the server. So you are more likely to instead see task based user interfaces, or protocols that clearly identify the semantics of the change.
When the outside world is the authority for some piece of data (a shopper's shipping address, for example), you are more likely to see the more traditional "just edit the existing representation" approach.
So because this is a async process and the user has fired and forgotten, how do you then show the user it has either failed or succeeded?
Fire and forget really doesn't work for a distributed protocol on an unreliable network. In most cases, at-least-once delivery is important, so Fire until verified is the more common option. The initial acknowledgement of the message might be something like 202 Accepted -- "We received your message, we wrote it down, here's our current progress, here are some links you can fetch for progress reports".
It doesnt seem to me that event-sourcing fits with the traditional REST model where you CRUD a resource.
Jim Webber's 2011 talk may help to prune away the noise. A REST API is a disguise that your domain model wears; you exchange messages about manipulating resources, and as a side effect your domain model does useful work.
One way you could do this that would look more "traditional" is to work with representations of the event stream. I do a GET /08ff2ec9-a9ad-4be2-9793-18e232dbe615 and it returns me a representation of a list of events. I append a new event onto the end of that list, and PUT /08ff2ec9-a9ad-4be2-9793-18e232dbe615, and interesting side effects happen. Or perhaps I instead create a patch document that describes my change, and PATCH /08ff2ec9-a9ad-4be2-9793-18e232dbe615.
But more likely, I would do something else -- instead of GET /08ff2ec9-a9ad-4be2-9793-18e232dbe615 to fetch a representation of the list of events, I'd probably GET /08ff2ec9-a9ad-4be2-9793-18e232dbe615 to fetch a representation of available protocols - which is to say, a document filled with hyper links. From there, I might GET /08ff2ec9-a9ad-4be2-9793-18e232dbe615/603766ac-92af-47f3-8265-16f003ce5a09 to obtain a representation of the data collection form. I fill in the details of my event, submit the form, and POST /08ff2ec9-a9ad-4be2-9793-18e232dbe615 the form data to the server.
You can, of course, use any spelling you like for the URI.
In the first case, we need something like an HTTP capable document editor; the second case uses something more like a web browser.
If there were lots of different kinds of events, then the second case might well have lots of different form resources, all submitting POST /08ff2ec9-a9ad-4be2-9793-18e232dbe615 requests.
(You don't have to have all of the forms submitting to the same URI, but there are advantages to consider).
In a non event sourcing pattern I guess that would be first put into the database, then the event gets risen.
Even when you aren't event sourcing, there may still be some advantages to committing events to your durable store before emitting them. See Pat Helland: Data on the Outside versus Data on the Inside.
So you want clients to interact with the system, but they need to do this now by events.
Clients don't have to. Client may even not be aware of the underlying event store.
There are a number of trade-offs to consider and decisions to take when implementing an event-sourced system. To start with you can try to name a few pre computer era examples of event-sourced systems and look at their non-functional characteristics.
So the question is how do clients fire off specific event
Clients don't send events. They rather should express an intent (a command). Then it is the responsibility of the event-sourced system to validate the intent and either reject it or accept and store the corresponding event. It would mean that an intent to change the system's state was accepted and the stored event confirms the change.
My understanding is that you no longer use the rest api as resources
REST is one of the options. You just consider different things as resources. A command can be a REST resource. An event-sourced entity can be a resource, to which you POST a command. If you like it async - you can later GET the command to check its status. You can GET an entity to know its current state. You cant GET events from a class of entities as a means of subscription.
If we are talking about an end user, then most likely it doesn't deal with the event store directly. There is some third tier in between, which does CQRS. From a user client perspective it can be provided with REST, GraphQL, SOAP, gRPC or event e-mail. Whatever transport solution you find suitable. Command-processing part from CQRS is what specifically domain-driven. It decides which intent to accept and which to reject.
Event store itself is responsible for the data consistency. I.e. it should not allow two concurrent event leading to invalid state be published. This is what pre-computer event-sourced systems are good at. You usually have some physical object as an entity, so you lock for update by just getting hand of it.
Then an end-user client usually reads from some prepared read model. The responsibility of a read (R in CQRS) component is to prepare read-optimised data for clients. This data may come from multiple event-sourced of the same or different classes. Again, client may interact with a read model with whatever transport is suitable.
While an event-store is consistent and consistent immediately, a read model is eventually consistent. But it's up to you to tune this eventuality.
Just try to throw REST out of the architecture for a while. Consider it a one of available transport options - that may help to look at the root.
I'm currently stuck with designing my endpoints in a way so that they are conform with the REST principles but also ensure the integrity of the underlying data.
I have two resources, ShadowUser and RealUser whereas the first one only has a first name, last name and an e-mail.
The second user has much more properties such like an Id under which the real user can be addressed at other place in the system.
My use-case it to convert specific ShadowUsers into real users.
In my head the flow seems pretty simple:
get the shadow users /GET api/ShadowUsers?somePropery=someValue
create new real users with the data fetched /POST api/RealUsers
delete the shadow-users /DELETE api/ShadowUSers?somePropery=someValue
But what happens when there is a problem between the creation of new users and the deletion of the shadow ones? The data would now be inconsistent.
The example is even easier when there is only one single user, but the issue stays the same as there could be something between step 2 and 3 leaving the user existing as shadow and real.
So the question is, how this can be done in a "transactional" manner where anything is good and persisted or something went wrong and nothing has been changed in the underlying data-store?
Are there any "best practices" or "design-patterns" which can be used?
Perhaps the role of the RESTful API GETting and POSTing those real users in batch (I asked a question some weeks ago about a related issue: Updating RESTful resources against aggregate roots only).
In the API side, POSTed users wouldn't be handled directly but they would be enqueued in a reliable messaging queue (for example RabbitMQ). A background process would be subscribed to the whole queue and it would process both the creation and removal of real and shadow users respectively.
The point of using a reliable messaging system is that you can implement retry policies. If the operation is interrupted in the middle of finishing its work, you can retry it and detect which changes are still pending to complete the task.
In summary, using this approach you can implement that operation in a transactional way.
Usually when talk implementing CQRS it is supposed that commands are sorces for events. But can queries made by user be source of created events in event store? Or such actions (when we need an event that reflects query) should be implemented using command still?
But can queries made by user be source of created events in event store?
Go not to the elves for counsel, for they will answer both no and yes.
So, the "no" part: queries are distinguished by the fact that they don't change the domain model. In a CQRS implementation, the queries are being served by the read model, which may not even have access to your event store at all.
when we need an event that reflects query
The yes part: there's no law that says you can't assemble a history of queries, and stick that in your event store.
But I'm stumped, in that I don't see a clear case where the domain needs an event that reflects a query. That's really weird. My guess would be that needing an event that reflects a query is a hint that your model is broken.
You may be able to make some progress with this by exploring the source of the requirement.
If the requirement is coming from operations, analytics, reporting, usability... then the domain model probably isn't the right place for that information.
If the requirement is coming from your domain experts ("we need to capture these queries so that the model supports the right changes later"), then you should be looking to identify what entity is responsible for tracking that the query happened, and sending an appropriate command to that entity.
Still pretty young at ES and CQRS, I understand that they are tightly related to eventual consistency of data.
Eventual consistency can be problematic when we should perform validation before writing to the store, like checking that an email address isn't already used by an existing user. The only way to do that in a strongly consistent way would be to stop accepting new events, finish processing the remaining events against our view and then querying the view. We obviously don't want to go that far and Greg Young actually recommends to embrace eventual consistency and deal with (rare) cases where we break constraints.
Pushing this approach to the limits, my understanding is that this would mean, when developing a web API for example, to respond 'OK' to every request because it is impossible, at the time of the request, to validate it... Am I on the right track, or missing something here?
As hinted in my comment above, a RESTful API can return 202 Accepted.
This provides a way for a client to poll for status updates if that's necessary.
The client can monitor for state if that's desirable, but alternatively, it can also simply fire and forget, assuming that if it gets any sort of 200-range response, the command will eventually be applied. This can be a good alternative if you have an alternative channel on which you can propagate errors. For example, if you know which user submitted the command, and you have that user's email address, you can send an email in the event of a failure to apply the command.
One of the points of a CQRS architecture is that the edge of the system should do whatever it can to validate the correctness of a Command before it accepts it. Based on the known state of the system (as exposed by the Query side), the system can make a strong effort to validate that a given Command is acceptable. If it does that, the only permanent error that should happen if you accept a Command is a concurrency conflict. Depending on how fast your system approaches consistent states, such concurrency conflicts may be so few that e.g. sending the user an email is an appropriate error-handling strategy.
I am currently drafting a concept for a (mostly) HTML-based collaboration suite which I plan to implement using CQRS. This software will contain messages that can be sent to the user (which can either be read or unread, obviously) and other elements which shall be marked "new" if they were created after the last user login.
Hardly something new, but I am not quite sure how that would be correctly implemented using CQRS. As I understand it, Change of any kind should, without exception, only be possible via Commands. But creating commands for every single (new) element that is being accessed seems a bit too much, not to mention the overhead.
I don't know if I need it, but what would be the best way to implement a Last-Accessed Timestamp on elements. Basically the same problem like the above, with the difference that the change happens EVERY time the element is accessed, not only the first time for each user.
CQRS seems to be an awesome concept but it really needs more learning material. Can't wait till a book is released :)
Regards
[Edit] No one? Wouldn't have thought that this is such a complicated issue..
I assume you're using event-sourcing in which case once you allow your query-service/event-handlers to raise appropriate events then this becomes fairly easy to solve.
For your messages/elements; when handling the specific creation events of your elements either add to existing or create additional event-handlers, to store to a messages read-model with a status of new and appropriate information about the element.
As part of you're user login I don't see why you can't raise a user-logged-in event (from the security/query service depending on how your implementing authentication) to say the user has logged in. An event-handler could capture this and write the last-login timestamp to a specific user-last-login read-model.
In addition the user-logged-in event-handler would need to update all the new messages (for that user) to an unread status. Seeing as we're changing the status of the messages as the user logs in do you still need to store the last-login timestamp?
For your last-accessed timestamp, perhaps you could just work this into your query service as queries for your different elements complete. Raise a query-completed event with element id/type information.