I first heard about CQRS/ES in a ScalaDays San Francisco presentation some months ago. Since then, I have noticed occasional mentions of the terms in technical writing, most recently Fiverr's blog post on the topic.
It's seemed very interesting so far, but I have encountered a number of related issues, and I am wondering if there are any common solutions:
How do you deal with eventual consistency and potentially inconsistent views in the short term?
How do you deal with business logic which touches multiple resources a.k.a. atomic reads/writes a.k.a. transactions?
How do you deal with use cases where the user interface requires an immediate result to the request?
Moreover, what are some advantages of adopting this pattern, and how do they exceed the cost of the extra layer of complexity?
Thanks in advance!
How do you deal with eventual consistency and potentially inconsistent
views in the short term?
How do you deal with use cases where the user interface requires an
immediate result to the request?
It is a common misconception that using Event Sourcing automatically implies an eventually consistent read model. Many ES practitioners (up to and including Udi Dahan and Greg Young) recommend starting with a synchronous, immediately consistent model and moving toward eventual consistency only when it is needed.
How do you deal with business logic which touches multiple resources a.k.a. atomic reads/writes a.k.a. transactions?
This is a complex question that doesn't have a single answer, and may be more appropriate for a discussion forum vs a Q/A forum like SO. I would recommend poking around (and posting) in the DDD/CQRS discussion group.
Additionally, here are some slides I recently found on the topic. They cover a lot more of the "why"/"why not" aspects from a team which used CQRS/ES in practice.
http://ookami86.github.io/event-sourcing-in-practice/
Related
In the past I worked with what I believe were "microservices". We had a service discovery and the applications communicated with each other through REST sync calls, which I believe is called request / response.
Now we are working on an application that is simply broken down into multiple small applications which work together through publish / subscribe using Kafka, there's no direct communication between them nor a service discovery, per say.
Is it safe to say that these are "Microservices" too or what should I call them?
As one commenter pointed out- An answer will always be opinionated (and stackoverflow is generally not the best place for opinion based questions), but I am not afraid to give mine in an attempt to help.
My view goes along the lines of what Martin Fowler and others have taught us in last decade, as well as the insight from the pioneering work of Werner Vogels at Amazon. Their definitions are that microservices are not a well defined single standard, but an architectural style that can encompass a range of different approaches in distributed computing. What they have in common are the goals of providing scalability in many different dimensions, but mostly system complexity, organizational productivity, throughput and resilience - All while being affordable. To achieve that the designs should be centered around:
Service boundaries cut according to business goals (a type of Domain Driven Design).
Sizing / Scope of services to be limited (the famous two-pizza team size idea).
Clear separation in terms of organizational responsibilities and maintenance. This includes independent teams and release-/ maintenance cycles for each service.
Embrace of automation on all layers (how cloud virtualization, devops, CICD, infrastructure as code, etc. became popular tools).
Design focus on failure-resilient operation, e.g. avoid cascading failures, defined failure states, fallbacks, etc.
A consequence of those goals is also the embrace of distributed system architectures and design for horizontal scalability (statelessness, separation of compute and storage, etc.).
So if you feel your designs are along those lines then in my opinion you would have applied the microservices architectural style successfully.
If you want to learn more about this school of thought I recommend this read.
I'm searching for a topic for my bachelor thesis. I talked to a research assistant and he told me that there might be a lot of other operations (e.g. BI-functions like trigger actions or calculate something) except of CRUD that you use REST for. He offered me two topic. One of them is to expand a benchmarking framework to be able to handle these operations.
I did some pre-research but couldn't find anything in this direction.
Since the topic sounds really cool I wouldn't like to reject it that fast.
I need a hint what kind of abstract operations there might be.
Thanks
the rest operations are based on HTTP Protocol which support several methods (OPTIONS,HEAD,GET,POST,PUT,DELETE also TRACE and CONNECT but not used in REST API)
see https://www.rfc-editor.org/rfc/rfc2616 for more details
Can I do with REST any other Operations than CRUD, e.g. BI-functions?
Especially in the context of thesis work, you are going to want to start from chapter 5 of Fielding's disseration.
If you read carefully, you'll discover that REST is an architectural style, and the "killer app" built using that architectural style (mostly) is the World Wide Web.
So the question "Can I do BI-functions with REST?" is closely analogous to "Can I create a web interface that implements the application protocol for BI-functions?"
Given that the arrival of the Web was a catastrophic event for legacy information systems generally, I'm going to guess that the answer is "yes, of course". But what form that might take, and what sorts of standards might emerge from the work is not obvious to me.
After being exposed to scala's Actors and Clojure's Futures, I feel like both languages have excellent support for multi core data processing.
However, I still have not been able to determine the real engineering differences between the concurrency features and pros/cons of the two models. Are these languages complimentary, or opposed in terms of their treatment of concurrent process abstractions?
Secondarily, regarding big data issues, it's not clear wether the scala community continues to support Hadoop explicitly (whereas the clojure community clearly does ). How do Scala developers interface with the hadoop ecosystem?
Some solutions are well solved by agents/actors and some are not. This distinction is not really about languages more than how specific problems fit within general classes of solutions. This is a (very short) comparason of Actors/agents vs. References to try to clarify the point that the tool must fit the concurrency problem.
Actors excel in distributed situation where no data needs to be concurrently modified. If your problem can be expressed purely by passing messages then actors will do the trick. Actors work poorly where they need to modify several related data structures at the same time. The canonical example of this being moving money between bank accounts.
Clojure's refs are a great solution to the problem of many threads needing to modify the same thing at the same time. They excel at shared memory multi-processor systems like today's PCs and Servers. In addition to the Bank account example, Rich Hickey (the author of clojure) uses the example of a baseball game to explain why this is important. If you wanted to use actors to represent a baseball game then before you moved the ball, all the fans would have to send it a message asking it where it was... and if they wanted to watch a player catching the ball things get even more complex.
Clojure has cascalog which makes writing hadoop jobs look a lot like writing clojure.
Actors provide a way of handling the potential interleaving and synchronization control that inevitably comes when trying to get multiple threads to work together. Each actor has a queue of messages that it processes in order one at a time so as to avoid the need to include explicit locks. In this case a Future provides a way of waiting for a response from an actor.
As far as Hadoop is concerned, Twitter just released a library specifically for Hadoop called Scalding but as long as the library is written for the JVM, it should work with either language.
Besides missing some of the benefits of Event Sourcing, are there any other drawbacks to adapting an existing architecture to CQRS without the Event Sourcing piece?
I'm working on large application and the developers should be able to handle separating the existing architecture into Commands and Queries over the next few months, but asking them to also add in the Event Sourcing at this stage would be a HUGE problem from a resourcing perspective. Am I committing sacrilege by not including Event Sourcing?
Event Sourcing is optional and in most cases complicates things more than it helps if introduced too early. Especially when transitioning from a legacy architecture and even more when the team has no experience with CQRS.
Most of the advantages being attributed to ES can be obtained by storing your events in a simple Event Log. You don't have to drop your state-based persistence, (but in the long run you probably will, because at some point it will become the logical next step).
My recommendation: Simplicity is the key. Do one step at a time, especially when introducing such a dramatic paradigm shift. Start with simple CQRS, then introduce an Event Log when you (and your team) have become used to the new concepts. Then, if at all required, change your persistence to Event Sourcing and fire the DBA ;-)
I completely agree with Dennis, ES is no precondition for CQRS, in fact CQRS on its own is pretty easy to implement and has the potential to really simplify your design.
You can find a smooth introduction to it here
Secondly what benefits does CQRS on its own bring to the table?
Simplifies your domain objects, by sucking out all the query concerns
Makes code scalable, your queries are separated and can be easily tuned
As you iterate over your product design you can add/remove/change
individual commands/queries, instead of dealing with larger
structures as a whole (e.g. entities, aggregates, modules).
Commands and queries produce a well-known vocabulary to talk with
domain experts. Other architectural patterns (e.g. pipes and filters,
actors) use terms and concepts that may be harder to grasp by
non-programmers.
Limits the use of ORM (if you use one), I feel ORM's bring in
unwarranted complexity if you try and use them for querying, the
abstractions are leaky and heavy, trying to tune them is a nightmare :). Using an ORM only on the command side makes things much
easier, plain old SQL is the best for queries, probably using a simple library to convert result sets into DTO's is the most you need.
More on how CQRS benefits design can be found here
Also do not forget about the non tangible benefits of CQRS
If you still have your doubts, you may want to read this
We currently use CQRS for projects with medium complexity and have found it be very suitable. We started out using a custom bootstrap code and have now moved on to using the Axon Framework to give us some of the infrastructure components
Feel free to PM me in case you want to know anything more specific.
There is a fundamental problem of Event Sourcing pattern, that is the events in the Event Store may not be compatible with your event handlers in your application any more due to code change.
That being said, whenever you modify the event handler by adding new features, you need to think about the backward compatibility. You have to make sure your code can always handle the same event in different stage created by different version of your code.
When your application becomes more complex, you will find it really pain in the ass to make it backward compatible.
I think Event Sourcing is what makes people to be afraid of CQRS. And thats for a reason. Its not natural - when you interact with something in real world you don't need to get whole history about this object.
“event sourcing is a completely orthogonal concept to CQRS” (source) - technically if you don't use ES you loose nothing from CQRS features.
I have no idea why Event Sourcing is considered as the only foundation for solving of some "messaging" related problems like: duplication/missing of messages, reordering of messages and data collisions, etc. Its not true that if you don't use Event Sourcing you cant create encapsulated means to solve such problems other way.
How i see alternative ways to implement CQRS's messaging using another data-organizing principle you can read here.
I propose "signed documents" approach where you treat your data not as composition of modification events, but as composition of immutable parts signed by responsible users. Im sure there can be a lot of other solutions to implement message flow and data storage. And you need to take your business model into account when selecting which one you like to use.
The best CQRS pattern based framework in my opinion is MediatR by Jimmy Bogard, If you don't need Event Sourcing in the beginning of your application development, MediatR is the right choice. Here is the repository- https://github.com/jbogard/MediatR
In my opinion, you're making a big mistake by not using event sourcing with CQRS.
First up, you'll almost certainly have issues synchronising your Query model with the Command model. With an event store, if the query side ever gets out of synch, you simply need to replay your events to correct it. That's the theory anyway!
But with Event Sourcing, you also get to store the complete history of all entity transactions. And this means you can decide to create new queries and views after implementation. These are very often views that would not be possible with non-Event Sourced CQRS. I've heard Greg Young give the example of querying items that have been added, and then removed, from a shopping cart. With Event Sourcing this is possible. Without ES it's not possible because you only store the final state of the cart.
I'm working on a simple eCart system using .net4 (c#). I've been doing a lot of reading about Unit of Work Pattern, Repository Pattern, and Persistence Ignorance. I think I have a grasp on the strategy and benefits to building my layers this way, but for my simple app I'm wondering if it's necessary and if anyone can point me towards good architecture for my scope.
Please correct me if I'm wrong - the main benefits to using repositories are to create fewer trips to the DB and to separate application architecture from DB architecture. IE - what's good for DB performance isn't always good for application design so it's best to design what's best for both and then create an interface between the two.
So here's the question - I want any business transaction that occurs to be saved to the DB as soon as it occurs, so there doesn't seem to be a point in queuing data in repositories and then saving it immediately. Why not just save it directly?
Are there other benefits of DDD that I'm missing or would it be over engineering to build out such a robust architecture for every simple project that comes along? Thanks for any help.
Do you need to use [insert pattern here]: Nope When it comes right down to it, the best practice is always the one that gets your application done, and meets the time, monetary, and technical requirements.
But if you take the "lets just get it done" approach, then be aware of the Technical Debt you might be incurring.
Also there are a lot of reasons to use some of these patterns (and they don't always have to do with performance), particularly the Unit Of Work pattern. This has more to do with the requirements and restrictions that often come with ORM's and such. These issues can be a bit complex, but I suspect as you begin to implement some of these things you'll start to realize what those issues are and come to understand why these patterns are useful.
Agree with CodingGorilla. Patterns are great unless they conflict with YAGNI.
If every transaction needs to be written immediately (that is, if you have potential contention between the actions of two users) then you will need a queuing mechanism or you can use the underlying transactional mechanism of whatever data repository you might be using (e.g. SQL)