I'am wokring on a project where I have implemented CQRS without event sourcing. I am using Mongo for the read database and also for the write database. When something has to be changed it is first changed in the write db and then the read db is synchronized.
Later I introtoduced somehting like Event Store, also a MongoDB instace. I am making history of all the events that changed the other databases in some way. The read db does not synchronize with the Event Store, so I have no way to read the events.
I've reached a situation where I need the information that's inside the events from the Event Store. Should I connect directly to the Event Store and read from there or I should make the read db synchronize with the Event Store and basically hold a duplicate of the Event Store?
Thank you in advance guys! I am using C# .NET Core if someone needs this kind of info.
There's no functional reason why you shouldn't just read events directly from the event store.
The motivation for read models is usually low latency queries; we take the representation of data in the book of record (the event streams), and reshape it into a data structure that can answer queries. We accept the consequences of eventual consistency to get fast response times.
But if the shape we need for a query is an event stream, then we can just use the source data.
If the queries of the event store are having a negative performance impact on the writes to the store, then we might redesign the system to direct queries to a cached copy of the events instead.
Related
Can a Firestore client detect subscriptions to a document? I want a client to only publish (write) document changes if at least one other client is listening to that document. In my case, this would dramatically reduce the number of Firestore writes across the database.
Scenario:
a client is sampling a measured value every 3 seconds and publishing this to Firestore
another client (an app) is listening to this Firestore document and displaying the value updates
the app is only open on occasion, when the user wants to view the data. Most of the time it is not open, and thus no clients are listening to that document
Is there any way in the Firestore API to detect if a document is being listened to?
No, that's not possible. Code that writes a document will do so unconditionally. You can't arrange for a write to happen if there are no listeners, and you can't find out if there are any listeners at all. You will need some other (much more complex system) set up to do something like that.
As Doug said, there is nothing built in to Firestore to detect whether there are listeners on the data. The closest I can think of is the presence system that is (sort of) built into Firebase's other database (Realtime Database), but that doesn't exist on Firestore (yet, an Extension is in the works for it).
But even on that you'd probably need to do some extra work, to track specifically what collection(s) each client is listening to. It'd boil down to:
When attaching an observer, also record that observer in the Realtime Database. At the same time register an onDisconnect handler for this node, so that it automatically gets deleted when the client disconnects.
When detaching an obersver, remove its record from the Realtime Database and remove the onDisconnect handler.
Now in your publisher app, you can detect how many observers there are to each collection.
I need to implement an auditing log for GDPR compliance so that we have a record of every consent given or revoked (an event) per user of our system. It has to store how & when it happened alongside things like what the wording of the consent actually was at the time.
So that we can recover from a backup restore, this log will be stored separately from our main DB. We will then need to be able to update the state of the user consent so that it accurately reflects the event log (i.e. the last known value (true/false) of each consent question per user)
I could simply do this using a second postgres instance (our main DB is postgres) with a single table to store the information and then some simple application code to log each event as well as update the main DB. There could also be some simple application logic to find the last known states of each consent from the event log and update the master DB.
To me it seems like a bit of overkill using postgres to store this info? though adding a new technology to store this also seems overkill. Are there any technologies that are more suitable for this sort of thing? It sounds a lot like Event Sourcing to me.
If you're already running postgres, it doesn't seem like overkill, given that it needs to be online and queryable. Something like kafka is often a natural fit for this kind of problem, but that's even more overkill.
This bears a passing resemblance to event sourcing, but on a really small scale. Event sourcing usually means that all your data is expressed in terms of events, and replayed from beginning to end to materialize the current state.
Could you elaborate on this?:
So that we can recover from a backup restore, this log will be stored separately from our main DB.
Doesn't your main database recover from a backup / restore?
Usually when talk implementing CQRS it is supposed that commands are sorces for events. But can queries made by user be source of created events in event store? Or such actions (when we need an event that reflects query) should be implemented using command still?
But can queries made by user be source of created events in event store?
Go not to the elves for counsel, for they will answer both no and yes.
So, the "no" part: queries are distinguished by the fact that they don't change the domain model. In a CQRS implementation, the queries are being served by the read model, which may not even have access to your event store at all.
when we need an event that reflects query
The yes part: there's no law that says you can't assemble a history of queries, and stick that in your event store.
But I'm stumped, in that I don't see a clear case where the domain needs an event that reflects a query. That's really weird. My guess would be that needing an event that reflects a query is a hint that your model is broken.
You may be able to make some progress with this by exploring the source of the requirement.
If the requirement is coming from operations, analytics, reporting, usability... then the domain model probably isn't the right place for that information.
If the requirement is coming from your domain experts ("we need to capture these queries so that the model supports the right changes later"), then you should be looking to identify what entity is responsible for tracking that the query happened, and sending an appropriate command to that entity.
I am writing an app for iOS that uses data provided by a web service. I am using core data for local storage and persistence of the data, so that some core set of the data is available to the user if the web is not reachable.
In building this app, I've been reading lots of posts about core data. While there seems to be lots out there on the mechanics of doing this, I've seen less on the general principles/patterns for this.
I am wondering if there are some good references out there for a recommended interaction model.
For example, the user will be able to create new objects on the app. Lets say the user creates a new employee object, the user will typically create it, update it and then save it. I've seen recommendations that updates each of these steps to the server --> when the user creates it, when the user makes changes to the fields. And if the user cancels at the end, a delete is sent to the server. Another different recommendation for the same operation is to keep everything locally, and only send the complete update to the server when the user saves.
This example aside, I am curious if there are some general recommendations/patterns on how to handle CRUD operations and ensure they are sync'd between the webserver and coredata.
Thanks much.
I think the best approach in the case you mention is to store data only locally until the point the user commits the adding of the new record. Sending every field edit to the server is somewhat excessive.
A general idiom of iPhone apps is that there isn't such a thing as "Save". The user generally will expect things to be committed at some sensible point, but it isn't presented to the user as saving per se.
So, for example, imagine you have a UI that lets the user edit some sort of record that will be saved to local core data and also be sent to the server. At the point the user exits the UI for creating a new record, they will perhaps hit a button called "Done" (N.B. not usually called "Save"). At the point they hit "Done", you'll want to kick off a core data write and also start a push to the remote server. The server pus h won't necessarily hog the UI or make them wait till it completes -- it's nicer to allow them to continue using the app -- but it is happening. If the update push to server failed, you might want to signal it to the user or do something appropriate.
A good question to ask yourself when planning the granularity of writes to core data and/or a remote server is: what would happen if the app crashed out, or the phone ran out of power, at any particular spots in the app? How much loss of data could possibly occur? Good apps lower the risk of data loss and can re-launch in a very similar state to what they were previously in after being exited for whatever reason.
Be prepared to tear your hair out quite a bit. I've been working on this, and the problem is that the Core Data samples are quite simple. The minute you move to a complex model and you try to use the NSFetchedResultsController and its delegate, you bump into all sorts of problems with using multiple contexts.
I use one to populate data from your webservice in a background "block", and a second for the tableview to use - you'll most likely end up using a tableview for a master list and a detail view.
Brush up on using blocks in Cocoa if you want to keep your app responsive whilst receiving or sending data to/from a server.
You might want to read about 'transactions' - which is basically the grouping of multiple actions/changes as a single atomic action/change. This helps avoid partial saves that might result in inconsistent data on server.
Ultimately, this is a very big topic - especially if server data is shared across multiple clients. At the simplest, you would want to decide on basic policies. Does last save win? Is there some notion of remotely held locks on objects in server data store? How is conflict resolved, when two clients are, say, editing the same property of the same object?
With respect to how things are done on the iPhone, I would agree with occulus that "Done" provides a natural point for persisting changes to server (in a separate thread).
Does anyone use ElasticSearch for building read model in CQRS approach? I have some questions related to such solution:
Where do you store your domain
events? In JDBC database? In
ElasticSearch?
Do you build indexes by event handlers that processes domain events or using ElasticSearch River functionality?
How do you handle complete rebuild of view model - for example in case when view is corrupted? Do you process all events to rebuid view?
Where the authoritative repository for your domain event is located is an implementation detail. For example, you could store serialized versions on S3 or CouchDB or any other number of storage implementations. The easiest if you're just getting started is a relational database.
Typically people use event handlers that understand the business intent behind each message and can then properly denormalize the message into the read model structure appropriate for the needs of your views.
If the view model is ever corrupted or perhaps you have a bug in a view model handler, there are a few simple steps to follow after fixing the bug:
Temporarily enqueue the flow of events arriving from the domain--these are the typical messages that are being published as your domain is doing work. We still want these messages, but not just yet. This could be done by turning off any message bus or not connecting to your queuing infrastructure if you use one.
Read all events from event storage. As each event is received (this can be done through a simple DB query), run each message through the appropriate message handler. Make sure that you keep track of the last 10,000 (or so) identifiers for all messages processed.
Now reconnect to your queues and start processing normally. If the identifier for the message has been seen, drop the message. Otherwise, process it normally.
The reason for tracking identifiers is to avoid a race condition where you're getting all events from the event store but the same message is coming across through the message queue.
Another technique that's highly related, but involves keeping track of all message identifiers can be found here: http://blog.jonathanoliver.com/2011/03/removing-2pc-two-phase-commit.html