I want to achieve the use case where I have :
1. A task that calls an API to get some data and store to DB
2. Fetch all the records stored in DB from step#1 and call a second API with each of the IDs separately and store to DB
3. Fetch all the records stored in DB from step#2 and call a third API with each of the IDs separately and store to DB```
Something like -
1. GET ALL CONTINENTS AND SAVE in DB -
2. FOR EACH CONTINENTs from DB GET ALL COUNTRIES and store in DB -
3. FOR EACH COUNTRY in DB GET ALL STATES in DB
Task A is simple - I can create a Job with a Step along with Reader/Processor/Writer.
Task B - I am confused how to fetch the DB records and then pass them to Step2's reader one by one.
That requirement does not require running a step in a loop, it is not the same logic to repeat. Those are different steps operating on different data sets and doing different things. I see at least three ways to address this:
1. create a single chunk-oriented step following the "driving query pattern"
In this pattern, you would iterate over continent IDs using the item reader, and use an item processor to enrich items with countries and states, then write fully-loaded items to the database. This requires a single-step job.
2. create a job with a sequence of three chunk-oriented steps where steps share data through the job execution context
There are different ways to share data between steps. The typical way to do that is through the job execution context. Each step can add data required by the next step in the job execution context, and the next step can get it from there. In your example, this would be something like:
step 1: read continents and save them to the database, and store their IDs in the EC
step 2: get continent IDs from the EC, fetch/save countries, and store countries IDs in the EC.
step 3: gets countries IDs from the EC and fetch/save states
3. Join data on the database side
A third way to address that requirement is to write a SQL query to join and grab the required data and save it to the database. This would require a single-step job.
Trying to implement Event Sourcing and CQRS for the first time, but got stuck when it came to persisting the aggregates.
This is where I'm at now
I've setup "EventStore" an a stream, "foos"
Connected to it from node-eventstore-client
I subscribe to events with catchup
This is all working fine.
With the help of the eventAppeared event handler function I can build the aggregate, whenever events occur. This is great, but what do I do with it?
Let's say I build and aggregate that is a list of Foos
[
{
id: 'some aggregate uuidv5 made from barId and bazId',
barId: 'qwe',
bazId: 'rty',
isActive: true,
history: [
{
id: 'some event uuid',
data: {
isActive: true,
},
timestamp: 123456788,
eventType: 'IsActiveUpdated'
}
{
id: 'some event uuid',
data: {
barId: 'qwe',
bazId: 'rty',
},
timestamp: 123456789,
eventType: 'FooCreated'
}
]
}
]
To follow CQRS I will build the above aggregate within a Read Model, right? But how do I store this aggregate in a database?
I guess just a nosql database should be fine for this, but I definitely need a db since I will put a gRPC APi in front of this and other read models / aggreates.
But what do I actually go from when I have built the aggregate, to when to persist it in the db?
I once tried following this tutorial https://blog.insiderattack.net/implementing-event-sourcing-and-cqrs-pattern-with-mongodb-66991e7b72be which was super simple, since you'd use mongodb both as the event store and just create a view for the aggregate and update that one when new events are incoming. It had it's flaws and limitations (the aggregation pipeline) which is why I now turned to "EventStore" for the event store part.
But how to persist the aggregate, which is currently just built and stored in code/memory from events in "EventStore"...?
I feel this may be a silly question but do I have to loop over each item in the array and insert each item in the db table/collection or do you somehow have a way to dump the whole array/aggregate there at once?
What happens after? Do you create a materialized view per aggregate and query against that?
I'm open to picking the best db for this, whether that is postgres/other rdbms, mongodb, cassandra, redis, table storage etc.
Last question. For now I'm just using a single stream "foos", but at this level I expect new events to happen quite frequently (every couple of seconds or so) but as I understand it you'd still persist it and update it using materialized views right?
So given that barId and bazId in combination can be used for grouping events, instead of a single stream I'd think more specialized streams such as foos-barId-bazId would be the way to go, to try and reduce the frequency of incoming new events to a point where recreating materialized views will make sense.
Is there a general rule of thumb saying not to recreate/update/refresh materialized views if the update frequency gets below a certain limit? Then the only other a lternative would be querying from a normal table/collection?
Edit:
In the end I'm trying to make a gRPC api that has just 2 rpcs - one for getting a single foo by id and one for getting all foos (with optional field for filtering by status - but that is not so important). The simplified proto would look something like this:
rpc GetFoo(FooRequest) returns (Foo)
rpc GetFoos(FoosRequest) returns (FooResponse)
message FooRequest {
string id = 1; // uuid
}
// If the optional status field is not specified, return all foos
message FoosRequest {
// If this field is specified only return the Foos that has isActive true or false
FooStatus status = 1;
enum FooStatus {
UNKNOWN = 0;
ACTIVE = 1;
INACTIVE = 2;
}
}
message FoosResponse {
repeated Foo foos;
}
message Foo {
string id = 1; // uuid
string bar_id = 2 // uuid
string baz_id = 3 // uuid
boolean is_active = 4;
repeated Event history = 5;
google.protobuf.Timestamp last_updated = 6;
}
message Event {
string id = 1; // uuid
google.protobuf.Any data = 2;
google.protobuf.Timestamp timestamp = 3;
string eventType = 4;
}
The incoming events would look something like this:
{
id: 'some event uuid',
barId: 'qwe',
bazId: 'rty',
timestamp: 123456789,
eventType: 'FooCreated'
}
{
id: 'some event uuid',
isActive: true,
timestamp: 123456788,
eventType: 'IsActiveUpdated'
}
As you can see there is no uuid to make it possible to GetFoo(uuid) in the gRPC API, which is why I'll generate a uuidv5 with the barId and bazId, which will combined, be a valid uuid. I'm making that in the projection / aggregate you see above.
Also the GetFoos rpc will either return all foos (if status field is left undefined), or alternatively it'll return the foo's that has isActive that matches the status field (if specified).
Yet I can't figure out how to continue from the catchup subscription handler.
I have the events stored in "EventStore" (https://eventstore.com/), using a subscription with catchup, I have built an aggregate/projection with an array of Foo's in the form that I want them, but to be able to get a single Foo by id from a gRPC API of mine, I guess I'll need to store this entire aggregate/projection in a database of some sort, so I can connect and fetch the data from the gRPC API? And every time a new event comes in I'll need to add that event to the database also or how is this working?
I think I've read every resource I can possibly find on the internet, but still I'm missing some key pieces of information to figure this out.
The gRPC is not so important. It could be REST I guess, but my big question is how to make the aggregated/projected data available to the API service (possible more API's will need it as well)? I guess I will need to store the aggregated/projected data with the generated uuid and history fields in a database to be able to fetch it by uuid from the API service, but what database and how is this storing process done, from the catchup event handler where I build the aggregate?
I know exactly how you feel! This is basically what happened to me when I first tried to do CQRS and ES.
I think you have a couple of gaps in your knowledge which I'm sure you will rapidly plug. You hydrate an aggregate from the event stream as you are doing. That IS your aggregate persisted. The read model is something different. Let me explain...
Your read model is the thing you use to run queries against and to provide data for display to a UI for example. Your aggregates are not (directly) involved in that. In fact they should be encapsulated. Meaning that you can't 'see' their state from the outside. i.e. no getter and setters with the exception of the aggregate ID which would have a getter.
This article gives you a helpful overview of how it all fits together: CQRS + Event Sourcing – Step by Step
The idea is that when an aggregate changes state it can only do so via an event it generates. You store that event in the event store. That event is also published so that read models can be updated.
Also looking at your aggregate it looks more like a typical read model object or DTO. An aggregate is interested in functionality, not properties. So you would expect to see void public functions for issuing commands to the aggregate. But not public properties like isActive or history.
I hope that makes sense.
EDIT:
Here are some more practical suggestions.
"To follow CQRS I will build the above aggregate within a Read Model, right? "
You do not build aggregates in the read model. They are separate things on separate sides of the CQRS side of the equation. Aggregates are on the command side. Queries are done against read models which are different from aggregates.
Aggregates have public void functions and no getter or setters (with the exception of the aggregate id). They are encapsulated. They generate events when their state changes as a result of a command being issued. These events are stored in an event store and are used to recover the state of an aggregate. In other words, that is how an aggregate is stored.
The events go on to be published so the event handlers and other processes can react to them and update the read model and or trigger new cascading commands.
"Last question. For now I'm just using a single stream "foos", but at this level I expect new events to happen quite frequently (every couple of seconds or so) but as I understand it you'd still persist it and update it using materialized views right?"
Every couple of seconds is very likely to be fine. I'm more concerned at the persist and update using materialised views. I don't know what you mean by that but it doesn't sound like you have the right idea. Views should be very simple read models. No need to complex relations like you find in an RDMS. And is therefore highly optimised fast for reading.
There can be a lot of confusion on all the terminologies and jargon used in DDD and CQRS and ES. I think in this case, the confusion lies in what you think an aggregate is. You mention that you would like to persist your aggregate as a read model. As #Codescribler mentioned, at the sink end of your event stream, there isn't a concept of an aggregate. Concretely, in ES, commands are applied onto aggregates in your domain by loading previous events pertaining to that aggregate, rehydrating the aggregate by folding each previous event onto the aggregate and then applying the command, which generates more events to be persisted in the event store.
Down stream, a subscribing process receives all the events in order and builds a read model based on the events and data contained within. The confusion here is that this read model, at this end, is not an aggregate per se. It might very well look exactly like your aggregate at the domain end or it could be only creating a read model that doesn't use all the events and or the event data.
For example, you may choose to use every bit of information and build a read model that looks exactly like the aggregate hydrated up to the newest event(likely your source of confusion). You may instead have another process that builds a read model that only tallies a specific type of event. You might even subscribe to multiple streams and "join" them into a big read model.
As for how to store it, this is really up to you. It seems to me like you are taking the events and rebuilding your aggregate plus a history of events in a memory structure. This, of course, doesn't scale, which is why you want to store it at rest in a database. I wouldn't use the memory structure, since you would need to do a lot of state diffing when you flush to the database. You should be modify the database directly in response to each individual event. Ideally, you also transactionally store the stream count with said modification so you don't process the same event again in the case of a failure.
Hope this helps a bit.
I have a method that executes a paginated call to a database to collect data and return an observable.
public Observable search(string query, int limit, int offset)
I want to build a method that executes all paginated search calls to load all pages for my query and return a Completable. In this method I would like to concatenate all pages in one stream, executes reactive transformations and actions and return a completable.
However, as I have to deal with a huge quantity of data, I do not want to load all pages at the same time , put a huge quantity of data in memory and flood my reactive stream with a lot of data because my subscriber processes the data slower than the publisher.
So, I would like to process the first page of data, then load another page through a call to search method, process it and continue until I process the last page.
As I am in a reactive flux, I am not sure that it's a good idea to write a loop to : fetch the data, subscribe to the stream that will transform and process data.
My approach is to concatenate all the pages in one stream (using a method such as Observable.concat) and then process the stream. However I do not want to load a huge quantity of data in memory and get an outOfMemoryException.
Should I use a Flowable and use backpressure in a such situation?
What is the best way to do this with rxjava2?
I am periodically doing full backups of an IReliableDictionary is Azure Service Fabric. Every X minutes I check to see if the number of elements in the collection has changed and if so, I create a new backup.
This obviously has the downside that if I just change the value of an item in the dictionary, the collection size does not change and no backup occurs.
One solution would be to enumerate the entire dictionary and calculate an overall hash for the collection (assuming the order of the items can be guaranteed). Is there a better way to identify if an IReliableDictionary has changed?
There are a few events generated by the StateManager that may be helpful.
TransactionChanged events are raised if the transaction is committed.
For example:
public MyService(StatefulServiceContext context)
: base(MyService.EndpointName, context, CreateReliableStateManager(context))
{
this.StateManager.TransactionChanged += this.OnTransactionChangedHandler;
}
I have the following problem, I am implementing a queue abstraction with RavenDB in a stateless REST server.
Suppose I have 2 REST calls A and B
When call A is happening I query the queue for the last item ( item A ) and give it to call A.
If call B is called at the same time as call A - which can happen with REST calls, I need to prevent the program from giving call B the same item as A meaning item A should be "locked" by the call A.
A standard multi-threaded protection here would be a simple lock, how do I translate this idea to my situation with RavenDB and REST?
P.S I use Nancy for the REST server
That should not be too hard unless I missed something:
Introduce a flag (boolean property) on your items, e.g. "Processed".
Inside your action, open up a new RavenDB session and enable optimistic concurrency on it (DocumentSession.Advanced .UseOptimisticConcurrency).
Get the next unprocessed item and then immediately update its processed flag to true.
Call .SaveChanges on your session -> if that succeeds (you don't get a ConcurrencyException), you can safely return the item as the result of the request. If not, load the next item.