I know that etcd uses the Raft protocol to achieve consensus among multiple nodes. Raft is usually presented as storing data as key-value pairs in the replicated log; does etcd store its key-value data directly in that Raft log, or is there some sort of abstraction/indirection between the Raft log and etcd's data store?
The indirection is the StateMachine.
etcd looks to be using bbolt-db as its storage manager; see the source.
The final store for etcd is the bbolt database which is the MVCC store. The raft log is a write-ahead log (aka WAL) that is used to achieve consensus among leader and followers and prevent loss of information received from client.
Lets say a write request comes to a follower. It is re-directed to leader. Leader writes the request to its WAL and sends it to all followers using Raft. Once quorum confirms that the write request has been written to their respective WAL, then leader commits the same to bbolt database and sends response to client.
So yes, you can think that there is "some sort of" indirection between Raft log (WAL) and etcd data store (bbolt database).
Related
Many articles tell me that Kafka writes data to the PageCache first, which improves write performance.
However, I have a doubt, when ack=-1, when copy=2, the data does already exist in the PageCache of both nodes.
If Kafka responds to acks at this time, and immediately, both nodes experience a power outage or system crash at the same time, at this time, neither node's data is yet persistent on disk.
In this extreme case, data loss can still occur?
Data loss can occur in the situation outlined.
Related reading:
this other answer
Confluent blog post: "Since the log data is not flushed from the page cache to disk synchronously, Kafka relies on replication to multiple broker nodes, in order to provide durability. By default, the broker will not acknowledge the produce request until it has been replicated to other brokers."
I'm a little confused with the RAFT log driving the state machine of my system
Let's take Kafka as an example,
Every partition has a leader & a replica, does that mean there's a RAFT instance per partition?
IF yes, then every time a producer sends a record, should it be persisted in the RAFT log before appending to the Kafka partition?
IF no, then how does Kafka maintain leader & replica to be in sync during failures?
Consensus is confusing me
There are multiple consensus points IMO and I'm unsure where to use RAFT and where it's not required. e.g. the cluster leader election, partition leader, even replication is a form of consensus to me
Other doubts regarding RAFT
It is said that followers should update their state machine only after a commit is done, what if a follower is isolated and ends up committing the wrong event? Is this even possible? IF yes, should the state machine have logic for rollbacks? From my experience with elastic search, there always should be a rollback mechanism for obsolete events
I'm trying to implement something similar to Kafka for learning purpose and I'm confused at the point where I don't even know how to begin implementing one
As I got right from book, Kafka Streams state store it is a memory key/value storage to store data to Kafka or after filtering.
I am confused by some theoretical questions.
What is differenct Kafka streams state from another memory storage like Redis etc?
What is real case to use state storage in Kafka Streams?
Why topic is not alternative for state storage?
Why topic is not alternative for state storage?
A topic contains messages in a sequential order that typically represents a log.
Sometimes, we would want to aggregate these messages, group them and perform an operation, like sum, for example and store it in a place which we can retrieve later using a key. In this case, an ideal solution would be to use a key-value store rather than a topic that is a log-structure.
What is real case to use state storage in Kafka Streams?
A simple use-case would be word count where we have a word and a counter of how many times it has occurred. You can see more examples at kafka-streams-examples on github.
What is difference between Kafka streams state from another memory storage like Redis etc?
State can be considered as a savepoint from where you can resume your data processing or it might also contain some useful information needed for further processing (like the previous word count which we need to increment), so it can be stored using Redis, RocksDB, Postgres etc.
Redis can be a plugin for Kafka streams state storage, however the default persistent state storage for Kafka streams is RocksDB.
Therefore, Redis is not an alternative to Kafka streams state but an alternative to Kafka streams' default RocksDB.
-Why topic is not alternative for state storage?
Topic is the final statestore storage under the hood (everything is topic in kafka)
If you create a microservice with name "myStream" and a statestore named "MyState", you'll see appear a myStream-MyState-changelog with has an history of all changes in the statestore.
RocksDB is only the local cache to improve performances, with a first layer of local backup on the local disk, but at the end the real high availability and exactly-once processing guarantee is provided by the underlying changelog topic.
What is differenct Kafka streams state from another memory storage like Redis etc?
What is real case to use state storage in Kafka Streams?
It not a storage, it's a just local, efficient, guaranteed memory state to manage some business case is a fully streamed way.
As an example :
For each Incoming Order (Topic1), i want to find any previous order (Topic2) to the same location in the last 6 hours
I desire to provide a fast ability to get status of user his availability.
It must be fastest reading data from storage.
Thus I chosed Redis storage for storing available status of each users.
So, besides that I need store more extended information about available users, such as region, time of login, etc.
For this purpose I got a Kafka, where this data is stored.
Question is, how to synchronise Kafka and Redis?
Which sequence should be, first store event online users in Kafka, then sink it to Redis?
Second is store in Redis and asynchronously in Kafka.
I afraid a latency between Kafka and Redis for sink operation.
As I understood from the question, you want to store only user and userstatus in Redis and complete profile on Kafka.
I am not sure about the reason of choosing Kafka as your primary source of all data. Also, how are you planning to use the data stored there.
If data storage in Kafka is really important to you, then I'd suggest to update your primary database first(Kafka or any) and then update cache.
In this case, you need to do a sync operation on Kafka producer and once its successful, update ur cache.
As your readd operations are only from redis - performance will not be impacted.
But if opting sync producer might add little bit overhead beacuse of acknowledgement when compared to async.
Been reading a lot about kafka's use as an event store and a potential good candidate for CQRS.
I was wondering, since messages in kafka have a limited retention time, how will events be replayed after the messages were deleted from the disk where kafka retains messages?
Logically, when these messages are stored externally from kafka (after reading messages from kafka topics) in a db (sql/nosql), that would make more sense from an event store standpoint than kafka.
In lieu of above, given my understanding is correct, what is the real use case of kafka being used in CQRS even though the actual intent of kafka was just a high throughput messaging system?
You can use Kafka of event store and CQRS. You can use Kafka Stream to process all events generated by commands and store a snapshot of your entities in a changelog topic and store the changelog topic in a NOSQL one or more databases that meets your requirement. Also, all event can be store in a database(PostgresSql). What's important to know is that Kafka can be used as a store(its store files in high available way) or as a message query.
Retention time: You can set the retention time as long as you want or even keep messages forever in the topic.
Using Kafka as the data store: Sure, you can. There is a feature named Log Compaction. Let say the following scenario:
Insert product with ID=10, Name=Apple, Price=10
Insert product with ID=20, Name=Orange, Price=20
Update product with ID=10, Price becomes 30
When one topic is turned on the log compaction, a background job will periodically clean up messages on that topic. This job will check if any message has the same key then only keeps the final. With the above scenario, messages which are written to Kafka will the following format:
Message 1: Key=1, Name=Apple, Price=10
Message 2: Key=2, Name=Orange, Price=20
Message 3: Key=1, Name=Apple, Price=30 (Every update now includes all fields so it can self-contained)
After the log compaction, the topic will become:
Message 1: Key=2, Name=Orange, Price=20
Message 2: Key=1, Name=Apple, Price=30 (Keep the lastest record with the ID=1)
In reality, Kafka uses log compaction feature to make Kafka as the persistent data storage.