where does kafka save the local state store? - apache-kafka

Question

I created a Kafka topic and sent some messages to it.
I created an application which had the stream topology. Basically, it was doing the sum and materialized it to a local state store.
I saw the new directory was created in the state folder I configured.
I could read the sum from the local state store.
Everything was good so far.
Then, I turned off my application which was running the stream.
I removed the directory created in my state folder.
I restart the Kafka cluster.
I restart my application which has the stream topology.
In my understanding, the state was gone. Kafka needs to do the aggregation again. But it did not. I was still able to get the previous sum result.
How comes? Where did Kafka save the local state store?
Here is my code
Reducer<Double> reduceFunction = (subtotal, amount) -> {
// detect when the reducer is triggered
System.out.println("reducer is running to add subtotal with amount..." + amount);
return subtotal + amount;
};
groupedByAccount.reduce(reduceFunction,
Materialized.<String, Double, KeyValueStore<Bytes, byte[]>>as(BALANCE).withValueSerde(Serdes.Double()));
I explicitly put the System.out in the reduceFunction. Whenever it is executed, I shall see it on the console.
But I did not see any after restart kafka cluster and my application.
Does Kafka really recover the state? Or it saves state somewhere else?

Answer 1

If I'm not mistaken and according to Designing Event-Driven Systems by Ben Stopford (free book) on page 137, it states:
We could store, these
stats in a state store and they’ll be saved locally as well as being backed up to
Kafka, using what’s called a changelog topic, inheriting all of Kafka’s durability
guarantees.
It seems like a copy of your state store is also backed up in Kafka itself (i.e. changelog topic). I don't think restarting a cluster will flush out (or remove) messages already in a topic as they're kept track in the Zookeeper.
So once you restart your cluster and application again, the local state store is recovered from Kafka.