Apache Flink - implementing a stream processor with potentially very large state - scala

Question

I wish to project a potentially very large state from a stream of events. This is how I might implement this in an imperative fashion:
class ImperativeFooProcessor {
val state: mutable.Map[UUID, BarState] = mutable.HashMap.empty[UUID, BarState]
def handle(event: InputEvent) = {
event match {
case FooAdded(fooId, barId) => {
// retrieve relevant state and do some work on it
val barState = state(barId)
// let the world know about what may have happened
publish(BarOccured(fooId, barId))
// or maybe rather
publish(BazOccured(fooId, barId))
}
case FooRemoved(fooId, barId) => {
// retrieve relevant state and do some work on it
val barState = state(barId)
// let the world know about what may have happened
publish(BarOccured(fooId, barId))
// or maybe rather
publish(BazOccured(fooId, barId))
}
}
}
private def publish(event: OutputEvent): Unit = {
// push event to downstream sink
}
}
In the worst case the size of BarState will grow with the number of times its been mentioned by FooAdded
The number of unique barId's is very small relative to the total number of events for each barId.
How would I begin to represent this processing structure in Flink?
How do I work with the fact that each BarState can potentially get very large?

Answer 1

Flink maintains state in so-called state backends. There are state backends (MemoryStateBackend and FsStateBackend) that operated on the JVM heap of the worker processes. These backends are not suited to handle large state.
Flink also features a RocksDBStateBackend, which is based on RocksDB. RocksDB is used as a local database (no need to set it up as an external service) on each worker node and writes state data to disk. Hence, it can handle very large state exceeding the memory.
Flink offers a KeyedStream which is a stream which is partitioned on a certain attribute. In your case, you probably want that all access to the same id go to the same state instance, so you would use barId as a key. Then, the state is partitioned across all parallel worker threads based on the barId. This is basically a distributed key-value store or map. So you would not need to represent the state as a map, because it is automatically distributed by Flink.