I'm not sure if it's already answered. As I didn't get proper explanation, posting my question here.
Why kafka streams state.dir is stored under /tmp/kafka-streams?
I know I can change the path by providing the state dir config in the stream code like below
StreamsConfig.STATE_DIR_CONFIG,"/var/abc-Streams"
But will there be any impact of changing the directory?
or
Can I configure the state DB in an application directory and not in /tmp.
As per the confluent documentation, for :
Stateful operations :
automatically creates and manages such state stores when you are calling stateful operators such as count() or aggregate(), or when you are windowing a stream
but didn't specify where exactly it's being stored.
ANy thoughts?
Why kafka streams state.dir is stored under /tmp/kafka-streams?
There are several reasons.
Usually /tmp directory has a default write permission. So you don't have to struggle with write permissions as a beginner.
/tmp directory is short lived directory. On each system reboot, it is cleaned, hence you don't experience the over flooded disk storage in case you forgot to delete the state.dir. Downside is, you lose the states from previous run hence you need to rebuild the states from scratch.
If you want to reuse the states stored in state.dir, you should store it somewhere except /tmp.
All the state-stores are stored in the location specified in state.dir. If not specified, it is /tmp/kafka-streams/<app-id> directory.
Related
The question is simple. What will happen if you delete the Kafka snapshot files in the Kafka log dir. Will Kafka be able to start? Will it have to do a slow rebuild of something?
Bonus question what exactly do the snapshot files contain?
Background for this question
I have a cluster that has been down for a couple of days due to simultaneous downtime on all brokers and a resulting corrupted broker. Now when it starts it is silent for hours (in the log file no new messages). By inspecting the JVM I have found that all the (very limited) cpu usage is spent in loadproducersfromlog function/method. By reading the comments above it is suggested that this is an attempt to recover producer state from the snapshots. I do not care about this. I just want my broker back so I am thinking if I can simply delete the snapshots to get Kafka started again.
If snapshot files are deleted, during start up method log.loadSegmentFiles(), all messages in the partition will have to be read to recreate the snapshot even if log and index files are present. This will increase the time to load partition.
For contents of snapshot file, please refer writeSnapshot() in ProducerStateManager.
https://github.com/apache/kafka/blob/980b725bb09ee42469534bf50d01118ce650880a/core/src/main/scala/kafka/log/ProducerStateManager.scala
Parameter log.dir defines where topics (ie, data) is stored (supplemental for log.dirs property).
A snapshot basically gives you a copy of your data at one point in time.
In a situation like yours, instead of waiting for a response, you could:
change the log.dirs path, restart everything and see how it goes;
backup the snapshots, saving them in a different location, delete them all from the previous one and see how it goes.
After that you're meant to be able to start up Kafka.
What it means to set some topic for compaction?
What are pros/cons if that is not done for a specific topic?
Just trying to understand terms and how it behaves.
The simple log retention i.e log retention based out of time/size will not work for all cases. Definitely, this method proves to save you the space but it does not guarantee you the current state of the data.
Please allow me to explain this with a scenario.
Let us imagine we are having a stream of messages or logs with respect to a user in stackoverflow and you want to manage the state of the user. The key of the message is UserID and the value is the UserInformation(NickName,Address,Email,Telephone etc).
A user can change his nicknames as many times he want. Similarly, he can also change the Contact or any information. Log compaction will help you to retain the latest user based information by deleting or cleaning the redundant data. But, time based or size based retention may not give you a guarantee to retain the state of the user.
Log compaction retains last known value it is a full snapshot of the
latest records it is useful for restoring state after a crash or
system failure for an in-memory service, a persistent data store, or
reloading a cache. It allows downstream consumers to restore their
state.
Hope this helps.
I've read the stateful stream processing overview and if I understand correctly, one of the main reasons why the RocksDB is being used as a default implementation of the key value store is a fact, that unlike in-memory collections, it can handle data larger than the available memory, because it can flush to disk. Both types of stores can survive application restarts, because the data is backed up as a Kafka topic.
But are there other differences? For example, I've noticed that my persistent state store creates some .log files for each topic partition, but they're all empty.
In short, I'm wondering what are the performance benefits and possible risks of replacing persistent stores with in-memory ones.
I've got a very limited understanding of the internals of Kafka Streams and the different use cases of state stores, esp. in-memory vs persistent, but what I managed to learn so far is that a persistent state store is one that is stored on disk (and hence the name persistent) for a StreamTask.
That does not give much as the names themselves in-memory vs persistent may have given the same understanding, but something that I found quite refreshing was when I learnt that Kafka Streams tries to assign partitions to the same Kafka Streams instances that had the partitions assigned before (a restart or a crash).
That said, an in-memory state store is simply recreated (replayed) every restart which takes time before a Kafka Streams application is up and running while a persistent state store is something already materialized on a disk and the only time the Kafka Streams instance has to do to re-create the state store is to load the files from disk (not from the changelog topic that takes longer).
I hope that helps and I'd be very glad to be corrected if I'm wrong (or partially correct).
I don't see any real reason to swap current RocksDB store. In fact RocksDB one of the fastest k,v store:
Percona benchmarks (based on RocksDB)
with in-memory ones - RocksDB already acts as in-memory with some LRU algorithms involved:
RocksDB architecture
The three basic constructs of RocksDB are memtable, sstfile and logfile. The memtable is an in-memory data structure - new writes are inserted into the memtable and are optionally written to the logfile.
But there is one more noticeable reason for choosing this implementation:
RocksDB source code
If you will look at source code ratio - there are a lot of Java api exposed from C++ code. So, it's much simpler to integrate this product in existing Java - based Kafka ecosystem with comprehensive control over store, using exposed api.
I have a Web API stateless-service that bring a file from a client and transfers it to an actor-service (for deferred ETL operations). File size limited up to 20MB.
Is it good idea to tranfer file directly (in-memory as a byte array) from one service to another? Or there any feature like a file-based state to replicate file within the cluster and further processing?
P.S. It is impossible (due to legal reasons) to upload it anywhere before processing.
P.P.S. SF cluster is on-premises installation.
It is not a good idea to do that for a few reasons:
1 - If you store your files in Reliable Collections, it would make your collections too big and slow down the replication, as every update to your collections would be replicated to other nodes, it would be also expensive(time) to move services around the cluster.
2 - if you don't store it on any collection and leave it in memory, Service Fabric could try to move your service around the cluster and you risk to loose the data.
3 - When you upload the file, you must return a confirmation to the user, to avoid let them waiting until the processing is complete, locking server resources is a bad idea.
4 - Saving it to the disk won't replicate the file, and if your service move to other nodes, you loose access to the file.
There are many reasons, if can't save it somewhere (like a file share), you can take these risks.
If you still prefer to go this route, I would suggest:
Send the content to the actor that will process it, the actor will save it to the actor state.
Register a timer (or reminder, dependning on your requirementts) in the actor for triggering the processing of this file.
Create the logic after the processing to deactivate the timer and them save the output to somewhere after processed.
Deactivate the actor and delete the state.
Using the actor state to store each file will make it more flexible, you might register the file in your actor at one node, and if the actor is moved, the actor state will be still available when it get activated on other node.
Keep in mind that your cluster has nodes and they may fail, so you
should not rely on their memory or disks to save state, unless
replicated to others with different reliability guarantees, like azure
storage.
I want a external config store for some of my services , and the data can be in following format like JSON,YML,XML. The use case I want is that I can save my configs , change them dynamically , and the read for these configs will be very frequent. So, for this is Zookeeper a good solution. Also my configs are of atmost 500MB.
The reason that Zookeeper is under consideration as it has synchronization property, version (as I will be changing configs a lot) ,can provide notifications to the depending service of changes to config. Kindly tell if Zookeeper can be data store and will be best for this use case,any other suggestion if possible.
Zookeeper may be used as data store but
Size of single node should not be longer than 1MB
Getting huge amount of nodes from zookeeper will take time, so you need to use caches. You can use Curator PathChildrenCache recipe. If you have tree structure in your zNodes you can use TreeCache, but be aware that TreeCache had memory leaks in various 2.x versions of Curator.
Zookeeper notifications is a nice feature, but if you have pretty big cluster you might have too many watchers which brings stress on your zookeeper cluster.
Please find more information about zookeeper failure reasons.
So generally speaking Zookeeper can be used as a datastore if the data is organized as key/value and value doesn't exceed 1MB. In order to get fast access to the data you should use caches on your application side: see Curator PathChildrenCache recipe.
Alternatives are Etcd and consul