I use KafkaAdminClient to delete Kafka topics in my Java project. I delete my topics then I produce a message to a new topic. Kafka creates my old topics again. kafka server.log
I have "allow.auto.create.topics" : "false" configuration on consumer instance.
You must send auto.create.topics.enable=false in the configuration of the broker. It also depends on how your producer is implemented. In case of Kafka Streams this property is not applied.
I just ran into the same issue and this is how I solved it in my situation:
You no longer have any active subscriptions to that topic, thus you should unsubscribe before deleting.
You have to close the producer that produced messages into that topic.
Apparently, if you don't do all that, Kafka kinda holds some connection to the topic and doesn't let it get removed.
Related
I am using java Spring-Boot framework and trying to prevent our consumer from creating topic in kafka by setting the config properties.
where Configurations are:
From broker side:
auto.create.topics.enable=true
From consumer side
auto.create.topics.enable=false
for consumer we made auto creation topic false where on broker it is true.
Above configs are not working for us,
and Also if We have any other ways to archive the same we can discuss.
auto.create.topics.enable is not a consumer config. It needs to be allow.auto.create.topics, but is only a valid option in kafka-clients version 2.3+
There may be other Spring related settings; refer latest comment thread here. Disable auto topic creation from Spring Kafka Consumer
I'm using Zookeeper and Kafka for messaging use case using Java. I thought consumer group details will be removed when you restart Zookeeper and Kafka servers. But they don't. Does zookeeper keeps consumer groups details in some kind of a file?
Should I remove consumer group details manually if I want to reset the consumer groups?
Can anyone clarify this to me?
Since Kafka 0.9, Consumer Offsets are stored directly in Kafka in an internal topic called __consumer_offsets.
Consumer Offsets are preserved across restarts and are kept at least for offsets.retention.minutes (7 days by default).
If you want to reset a Consumer Group, you can:
use the kafka-consumer-groups.sh tool with the --reset-offsets option
use AdminClient.deleteConsumerGroups() to fully delete the Consumer group
Recently we are doing some streaming jobs with apache flink. There are multiple KAFKA topics with a format as xxxxxx_yy-mm-dd. We used a topic regex pattern to let a consumer to consume those topics. However, if we delete some older topics, it seems that the metadata in consumer does not update properly. It still remember those outdated topic in its topic list? so the UNKNOWN_TOPIC_EXCEPTIOIN. We must restart the consumer job to recovery. It seems to occur in producer as well. Any idea to solve this problem? Thank you very much!
I am very new to Kafka and I am dabbling about with it.
Say I have Kafka running on a Debian machine and I have managed to create a topic with a 100 messages on it.
After that initial burst of activity (i.e. placing a 100 messages onto the topic via some Kafka Producer) the Topic is just sat there idle with nothing happening (no consumers consuming and no producers producing)
I am aware of a Message Retention Policy setting, which I believe has a default value of 7 days. Let's say those 7 days pass, and the messages are indeed removed from the Topic, but what about the Topic itself?
Will Kafka eventually kill that Topic?
Also, what happens when I manually go and pull out the power cord for the machine that Kafka is running on? Will the Topic be discarded? Or will I still have my topic after I start up the machine, run ZooKeeper and create a Kafka Broker?
Any light on this matter would be appreciated.
Thank you
No, Kafka will keep the topic. It sounds like a bad idea that Kafka deletes topics by itself.
Before version 1.0.0 the topic deletion option (delete.topic.enable) was set to false by default. So it wasn't even possible to delete it without changing the config.
So the answer for you question would be Kafka never deletes topics.
There is following consumer code:
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
kafka = KafkaClient("localhost", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic")
consumer.seek(0, 2)
for message in consumer:
print message
kafka.close()
Then I produce message with script:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
The thing is that when I start consumers as two different processes then I receive new messages in each process. However I want it to be sent to only one consumer, not broadcasted.
In documentation of Kafka (https://kafka.apache.org/documentation.html) there is written:
If all the consumer instances have the same consumer group, then this
works just like a traditional queue balancing load over the consumers.
I see that group for these consumers is the same - my-group.
How to make it so that new message is read by exactly one consumer instead of broadcasting it?
the consumer-group API was not officially supported until kafka v. 0.8.1 (released Mar 12, 2014). For server versions prior, consumer groups do not work correctly. And as of this post the kafka-python library does not currently attempt to send group offset data:
https://github.com/mumrah/kafka-python/blob/c9d9d0aad2447bb8bad0e62c97365e5101001e4b/kafka/consumer.py#L108-L115
Its hard to tell from the example above what your Zookeeper configuration is or if there's one at all. You'll need a Zookeeper cluster for the consumer group information to be persisted WRT what consumer within each group has consumed to a given offset.
A solid example is here:
Official Kafka documentation - Consumer Group Example
This should not happen - make sure that both of the consumers are being registered under the same consumer group in the zookeeper znodes. Each message to a topic should be consumed by a consumer group exactly once, so one consumer out of everyone in the group should receive the message, not what you are experiencing. What version of Kafka are you using?