When I check for consumer lag, it shows that a particular consumer-id is running from a particular host consuming from a topic.
But when I go to that host there is no such consumer running.
How do I kill this consumer-id, so that I can reset consumer offset for the group that its part of.
Kafka server version: 0.11.0.1
Kafka client version(stuck): 0.10.0.2
This consumer-id got stuck in the first place as it was not able to consume messages because of some messages having headers in Kafka.
I've already tried the following:
Consuming from a different host and different Kafka version, it consumes messages but the consumer-id, host does not change.
Restarting kafka broker which is the leader for that topic.
Changing security groups to prevent the host from connecting to my broker.
Perhaps what you see is not a consumer id, but a consumer group, see Kafka docs, consumer config to learn about the difference.
Kafka uses consumer groups to keep track of the last consumed message (consumer offset), so when talking about the consumer lag this is probably the explanation.
This means there is no consumer running and you only need to get rid of the consumer offset for this group. See e.g. How do I delete a Kafka Consumer Group to reset offsets?
Related
Hope you are having good day.
I have an issue with kafka consumers on kubernetes. I am running 3 replicas inside a consumer group
I have a topic with 3 partitions and 3 brokers with offsets replication factor set to 3. My offset in consumer group is set to earliest.
When I start the consumer group, all are working fine with each consumer replica taking different partition and processing the data.
Issue: When by any means if a consumer replica inside the consumer group "abc-consumer-group" restarts OR if a broker(leader) restarts, it is not resuming from the point where it stopped. It states that I am up to date and no messages I have to process.
Any suggestions please where to look at?
Tried increasing rebalance, heartbeat, session timeout on broker level, no luck.
And yes whenever any new consumer is added or removed to the consumer group rebalacing is taken care by kafka. I do see it happening but still not consumers are not resuming messages. It states nothing to process.
I'm very much new in Apache Kafka. So, I'm facing some issues.
If one consumer Auto-Commit is enabled and that machine is consumed a data then other new consumers on that group is not receiving those message.
I have 3 consumer, 1 producer. Here, every single consumer is running on different machine. I'm consuming messages from a single machine with auto-commit enabled for testing purpose. And after that when I'm running another two consumer machine then they are not receiving those messages.
As other two consumers did not received those messages so I want to receive those message for this two consumer.
So, How Kafka commit works for a consumer group?
kafka consumer using librdkafka (high level consumer) connected to kafka cluster and subscribed to 10 topics and consuming data. There was assign partitions event.
There was network issue due to which cluster was not reachable. Lost connection with group co-ordinator and heartbeats got stuck. There was revoked partitions event where the code calls unassign on consumer.
When cluster came back up, the consumer was not consuming any data although its in the while true loop calling consume with timeout of 1 sec.
Does consumer needs to resubscribe to topics again once it is connected to cluster? What is a reliable way to detect the consumer is connected to cluster in code?
Does consumer needs to resubscribe to topics again once it is connected to cluster?
Yes. New group members will cause a rebalance amongst existing members, and they need to resubscribe
What is a reliable way to detect the consumer is connected to cluster in code?
You could describe the consumer group and see if there are active clients for the consumer group you are interested in
I'm using Zookeeper and Kafka for messaging use case using Java. I thought consumer group details will be removed when you restart Zookeeper and Kafka servers. But they don't. Does zookeeper keeps consumer groups details in some kind of a file?
Should I remove consumer group details manually if I want to reset the consumer groups?
Can anyone clarify this to me?
Since Kafka 0.9, Consumer Offsets are stored directly in Kafka in an internal topic called __consumer_offsets.
Consumer Offsets are preserved across restarts and are kept at least for offsets.retention.minutes (7 days by default).
If you want to reset a Consumer Group, you can:
use the kafka-consumer-groups.sh tool with the --reset-offsets option
use AdminClient.deleteConsumerGroups() to fully delete the Consumer group
I want to check the connection of the Kafka consumer at a remote location.
It is possible to determine whether the consumer is allocated to the partition.
At a remote location, i can get detailed information about the topic from the Kafka broker.
But can the consumer guarantee that the consumer is able to receive the message that the consumer is matched to the topic's partition?
You can use Kafka Rest Proxy to determine whether a partition has been assigned to the consumer or not.
I think, Kafka Rest Proxy have all functionality which you are looking for.