I have a single Kafka node (v 0.10.2) running.We have configured log.retention.hours=480 but messages are not available to the consumers before the expiry time of the messages.(Ex message is pushed on 1st July and I have started my consumer on 4th July. Before starting the consumer I have verified messages are there through yahoo Kafka monitoring service. But on starting the consumer it keeps on polling and waiting for the messages).
Below mentioned are broker configuration :
broker.id=1
delete.topic.enable=true
num.network.threads=5
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=xxx
num.partitions=1
num.recovery.threads.per.data.dir=5
log.retention.hours=480
offsets.retention.minutes=43200
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=x.x.x.x:2181
zookeeper.connection.timeout.ms=30000
zookeeper.session.timeout.ms=30000
I have googled it but was not able to find the reason. Please let me know why this is happening and how to fix it.
There are some reasons why a message is not being consumed. Here are some ways you can debug:
One is as said, you might be consuming from the latest offset and hence you will be waiting. In this case, try producing the messages to the topic and check if they are consumed.
Next, ensure that your consumer subscribes to the partitions to which the messages are produced. If you are using Streams API then you need not worry about this since you will subscribe to all the partitions.
There could be some consumer configuration problem. Try consuming using the kafka-console-consumer with --from-beginning flag.
If there is no output means that the messages are probably not there or you have some difficulty connecting to it.
You can repeat this test both on the Kafka machine and from outside.
Check your connection from your consumer client to your broker which is the leader of the topic partitions you are consuming from. If there is a problem connecting to it, you must be getting an exception (like timeout for fetching the data).
For this, you can use telnet broker_ip:port
Sometimes, it may happen that your bootstrap-server may be accessible, but not the other brokers on which your topic partitions are lead.
Check your iptables rules to see if the brokers port is blocked or not. See
What happens if the leader is not dead but unable to receive messages in Kafka? SPoF?
Related
I'm testing Kafka on Linux, but I don't know what's wrong because the test results are different from what I understand.
Let me explain the setting.
Currently, three brokers were configured with kafka version 2.8.1 in centos7 using 9092, 9093, and 9094 ports, respectively.
In the case of producers, all three ports were connected to the bootstrap-server setting and then executed.
kafka-console-producer.bat --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test
In the case of consumers, three were set up so that they could be attached to each of the three ports.
1. kafka-console-consumer.bat --bootstrap-serverlocalhost:9092 --topic test
2. kafka-console-consumer.bat --bootstrap-serverlocalhost:9093 --topic test
3. kafka-console-consumer.bat --bootstrap-serverlocalhost:9094 --topic test
If I were to explain what I understood here,
In the case of Kafka, the leader broker acts as a controller, and the follow brokers copy the leader broker's data.
If one of the follower brokers dies, a disconnection message simply appears on the consumer connected to the broker.Other brokers operate normally.
If a leader broker dies, one of the follow brokers will be changed to a leader broker, and the changed leader broker will act as a controller.
If I were to explain the problem,
If you kill a leader broker, check the describe option, and the other follow broker has changed to a leader, but both producers and consumers cannot find a new leader and fail.
Even if a broker running on 9092 ports kills the broker without being a leader, the producer and consumer will fail.
Question.
If the leader broker dies, should the producer and consumer also set up a new connection?
Am I misunderstanding the producer and consumer?
Is there anything wrong with the setting?
I'm testing Kafka on Linux
But you're using Batch files, and connecting to localhost, which are for Windows...
so that they could be attached to each of the three ports.
This isn't how Kafka distributes load. You can only have one consumer thread active per topic partition. Unclear how many partitions your topic has, but if you have only one and that specific broker died (it is the only replica and leader), this explains why your clients would stop working.
Besides that, Kafka is generally on the same port, on mulitple hosts. Using one host is not truly fault-tolerant, and is a waste of resources (CPU, RAM, and disk).
Regarding producers, there is a property for retries that can be configured; I'm not sure if the console producer overrides the default or not, but it should connect to the next available broker upon a new request.
For consumers, the same, however, you'll want to make sure your offsets.topic.replication.factor (and transactions topic factor, if you use them) is higher than 1; otherwise, consumers will be unable to read anything (or transactions will not work, which are enabled by default in newer versions)
When I check for consumer lag, it shows that a particular consumer-id is running from a particular host consuming from a topic.
But when I go to that host there is no such consumer running.
How do I kill this consumer-id, so that I can reset consumer offset for the group that its part of.
Kafka server version: 0.11.0.1
Kafka client version(stuck): 0.10.0.2
This consumer-id got stuck in the first place as it was not able to consume messages because of some messages having headers in Kafka.
I've already tried the following:
Consuming from a different host and different Kafka version, it consumes messages but the consumer-id, host does not change.
Restarting kafka broker which is the leader for that topic.
Changing security groups to prevent the host from connecting to my broker.
Perhaps what you see is not a consumer id, but a consumer group, see Kafka docs, consumer config to learn about the difference.
Kafka uses consumer groups to keep track of the last consumed message (consumer offset), so when talking about the consumer lag this is probably the explanation.
This means there is no consumer running and you only need to get rid of the consumer offset for this group. See e.g. How do I delete a Kafka Consumer Group to reset offsets?
I had a set up of 3 Node Zookeeper and 3 Broker Cluster when one of my brokers goes down in Cluster, the producer is not giving any Error but, consumers will throw an error saying that...
Marking coordinator Dead for the group... Discovered coordinator for
the group.
According to my knowledge if any one Broker available across the cluster I should not be stopped consuming messages.
But, as of now Server.1, server.2, server.3 if my server.2 goes down my all consumers stops consuming messages.
What are the exact parameters to set to achieve failover of producers and as well as consumers?
if my server.2 goes down my all consumers stops consuming messages.
For starters, you disable unclear leader election in the brokers, and create your topics with --replication-factor=3 and a configuration of min.insync.replicas=2.
To ensure that a producer has at least two durable writes (as set by the in-sync replcicas), then set acks=all
Then, if any broker fails, and assuming a leader election does not have any error, a producer and consumer should seemlessly re-connect to the new leader TopicPartitions.
If I have multiple brokers, which broker should my producer use? Do I need to manually switch the broker to balance the load? Also why does the consumer only need a zookeeper endpoint instead of a broker endpoint?
quick example from tutorial:
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
which broker should my producer use? Do I need to manually switch the broker to balance the load?
Kafka runs on cluster, meaning set of nodes, so while producing anything you need to tell him the LIST of brokers that you've configured for your application, below is a small note taken from their documentation.
“metadata.broker.list” defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster but should include at least two in case the first Broker is not available. No need to worry about figuring out which Broker is the leader for the topic (and partition), the Producer knows how to connect to the Broker and ask for the meta data then connect to the correct Broker.
Hope this clear some of your confusion
Also why does the consumer only need a zookeeper endpoint instead of a
broker endpoint
This is not technically correct, as there are two types of APIs available, High level and Low level consumer.
The high level consumer basically takes care of most of the thing like leader detection, threading issue, etc. but does not provide much control over messages which exactly the purpose of using the other alternatives Simple or Low level consumer, in which you will see that you need to provide the brokers, partition related details.
So Consumer need zookeeper end point only when you are going with the high level API, in case of using Simple you do need to provide other information
Kafka sets a single broker as the leader for each partition of each topic. The leader is responsible for handling both reads and writes to that partition. You cannot decide to read or write from a non-Leader broker.
So, what does it mean to provide a broker or list of brokers to the kafka-console-producer ? Well, the broker or brokers you provide on the command-line are just the first contact point for your producer. If the broker you list is not the leader for the topic/partition you need, your producer will get the current leader info (called "topic metadata" in kafka-speak) and reconnect to other brokers as necessary before sending writes. In fact, if your topic has multiple partitions it may even connect to several brokers in parallel (if the partition leaders are different brokers).
Second q: why does the consumer require a zookeeper list for connections instead of a broker list? The answer to that is that kafka consumers can operate in "groups" and zookeeper is used to coordinate those groups (how groups work is a larger issue, beyond the scope of this Q). Zookeeper also stores broker lists for topics, so the consumer can pull broker lists directly from zookeeper, making an additional --broker-list a bit redundant.
Kafka Producer API does not interact directly with Zookeeper. However, the High Level Consumer API connects to Zookeeper to fetch/update the partition offset information for each consumer. So, the consumer API would fail if it cannot connect to Zookeeper.
All above answers are correct in older versions of Kafka, but things have changed with arrival of Kafka 0.9.
Now there is no longer any direct interaction with zookeeper from either the producer or consumer. Another interesting things is with 0.9, Kafka has removed the dissimilarity between High-level and Low-level APIs, since both follows a uniformed consumer API.
There is following consumer code:
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
kafka = KafkaClient("localhost", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic")
consumer.seek(0, 2)
for message in consumer:
print message
kafka.close()
Then I produce message with script:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
The thing is that when I start consumers as two different processes then I receive new messages in each process. However I want it to be sent to only one consumer, not broadcasted.
In documentation of Kafka (https://kafka.apache.org/documentation.html) there is written:
If all the consumer instances have the same consumer group, then this
works just like a traditional queue balancing load over the consumers.
I see that group for these consumers is the same - my-group.
How to make it so that new message is read by exactly one consumer instead of broadcasting it?
the consumer-group API was not officially supported until kafka v. 0.8.1 (released Mar 12, 2014). For server versions prior, consumer groups do not work correctly. And as of this post the kafka-python library does not currently attempt to send group offset data:
https://github.com/mumrah/kafka-python/blob/c9d9d0aad2447bb8bad0e62c97365e5101001e4b/kafka/consumer.py#L108-L115
Its hard to tell from the example above what your Zookeeper configuration is or if there's one at all. You'll need a Zookeeper cluster for the consumer group information to be persisted WRT what consumer within each group has consumed to a given offset.
A solid example is here:
Official Kafka documentation - Consumer Group Example
This should not happen - make sure that both of the consumers are being registered under the same consumer group in the zookeeper znodes. Each message to a topic should be consumed by a consumer group exactly once, so one consumer out of everyone in the group should receive the message, not what you are experiencing. What version of Kafka are you using?