I have an kafka environment which has 3 brokers and 1 zookeeper. I had pushed around >20K message in my topic. Apache Storm is computing the data in topic which is added by producer.
After few hours passed, While I am trying to produce messages to kafka, its showing the following exception
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
After restarting the kafka servers its working fine.
but on production i can't restart my server everytime.
so can any one help me out to figure out my issue.
my kafka configuration are as follows :
prodProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"list of broker");
prodProperties.put(ProducerConfig.ACKS_CONFIG, "1");
prodProperties.put(ProducerConfig.RETRIES_CONFIG, "3");
prodProperties.put(ProducerConfig.LINGER_MS_CONFIG, 5);
Although Kafka producer tuning is a quite hard topic, I can imagine that your producer is trying to generate records faster than it can transfer to your Kafka cluster.
There is a producer setting buffer.memory which defines how much memory producer can use before blocking. Default value is 33554432 (33 MB).
If you increase the producer memory, you will avoid blocking. Try different values, like 100MB.
Related
Running some perf test on Kafka, we are having bad latency in the producer.
Checking the kafka broker logs I can see this log
[Topic] Wrote producer snapshot at offset 331258 with 62 producer ids in 860 ms. (kafka.utils.Logging)
I dont know if this is the time that it takes to write in disk or replicas before ack to the producer(ack=all) but those 800ms it seems a lot to me.
Regards
This needs a detailed analysis. Here are few things that you can do:
Monitor kafka broker/producer resources like CPU/Memory to see if any particular resource is reaching near 100% usage.
If kafka broker resource is saturating, then for your load you might need to provide more resources to your kafka broker. Same logic is applicable on your kafka producer.
If resources are not saturating, you would need to play around with your kafka producer configuration. Calculate rough throughput of your kafka producers (in messages/sec as well as bytes/sec). Check kafka producer config defaults to see if you can find a probable cause. There are a lot of producer configs like: batch.size, buffer.memory, linger.ms, max.request.size etc any of which might be the reason for your producer to not perform in an optimum way.
I have a problem to detect kafka partition changes in runtime. I configured kafka using spring integration and i could not figure out how to detect number of partition changes during the application is running.
The main problem is that the kafka topic has 10 partitions and my kafka config is below(it has 10 threads. 1 to 1 relationship between partition and thread). when i increase the kafka topic partition number(assuming 20 partitions), application could not read messages which come into newly created partitions without restarting.
is there any way to configure spring integration to be aware of this kind of changes?
Thanks in advance.
IntegrationFlow flow = IntegrationFlows.from(Kafka.messageDrivenChannelAdapter(kafkaConsumerFactory, topic)
.configureListenerContainer(c-> c.concurrency(10))))
.transform(transformer)
.get();
....
This happens automatically, every metadata.max.age.ms Consumer property (default 5 minutes)
We have a kafka producer configured as -
metadata.broker.list=broker1:9092,broker2:9092,broker3:9092,broker4:9092
serializer.class=kafka.serializer.StringEncoder
request.required.acks=1
request.timeout.ms=30000
batch.num.messages=25
message.send.max.retries=3
producer.type=async
compression.codec=snappy
Replication Factor is 3 and total number of partition currently is 108
Rest of the properties are default.
This producer was running absolutely fine. Then, due to some reason, one of the broker went down. Then, our producer started to show the log as -
"Failed to update metadata after 60000 ms". Nothing else was there in the log and we were seeing this error. In some interval, few requests were getting blocked, even if producer was async.
This issue was resolved when the broker was again up and running.
What can be the reason of this? One broker down should not affect the system as a whole as per my understanding.
Posting the answer for someone who might face this issue -
The reason is older version of Kafka Producer. The kafka producers take bootstrap servers as list. In older versions, for fetching metadata, producers will try to connect with all the servers in Round Robin fashion. So, if one of the broker is down, the requests going to this server will fail and this message will come.
Solution:
Upgrade to newer producer version.
can reduce metadata.fetch.timeout.ms settings: This will ensure the main thread is not getting blocked and send will fail soon. Default value is 60000ms. Not needed in higher version
Note: Kafka send method is blocked till the producer is able to write to buffer.
I got the same error because I forgot to create the topic. Once I created the topic the issue was resolved.
I have written Java Program which is making use of Kafka libraries, I heard Kafka Producer is having internal buffer to hold the message, so that it can retry it later. So i created Idempotent Kafka Producer with the retry properties.
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, System.getenv(KafkaConstants.KAFKA_URL));
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringSerializer");
props.put("linger.ms", 1000);
props.put("acks", "all");
props.put("request.timeout.ms",60000);
props.put("retries",3);
props.put("retry.backoff.ms",1000);
props.put("max.in.flight.requests.per.connection",1);
props.put("enable.idempotence",true);
Before Running the Program, I am keeping Kafka Server (only one broker) down. When i ran the program i am getting an exception "Failed to update metadata after 60000ms". But When i restart the Kafka Server then it should push the data to kafka topics as i have given retry properties.
Please help in this regard.
Thanks,
Priyam Saluja
One of the first requests a Kafka client sends is about getting metadata. Remember that the client tries to connect to the brokers in the bootstrap servers list but the topic to which it could want to send could be not one of them. For example, consider to have 3 brokers B01, B02, B03 and the bootstrap servers is just B01 but producer wants to send messages to a topic partition with B02 as leader : the first metadata requests is needed by the producer to get this information and then opening a connection to the B02 for sending messages. I guess that the retry mechanism come into play after this step because the batching inside the producer leverages on known partitions and where they are located. You should check if the retry work shutting down the server after the getting metadata step is done properly and the producer knows who the partition leader is.
I found out the problem,Every time when Kafka Producer tries to produce a message,first it goes to update the metadata(to check the leader and partitions in Kafka Cluster). If it is not able to get the information then it'll throw the error saying "Failed to update metadata after 60000 ms".
The Second part was retry, Kafka Producer will try if messages failed because of transient errors.
I'm a new user of Apache Kafka and I'm still getting to know the internals.
In my use case, I need to increase the number of partitions of a topic dynamically from the Kafka Producer client.
I found other similar questions regarding increasing the partition size, but they utilize the zookeeper configuration. But my kafkaProducer has only the Kafka broker config, but not the zookeeper config.
Is there any way I can increase the number of partitions of a topic from the Producer side? I'm running Kafka version 0.10.0.0.
As of Kafka 0.10.0.1 (latest release): As Manav said it is not possible to increase the number of partitions from the Producer client.
Looking ahead (next releases): In an upcoming version of Kafka, clients will be able to perform some topic management actions, as outlined in KIP-4. A lot of the KIP-4 functionality is already completed and available in Kafka's trunk; the code in trunk as of today allows client to create and to delete topics. But unfortunately, for your use case, increasing the number of partitions is still not possible yet -- this is in scope for KIP-4 (see Alter Topics Request) but is not completed yet.
TL;DR: The next versions of Kafka will allow you to increase the number of partitions of a Kafka topic, but this functionality is not yet available.
It is not possible to increase the number of partitions from the Producer client.
Any specific use case use why you cannot use the broker to achieve this ?
But my kafkaProducer has only the Kafka broker config, but not the
zookeeper config.
I don't think any client will let you change the broker config. You can only access (read) the server side config at max.
Your producer can provide different keys for ProducerRecord's. The broker would place them in different partitions. For example, if you need two partitions, use keys "abc" and "xyz".
This can be done in version 0.9 as well.