hi please when i run kafka producer im displays this error
is what you need to delete the topics??
[2019-04-09 01:48:29,011] WARN [Producer clientId=console-producer]
Received invalid metadata error in produce request on partition
test2-0 due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition.. Going to request
metadata update now
(org.apache.kafka.clients.producer.internals.Sender) [2019-04-09
01:48:29,124] ERROR Error when sending message to topic test2 with
key: null, value: 0 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition. [2019-04-09
01:48:29,132] WARN [Producer clientId=console-producer] Received
invalid metadata error in produce request on partition test2-0 due to
org.apache.kafka.common.errors.NotLeaderForPartitionException: This
server is not the leader for that topic-partition.. Going to request
metadata update now
(org.apache.kafka.clients.producer.internals.Sender)
Related
We are using Apache flink kafka consumer to consume the payload . We are facing the delay in processing intermittently. We have added the logs in our business logic and everything looks good. But keep on getting the below error.
[kafka-producer-network-thread | producer-44] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-44] Got error produce response with correlation id 82 on topic-partition topicname-ingress-0, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
[kafka-producer-network-thread | producer-44] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-44] Received invalid metadata error in produce request on partition topicnamae-ingress-0 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
We use spring kafka stream producer to produce data to kafka topic. when we did resiliency test, we got the the below error.
`2020-08-28 16:18:35.536 WARN [,,,] 26 --- [ad | producer-3] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-3] Received invalid metadata error in produce request on partition topic1-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now
log: 2020-08-28 16:18:35.536 WARN [,,,] 26 --- [ad | producer-3] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-3] Got error produce response with correlation id 80187 on topic-partition topic1-0, retrying (4 attempts left). Error: NOT_LEADER_FOR_PARTITION
[Producer clientId=producer-3] Received invalid metadata error in produce request on partition topic1-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now.
The warn should be coming only for the period of time we are running resiliency(broker down/up testing) but these warning happening even after the resiliency test period and happening only for the particular partition(here topic1-0). all the other partitions are working fine.`
this is the producer config we have:
spring.cloud.stream.kafka.binder.requiredAcks=all spring.cloud.stream.kafka.binder.configuration.retries=5 spring.cloud.stream.kafka.binder.configuration.metadata.max.age.ms=3000 spring.cloud.stream.kafka.binder.configuration.max.in.flight.requests.per.connection=1 spring.cloud.stream.kafka.binder.configuration.retry.backoff.ms=10000
we have retry config too and it is retrying to get the proper metadata which you can see the above log but keep getting the same warning for that particular partition. Our kafka team also analyzing this issue. I checked google for any solution but nothing i could find to be useful.
is there any config or anything else missing?
please help me.
Thanks in advance.
This error comes when Kafka is down. Restarting Kafka worked for me! :)
I am using spring boot 2.1.9 and spring Kafka 2.2.9 with Kafka chained transactions.
I am getting some warning from Kafka producer every time. due to this some time functionality will not work.
I want to know why these errors are coming? is there in config problem?
2020-05-04 09:12:35.216 WARN [xxxxx-order-service,,,] 10 --- [ad | producer-8] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-8, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-2] Got error produce response with correlation id 1946 on topic-partition process_event-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:35.327 WARN [xxxxx-order-service,,,] 10 --- [ad | producer-8] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-8, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-2] Got error produce response with correlation id 1950 on topic-partition audit-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.512 WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5807 on topic-partition process_submitted_page_count-2, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.632 WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5811 on topic-partition process_event-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.752 WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5816 on topic-partition audit-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
I assume that you might be hitting this issue.
When a streams application has little traffic, then it is possible
that consumer purging would delete even the last message sent by a
producer (i.e., all the messages sent by this producer have been
consumed and committed), and as a result, the broker would delete that
producer's ID. The next time when this producer tries to send, it will
get this UNKNOWN_PRODUCER_ID error code, but in this case, this error
is retriable: the producer would just get a new producer id and
retries, and then this time it will succeed.
Proposed Solution: Upgrade Kafka
Now this issue has been solved for versions 2.4.0+ so if you are still hitting this you need to upgrade to a newer Kafka version.
Alternative Solution: Increase retention time & transactional.id.expiration.ms
Alternatively, if you cannot (or don't want to) upgrade then you can increase the retention period (log.retention.hours) as well as transactional.id.expiration.ms that defines the amount of inactivity time that needs to pass in order for a producer to be considered as expired (defaults to 7 days).
The following warning message observed in Kafka producer logs. I have a callback to notify failures. But didn't see failure logs. Is this WARN can cause a message failures to topic?
2020-02-17 10:23:47|[kafka-producer-network-thread | producer-1]|WARN |[o.a.k.c.producer.internals.Sender.warn(251)]|[Producer clientId=producer-1] Got error produce response with correlation id 2031 on topic-partition queue-1, retrying (0 attempts left). Error: NETWORK_EXCEPTION
retrying (0 attempts left).
Is saying that the producer is going to retry one final time to send the message
If that fails, I suspect you're going to see another log message, at which point you're losing events due to your network exceptions
I have been creating a Kafka Producer example using Java. I have been
sending normal data which is just "Test" + Integer as value to Kafka. I
have used the below properties and after I have started the Producer
Client and messages are on the way, during this I am killing the broker
and suddenly receiving the below error message instead of retrying.
Using 3 brokers and topic with 3 partitions and replication factor as 3
and no min-insync-replicas
Below are the properties configured config.put(ProducerConfig.ACKS_CONFIG, "all");
config.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1");
config.put(CommonClientConfigs.RETRIES_CONFIG, 60);
config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
config.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG ,10000);
config.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG ,30000);
config.put(ProducerConfig.MAX_BLOCK_MS_CONFIG ,10000);
config.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG , 1048576);
config.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
config.put(ProducerConfig.LINGER_MS_CONFIG, 0);
config.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 1073741824); // 1GB
and the result when I have killed all my brokers or sometimes one of the
broker is as below
**Error:**
WARN org.apache.kafka.clients.producer.internals.Sender - [Producer
clientId=producer-1] Got error produce response with correlation id 124
on topic-partition testing001-0, retrying (59 attempts left). Error:
NETWORK_EXCEPTION
27791 [kafka-producer-network-thread | producer-1] WARN
org.apache.kafka.clients.producer.internals.Sender - [Producer
clientId=producer-1] Received invalid metadata error in produce request
on partition testing001-0 due to
org.apache.kafka.common.errors.NetworkException: The server disconnected
before a response was received.. Going to request metadata update now
28748 [kafka-producer-network-thread | producer-1] ERROR
org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread
'kafka-producer-network-thread | producer-1':
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate
(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom
(NetworkReceive.java:112)
at org.apache.kafka.common.network.KafkaChannel.receive
(KafkaChannel.java:335)
at org.apache.kafka.common.network.KafkaChannel.read
(KafkaChannel.java:296)
at org.apache.kafka.common.network.Selector.attemptRead
(Selector.java:560)
at org.apache.kafka.common.network.Selector.pollSelectionKeys
(Selector.java:496)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.producer.internals.Sender.run
(Sender.java:239)
at org.apache.kafka.clients.producer.internals.Sender.run
(Sender.java:163)
at java.lang.Thread.run(Unknown Source)
I assume you are testing the producer. When a producer connect to the Kafka cluster you will pass all broker IPs and ports as a comma separated string. In your case there are three brokers. When producer try to connect to cluster, as part of initialization cluster controller responds with cluster metadata. Assume your producer only populating message to a single topic. Cluster maintains a leader among brokers for each and every topic. After identify the leader for the topic, your producer only going to communicate to the leader until it is live.
In your testing scenario, you are deliberately killing the broker instances. When it happens kafka cluster need to identify a new leader for your topic and controller has to pass the new meta data to your producer. If the metadata change quite frequently( in your case you may kill another broker mean while) producer may receive invalid metadata.