I do not understand how messages that could not be de-serialized can be written to a DLT topic with spring kafka.
I configured the consumer according to the spring kafka docs and this works well for exceptions that occur after de-serialization of the message.
But when the message is not de-serializable a org.apache.kafka.common.errors.SerializationExceptionis thrown while polling for messages.
Subsequently, SeekToCurrentErrorHandler.handle(Exception thrownException, List<ConsumerRecord<?, ?>> records, ...) is called with this exception but with an empty list of records and is therefore unable to write something to DLT topic.
How can I write those messages to DLT topic also?
The problem is that the exception is thrown by the Kafka client itself so Spring doesn't get to see the actual record that failed.
That's why we added the ErrorHandlingDeserializer2 which can be used to wrap the actual deserializer; the failure is passed to the listener container and re-thrown as a DeserializationException.
See the documentation.
When a deserializer fails to deserialize a message, Spring has no way to handle the problem, because it occurs before the poll() returns. To solve this problem, version 2.2 introduced the ErrorHandlingDeserializer2. This deserializer delegates to a real deserializer (key or value). If the delegate fails to deserialize the record content, the ErrorHandlingDeserializer2 returns a null value and a DeserializationException in a header that contains the cause and the raw bytes. When you use a record-level MessageListener, if the ConsumerRecord contains a DeserializationException header for either the key or value, the container’s ErrorHandler is called with the failed ConsumerRecord. The record is not passed to the listener.
The DeadLetterPublishingRecoverer has logic to detect the exception and publish the failed record.
Related
I have a Flink (v1.15) pipeline implementing a custom AbstractDeserializationSchema.
An exception is thrown for a bad message in the deserialize(byte[] message) method whereby I catch the exception and simply return null.
According to the Flink docs (see below) returning null should cause Flink to ignore this message and move onto the next. My example doesn't and reprocesses the message continuously.
Here is a sample of my deserializer. I did some debugging and it does indeed step into the if block and it returns null. The message is then simply reprocessed over and over.
According to the official Flink docs (v1.13 and below) returning null should cause Flink to ignore this message.
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/
This exceprt below is from the above link (v1.13) and I noticed from v1.14 / v1.15 on this has been removed (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/)
This is the repeated flow of the task after the XML parsing error
Thank you
I am using kafka-avro-serializer-6.0.0.jar. When I hit exceptions deserializing events my consumer stops and does not move to the next event. These are usually caused by errors at the producer and have happened because of issues using a new avro schema registry server.
Example:
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 58
Caused by: java.lang.ClassCastException:
I can fix the exceptions, that's not the issue. But to fix the consumers I need to reset each offset manually to latest. This is a lot of hassle in my scenario and I have a lot of consumer groups.
Is there a way for me to ignore these exceptions and move the offset at the consumer? I guess because I am using manual offset commit I have this issue. Anyone knows of ways to configure kafka-avro-serializer-6.0.0.jar to do what I want?
Thanks.
You have mainly two options:
Override the deserializer deserialize method and reimplement it by catching the ClassCastException exception and returning a null Object instead of the deserialized record. These null objects will then be dealt with in your consumer code.
Catch the SerializationException exception on your consumer code and seek your consumer past the bad record offset.
Both solutions are very well explained in this article by Jon Boulineau.
I am trying to implement a retry logic within kafka streams processor topology in the event there was an exception producing to a sink topic.
I am using a custom ProductionExceptionHandler to be able to catch exception that happen on "producer.send" to the sink topic upon context.forward
What criteria should I use to be able resend the message to an alternate sink topic if there was an exception sending to original sink topic. Could this be deduced from type of exception in producer exception handler without compromising the transactional nature of the internal producer in Kafka streams.
If we decide to produce to a dead letter queue from production exception handler in some unrecoverable errors, could this be done within the context of "EOS" guarantee or it has to be a custom producer not known to the topology.
Kafka Streams has not built-in support for dead-letter-queue. Hence, you are "on your own" to implement it.
What criteria should I use to be able resend the message to an alternate sink topic if there was an exception sending to original sink topic.
Not sure what you mean by this? Can you elaborate?
Could this be deduced from type of exception in producer exception handler
Also not sure about this part.
without compromising the transactional nature of the internal producer in Kafka streams.
That is not possible. You have no access to the internal producer.
If we decide to produce to a dead letter queue from production exception handler in some unrecoverable errors, could this be done within the context of "EOS" guarantee or it has to be a custom producer not known to the topology.
You would need to maintain your own producer and thus it's out-of-scope for EOS.
I am using Spring Kafka's KafkaTemplate for sending message using the async way and doing proper error handing using callback.
Also, I have configured the Kafka producer to have maximum of retries (MAX_INTEGER).
However there maybe some errors which is related with avro serialization, but for those retry wouldn't help. So how can I escape those error without retries but for other broker related issues I want to do retry?
The serialization exception will occur before the message is sent, so the retries property is irrelevant in that case; it only applies when the message is actually sent.
I have a service that is producing Kafka messages with a payload of type com.fasterxml.jackson.databind.JsonNode. When I consume this message I want it to be serialized into a POJO, but I'm getting the following message:
IllegalArgumentException: Incorrect type specified for header
'kafka_receivedMessageKey'. Expected [class com.example.Person] but
actual type is [class com.fasterxml.jackson.databind.node.ObjectNode]
How do I configure either the producer or the consumer parts to work as I intend it to?
I think these are the approaches I can take:
Kafka Consumer should ignore __Key_TypeId__ if it is a subclass of JsonNode
OR
Kafka Producer should produce a message without __Key_TypeId__ header if it is a subclass of JsonNode
But how do I implement either of these approaches? Or is there another option?
See the reference manual.
You can either set JsonSerializer.ADD_TYPE_INFO_HEADERS to false on the producer or JsonDeserializer.USE_TYPE_INFO_HEADERS to false on the consumer.