When I am creating consumer and trying to deserialize object I got error
Caused by: IllegalArgumentException: The class 'com.domain.project2.package2.SomeEvent' is not in the trusted packages: [java.util, java.lang, com.domain.project2.package1, com.domain.project2.package2]. If you belive this class is....
My .yml config:
spring:
kafka:
bootstrap-servers: localhost:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.support.serializer.JsonSerializer
consumer:
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.support.serializer.JsonDeserializer
properties:
spring:
json:
trusted:
packages: 'com.domain.project2.package1, com.domain.project2.package2'
I presume you mean
spring:
kafka:
bootstrap-servers: localhost:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
consumer:
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
properties:
spring:
json:
trusted:
packages: 'com.domain.project2.package1, com.domain.project2.package2'
Since you are using the Spring deserializer, not the apache JsonDeserializer.
The problem is the space after the comma.
Use 'com.domain.project2.package1,com.domain.project2.package2'.
We should probably trim the packages to remove extraneous spaces.
Related
I am using kafka for consuming messages. While consuming messages, there are possibility that I may get different messages which would cause DeserializationException. I want to skip the records that causes DeserializationException and process the one which is not causing any issues.
All Kafka related properties are configured through properties like below,
kafka:
producer:
bootstrap-servers:
- PRODUCER_BROKERS
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
consumer:
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
bootstrap-servers:
- CONSUMER_BROKERS
properties:
key.deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
value.deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
spring.deserializer.value.delegate.class: io.confluent.kafka.serializers.KafkaAvroDeserializer
When I googled I get some solution by implementing ErrorHandler from How to catch deserialization error in Kafka-Spring? but since I am using properties I am not sure, how can I bind it to ConcurrentKafkaListenerContainerFactory. What is a better approach?
Your configuration looks correct.
The default error handler (DefaultErrorHandler) will discard (log) the records with failed deserialization errors.
My consumer is going to get data from Cluster A - which is non secure and does PLAINTEXT based communication.
Once consumer received that message from cluster A , application split the message into 3 parts based on business logic and send ( Producer) the message to the cluster B ( SASL_SSL) to 3 topics.
Consumer
-> Cluster A (PLAINTEXT )
-> Topic : raw-item
Split message into three parts
Producer
-> Cluster B (SASL_SSL)
-> Topics : item, price, inventory
If any thing goes wrong with any of the 3 destination topics ( item, price, inventory) - then entire Transaction will roll backed.
Does Multi cluster Transaction supported by spring - Kafka configuration?
I get below exception -
2022-04-18 12:28:55.827 INFO 26312 --- [ntainer#0-0-C-1] o.a.k.clients.producer.KafkaProducer : [Producer clientId=multi-topic-tx-producer-1, transactionalId=tx-432eeb17-7af0-43a9-bfe9-d757234faca4raw-item-consumer-grp.OSMI_C02_CATALOG_MKPDOMAIN.0] Aborting incomplete transaction
2022-04-18 12:28:55.835 ERROR 26312 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : Authentication/Authorization Exception and no authExceptionRetryInterval set
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: raw-item-consumer-grp
raw-item-consumer-grp ( consumer ) getting data from cluster A where No ACL added then why it ask Group permission
Is this due to Non supported Multi cluster Transaction for by spring - Kafka configuration?
below configuration clearly shows my consumer is getting data from cluster A and Producer will send data to SASL_SSL enabled cluster B
spring:
profiles : local
kafka:
consumer:
bootstrap-servers: localhost:9192,localhost:9193,localhost:9194
groupId: raw-item-consumer-grp
client-id: raw-item-consumer-client
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
enable-auto-commit: false
auto-offset-reset: earliest
isolation-level: read-committed
producer:
client-id: multi-topic-tx-producer
bootstrap-servers: localhost:9092,localhost:9093,localhost:9094
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
ssl:
trust-store-location: kafka.producer.truststore.jks
trust-store-password: password
transaction-id-prefix: tx-${random.uuid}
properties:
sasl:
jaas:
config: org.apache.kafka.common.security.scram.ScramLoginModule required username="rawitem-multitopic-sasl-producer" password="Dem12345";
mechanism: SCRAM-SHA-512
security:
protocol: SASL_SSL
ssl.endpoint.identification.algorithm:
enable.idempotence: true
acks: all
retries: 10
in my application kafka messages were consumed from multiple topics by streamlistner but i want to add functional consumer bean for one topic is this possible some topics consuming by streamlistner and some with consumer bean in same application?
when i tried topics were created only which are consumed by streamlistner but not for consumer bean?
application.yml
spring:
cloud:
function:
definition: printString
stream:
kafka:
binder:
brokers: localhost:9092
zkNodes: localhost:2181
autoCreateTopics: true
autoAddPartitions: true
bindings:
printString-in-0:
destination: function-input-topic
group: abc
consumer:
maxAttempts: 1
partitioned: true
concurrency: 2
xyz-input:
group: abc
destination: xyz-input-input
consumer:
maxAttempts: 1
partitioned: true
concurrency: 2
topic was created for xyz-input topic but not for function-input-topic
consumer bean
import java.util.function.Consumer;
#Component
public class xyz {
#Bean
Consumer<String> printString() {
return System.out::print;
}
}
kafkaConfig Interface
public interface KafkaConfig {
#Input("xyz-input")
SubscribableChannel inbound();
}
No, we tried in the past but there are subtle issues when the two programming modes collide. It is quite simple to factor out any StreamListener into a functional way (mostly removing code) and the app actually becomes much simpler.
Just get rid of KafkaConfig interface all together, get rid of #EnableBinding and you should be fine.
I have below property set in spring boot kafka producer application.yaml
consumer-properties:
key.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
producer-properties:
key.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
I have to produce message from kafka console producer eg-
kafka-console-producer --bootstrap-server confluent-cp-kafka:9092 --topic TSTTOPIC --producer-property key.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
but its not working and whn I produce message from console producer I get error in consumer log as below
You cannot use colons on the CLI.
If you want to use your property file, then pass --producer.config with the producer.properties file
Otherwise, you can use kafka-avro-console-producer along with --producer-property key.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
As for the Avro serializers, you appear to be missing any key.schema or value.schema + schema.registry.url, which are only properties read by the kakfa-avro-console-producer and would explain why your Avro consumer would be unable to read the data (it was sent as plaintext)
We have a micro-services that produces and consumes messages from Kafka using spring-boot and spring-cloud-stream.
versions:
spring-boot: 1.5.8.RELEASE
spring-cloud-stream: Ditmars.RELEASE
Kafka server: kafka_2.11-1.0.0
EDIT:
We are working in a Kubernetes environment using StatefulSets cluster of 3 Kafka nodes and a cluster of 3 Zookeeper nodes.
We experienced several occurrences of old messages that are reprocessed when those messages where already processed few days ago.
Several notes:
Before that happens the following logs were printed (there are more similar lines this is just a summary)
Revoking previously assigned partitions [] for group enrollment-service
Discovered coordinator dev-kafka-1.kube1.iaas.watercorp.com:9092 (id: 2147483646 rack: null)
Successfully joined group enrollment-service with generation 320
The above-mentioned incidents of revoking and reassigning of partitions happens every few hours. And just in few of those incidents old messages are re-consumed. In most cases the reassigning doesn't triggers message consumption.
The messages are from different partitions.
There are more than 1 message per partition that is being reprocessed.
application.yml:
spring:
cloud:
stream:
kafka:
binder:
brokers: kafka
defaultBrokerPort: 9092
zkNodes: zookeeper
defaultZkPort: 2181
minPartitionCount: 2
replicationFactor: 1
autoCreateTopics: true
autoAddPartitions: true
headers: type,message_id
requiredAcks: 1
configuration:
"[security.protocol]": PLAINTEXT #TODO: This is a workaround. Should be security.protocol
bindings:
user-enrollment-input:
consumer:
autoRebalanceEnabled: true
autoCommitOnError: true
enableDlq: true
user-input:
consumer:
autoRebalanceEnabled: true
autoCommitOnError: true
enableDlq: true
enrollment-mail-output:
producer:
sync: true
configuration:
retries: 10000
enroll-users-output:
producer:
sync: true
configuration:
retries: 10000
default:
binder: kafka
contentType: application/json
group: enrollment-service
consumer:
maxAttempts: 1
producer:
partitionKeyExtractorClass: com.watercorp.messaging.PartitionKeyExtractor
bindings:
user-enrollment-input:
destination: enroll-users
consumer:
concurrency: 10
partitioned: true
user-input:
destination: user
consumer:
concurrency: 5
partitioned: true
enrollment-mail-output:
destination: send-enrollment-mail
producer:
partitionCount: 10
enroll-users-output:
destination: enroll-users
producer:
partitionCount: 10
Is there any configuration that I might be missing? What can cause this behavior?
So the actual problem is the one that is described in the following ticket: https://issues.apache.org/jira/browse/KAFKA-3806.
Using the suggested workaround fixed it.