Kafka Producer stopped working after upgrading the springboot and Kafka Client version - apache-kafka

Question

We have a Kafka producer which worked just fine for many years with the Kafka client version 2.8.2. We use the class "KafkaJsonSchemaSerializer" for the value serialization. Recently we upgraded to the Kafka client version 3.3.0 and all hell broke loose. The application builds and deploys fine but when it tries to publish the message, we are getting the below runtime exception.
<code>
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:468)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:291)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.createRawProducer(DefaultKafkaProducerFactory.java:937)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.createKafkaProducer(DefaultKafkaProducerFactory.java:788)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.doCreateProducer(DefaultKafkaProducerFactory.java:758)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:733)
at org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:727)
at org.springframework.kafka.core.KafkaTemplate.getTheProducer(KafkaTemplate.java:761)
at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:645)
at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:410)
at com.moneygram.dca.receiptprocessing.document.DocumentPublisher.send(DocumentPublisher.java:35)
at com.moneygram.dca.receiptprocessing.document.DocumentService.publishDocument(DocumentService.java:27)
at com.moneygram.dca.receipts.DownloadableReceiptsFacade.publishKafkaDocument(DownloadableReceiptsFacade.java:4773)
at com.moneygram.dca.receipts.DownloadableReceiptsFacade.getDownloadable(DownloadableReceiptsFacade.java:2708)
at com.moneygram.service.DCAServiceEndPoint.getContent(DCAServiceEndPoint.java:320)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.performInvocation(JAXWSMethodInvoker.java:66)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232)
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:74)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkState(ZLjava/lang/String;J)V at com.google.common.cache.CacheBuilder.maximumSize(CacheBuilder.java:433) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:195) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:174) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:157) at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.configureClientProperties(AbstractKafkaSchemaSerDe.java:83) at io.confluent.kafka.serializers.json.AbstractKafkaJsonSchemaSerializer.configure(AbstractKafkaJsonSchemaSerializer.java:54) at io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer.configure(KafkaJsonSchemaSerializer.java:64) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:396) ... 79 more
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:696)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
</code>
Here is the snippet of our KafkaProducerConfig class
<code> public Map<String, Object> producerJsonConfigs() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, getBootstrapServers()); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaJsonSchemaSerializer.class.getName());
props.put(ProducerConfig.CLIENT_ID_CONFIG, getClientIdConfig());
props.put(ProducerConfig.ACKS_CONFIG, getAcksConfig());
props.put(ProducerConfig.RETRIES_CONFIG, getRetriesConfig());
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, getRetryBackOffMsConfig());
props.put("schema.registry.url", getSchemaRegistryUrl());
props.put(ProducerConfig.BATCH_SIZE_CONFIG, getBatchSizeConfig());
props.put(ProducerConfig.LINGER_MS_CONFIG, getLingerMsConfig());
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, getCompressionTypeConfig());
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, getMaxInFlightRequestsPerConnConfig());
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, getBufferMemoryConfig());
props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, getProducerInterceptorClassesConfig());
props.put("confluent.monitoring.interceptor.bootstrap.servers", getConfluentMonitoringInterceptorServerConfig());
props.put("auto.register.schemas", true);
log.info("Enabled Confluent Cloud : {}",enableConfluentCloud);
if (enableConfluentCloud) {
props.put("schema.registry.url", getConfluentCloudSchemaRegistryUrl());
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, getConfluentCloudBootstrapServers());
props.put("security.protocol", getSecurityProtocolConfig());
props.put("sasl.jaas.config", getSaslJaasConfig());
props.put("sasl.login.connect.timeout.ms","300");
props.put("sasl.login.read.timeout.ms","300");
props.put("ssl.endpoint.identification.algorithm", getSslEndpointConfig());
props.put("sasl.mechanism", getSaslMechanismConfig());
props.put("basic.auth.credentials.source", getBasicAuthCredentialsConfig());
props.put("schema.registry.basic.auth.user.info", getSchemaRegistryBasicAuthConfig());
}
return props;
}
</code>
We tried downgrading the kafka client and JSON serializer classes in our POM but it didnt help.
Here are our POM entries for Kafka
org.springframework.kafka
spring-kafka
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-json-schema-serializer</artifactId>
</dependency>