spring-cloud-stream and kafka-clients backward compatibility

spring-cloud-stream and kafka-clients backward compatibility - apache-kafka

We have a micro services that currently uses spring-cloud-stream Ditmars.RELEASE that in turn uses kafka-clients 0.10.1.1.
We are interested in upgrading to spring-cloud-stream 2.0.0.RC3 that in turn uses kafka-clients 1.0.0 in order to maybe solve a problem that we have: Kafka messages are reprocessed.
After we upgraded only one of our services to spring-boot 2.0.0.RELEASE and spring-cloud-stream 2.0.0.RC3 we encountered strange behavior:
The service that was upgraded (from now I will call it service-1) is a producer of a certain topic send-enrollment-mail. Also, we have another (service-2), that uses the older versions of spring-cloud-stream and kafka-clients, and that is a consumer of this topic. When we are booting service-1 it creates this topic with the appropriate partitions, then we are booting service-2 (that uses the with the new versions) and finally produce a message from service-1 we are receiving the following exception
org.springframework.messaging.MessageHandlingException: nested exception is java.util.concurrent.ExecutionException: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.handleRequestMessage(KafkaProducerMessageHandler.java:406) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder$SendingHandler.handleMessageInternal(AbstractMessageChannelBinder.java:725) ~[spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:463) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:407) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at com.watercorp.app.messaging.producer.MessageProducer.send(MessageProducer.java:39) ~[classes/:na]
at com.watercorp.app.service.EnrollmentServiceImpl.sendEnrollmentMail(EnrollmentServiceImpl.java:56) ~[classes/:na]
at com.watercorp.app.service.EnrollmentServiceImpl.enrollUsers(EnrollmentServiceImpl.java:50) ~[classes/:na]
at com.watercorp.app.messaging.handler.UsersEnrollmentMessageHandler.handleMessage(UsersEnrollmentMessageHandler.java:60) ~[classes/:na]
at com.watercorp.app.messaging.handler.UsersEnrollmentMessageHandler.handleMessage(UsersEnrollmentMessageHandler.java:24) ~[classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.lambda$handleMessageWithRetry$0(MessageConsumer.java:84) [classes/:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) ~[spring-retry-1.2.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164) ~[spring-retry-1.2.2.RELEASE.jar:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.handleMessageWithRetry(MessageConsumer.java:77) [classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.handleUsersEnrollmentMessage(MessageConsumer.java:65) [classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer$$FastClassBySpringCGLIB$$8b03a437.invoke() [classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:112) [spring-context-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at com.watercorp.app.messaging.consumer.MessageConsumer$$EnhancerBySpringCGLIB$$a3ae110f.handleUsersEnrollmentMessage() [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:181) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:114) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:55) [spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.cloud.stream.binding.DispatchingStreamListenerMessageHandler.handleRequestMessage(DispatchingStreamListenerMessageHandler.java:87) [spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:157) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:463) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:407) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:181) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:160) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:108) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:203) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:70) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:387) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:364) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1001) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:981) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:932) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:801) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:689) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.util.concurrent.ExecutionException: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_121]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) [na:1.8.0_121]
at org.springframework.util.concurrent.SettableListenableFuture.get(SettableListenableFuture.java:133) ~[spring-core-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.processSendResult(KafkaProducerMessageHandler.java:507) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.handleRequestMessage(KafkaProducerMessageHandler.java:398) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
... 62 common frames omitted
Caused by: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at org.springframework.kafka.core.KafkaTemplate$1.onCompletion(KafkaTemplate.java:354) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) ~[kafka-clients-1.0.0.jar:na]
... 1 common frames omitted
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
And also the following error is printed:
ERROR 22159 --- [ad | producer-5] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='{123, 34, 109, 101, 115, 115, 97, 103, 101, 84, 121, 112, 101, 34, 58, 34, 69, 78, 82, 79, 76, 76, 7...' to topic send-enrollment-mail and partition 7:
Some notes:
1. The message get sent to kafka server and stored there - when I open a console consumer for this topic I see the message that was by service-1 sent.
2. The message is not consumed by the service-2 that subscribed to this topic.
3. After some time service-2 starts spitting the following errors every millisecond
WARN 11837 --- [ -C-1] o.a.k.c.consumer.internals.Fetcher : Unknown error fetching data for topic-partition send-enrollment-mail-5
4. When I stop service-2 and then produce another message from service-1, service-1 not receiving the UnknownServerException error.
5. Then when I start service-2 it spitting the
WARN 11837 --- [ -C-1] o.a.k.c.consumer.internals.Fetcher : Unknown error fetching data for topic-partition send-enrollment-mail-5
for each partition that has at least one message that was sent from the producer.
I would appreciate any help with this issue

You have to use headerMode=embeddedHeaders or none for compatibility with older (Ditmars) SCSt apps (depending on which header more those apps use). The native header mode for 2.0 apps is native - since Kafka now supports headers.

Related

Error for Describe Topics Offsets error in AKHQ

Connection to kafka server is fine but when I try to load the topics in AKHQ getting the following error
2022-04-11 09:20:06,309 ERROR r-thread-6 o.a.c.ErrorController Error for Describe Topics Offsets {}
java.lang.RuntimeException: Error for Describe Topics Offsets {}
at org.akhq.utils.Logger.call(Logger.java:31)
at org.akhq.modules.AbstractKafkaWrapper.describeTopicsOffsets(AbstractKafkaWrapper.java:121)
at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Exec.dispatch(Unknown Source)
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:128)
at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Intercepted.describeTopicsOffsets(Unknown Source)
at org.akhq.repositories.TopicRepository.findByName(TopicRepository.java:108)
at org.akhq.repositories.TopicRepository.findByName(TopicRepository.java:94)
at org.akhq.controllers.TopicController.data(TopicController.java:191)
at org.akhq.controllers.$TopicController$Definition$Exec.dispatch(Unknown Source)
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:656)
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$null$0(ReactorInstrumentation.java:62)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:68)
at io.micrometer.core.instrument.Timer.lambda$wrap$1(Timer.java:171)
at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 1600ms

Increasing the default.api.timeout.ms value in AKHQ application.yml solved the problem
akhq:
clients-defaults:
consumer:
properties:
default.api.timeout.ms: 15000000

pyflink on yarn, kafka, NoClassDefFoundError

I deployed my pyflink job on yarn, it include kafka consume.
my run command:
/opt/flink/bin/flink run -m yarn-cluster -yid application_1634021687380_0009 --jarfile=/opt/flink/lib/flink-sql-connector-kafka_2.11-1.13.2.jar -pyarch venv.zip -pyexec venv.zip/venv/bin/python -py demo.py
when I submit my job, it get this output errors：
<pre>SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/ impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.ja r!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021-10-13 14:41:14,387 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink.
2021-10-13 14:41:14,387 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-flink.
PID 32298: initializing main ...
Traceback (most recent call last):
File "/opt/demo.py", line 92, in <module>
GenUniqueIPAssest()
File "/opt/demo.py", line 84, in GenUniqueIPAssest
source_table.select(
File "/opt/flink/opt/python/pyflink.zip/pyflink/table/table.py", line 1056, in execute _insert
File "/opt/flink/opt/python/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1285, in __call__
File "/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 146, in deco
File "/opt/flink/opt/python/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 326, in get_ return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o64.executeInsert.
: java.lang.NoClassDefFoundError: org/apache/kafka/common/errors/InvalidTxnStateExceptio n
at org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSink.createKafk aProducer(KafkaDynamicSink.java:329)
at org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSink.getSinkRun timeProvider(KafkaDynamicSink.java:175)
at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.createSi nkTransformation(CommonExecSink.java:118)
at org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink.translat eToPlanInternal(StreamExecSink.java:130)
at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(E xecNodeBase.java:134)
at org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToP lan$1(StreamPlanner.scala:70)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.flink.table.planner.delegation.StreamPlanner.translateToPlan(Strea mPlanner.scala:69)
at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.s cala:165)
at org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvir onmentImpl.java:1518)
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(Tabl eEnvironmentImpl.java:740)
at org.apache.flink.table.api.internal.TableImpl.executeInsert(TableImpl.java:57 2)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl. java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(Metho dInvoker.java:244)
at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(Re flectionEngine.java:357)
at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod (AbstractCommand.java:132)
at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallComm and.java:79)
at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnecti on.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.errors.InvalidTxnSt ateException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 32 more
org.apache.flink.client.program.ProgramAbortException: java.lang.RuntimeException: Pytho n process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl. java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgra m.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecu tion(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.jav a:1730)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(H adoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.RuntimeException: Python process exits with code: 1
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:124)
... 16 more
</pre>
My Environment:
jdk1.8.0_211
flink 1.13.2
centos 7.6
my jar classpath : /opt/flink/lib
flink-connector-kafka_2.12.1.13.2.jar
flink-sql-connector-kafka_2.11.1.13.2.jar
I suspect it can't find the kafka jar libiary in my environment, while i use --jarfile and venv.zip to declare , how to slove this? many thanks

Kafka Connect Debezium error while filtering

For config:
curl -X POST "${KAFKA_CONNECT_HOST}/connectors" -H "Content-Type: application/json" -d '{
"name": "DebeziumSMS",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": 1,
"database.hostname": "bill-srv02.corp.oblakowifi.ru",
"database.port": 1433,
"database.user": "sa",
"database.password": "******",
"database.dbname": "sms",
"database.server.name": "server-test",
"database.history.kafka.bootstrap.servers": "192.168.26.142:9092",
"database.history.kafka.topic": "schema-changes.sms",
"errors.log.enable": "true",
"database.history.skip.unparseable.ddl": "true",
"time.precision.mode": "connect",
"transforms": "filter",
"transforms.filter.type": "io.debezium.transforms.Filter",
"transforms.filter.language": "jsr223.groovy",
"transforms.filter.condition": "value.op == ''c''"
}
}'
this error:
[2021-10-19 12:50:46,234] ERROR Error encountered in task DebeziumSMS-0. Executing stage 'TRANSFORMATION' with class 'io.debezium.transforms.Filter'. (org.apache.kafka.connect.runtime.errors.LogReporter:66)
io.debezium.DebeziumException: Error while evaluating expression 'value.op == 'c'' for record 'SourceRecord{sourcePartition={server=server-test}, sourceOffset={transaction_id=null, event_serial_no=1, commit_lsn=00000064:0000420c:0003, change_lsn=00000064:0000420c:0002}} ConnectRecord{topic='server-test', kafkaPartition=0, key=Struct{databaseName=sms}, keySchema=Schema{io.debezium.connector.sqlserver.SchemaChangeKey:STRUCT}, value=Struct{source=Struct{version=1.5.0.Final,connector=sqlserver,name=server-test,ts_ms=1634637045699,db=,schema=,table=,change_lsn=00000064:0000420c:0002,commit_lsn=00000064:0000420c:0003},databaseName=sms,schemaName=dbo,ddl=N/A,tableChanges=[Struct{type=ALTER,id="sms"."dbo"."sms",table=Struct{primaryKeyColumnNames=[id],columns=[Struct{name=id,jdbcType=1,typeName=uniqueidentifier,typeExpression=uniqueidentifier,length=36,position=1,optional=false,autoIncremented=false,generated=false}, Struct{name=creator_login,jdbcType=12,typeName=varchar,typeExpression=varchar,length=64,position=2,optional=false,autoIncremented=false,generated=false}, Struct{name=system_ip_address,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=3,optional=false,autoIncremented=false,generated=false}, Struct{name=system_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=4,optional=false,autoIncremented=false,generated=false}, Struct{name=system_sub_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=5,optional=false,autoIncremented=false,generated=false}, Struct{name=type_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=6,optional=true,autoIncremented=false,generated=false}, Struct{name=date_create,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=7,optional=false,autoIncremented=false,generated=false}, Struct{name=date_send,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=8,optional=false,autoIncremented=false,generated=false}, Struct{name=date_state,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=9,optional=true,autoIncremented=false,generated=false}, Struct{name=time_to_live,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=10,optional=false,autoIncremented=false,generated=false}, Struct{name=state_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=11,optional=false,autoIncremented=false,generated=false}, Struct{name=phone,jdbcType=12,typeName=varchar,typeExpression=varchar,length=32,position=12,optional=false,autoIncremented=false,generated=false}, Struct{name=target_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=13,optional=true,autoIncremented=false,generated=false}, Struct{name=provider_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=14,optional=false,autoIncremented=false,generated=false}, Struct{name=message,jdbcType=12,typeName=varchar,typeExpression=varchar,length=2147483647,position=15,optional=false,autoIncremented=false,generated=false}, Struct{name=linked_file,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=16,optional=true,autoIncremented=false,generated=false}, Struct{name=error_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=17,optional=true,autoIncremented=false,generated=false}, Struct{name=error_msg,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=18,optional=true,autoIncremented=false,generated=false}]}}]}, valueSchema=Schema{io.debezium.connector.sqlserver.SchemaChangeValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}'
at io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:116)
at io.debezium.transforms.Filter.doApply(Filter.java:33)
at io.debezium.transforms.ScriptingTransformation.apply(ScriptingTransformation.java:189)
at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)
at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)
at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:323)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.script.ScriptException: org.apache.kafka.connect.errors.DataException: op is not a valid field name
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:320)
at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71)
at javax.script.CompiledScript.eval(CompiledScript.java:92)
at io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:107)
... 16 more
Caused by: org.apache.kafka.connect.errors.DataException: op is not a valid field name
at org.apache.kafka.connect.data.Struct.lookupField(Struct.java:254)
at org.apache.kafka.connect.data.Struct.get(Struct.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetMethodMetaProperty.getProperty(MethodMetaProperty.java:62)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:329)
at Script1.run(Script1.groovy:1)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
... 19 more
[2021-10-19 12:50:46,237] INFO WorkerSourceTask{id=DebeziumSMS-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:487)
[2021-10-19 12:50:46,237] ERROR WorkerSourceTask{id=DebeziumSMS-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:184)
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)
at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)
at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:323)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.debezium.DebeziumException: Error while evaluating expression 'value.op == 'c'' for record 'SourceRecord{sourcePartition={server=server-test}, sourceOffset={transaction_id=null, event_serial_no=1, commit_lsn=00000064:0000420c:0003, change_lsn=00000064:0000420c:0002}} ConnectRecord{topic='server-test', kafkaPartition=0, key=Struct{databaseName=sms}, keySchema=Schema{io.debezium.connector.sqlserver.SchemaChangeKey:STRUCT}, value=Struct{source=Struct{version=1.5.0.Final,connector=sqlserver,name=server-test,ts_ms=1634637045699,db=,schema=,table=,change_lsn=00000064:0000420c:0002,commit_lsn=00000064:0000420c:0003},databaseName=sms,schemaName=dbo,ddl=N/A,tableChanges=[Struct{type=ALTER,id="sms"."dbo"."sms",table=Struct{primaryKeyColumnNames=[id],columns=[Struct{name=id,jdbcType=1,typeName=uniqueidentifier,typeExpression=uniqueidentifier,length=36,position=1,optional=false,autoIncremented=false,generated=false}, Struct{name=creator_login,jdbcType=12,typeName=varchar,typeExpression=varchar,length=64,position=2,optional=false,autoIncremented=false,generated=false}, Struct{name=system_ip_address,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=3,optional=false,autoIncremented=false,generated=false}, Struct{name=system_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=4,optional=false,autoIncremented=false,generated=false}, Struct{name=system_sub_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=5,optional=false,autoIncremented=false,generated=false}, Struct{name=type_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=6,optional=true,autoIncremented=false,generated=false}, Struct{name=date_create,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=7,optional=false,autoIncremented=false,generated=false}, Struct{name=date_send,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=8,optional=false,autoIncremented=false,generated=false}, Struct{name=date_state,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=9,optional=true,autoIncremented=false,generated=false}, Struct{name=time_to_live,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=10,optional=false,autoIncremented=false,generated=false}, Struct{name=state_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=11,optional=false,autoIncremented=false,generated=false}, Struct{name=phone,jdbcType=12,typeName=varchar,typeExpression=varchar,length=32,position=12,optional=false,autoIncremented=false,generated=false}, Struct{name=target_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=13,optional=true,autoIncremented=false,generated=false}, Struct{name=provider_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=14,optional=false,autoIncremented=false,generated=false}, Struct{name=message,jdbcType=12,typeName=varchar,typeExpression=varchar,length=2147483647,position=15,optional=false,autoIncremented=false,generated=false}, Struct{name=linked_file,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=16,optional=true,autoIncremented=false,generated=false}, Struct{name=error_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=17,optional=true,autoIncremented=false,generated=false}, Struct{name=error_msg,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=18,optional=true,autoIncremented=false,generated=false}]}}]}, valueSchema=Schema{io.debezium.connector.sqlserver.SchemaChangeValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}'
at io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:116)
at io.debezium.transforms.Filter.doApply(Filter.java:33)
at io.debezium.transforms.ScriptingTransformation.apply(ScriptingTransformation.java:189)
at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
... 11 more
Caused by: javax.script.ScriptException: org.apache.kafka.connect.errors.DataException: op is not a valid field name
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:320)
at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71)
at javax.script.CompiledScript.eval(CompiledScript.java:92)
at io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:107)
... 16 more
Caused by: org.apache.kafka.connect.errors.DataException: op is not a valid field name
at org.apache.kafka.connect.data.Struct.lookupField(Struct.java:254)
at org.apache.kafka.connect.data.Struct.get(Struct.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetMethodMetaProperty.getProperty(MethodMetaProperty.java:62)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:329)
at Script1.run(Script1.groovy:1)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
... 19 more
[2021-10-19 12:50:46,237] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask:238)
After recreating Debezium Connector by DELETE and next POST, Debezium works fine a few days. But then error repeats, and connector stopps with this error.
And after reboot Kafka Connect (distributed) service this error again.
Trace for connector error by REST :
"trace":"org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)\n\tat org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:323)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:182)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:231)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: io.debezium.DebeziumException: Error while evaluating expression 'value && value.op == 'c'' for record 'SourceRecord{sourcePartition={server=server-test}, sourceOffset={transaction_id=null, event_serial_no=1, commit_lsn=00000064:0000420c:0003, change_lsn=00000064:0000420c:0002}} ConnectRecord{topic='server-test', kafkaPartition=0, key=Struct{databaseName=sms}, keySchema=Schema{io.debezium.connector.sqlserver.SchemaChangeKey:STRUCT}, value=Struct{source=Struct{version=1.5.0.Final,connector=sqlserver,name=server-test,ts_ms=1634805618423,db=,schema=,table=,change_lsn=00000064:0000420c:0002,commit_lsn=00000064:0000420c:0003},databaseName=sms,schemaName=dbo,ddl=N/A,tableChanges=[Struct{type=ALTER,id=\"sms\".\"dbo\".\"sms\",table=Struct{primaryKeyColumnNames=[id],columns=[Struct{name=id,jdbcType=1,typeName=uniqueidentifier,typeExpression=uniqueidentifier,length=36,position=1,optional=false,autoIncremented=false,generated=false}, Struct{name=creator_login,jdbcType=12,typeName=varchar,typeExpression=varchar,length=64,position=2,optional=false,autoIncremented=false,generated=false}, Struct{name=system_ip_address,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=3,optional=false,autoIncremented=false,generated=false}, Struct{name=system_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=4,optional=false,autoIncremented=false,generated=false}, Struct{name=system_sub_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=5,optional=false,autoIncremented=false,generated=false}, Struct{name=type_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=6,optional=true,autoIncremented=false,generated=false}, Struct{name=date_create,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=7,optional=false,autoIncremented=false,generated=false}, Struct{name=date_send,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=8,optional=false,autoIncremented=false,generated=false}, Struct{name=date_state,jdbcType=93,typeName=datetime,typeExpression=datetime,length=23,scale=3,position=9,optional=true,autoIncremented=false,generated=false}, Struct{name=time_to_live,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=10,optional=false,autoIncremented=false,generated=false}, Struct{name=state_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=11,optional=false,autoIncremented=false,generated=false}, Struct{name=phone,jdbcType=12,typeName=varchar,typeExpression=varchar,length=32,position=12,optional=false,autoIncremented=false,generated=false}, Struct{name=target_id,jdbcType=12,typeName=varchar,typeExpression=varchar,length=128,position=13,optional=true,autoIncremented=false,generated=false}, Struct{name=provider_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=14,optional=false,autoIncremented=false,generated=false}, Struct{name=message,jdbcType=12,typeName=varchar,typeExpression=varchar,length=2147483647,position=15,optional=false,autoIncremented=false,generated=false}, Struct{name=linked_file,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=16,optional=true,autoIncremented=false,generated=false}, Struct{name=error_id,jdbcType=4,typeName=int,typeExpression=int,length=10,scale=0,position=17,optional=true,autoIncremented=false,generated=false}, Struct{name=error_msg,jdbcType=12,typeName=varchar,typeExpression=varchar,length=4000,position=18,optional=true,autoIncremented=false,generated=false}]}}]}, valueSchema=Schema{io.debezium.connector.sqlserver.SchemaChangeValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}'\n\tat io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:116)\n\tat io.debezium.transforms.Filter.doApply(Filter.java:33)\n\tat io.debezium.transforms.ScriptingTransformation.apply(ScriptingTransformation.java:189)\n\tat org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)\n\t... 11 more\nCaused by: javax.script.ScriptException: org.apache.kafka.connect.errors.DataException: op is not a valid field name\n\tat org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:320)\n\tat org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71)\n\tat javax.script.CompiledScript.eval(CompiledScript.java:92)\n\tat io.debezium.transforms.scripting.Jsr223Engine.eval(Jsr223Engine.java:107)\n\t... 16 more\nCaused by: org.apache.kafka.connect.errors.DataException: op is not a valid field name\n\tat org.apache.kafka.connect.data.Struct.lookupField(Struct.java:254)\n\tat org.apache.kafka.connect.data.Struct.get(Struct.java:74)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)\n\tat groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)\n\tat org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetMethodMetaProperty.getProperty(MethodMetaProperty.java:62)\n\tat org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:329)\n\tat Script3.run(Script3.groovy:1)\n\tat org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)\n\t... 19 more\n"}
Caused by: org.apache.kafka.connect.errors.DataException: op is not a valid field name
Why?
What can be wrong ? Thank you.

Facing issues while using camel-sftp-kafka-connector in Apache kafka?

i am trying to create a sftp connector in Apache kafka using camel-sftp-kafka-connector.
i just downloaded the sftp-connector jar and pasted into the kafka's lib folder.
while hitting /connector-plugins endpoint i can able to see the sftp-connector is available
so below is my sample connector configuration
{
"name": "camel-sftp-connector-1",
"config": {
"connector.class": "org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"transforms": "RemoteTransformer",
"transforms.RemoteTransformer.type": "org.apache.camel.kafkaconnector.RemoteFileTransforms",
"topics": "came-sftp-test-1",
"camel.source.path.host": "192.168.30.167",
"camel.source.path.port": "66",
"camel.source.path.directoryName": "/csv/sample",
"camel.source.endpoint.recursive": "true",
"camel.source.endpoint.username": "sftpuser",
"camel.source.endpoint.password": "sftppass",
"camel.source.endpoint.noop": "/false",
"camel.source.endpoint.move": "/done"
}
}
I got the below exception and dont know how to fix it.
[2021-05-05 19:20:58,560] WARN /connectors (org.eclipse.jetty.server.HttpChannel)
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:563)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:254)
at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:236)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:436)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
... 30 more
Caused by: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.apache.camel.kafkaconnector.CamelSourceConnectorConfig.<clinit>(CamelSourceConnectorConfig.java:43)
at org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector.config(CamelSftpSourceConnector.java:29)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:366)
at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$1(AbstractHerder.java:326)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
Caused by: java.lang.ClassNotFoundException: org.jctools.queues.MessagePassingQueue$Supplier
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 9 more
[2021-05-05 19:20:58,562] WARN unhandled due to prior sendError (org.eclipse.jetty.server.HttpChannelState)
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:563)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:254)
at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:236)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:436)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
... 30 more
Caused by: java.lang.NoClassDefFoundError: org/jctools/queues/MessagePassingQueue$Supplier
at org.apache.camel.kafkaconnector.CamelSourceConnectorConfig.<clinit>(CamelSourceConnectorConfig.java:43)
at org.apache.camel.kafkaconnector.sftp.CamelSftpSourceConnector.config(CamelSftpSourceConnector.java:29)
at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:366)
at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$1(AbstractHerder.java:326)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
Caused by: java.lang.ClassNotFoundException: org.jctools.queues.MessagePassingQueue$Supplier
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 9 more
could anyone tell me what is the issue ?
whether i am configuring in wrong way or else any extra jars need to be added?

I've not used Camel connectors, but based on the error, you'll need these jars for that specific package. There's no guarantee you'll be missing other classes needed for the connector
https://github.com/JCTools/JCTools#get-it-now
JARs pasted into the kafka's lib folder.
Modifying the broker classpath is not the appropriate way to configure Connect's plugin.path setting. Connector plugin JARs should be in isolated directories per connector

Storm Trident Topology with Kafka: Received unexpected tuple error

We have a Trident topology that both consumes from and produces to kafka using the kafka-storm OpaqueTridentKafkaSpout and TridentKafkaState. Everything works fine when running on our production storm cluster, but when run in local mode we very often receive the following error:
java.lang.RuntimeException: java.lang.RuntimeException: Received unexpected tuple source: $mastercoord-bg1:2, stream: $commit, id: {-4957901903366351898=6364388931843393707}, [1:0]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.executor$fn__4606$fn__4619$fn__4670.invoke(executor.clj:806) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by: java.lang.RuntimeException: Received unexpected tuple source: $mastercoord-bg1:2, stream: $commit, id: {-4957901903366351898=6364388931843393707}, [1:0]
at storm.trident.planner.SubtopologyBolt.execute(SubtopologyBolt.java:144) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:369) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.executor$fn__4606$tuple_action_fn__4608.invoke(executor.clj:668) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.executor$mk_task_receiver$fn__4529.invoke(executor.clj:424) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.disruptor$clojure_handler$reify__1229.onEvent(disruptor.clj:58) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 6 common frames omitted
We also see this error as part of the same error dump:
java.lang.RuntimeException: org.apache.storm.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /errors/<topology-name>/<bolt-name>-last-error
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:48) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.zookeeper$create_node.invoke(zookeeper.clj:92) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.cluster$mk_distributed_cluster_state$reify__2234.set_data(cluster.clj:104) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.cluster$mk_storm_cluster_state$reify__2774.report_error(cluster.clj:450) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.executor$throttled_report_error_fn$fn__4385.invoke(executor.clj:191) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.executor$mk_executor_data$fn__4439$fn__4440.invoke(executor.clj:253) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.util$async_loop$fn__543.invoke(util.clj:485) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.6.0.jar:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_31]
Caused by: org.apache.storm.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /errors/<topology-name>/<bolt-name>-last-error
at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:119) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:51) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:676) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:660) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:656) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:441) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:431) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:239) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at org.apache.storm.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:193) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:na]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:na]
at backtype.storm.zookeeper$create_node.invoke(zookeeper.clj:91) ~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
... 7 common frames omitted
Currently using the latest storm 0.10.0-SNAPSHOT built from github, but the same issue was occuring with the stable 0.9.3 release. Any help would be greatly appreciated, thanks.

In LocalCluster mode, if you're deploying multiple topologies then name the Spouts uniquely across topologies.
TestTopology: Spout1 -> Bolt1 -> Bolt2...
DummyTopology: Spout1 -> Bolt1 -> Bolt2...
The aforementioned leads to KeeperException as Storm gets confused to which Spout to refer to in case of re-emit attempt, batch failure etc.
The correct way:
TestTopology: TestTopology-Spout1 -> Bolt1 -> Bolt2...
DummyTopology: DummyTopology-Spout1 -> Bolt1 -> Bolt2...

I just met the KeeperException$NodeExistsException because I inited 2 spouts and gave them the same ID. Change one of the IDs and the problem will be resolved.