Occasional errors occur. You are only consuming Kafka messages.
org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000milliseconds while awaiting AddOffsetsToTxn
at org.springframework.kafka.listener.SeekUtils.seekOrRecover(SeekUtils.java:206) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.SeekToCurrentErrorHandler.handle(SeekToCurrentErrorHandler.java:112) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:2377) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2254) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access$2300(KafkaMessageListenerContainer.java:518) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$3.doInTransactionWithoutResult(KafkaMessageListenerContainer.java:2092) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36) ~[spring-tx-5.3.20.jar!/:5.3.20]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.20.jar!/:5.3.20]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeInTransaction(KafkaMessageListenerContainer.java:2083) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListenerInTx(KafkaMessageListenerContainer.java:2052) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2028) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1704) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1275) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1267) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1162) ~[spring-kafka-2.7.14.jar!/:2.7.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired after 60000milliseconds while awaiting AddOffsetsToTxn
When does this error occur?
What is AddOffsetsToTxn doing?
Sorry.
I speak a little English. So I'm not good at asking questions
Related
I am trying to implement ReplyingKafkaTemplate with spring batch integration.
It is actually running eagerly and throwing the "Dispatcher has no subscribers for channel" error before the subscriber bean is created. As you can see in the log.
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'kafka Cloud Task.gatewayinputchannel'.;
Adding {kafka:outbound-gateway:kafkaConfig.outGateway.serviceActivator} as a subscriber to the 'gatewayinputchannel' channel
2022-08-18 02:02:08.068 INFO 24232 --- [ main] o.s.integration.channel.DirectChannel : Channel 'kafka Cloud Task.gatewayinputchannel' has 1 subscriber(s).
2022-08-18 02:02:08.068 INFO 24232 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean 'kafkaConfig.outGateway.serviceActivator'
Is there a way to prevent it from running eagerly? and wait till all the beans are created.
My ReplyingKafkaTemplate is described as below.
#Bean
#ServiceActivator(inputChannel = "gatewayinputchannel")
public KafkaProducerMessageHandler<String, Message<?>> outGateway(
ReplyingKafkaTemplate<String, Message<?>, Message<?>> kafkaTemplate) {
KafkaProducerMessageHandler<String,Message<?>> kpmh = new KafkaProducerMessageHandler<String,Message<?>>(kafkaTemplate);
kpmh.setMessageKeyExpression(new LiteralExpression(kafka_processing_topic));
kpmh.setTopicExpression(new LiteralExpression(kafka_processing_topic));
kpmh.setOutputChannel(gatewayoutputchannel());
return kpmh;
}
Edited
#Configuration
#MessagingGateway
#EnableIntegration
#IntegrationComponentScan
public interface IntegrationGateway {
#Gateway(requestChannel = "gatewayinputchannel", replyChannel = "gatewayoutputchannel")
public Message<?> sendToKafka(Message<?> input);
}
We are calling it from the item processor of a batch job
Message<?> request= MessageBuilder.withPayload(inpMessage).copyHeaders(header).build();
Message<?> response = integrationGateway.sendToKafka(request);
-------------------------- Edit 3 --------------------------
How you start the batch job ? :
We are trying to start the batch job from a spring boot project with #Scheduled method programmatically using JobLuncher.
I have below flag to prevent the job starting automatically.
spring.batch.job.enabled=false
2022-08-19 16:00:24.781 ERROR 13732 --- [ task-1] o.s.batch.core.step.AbstractStep : Encountered an error executing step Synchronous Processing : Read -> Process -> Write in job fixedLengthFileJob
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'IMPF Cloud Task.gatewayinputchannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=10001,Bruce,S,Willis,M,16/07/1974,#1 1st main,1st cross,Moody,Alabama,#2 2nd main,2nd cross,Acmar,Alabama,Mobile,(205)710-2385,,Landline,(20,01, headers={RunGUID=00a04208-c0d8-4354-97f0-c5d8bfb5368d, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#27192c74, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#27192c74, IngestionPath=/opt/usr/conf/xyz/, id=d52d89f7-4ec0-5c3e-b84b-be4dc58223b5, kafka_replyTopic=impf_int_batch_success, timestamp=1660905024776}]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:76) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:233) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:47) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.AbstractMessagingTemplate.sendAndReceive(AbstractMessagingTemplate.java:46) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:97) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.MessagingGatewaySupport.doSendAndReceive(MessagingGatewaySupport.java:522) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.MessagingGatewaySupport.sendAndReceiveMessage(MessagingGatewaySupport.java:492) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.sendOrSendAndReceive(GatewayProxyFactoryBean.java:652) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:588) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:555) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:544) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) [spring-aop-5.3.16.jar:5.3.16]
at com.sun.proxy.$Proxy91.sendToKafka(Unknown Source) ~[na:na]
at com.cloudtask.batchconfig.CloudTaskBatchSyncConfig.lambda$syncProcessor$2(CloudTaskBatchSyncConfig.java:204) ~[classes/:na]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doProcess(SimpleChunkProcessor.java:134) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.transform(SimpleChunkProcessor.java:319) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:210) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.16.jar:5.3.16]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:137) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.3.16.jar:5.3.16]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) [spring-batch-core-4.3.5.jar:4.3.5]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_331]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_331]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_331]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_331]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) [spring-aop-5.3.16.jar:5.3.16]
at com.sun.proxy.$Proxy93.run(Unknown Source) [na:na]
at com.cloudtask.batchconfig.runner.JobRunner.runJob(JobRunner.java:71) [classes/:na]
at com.cloudtask.batchconfig.runner.JobRunner.runFlatFileBatchJob(JobRunner.java:43) [classes/:na]
at com.cloudtask.batchconfig.runner.JobRunner$$FastClassBySpringCGLIB$$e5597de2.invoke(<generated>) [classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) [spring-aop-5.3.16.jar:5.3.16]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_331]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_331]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_331]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_331]
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72) ~[spring-integration-core-5.5.9.jar:5.5.9]
... 64 common frames omitted
--------------------------- Edit 4 Job laucnhing code---------------
#Bean
#Scheduled( fixedRate = 150000L)
public void jobScheduled(){
logger.info("Job triggered");
getAllFileNames().stream().forEach( fileName-> {
jobRunner.runFlatFileBatchJob(fileName);
});
}
Thanks
Santrupta
It is not a problem of this outGateway bean and its gatewayinputchannel subscription. It is a problem of producer which sends to this gatewayinputchannel too early. So, share with us, please, what and how does that instead.
version:
cdh(6.2.1),flink(1.13.1), kafka(2.1.0-cdh6.2.1)
data pipeline:
kafka(source) -> flink -> kafka(sink)
The submitted Job is running normally. After I savepoint, an exception occurs when I restore it through savepoint.
2021-11-22 16:39:52,556 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink Streaming Job (daf7707813d79b884b2c7b1897801248) switched from state RUNNING to FAILING.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=0, backoffTimeMS=3000)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:207) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:197) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:188) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:677) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [data-access-1.0-SNAPSHOT-jar-with-dependencies5.jar:?]
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default producerId at version 1
Your Kafka version is rather old, maybe you need to downgrade the kafka-client version in your fat jar. Or even better: maybe you can try to upgrade your Kafka to something more recent. Kafka fixed quite a few things in their transaction handling in 2.5+.
I have long running tasklet which runs for 20hours and after the task finishes I am getting error in Spring batch because of mysql connection closed.
Below is my job configuration
#Configuration
#EnableBatchProcessing
#Slf4j
public class BatchConfig extends DefaultBatchConfigurer {
#Override
#Autowired
public void setDataSource(#Qualifier("batchDataSource") DataSource batchDataSource) {
log.info("DataSource : " + batchDataSource);
super.setDataSource(batchDataSource);
}
public DataSource batchDataSource() {
HikariDataSource dataSource = new HikariDataSource();
dataSource.setDriverClassName(driverClassName);
dataSource.setJdbcUrl(url);
dataSource.setUsername(userName);
dataSource.setPassword(password);
dataSource.setPoolName("mysql");
dataSource.setMaximumPoolSize(10);
dataSource.setMinimumIdle(5);
return dataSource;
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
#Bean("asyncJobLauncher")
public JobLauncher asyncJobLauncher(JobRepository jobRepository) {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
final SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
jobLauncher.setTaskExecutor(simpleAsyncTaskExecutor);
return jobLauncher;
}
}
Job Configuration is
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Autowired
private JobRepository jobRepository;
#Bean
public Job job(PimAutomationBatchServiceImpl pimAutomationBatchService) {
Step stepDownload = steps.get("download-pim-data")
.tasklet((stepContribution, chunkContext) -> {
pimAutomationBatchService.executeDownloadPimDataStep(chunkContext.getStepContext());
return RepeatStatus.FINISHED;
})
.listener(new StepListener())
.build();
Step stepTtlGeneration = steps.get("ttl-generation")
.tasklet((stepContribution, chunkContext)->{
pimAutomationBatchService.executeTTLGenerationStep(chunkContext.getStepContext());
return RepeatStatus.FINISHED;
}).listener(new StepListener()).build();
return jobs.get("PimSync")
.listener(new JobResultListener(jobRepository))
.start(stepDownload)
.next(stepExtract)
.next(stepTtlGeneration)
.build();
}
ttl generation step took 20hours and it has to process 330GB data .
After finishing this step I am getting below error. Please suggest.
com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1094)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1042)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1345)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1027)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$0(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:593)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1168)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:932)
... 52 common frames omitted
Caused by: java.net.SocketException: Connection timed out (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.cj.protocol.a.SimplePacketSender.send(SimplePacketSender.java:55)
at com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:50)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:584)
... 56 common frames omitted
24-06-2019 11:29:19.024 [SimpleAsyncTaskExecutor-1] WARN o.s.j.support.SQLErrorCodesFactory.getErrorCodes:220 - Error while extracting database name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Connection is closed
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:331)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:356)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:214)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:136)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:99)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:122)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1442)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:632)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy84.getMetaData(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.getMetaData(ProxyConnection.java:354)
at com.zaxxer.hikari.pool.HikariProxyConnection.getMetaData(HikariProxyConnection.java)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
... 51 common frames omitted
24-06-2019 11:29:19.028 [SimpleAsyncTaskExecutor-1] ERROR o.s.b.core.step.tasklet.TaskletStep.doInTransaction:463 - JobRepository failure forcing rollback
org.springframework.dao.RecoverableDataAccessException: PreparedStatementCallback; SQL [UPDATE BATCH_STEP_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE STEP_EXECUTION_ID = ?]; The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.; nested exception is com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:100)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1442)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:632)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1094)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1042)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1345)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1027)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$0(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616)
... 44 common frames omitted
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:593)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
By default, MySQL will close stale connections after 8 hours (See max_timeout parameter). Since your tasklet takes 20h, your connection gets closed, hence the error.
You would need to increase the max_timeout value on your MySQL server. This answer might also help.
We have a micro services that currently uses spring-cloud-stream Ditmars.RELEASE that in turn uses kafka-clients 0.10.1.1.
We are interested in upgrading to spring-cloud-stream 2.0.0.RC3 that in turn uses kafka-clients 1.0.0 in order to maybe solve a problem that we have: Kafka messages are reprocessed.
After we upgraded only one of our services to spring-boot 2.0.0.RELEASE and spring-cloud-stream 2.0.0.RC3 we encountered strange behavior:
The service that was upgraded (from now I will call it service-1) is a producer of a certain topic send-enrollment-mail. Also, we have another (service-2), that uses the older versions of spring-cloud-stream and kafka-clients, and that is a consumer of this topic. When we are booting service-1 it creates this topic with the appropriate partitions, then we are booting service-2 (that uses the with the new versions) and finally produce a message from service-1 we are receiving the following exception
org.springframework.messaging.MessageHandlingException: nested exception is java.util.concurrent.ExecutionException: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.handleRequestMessage(KafkaProducerMessageHandler.java:406) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder$SendingHandler.handleMessageInternal(AbstractMessageChannelBinder.java:725) ~[spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:463) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:407) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at com.watercorp.app.messaging.producer.MessageProducer.send(MessageProducer.java:39) ~[classes/:na]
at com.watercorp.app.service.EnrollmentServiceImpl.sendEnrollmentMail(EnrollmentServiceImpl.java:56) ~[classes/:na]
at com.watercorp.app.service.EnrollmentServiceImpl.enrollUsers(EnrollmentServiceImpl.java:50) ~[classes/:na]
at com.watercorp.app.messaging.handler.UsersEnrollmentMessageHandler.handleMessage(UsersEnrollmentMessageHandler.java:60) ~[classes/:na]
at com.watercorp.app.messaging.handler.UsersEnrollmentMessageHandler.handleMessage(UsersEnrollmentMessageHandler.java:24) ~[classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.lambda$handleMessageWithRetry$0(MessageConsumer.java:84) [classes/:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) ~[spring-retry-1.2.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164) ~[spring-retry-1.2.2.RELEASE.jar:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.handleMessageWithRetry(MessageConsumer.java:77) [classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer.handleUsersEnrollmentMessage(MessageConsumer.java:65) [classes/:na]
at com.watercorp.app.messaging.consumer.MessageConsumer$$FastClassBySpringCGLIB$$8b03a437.invoke() [classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:112) [spring-context-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) [spring-aop-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at com.watercorp.app.messaging.consumer.MessageConsumer$$EnhancerBySpringCGLIB$$a3ae110f.handleUsersEnrollmentMessage() [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:181) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:114) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:55) [spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:164) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.cloud.stream.binding.DispatchingStreamListenerMessageHandler.handleRequestMessage(DispatchingStreamListenerMessageHandler.java:87) [spring-cloud-stream-2.0.0.RC3.jar:2.0.0.RC3]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:157) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:463) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:407) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:181) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:160) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:108) [spring-messaging-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:203) [spring-integration-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:70) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:387) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:364) [spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1001) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:981) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:932) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:801) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:689) [spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.util.concurrent.ExecutionException: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_121]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) [na:1.8.0_121]
at org.springframework.util.concurrent.SettableListenableFuture.get(SettableListenableFuture.java:133) ~[spring-core-5.0.4.RELEASE.jar:5.0.4.RELEASE]
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.processSendResult(KafkaProducerMessageHandler.java:507) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.handleRequestMessage(KafkaProducerMessageHandler.java:398) ~[spring-integration-kafka-3.0.3.RELEASE.jar:3.0.3.RELEASE]
... 62 common frames omitted
Caused by: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
at org.springframework.kafka.core.KafkaTemplate$1.onCompletion(KafkaTemplate.java:354) ~[spring-kafka-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:204) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:187) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:627) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:596) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:557) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:481) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:692) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:482) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:474) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239) ~[kafka-clients-1.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163) ~[kafka-clients-1.0.0.jar:na]
... 1 common frames omitted
Caused by: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
And also the following error is printed:
ERROR 22159 --- [ad | producer-5] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='{123, 34, 109, 101, 115, 115, 97, 103, 101, 84, 121, 112, 101, 34, 58, 34, 69, 78, 82, 79, 76, 76, 7...' to topic send-enrollment-mail and partition 7:
Some notes:
1. The message get sent to kafka server and stored there - when I open a console consumer for this topic I see the message that was by service-1 sent.
2. The message is not consumed by the service-2 that subscribed to this topic.
3. After some time service-2 starts spitting the following errors every millisecond
WARN 11837 --- [ -C-1] o.a.k.c.consumer.internals.Fetcher : Unknown error fetching data for topic-partition send-enrollment-mail-5
4. When I stop service-2 and then produce another message from service-1, service-1 not receiving the UnknownServerException error.
5. Then when I start service-2 it spitting the
WARN 11837 --- [ -C-1] o.a.k.c.consumer.internals.Fetcher : Unknown error fetching data for topic-partition send-enrollment-mail-5
for each partition that has at least one message that was sent from the producer.
I would appreciate any help with this issue
You have to use headerMode=embeddedHeaders or none for compatibility with older (Ditmars) SCSt apps (depending on which header more those apps use). The native header mode for 2.0 apps is native - since Kafka now supports headers.
I am getting an no lock exception (caused by sleepycat) while trying to open TitanGraph instance below. Any leads what might be causing it ? Am i missing something ? I have checked for permission and ownership of that directory but no luck.
TitanGraph titanGraph;
Configuration config = new BaseConfiguration();
Configuration storage = config.subset("storage");
storage.setProperty("backend", "local");
storage.setProperty("directory", "data/titan");
titanGraph = TitanFactory.open(config);
Caused by:
java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager
at com.thinkaurelius.titan.diskstorage.Backend.instantiate(Backend.java:355)
at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:367)
at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:311)
at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:121)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1163)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:75)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
at com.narus.tst.graph.GraphDaoImpl.openDb(GraphDaoImpl.java:221)
at com.narus.tst.job.RunNazca.jobRun(RunNazca.java:267)
at com.narus.tst.job.RunNazca.runNazca(RunNazca.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:269)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:312)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.thinkaurelius.titan.diskstorage.Backend.instantiate(Backend.java:344)
... 18 more
Caused by: com.thinkaurelius.titan.diskstorage.PermanentStorageException: Error during BerkeleyJE initialization:
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.initialize(BerkeleyJEStoreManager.java:74)
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<init>(BerkeleyJEStoreManager.java:42)
... 23 more
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) data/titan java.io.IOException: No locks available LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. Environment is invalid and must be closed.
at com.sleepycat.je.log.FileManager.lockEnvironment(FileManager.java:2181)
at com.sleepycat.je.log.FileManager.<init>(FileManager.java:370)
at com.sleepycat.je.dbi.EnvironmentImpl.<init>(EnvironmentImpl.java:453)
at com.sleepycat.je.dbi.EnvironmentImpl.<init>(EnvironmentImpl.java:382)
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:178)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:246)
at com.sleepycat.je.Environment.<init>(Environment.java:227)
at com.sleepycat.je.Environment.<init>(Environment.java:170)
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.initialize(BerkeleyJEStoreManager.java:70)
... 24 more
Caused by: java.io.IOException: No locks available
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1063)
at com.sleepycat.je.log.FileManager.lockEnvironment(FileManager.java:2171)
... 32 more