Spring batch writing to and reading from two databases multiple times - spring-batch

I have gotten a batch job where I read frequently from databases, write to '.csv' and right to other databases and these include multiple steps of writing and reading. I have written steps for each phase of writing and reading. The job runs fine with few hundreds of records but when the record becomes large, I got an exception that the reader is unable to read from file as socket is closed. Is there any design strategy you can help me with or example which involves multiple writing and reading from database with csv. Here is my job.
#Bean
public Job transferDataJob(JobCompletionNotificationListener listener, Step step1, Step step2, Step step3, Step step4, Step step5, Step step6) {
return jobBuilderFactory
.get("transferDataJob")
.incrementer(new RunIdIncrementer())
.preventRestart()
.listener(listener)
.start(step1)
.next(step2)
.next(step3)
.next(step4)
.next(step5)
.next(step6)
.build();
}
Here are my steps.
#Bean
public Step step1() throws Exception{
return stepBuilderFactory.get("step1").<Ptrx, Ptrx> chunk(300)
.reader(databaseReader())
// .processor(processor())
.writer(fileWriter())
.listener(new StepExecutionListenerSupport())
.build();
}
#Bean
public Step step2() {
return stepBuilderFactory.get("step2")
.<Ptrx, Ptrx> chunk(300)
.reader(reader())
//.processor(processor())
.writer(writer())
.build();
}
#Bean
public Step step3() throws Exception{
return stepBuilderFactory.get("step3")
.<Ptrx, Ptrx> chunk(300)
.reader(matchedItemReader())
//.processor(processor())
.writer(matchedItemWriter())
.build();
}
#Bean
public Step step4() {
return stepBuilderFactory.get("step4")
.<Ptrx, Ptrx> chunk(300)
.reader(matchedItemFileReader())
//.processor(processor())
.writer(matchedItemDbWriter())
.build();
}
These are my readers and writers, sorry i had to remove d sql queries
#Bean
public JdbcCursorItemReader<Ptrx> databaseReader(){
JdbcCursorItemReader<Ptrx> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql();
reader.setRowMapper(new PtrxRowMapper());
return reader;
}
#Bean
public FlatFileItemWriter<Ptrx> fileWriter() throws Exception{
FlatFileItemWriter<Ptrx> writer = new FlatFileItemWriter<>();
String rootPath = Thread.currentThread().getContextClassLoader().getResource("").getPath();
String filePath = rootPath + "ptrx.csv";
Path path = Paths.get(new URI("file://"+filePath));
if(path.toFile().exists())
path.toFile().delete();
path.toFile().mkdir();
writer.setResource(new ClassPathResource("ptrx.csv"));
writer.setLineAggregator(new DelimitedLineAggregator<Ptrx>() {{
setDelimiter(",");
setFieldExtractor(new BeanWrapperFieldExtractor<Ptrx>() {{
setNames(new String[] { });
}});
}});
return writer;
}
#Bean
public JdbcBatchItemWriter<Ptrx> writer() {
return new JdbcBatchItemWriterBuilder<Ptrx>()
.itemSqlParameterSourceProvider(new
BeanPropertyItemSqlParameterSourceProvider<>())
.sql()
.dataSource(dataSource)
.build();
}
#Bean
public FlatFileItemReader<Ptrx> reader() {
return new FlatFileItemReaderBuilder<Ptrx>()
.name("ptrxItemReader")
.resource(new ClassPathResource("ptrx.csv"))
.delimited()
.names(new String[] {})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Ptrx>() {{
setTargetType(Ptrx.class);
}})
.build();
}
#Bean
public JdbcCursorItemReader<Ptrx> matchedItemReader(){
JdbcCursorItemReader<Ptrx> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql();
reader.setRowMapper(new PtrxRowMapper());
return reader;
}
#Bean
public FlatFileItemWriter<Ptrx> matchedItemWriter() throws Exception{
FlatFileItemWriter<Ptrx> writer = new FlatFileItemWriter<>();
String rootPath = Thread.currentThread().getContextClassLoader().getResource("").getPath();
String filePath = rootPath + "matched.csv";
Path path = Paths.get(new URI("file://"+filePath));
if(!path.toFile().exists())
path.toFile().mkdir();
else {
path.toFile().delete();
path.toFile().mkdir();
}
writer.setResource(new ClassPathResource("matched.csv"));
writer.setLineAggregator(new DelimitedLineAggregator<Ptrx>() {{
setDelimiter(",");
setFieldExtractor(new BeanWrapperFieldExtractor<Ptrx>() {{
setNames(new String[] { });
}});
}});
return writer;
}
#Bean
public FlatFileItemReader<Ptrx> matchedItemFileReader() {
return new FlatFileItemReaderBuilder<Ptrx>()
.name("matchedReader")
.resource(new ClassPathResource("matched.csv"))
.delimited()
.names(new String[] {})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Ptrx>() {{
setTargetType(Ptrx.class);
}})
.build();
}
#Bean
public JdbcBatchItemWriter<Ptrx> matchedItemDbWriter() {
return new JdbcBatchItemWriterBuilder<Ptrx>()
.itemSqlParameterSourceProvider(new
BeanPropertyItemSqlParameterSourceProvider<>())
.sql()
.dataSource(dataSource)
.build();
}
Here is my database configuration
#Configuration
public class DatabaseConfig {
#Bean(name = "datasource1")
#ConfigurationProperties("database1.datasource")
#Primary
public DataSource dataSource(){
return DataSourceBuilder.create()
.build();
}
#Bean(name = "datasource2")
#ConfigurationProperties("database2.datasource")
public DataSource dataSource2(){
return DataSourceBuilder.create()
.build();
}
Here is the stacktrace
2019-10-25 09:48:56.785 INFO 24323 --- [ restartedMain] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=transferDataJob]] launched with the following parameters: [{run.id=135}]
2019-10-25 09:48:56.803 INFO 24323 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Executing step: [step1]
2019-10-25 09:48:56.807 INFO 24323 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Starting...
2019-10-25 09:48:57.028 INFO 24323 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Start completed.
2019-10-25 09:48:57.420 INFO 24323 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Executing step: [step2]
2019-10-25 09:48:58.176 INFO 24323 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Executing step: [step3]
2019-10-25 09:48:58.296 INFO 24323 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Executing step: [step4]
2019-10-25 09:48:58.404 WARN 24323 --- [ Thread-2] o.s.b.f.support.DisposableBeanAdapter : Destroy method 'close' on bean with name 'matchedItemReader' threw an exception: org.springframework.batch.item.ItemStreamException: Error while closing item reader
2019-10-25 09:48:58.405 WARN 24323 --- [ Thread-2] o.s.b.f.support.DisposableBeanAdapter : Destroy method 'close' on bean with name 'databaseReader' threw an exception: org.springframework.batch.item.ItemStreamException: Error while closing item reader
2019-10-25 09:48:58.405 INFO 24323 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Shutdown initiated...
2019-10-25 09:48:58.412 INFO 24323 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Shutdown completed.
2019-10-25 09:48:58.412 INFO 24323 --- [ Thread-2] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2019-10-25 09:48:58.414 ERROR 24323 --- [ restartedMain] o.s.batch.core.step.AbstractStep : Encountered an error executing step step4 in job transferDataJob
org.springframework.batch.item.file.NonTransientFlatFileException: Unable to read from resource: [class path resource [matched.csv]]
at org.springframework.batch.item.file.FlatFileItemReader.readLine(FlatFileItemReader.java:220) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:173) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.read(AbstractItemCountingItemStreamItemReader.java:92) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:94) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:161) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:119) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:113) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:69) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at com.sun.proxy.$Proxy71.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:207) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:181) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:168) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:781) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:765) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at com.activedge.ReconReport.ReconReportApplication.main(ReconReportApplication.java:10) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.1.9.RELEASE.jar:2.1.9.RELEASE]
Caused by: java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:122) ~[na:1.8.0_171]
at java.io.BufferedReader.readLine(BufferedReader.java:317) ~[na:1.8.0_171]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[na:1.8.0_171]
at org.springframework.batch.item.file.FlatFileItemReader.readLine(FlatFileItemReader.java:201) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
... 53 common frames omitted
2019-10-25 09:48:58.414 INFO 24323 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2019-10-25 09:48:58.419 INFO 24323 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2019-10-25 09:48:58.419 ERROR 24323 --- [ restartedMain] o.s.batch.core.step.AbstractStep : Encountered an error saving batch meta data for step step4 in job transferDataJob. This job is now in an unknown state and should not be restarted.
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:306) ~[spring-jdbc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378) ~[spring-tx-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:475) ~[spring-tx-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289) ~[spring-tx-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at com.sun.proxy.$Proxy68.updateExecutionContext(Unknown Source) ~[na:na]
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at com.sun.proxy.$Proxy66.updateExecutionContext(Unknown Source) ~[na:na]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:249) ~[spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.1.1.RELEASE.jar:4.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.1.10.RELEASE.jar:5.1.10.RELEASE]
at com.sun.proxy.$Proxy71.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:207) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:181) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:168) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:781) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:765) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1204) [spring-boot-2.1.9.RELEASE.jar:2.1.9.RELEASE]
at com.activedge.ReconReport.ReconReportApplication.main(ReconReportApplication.java:10) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) [spring-boot-devtools-2.1.9.RELEASE.jar:2.1.9.RELEASE]
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96) ~[HikariCP-3.2.0.jar:na]
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:263) ~[spring-jdbc-5.1.10.RELEASE.jar:5.1.10.RELEASE]
... 51 common frames omitted

Related

org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel- occures before subscriber bean is created

I am trying to implement ReplyingKafkaTemplate with spring batch integration.
It is actually running eagerly and throwing the "Dispatcher has no subscribers for channel" error before the subscriber bean is created. As you can see in the log.
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'kafka Cloud Task.gatewayinputchannel'.;
Adding {kafka:outbound-gateway:kafkaConfig.outGateway.serviceActivator} as a subscriber to the 'gatewayinputchannel' channel
2022-08-18 02:02:08.068 INFO 24232 --- [ main] o.s.integration.channel.DirectChannel : Channel 'kafka Cloud Task.gatewayinputchannel' has 1 subscriber(s).
2022-08-18 02:02:08.068 INFO 24232 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean 'kafkaConfig.outGateway.serviceActivator'
Is there a way to prevent it from running eagerly? and wait till all the beans are created.
My ReplyingKafkaTemplate is described as below.
#Bean
#ServiceActivator(inputChannel = "gatewayinputchannel")
public KafkaProducerMessageHandler<String, Message<?>> outGateway(
ReplyingKafkaTemplate<String, Message<?>, Message<?>> kafkaTemplate) {
KafkaProducerMessageHandler<String,Message<?>> kpmh = new KafkaProducerMessageHandler<String,Message<?>>(kafkaTemplate);
kpmh.setMessageKeyExpression(new LiteralExpression(kafka_processing_topic));
kpmh.setTopicExpression(new LiteralExpression(kafka_processing_topic));
kpmh.setOutputChannel(gatewayoutputchannel());
return kpmh;
}
Edited
#Configuration
#MessagingGateway
#EnableIntegration
#IntegrationComponentScan
public interface IntegrationGateway {
#Gateway(requestChannel = "gatewayinputchannel", replyChannel = "gatewayoutputchannel")
public Message<?> sendToKafka(Message<?> input);
}
We are calling it from the item processor of a batch job
Message<?> request= MessageBuilder.withPayload(inpMessage).copyHeaders(header).build();
Message<?> response = integrationGateway.sendToKafka(request);
-------------------------- Edit 3 --------------------------
How you start the batch job ? :
We are trying to start the batch job from a spring boot project with #Scheduled method programmatically using JobLuncher.
I have below flag to prevent the job starting automatically.
spring.batch.job.enabled=false
2022-08-19 16:00:24.781 ERROR 13732 --- [ task-1] o.s.batch.core.step.AbstractStep : Encountered an error executing step Synchronous Processing : Read -> Process -> Write in job fixedLengthFileJob
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'IMPF Cloud Task.gatewayinputchannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=10001,Bruce,S,Willis,M,16/07/1974,#1 1st main,1st cross,Moody,Alabama,#2 2nd main,2nd cross,Acmar,Alabama,Mobile,(205)710-2385,,Landline,(20,01, headers={RunGUID=00a04208-c0d8-4354-97f0-c5d8bfb5368d, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#27192c74, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#27192c74, IngestionPath=/opt/usr/conf/xyz/, id=d52d89f7-4ec0-5c3e-b84b-be4dc58223b5, kafka_replyTopic=impf_int_batch_success, timestamp=1660905024776}]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:76) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:233) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:47) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.messaging.core.AbstractMessagingTemplate.sendAndReceive(AbstractMessagingTemplate.java:46) ~[spring-messaging-5.3.16.jar:5.3.16]
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:97) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.MessagingGatewaySupport.doSendAndReceive(MessagingGatewaySupport.java:522) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.MessagingGatewaySupport.sendAndReceiveMessage(MessagingGatewaySupport.java:492) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.sendOrSendAndReceive(GatewayProxyFactoryBean.java:652) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:588) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:555) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:544) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) [spring-aop-5.3.16.jar:5.3.16]
at com.sun.proxy.$Proxy91.sendToKafka(Unknown Source) ~[na:na]
at com.cloudtask.batchconfig.CloudTaskBatchSyncConfig.lambda$syncProcessor$2(CloudTaskBatchSyncConfig.java:204) ~[classes/:na]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doProcess(SimpleChunkProcessor.java:134) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.transform(SimpleChunkProcessor.java:319) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:210) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.3.16.jar:5.3.16]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:137) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.3.16.jar:5.3.16]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) [spring-batch-core-4.3.5.jar:4.3.5]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_331]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_331]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_331]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_331]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) [spring-batch-core-4.3.5.jar:4.3.5]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) [spring-aop-5.3.16.jar:5.3.16]
at com.sun.proxy.$Proxy93.run(Unknown Source) [na:na]
at com.cloudtask.batchconfig.runner.JobRunner.runJob(JobRunner.java:71) [classes/:na]
at com.cloudtask.batchconfig.runner.JobRunner.runFlatFileBatchJob(JobRunner.java:43) [classes/:na]
at com.cloudtask.batchconfig.runner.JobRunner$$FastClassBySpringCGLIB$$e5597de2.invoke(<generated>) [classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) [spring-aop-5.3.16.jar:5.3.16]
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) [spring-aop-5.3.16.jar:5.3.16]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_331]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_331]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_331]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_331]
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106) ~[spring-integration-core-5.5.9.jar:5.5.9]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72) ~[spring-integration-core-5.5.9.jar:5.5.9]
... 64 common frames omitted
--------------------------- Edit 4 Job laucnhing code---------------
#Bean
#Scheduled( fixedRate = 150000L)
public void jobScheduled(){
logger.info("Job triggered");
getAllFileNames().stream().forEach( fileName-> {
jobRunner.runFlatFileBatchJob(fileName);
});
}
Thanks
Santrupta
It is not a problem of this outGateway bean and its gatewayinputchannel subscription. It is a problem of producer which sends to this gatewayinputchannel too early. So, share with us, please, what and how does that instead.

SCDF + Spring Batch : JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={run.id=1}

SCDF is launching job as:
Command to be executed: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/bin/java -jar /opt/scdf-batches/foo/foo-1.0.26.jar --spring.cloud.task.executionid=23
Error:
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) [spring-boot-2.4.1.jar!/:2.4.1]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:785) [spring-boot-2.4.1.jar!/:2.4.1]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.4.1.jar!/:2.4.1]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) [spring-boot-2.4.1.jar!/:2.4.1]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) [spring-boot-2.4.1.jar!/:2.4.1]
at com.example.FooSsapApplication.main(FooSsapApplication.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [
Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={run.id=1}. If you want to run this job again, change the parameters.
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:136) ~[spring-batch-core-4.3.1.jar!/:4.3.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.2.jar!/:5.3.2]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.2.jar!/:5.3.2]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181) ~[spring-batch-core-4.3.1.jar!/:4.3.1]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.2.jar!/:5.3.2]
at com.sun.proxy.$Proxy150.createJobExecution(Unknown Source) ~[na:na]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137) ~[spring-batch-core-4.3.1.jar!/:4.3.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.1.jar!/:4.3.1]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.2.jar!/:5.3.2]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.2.jar!/:5.3.2]
at com.sun.proxy.$Proxy152.run(Unknown Source) ~[na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.4.1.jar!/:2.4.1]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795) [spring-boot-2.4.1.jar!/:2.4.1]
... 13 common frames omitted
Batch Code:
#Bean
public Job job() {
Step step = stepBuilderFactory.get("Foo")
.<Long, Long>chunk(chunkSize)
.reader(fooReader())
.processor(fooProcessor())
.writer(fooWriter())
.allowStartIfComplete(true)
.build();
return jobBuilderFactory.get("FooJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.start(step)
.build();
}
looking at A job instance already exists and is complete for parameters={-spring.cloud.task.executionid=2}. If you want to run this job again, change the parame response, modified the code, but seems its not working.
I am using Spring Boot 2.4.1 and SCDF - 2.7.1

Java parallelStream run task always wait a long time, why?

I use parallelStream to execute backgroud task in Spring #Async annotation. but the task always blocked then stop. the core code is:
#Async
public void task() {
List<ConfigSymbol> symbols = SymbolUtils.getSaasOpenSymbol();
symbols .parallelStream().forEach(symbol->{
//execute task. 1 minute per loop
}
}
I catch the thread stack to diagnosis the problem. but can not find reason.
why the ForkJoinTask status is waiting .
"SimpleAsyncTaskExecutor-1609" #1777 prio=5 os_prio=0 tid=0x00007fb968006800 nid=0x6f5 in Object.wait() [0x00007fba2d119000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
- locked <0x00000006428a0a38> (a java.util.stream.ForEachOps$ForEachTask)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
at com.chainup.stats.service.impl.StatsSymbolRateServiceImpl.getCalculationBTCExchangeRate(StatsSymbolRateServiceImpl.java:141)
at com.chainup.stats.service.impl.StatsSymbolRateServiceImpl.runThreadExchangeRate(StatsSymbolRateServiceImpl.java:67)
at com.chainup.stats.action.StatAct.insertStatsSymbolRate(StatAct.java:442)
at com.chainup.stats.action.StatAct$$FastClassBySpringCGLIB$$74ae31d4.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:66)
at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.chainup.stats.interceptor.StatsJobInterceptor.around(StatsJobInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor273.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$790/775437217.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:65)
at java.lang.Thread.run(Thread.java:748)

Spring batch long running tasklet error The last packet sent successfully to the server was

I have long running tasklet which runs for 20hours and after the task finishes I am getting error in Spring batch because of mysql connection closed.
Below is my job configuration
#Configuration
#EnableBatchProcessing
#Slf4j
public class BatchConfig extends DefaultBatchConfigurer {
#Override
#Autowired
public void setDataSource(#Qualifier("batchDataSource") DataSource batchDataSource) {
log.info("DataSource : " + batchDataSource);
super.setDataSource(batchDataSource);
}
public DataSource batchDataSource() {
HikariDataSource dataSource = new HikariDataSource();
dataSource.setDriverClassName(driverClassName);
dataSource.setJdbcUrl(url);
dataSource.setUsername(userName);
dataSource.setPassword(password);
dataSource.setPoolName("mysql");
dataSource.setMaximumPoolSize(10);
dataSource.setMinimumIdle(5);
return dataSource;
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
#Bean("asyncJobLauncher")
public JobLauncher asyncJobLauncher(JobRepository jobRepository) {
final SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
final SimpleAsyncTaskExecutor simpleAsyncTaskExecutor = new SimpleAsyncTaskExecutor();
jobLauncher.setTaskExecutor(simpleAsyncTaskExecutor);
return jobLauncher;
}
}
Job Configuration is
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Autowired
private JobRepository jobRepository;
#Bean
public Job job(PimAutomationBatchServiceImpl pimAutomationBatchService) {
Step stepDownload = steps.get("download-pim-data")
.tasklet((stepContribution, chunkContext) -> {
pimAutomationBatchService.executeDownloadPimDataStep(chunkContext.getStepContext());
return RepeatStatus.FINISHED;
})
.listener(new StepListener())
.build();
Step stepTtlGeneration = steps.get("ttl-generation")
.tasklet((stepContribution, chunkContext)->{
pimAutomationBatchService.executeTTLGenerationStep(chunkContext.getStepContext());
return RepeatStatus.FINISHED;
}).listener(new StepListener()).build();
return jobs.get("PimSync")
.listener(new JobResultListener(jobRepository))
.start(stepDownload)
.next(stepExtract)
.next(stepTtlGeneration)
.build();
}
ttl generation step took 20hours and it has to process 330GB data .
After finishing this step I am getting below error. Please suggest.
com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1094)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1042)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1345)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1027)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$0(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:593)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1168)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:932)
... 52 common frames omitted
Caused by: java.net.SocketException: Connection timed out (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.mysql.cj.protocol.a.SimplePacketSender.send(SimplePacketSender.java:55)
at com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:50)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:584)
... 56 common frames omitted
24-06-2019 11:29:19.024 [SimpleAsyncTaskExecutor-1] WARN o.s.j.support.SQLErrorCodesFactory.getErrorCodes:220 - Error while extracting database name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Connection is closed
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:331)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:356)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:214)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:136)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:99)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:122)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1442)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:632)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy84.getMetaData(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.getMetaData(ProxyConnection.java:354)
at com.zaxxer.hikari.pool.HikariProxyConnection.getMetaData(HikariProxyConnection.java)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
... 51 common frames omitted
24-06-2019 11:29:19.028 [SimpleAsyncTaskExecutor-1] ERROR o.s.b.core.step.tasklet.TaskletStep.doInTransaction:463 - JobRepository failure forcing rollback
org.springframework.dao.RecoverableDataAccessException: PreparedStatementCallback; SQL [UPDATE BATCH_STEP_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE STEP_EXECUTION_ID = ?]; The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.; nested exception is com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:100)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1442)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:632)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:861)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:916)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.persistSerializedContext(JdbcExecutionContextDao.java:236)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.updateExecutionContext(JdbcExecutionContextDao.java:163)
at org.springframework.batch.core.repository.support.SimpleJobRepository.updateExecutionContext(SimpleJobRepository.java:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy85.updateExecutionContext(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:452)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:399)
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1094)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1042)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1345)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1027)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$0(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:616)
... 44 common frames omitted
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 67,944,763 milliseconds ago. The last packet sent successfully to the server was 67,944,764 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:593)
at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:654)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:986)
By default, MySQL will close stale connections after 8 hours (See max_timeout parameter). Since your tasklet takes 20h, your connection gets closed, hence the error.
You would need to increase the max_timeout value on your MySQL server. This answer might also help.

Citrus integration tests fail randomly

Following up on How to access variables in a (citrus) static-response-adapter
Test layout:
Citrus -> SUT
SUT <-> MockServerOne
SUT <-> MockServerTwo
Citrus <- SUT
Applying the suggestions from the post above, I now see randomly occuring exceptions.
The test code:
#Test
#CitrusTest
public void testRequestNok() throws Exception {
setDefaultTestVariables();
http(builder -> builder
.client("fooClient")
.send()
.post("/foo/bar")
.payload(new ClassPathResource("foo/foo-sample-request.xml"))
.messageType(MessageType.XML)
.contentType("application/xml")
.accept("application/xml")
.fork(true));
applyBehavior(new MockOneServerBehavior());
applyBehavior(new MockTwoServerBehavior("bar/bar-error-response.xml"));
http(builder -> builder.client("fooClient").receive().response(HttpStatus.BAD_GATEWAY));
}
where the behaviors look like this
public class MockOneServerBehavior extends AbstractTestBehavior {
public void apply() {
http(builder -> builder.server("mockServerOne")
.receive()
.post());
http(builder -> builder.server("mockServerOne")
.send()
.response()
.payload(new ClasspathResource("foo/bar.xml"));
}
}
The test fails (roughly in 5% of all cases) and so far I have seen two different kinds of exceptions:
A ConcurrentModificationException:
2017-03-02 11:00:54,836 main [DEBUG ] citrus.channel.ChannelConsumer - Receiving message from: mockServerOne.inbound
2017-03-02 11:00:54,831 qtp1185618138-20 [DEBUG ] http.servlet.CitrusDispatcherServlet - Could not complete request
org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'mockServerOne.inbound'; nested exception is java.util.ConcurrentModificationException
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:449) ~[spring-integration-core-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373) ~[spring-integration-core-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:150) ~[spring-messaging-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSendAndReceive(GenericMessagingTemplate.java:45) ~[spring-messaging-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.messaging.core.AbstractMessagingTemplate.sendAndReceive(AbstractMessagingTemplate.java:42) ~[spring-messaging-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.integration.core.MessagingTemplate.sendAndReceive(MessagingTemplate.java:97) ~[spring-integration-core-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at com.consol.citrus.channel.ChannelSyncProducer.send(ChannelSyncProducer.java:76) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.channel.ChannelEndpointAdapter.handleMessageInternal(ChannelEndpointAdapter.java:68) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.endpoint.AbstractEndpointAdapter.handleMessage(AbstractEndpointAdapter.java:55) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.http.controller.HttpMessageController.handleRequestInternal(HttpMessageController.java:137) ~[citrus-http-2.7.jar:?]
at com.consol.citrus.http.controller.HttpMessageController.handlePostRequest(HttpMessageController.java:63) ~[citrus-http-2.7.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at com.consol.citrus.http.servlet.GzipServletFilter.doFilterInternal(GzipServletFilter.java:43) [citrus-http-2.7.jar:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at com.consol.citrus.http.servlet.RequestCachingServletFilter.doFilterInternal(RequestCachingServletFilter.java:41) [citrus-http-2.7.jar:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1228) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1130) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112) [jetty-io-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.1.v20170120.jar:9.4.1.v20170120]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.1.v20170120.jar:9.4.1.v20170120]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_112]
at java.util.AbstractMap.toString(AbstractMap.java:554) ~[?:1.8.0_112]
at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886) ~[?:1.8.0_112]
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763) ~[?:1.8.0_112]
at java.util.Formatter.format(Formatter.java:2520) ~[?:1.8.0_112]
at java.util.Formatter.format(Formatter.java:2455) ~[?:1.8.0_112]
at java.lang.String.format(String.java:2940) ~[?:1.8.0_112]
at com.consol.citrus.message.DefaultMessage.toString(DefaultMessage.java:108) ~[citrus-core-2.7.jar:?]
at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_112]
at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_112]
at org.springframework.messaging.support.GenericMessage.toString(GenericMessage.java:111) ~[spring-messaging-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_112]
at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_112]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:430) ~[spring-integration-core-4.3.7.RELEASE.jar:4.3.7.RELEASE]
... 56 more
and a problem with TypeConversion, which leads to a XMLParseException:
2017-03-02 11:11:59,543 main [DEBUG ] citrus.validation.MessageValidatorRegistry - Found 3 message validators for message type: XML
2017-03-02 11:11:59,543 main [DEBUG ] validation.xml.DomXmlMessageValidator - Start message validation ...
2017-03-02 11:11:59,543 main [DEBUG ] validation.xml.DomXmlMessageValidator - Start XML message validation
2017-03-02 11:11:59,544 main [WARN ] citrus.util.TypeConversionUtils - Using object toString representation: Failed to convert value of type 'java.io.ByteArrayInputStream' to required type 'java.lang.String'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.io.ByteArrayInputStream' to required type 'java.lang.String': no matching editors or conversion strategy found
2017-03-02 11:11:59,544 main [WARN ] citrus.util.TypeConversionUtils - Using object toString representation: Failed to convert value of type 'java.io.ByteArrayInputStream' to required type 'java.lang.String'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.io.ByteArrayInputStream' to required type 'java.lang.String': no matching editors or conversion strategy found
[Fatal Error] :1:1: Content is not allowed in prolog.
2017-03-02 11:11:59,553 main [INFO ] consol.citrus.Citrus -
2017-03-02 11:11:59,553 main [ERROR ] consol.citrus.Citrus - TEST FAILED FooTest.testBar <foo.bar.baz> Nested exception is:
com.consol.citrus.exceptions.CitrusRuntimeException: org.w3c.dom.ls.LSException: Content is not allowed in prolog.
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateMessagePayload(DomXmlMessageValidator.java:116) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateMessagePayload(DomXmlMessageValidator.java:68) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.AbstractMessageValidator.validateMessage(AbstractMessageValidator.java:65) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.AbstractMessageValidator.validateMessage(AbstractMessageValidator.java:50) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.actions.ReceiveMessageAction.validateMessage(ReceiveMessageAction.java:216) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.actions.ReceiveMessageAction.doExecute(ReceiveMessageAction.java:133) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:42) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.dsl.actions.DelegatingTestAction.doExecute(DelegatingTestAction.java:54) ~[citrus-java-dsl-2.7.jar:?]
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:42) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.TestCase.executeAction(TestCase.java:229) [citrus-core-2.7.jar:?]
at com.consol.citrus.dsl.runner.DefaultTestRunner.run(DefaultTestRunner.java:198) ~[citrus-java-dsl-2.7.jar:?]
at com.consol.citrus.dsl.runner.DefaultTestRunner.http(DefaultTestRunner.java:526) [citrus-java-dsl-2.7.jar:?]
at com.consol.citrus.dsl.testng.TestNGCitrusTestRunner.http(TestNGCitrusTestRunner.java:360) ~[citrus-java-dsl-2.7.jar:?]
at foo.bar.baz.FooTest.testFoo(FooTest.java:92) ~[integrationTest/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216) ~[spring-core-4.3.6.RELEASE.jar:4.3.6.RELEASE]
at com.consol.citrus.dsl.testng.TestNGCitrusTest.invokeTestMethod(TestNGCitrusTest.java:139) ~[citrus-java-dsl-2.7.jar:?]
at com.consol.citrus.dsl.testng.TestNGCitrusTest.run(TestNGCitrusTest.java:109) [citrus-java-dsl-2.7.jar:?]
at com.consol.citrus.dsl.testng.TestNGCitrusTest.run(TestNGCitrusTest.java:58) [citrus-java-dsl-2.7.jar:?]
at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:209) [testng-6.9.10.jar:?]
at org.testng.internal.Invoker.invokeMethod(Invoker.java:639) [testng-6.9.10.jar:?]
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) [testng-6.9.10.jar:?]
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128) [testng-6.9.10.jar:?]
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) [testng-6.9.10.jar:?]
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) [testng-6.9.10.jar:?]
at org.testng.TestRunner.privateRun(TestRunner.java:782) [testng-6.9.10.jar:?]
at org.testng.TestRunner.run(TestRunner.java:632) [testng-6.9.10.jar:?]
at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) [testng-6.9.10.jar:?]
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361) [testng-6.9.10.jar:?]
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319) [testng-6.9.10.jar:?]
at org.testng.SuiteRunner.run(SuiteRunner.java:268) [testng-6.9.10.jar:?]
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) [testng-6.9.10.jar:?]
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) [testng-6.9.10.jar:?]
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244) [testng-6.9.10.jar:?]
at org.testng.TestNG.runSuitesLocally(TestNG.java:1169) [testng-6.9.10.jar:?]
at org.testng.TestNG.run(TestNG.java:1064) [testng-6.9.10.jar:?]
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72) [testng-plugin.jar:?]
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:127) [testng-plugin.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) [idea_rt.jar:?]
Caused by: org.w3c.dom.ls.LSException: Content is not allowed in prolog.
at org.apache.xerces.parsers.DOMParserImpl.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at com.consol.citrus.util.XMLUtils.parseMessagePayload(XMLUtils.java:356) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateXMLSchema(DomXmlMessageValidator.java:148) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateMessagePayload(DomXmlMessageValidator.java:91) ~[citrus-core-2.7.jar:?]
... 45 more
Caused by: org.apache.xerces.xni.parser.XMLParseException: Content is not allowed in prolog.
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at org.apache.xerces.parsers.DOMParserImpl.parse(Unknown Source) ~[xercesImpl-2.11.0.jar:?]
at com.consol.citrus.util.XMLUtils.parseMessagePayload(XMLUtils.java:356) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateXMLSchema(DomXmlMessageValidator.java:148) ~[citrus-core-2.7.jar:?]
at com.consol.citrus.validation.xml.DomXmlMessageValidator.validateMessagePayload(DomXmlMessageValidator.java:91) ~[citrus-core-2.7.jar:?]
... 45 more
In case of the XMLParseException, I have verified on HTTP level that the response is identical to when the test actually passes.
Both exceptions look like threading issues to me.
I would be grateful for any feedback or alternative approach I might try.
Indeed this has been an issue located in the default message implementation when converting its payload to a String for logging purpose. The issue has been fixed with the mentioned github commit:
https://github.com/christophd/citrus/commit/2c0e70c4db9af102c2544ad60fb9447afb8a7b9c
Mentioned behavior is fixed since Citrus 2.7.2.