Issue with Spring Batch Application - spring-batch

Question

I am reading huge dataset from mysql and sending to one of the java micro-service which in turns dump the data into mongoDB.
But getting the following error:
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:203)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:313)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:144)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy87.run(Unknown Source)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:214)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:186)
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:172)
Checked the core table of Spring batch and it seems the tasks are NOT getting completed and hence may be running out of:
select STATUS,EXIT_CODE from BATCH_STEP_EXECUTION;
+-----------+-----------+
| STATUS | EXIT_CODE |
+-----------+-----------+
| FAILED | FAILED |
| STARTING | EXECUTING |
| COMPLETED | COMPLETED |
| COMPLETED | COMPLETED |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| COMPLETED | COMPLETED |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| COMPLETED | COMPLETED |
| COMPLETED | COMPLETED |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| COMPLETED | COMPLETED |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| COMPLETED | COMPLETED |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
| STARTING | EXECUTING |
Only 10% of tasks were in completed state.
Sample Code:
#Bean
public Step slaveStep() {
StepBuilder stepBuilder = stepBuilderFactory.get("slave-step");
return stepBuilder.<CarDekhoUser, ConnectoUserDto>chunk(20)
.reader(pagingItemReader(null, null))
.processor(dataProcessor())
.writer(mosaicApiWriter())
.faultTolerant()
.retryLimit(3)
.retry(ConnectionPoolTimeoutException.class)
.retry(ConnectTimeoutException.class)
.retry(DeadlockLoserDataAccessException.class)
.build();
}
#Bean
public Step masterStep() {
return stepBuilderFactory.get("master-step")
.partitioner(slaveStep().getName(), dataPartitioner())
.step(slaveStep())
.gridSize(100)
.taskExecutor(taskExecutor())
.build();
}
#Bean
public TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(10);
taskExecutor.setCorePoolSize(10);
taskExecutor.setQueueCapacity(10);
taskExecutor.setThreadNamePrefix("Slave-Task-Executor");
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
}
Partition code
public class CDPartitioner implements Partitioner {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
int min=5000000;
int max = 5050000;
LOG.info(" **** Start Id " + min + " End Id " + max);
int targetSize = (max - min) / gridSize + 1;
LOG.info(" **** Start Id " + min + " End Id " + max + " target size " + targetSize);
Map<String, ExecutionContext> result = new HashMap<>();
int number = 0;
int start = min;
int end = start + targetSize - 1;
while (start <= max) {
ExecutionContext value = new ExecutionContext();
result.put("partition" + number, value);
if (end >= max) {
end = max;
}
value.putInt("minValue", start);
value.putInt("maxValue", end);
start += targetSize;
end += targetSize;
number++;
}
LOG.info(" **** Partition Details " + result);
return result;
}
}