Spring Batch restart from last exception - spring-batch

I would like to restart a batch after it was terminated
I stop a batch when a particular exception throws :
public class IntegrationItemProcessorExceptionHandler implements ExceptionHandler {
private static final Logger LOG = LoggerFactory.getLogger(IntegrationItemProcessorExceptionHandler.class);
#Override
public void handleException(RepeatContext context, Throwable throwable) throws Throwable {
LOG.error("handleException", throwable);
if (throwable instanceof CustomResponseException) {
context.setTerminateOnly();
}
}
}
The input is a json I read with a FlatFileItemReader
#Bean
public ItemReader<UserDto> reader() {
FlatFileItemReader<UserDto> reader = new FlatFileItemReader<MerchantDTO>();
reader.setResource(new ClassPathResource("user.json"));
reader.setRecordSeparatorPolicy(new CustomJsonRecordSeparatorPolicy());
reader.setLineMapper(new CustomLineMapper());
return reader;
}
If CustomResponseException is throws by the ItemProcessor, I stop the batch.
After, I would like to restart the batch, but at the same line I stoped it.
What I need to to to have this behavor ??
And my job config :
#Bean
public Job importUserJob(JobBuilderFactory jobs, Step s1) {
return jobs.get("integrationJob")
.incrementer(new RunIdIncrementer())
.flow(s1)
.end()
.build();
}
#Bean
public Step step1(StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("step1")
.<UserDTO, User>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.listener(new IntegrationItemProcessorListener())
.exceptionHandler(new IntegrationItemProcessorExceptionHandler())
.build();
}

Related

How to quit a tasklet if error in Spring Batch?

I would like to quit a tasklet cleanly if I have an error on it and put and stop the batch without having to resort to a System.exit(1).
Here is my code:
/**
* execution de la tasklet
*
*/
#Override
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource), Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource), dateTraitement, Constantes.NAME_TRT}));
System.exit(1);
} else {
SuiviFluxBO suiviFluxBO = new SuiviFluxBO();
suiviFluxBO.setDateSysteme(Utils.getDateTodayFormatUS());
suiviFluxBO.setDateTrt(Utils.getDateFromStringFormatUS(dateTraitement));
suiviFluxBO.setLibelleTrt("Batch_Java");
suiviFluxBO.setNomficTrt(FilenameUtils.getName(resource));
suiviFluxBO.setNbrrecTrt(Utils.countNbFile(resource));
suiviFluxBO.setNomtabTrt(ConstantesNomsSql.TABLE_STCO_STAU_FIC_ADH);
suiviFluxBO.setNbrlignesTrt(0);
suiviFluxDao.insertSuiviBO(suiviFluxBO);
}
// fin de l'execution
return RepeatStatus.FINISHED;
}
The tasklet implements StepExecutionListener but how to indicate in the IF that contains the error to modify the execution status so that it is in FAILED?
Thank you for your leads.
Based on above requirement , we can build a flow using spring batch FlowBuilder object .
1 . Build a Tasklet which performs required validations and sets ExitStatus based on validation result.
#Component
public class TestTasklet implements StepExecutionListener, Tasklet {
// Any additional properties if required can be added here .
#Override
public void beforeStep(StepExecution stepExecution) {
// Any logic added here will execute before executing step
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// Any logic added here will execute after executing step
return null;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource),
Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource),
dateTraitement, Constantes.NAME_TRT}));
contribution.setExitStatus(ExitStatus.FAILED);
} else {
// any logic goes here .
contribution.setExitStatus(ExitStatus.COMPLETED);
}
return RepeatStatus.FINISHED;
}
}
2 . Below code snippet allows to configure job using flow builder :
#Configuration
public class JobConfigurations {
private StepBuilderFactory stepBuilderFactory;
private JobBuilderFactory jobBuilderFactory;
#Autowired
public JobConfigurations(StepBuilderFactory stepBuilderFactory,
JobBuilderFactory jobBuilderFactory) {
this.stepBuilderFactory = stepBuilderFactory;
this.jobBuilderFactory = jobBuilderFactory;
}
#Bean
public Job job(TestTasklet testTasklet) {
Step validationStep = stepBuilderFactory.get("validationTasklet")
.tasklet(testTasklet).build();
//create another step where you want to perform business logic
//for sake of brevity let us assume it to be businessValidationStep
//Step businessValidationStep = stepBuilderFactory.get("businessvalidationstep")
// .chunk().reader().processor().writer();
return jobBuilderFactory.get("JOB_NAME").incrementer(new RunIdIncrementer())
.start(validationStep)// start your job with validation step
.on(ExitStatus.FAILED.getExitCode()).end()// this will terminate your job cleanly
.from(validationStep)
.on(ExitStatus.COMPLETED.getExitCode())//.to("businessValidationStep")
.to(validationStep).build().build();
}
}
#lasnico37 Hope above code will solve the problem statement .

My spring batch step keeps looping after reaching build

This section reads in the file from our server, processes it, writes it out and archives it.
#Bean
public Step step1() {
log.info("Made if to step1");
System.out.println("Made it to Step 1");
return this.stepBuilderFactory.get("step1")
.<PaymentTransaction, PaymentTransaction>chunk(10)
.reader(paymentTransactionItemReader())
.writer(paymentTransactionItemWriter())
.build();
}
#Bean
public JobExecutionDecider decider() {
System.out.println("Made it to the decider");
return (jobExecution, stepExecution) -> new FlowExecutionStatus("Success"); }
#Bean
public FlowJobBuilder job() {
return jobBuilderFactory.get("BenefitIssuance")
.start(step1())
.next(decider())
.on("Success")
.end()
.build();
}
However when it reaches the build() step at the end, it loops back to the reader
As mentioned in the comments, I don't see why the job() method returns a FlowJobBuilder and not a Job. The following job definition does not loop back on the same step:
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(decider())
.on("Success")
.end()
.build()
.build();
}

How to check if a job is still running or is finished regardless of finalization status

Working in Spring Batch (3) with Spring Boot(1.5) project. I have an end of day job "endOfDayJob" that is asynchronously execute through a web controller, in the controller i am returning the job execution id.
Below the code for configuration class. Highlight here that i am implementing BatchConfigurer interface and creating a async JobLauncer with SimpleAsyncTaskExecutor.
#Configuration
#EnableBatchProcessing
#EnableAsync
public class BatchConfiguration implements BatchConfigurer {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DataSource dataSource;
#Bean
public Job endOfDayJob() throws Exception {
SimpleJobBuilder simpleJobBuilder = jobBuilderFactory.get("endOfDayJob")
.incrementer(new RunIdIncrementer())
.start(init())
.next(updateInventory())
.next(generateSalesReport())
.next(cleanup())
.next(sendReport());
return simpleJobBuilder.build();
}
#Bean
public Step init() {
return stepBuilderFactory.get("initStep").tasklet(initTasklet()).build();
}
#Bean
public Step updateInventory() {
return stepBuilderFactory.get("updateInventoryStep").tasklet(updateInventoryTasklet()).build();
}
#Bean
public Step generateSalesReport() {
return stepBuilderFactory.get("generateSalesReportStep").tasklet(generateSalesReportTasklet()).build();
}
#Bean
public Step cleanup() {
return stepBuilderFactory.get("cleanupStep").tasklet(cleanupTasklet()).build();
}
#Bean
public Step sendReport() {
return stepBuilderFactory.get("sendReportStep").tasklet(sendReportTasklet()).build();
}
#Override
public JobRepository getJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(getTransactionManager());
factory.setIsolationLevelForCreate("ISOLATION_READ_COMMITTED");
factory.setTablePrefix("BATCH_");
return factory.getObject();
}
#Override
public PlatformTransactionManager getTransactionManager() throws Exception {
return new DataSourceTransactionManager(dataSource);
}
#Override
public JobLauncher getJobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
#Override
public JobExplorer getJobExplorer() throws Exception {
JobExplorerFactoryBean jobExplorerFactoryBean = new JobExplorerFactoryBean();
jobExplorerFactoryBean.setDataSource(dataSource);
jobExplorerFactoryBean.setTablePrefix("BATCH_");
jobExplorerFactoryBean.afterPropertiesSet();
return jobExplorerFactoryBean.getObject();
}
}
Here the code for the web controller.
#RestController
#RequestMapping("/api/job")
public class WebController {
private static final Logger logger = LoggerFactory.getLogger(WebController.class);
#Autowired
private DataSource dataSource;
#Autowired
private BatchConfiguration batchConfiguration;
#Autowired
private JobLauncher jobLauncher;
#GetMapping("/endOfDayJob")
private Long kycrBatch(#RequestParam(value = "odate", required = true) String odate) {
logger.info("ExecutingendOfDayJob with odate = {}", odate);
if (odate == null || odate.isEmpty() || odate.trim().isEmpty()) {
return -1L;
}
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder.addString("odate", odate);
long jobExecutionId = -1L;
try {
Job endOfDayJob = this.batchConfiguration.endOfDayJob();
jobParametersBuilder.addDate("runtime", new Date());
jobExecutionId = jobLauncher.run(endOfDayJob, jobParametersBuilder.toJobParameters()).getId();
} catch (Exception e) {
logger.error("Error ocurred executing endOfDayJob with message: {}", e.getMessage());
return -1L;
}
return jobExecutionId;
}
}
Then i want to add new method in the controller to to know if the job ended or not. What is a possible way to check if a job is still running or is already finished regardless of finalization status??
Then i want to add new method in the controller to to know if the job ended or not.
You can inject the JobExplorer in your controller and write something like:
public boolean isRunning(long jobExecutionId) {
JobExecution jobExecution = jobExplorer.getJobExecution(jobExecutionId);
return jobExecution.isRunning();
}

How to use decider in Spring batch?

I'm new to Spring batch. I've created a decider which returns a FlowExecutionStatus as "YES" / "NO". Based on the FlowExecutionStatus, I need to call step2() or step3().
In my below code, step2() is getting called before decider. How to modify the code so that decider will be called and based on the FlowExecutionStatus returned bu the decider, either step2() or step3() should be called. Please help.
#Autowired
private NumberDecider decider;
#Bean
public Job NumberLoaderJob() throws NumberFormatException, IOException {
return jobBuilderFactory.get("numberLoaderJob").start(step1()).listener(new MyNumberJobListener())
.next(decider).on("YES").to(step2())
.from(decider).on("NO").to(step3()).end().build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").tasklet(new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
System.out.println("Exiting step1() execute()");
return RepeatStatus.FINISHED;
}
}).build();
}
/**
* Step 2
*
* #return
* #throws NumberFormatException
* #throws IOException
*/
#Bean
public Step step2() throws NumberFormatException, IOException {
return stepBuilderFactory.get("step2").listener(new MyStepListener())
.<OrderNumber, OrderNumber>chunk(Integer.valueOf(chunkSize)).faultTolerant()
.listener(new MyChunkListener()).reader(new MyItemReader())
.listener(new MyItemReaderListener()).writer(customItemWriter())
.listener(new MyWriteListener()).build();
}
You would need to to set the exit status in step1 so that the decider picks it up and make the decision:
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").tasklet(new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
System.out.println("Exiting step1() execute()");
chunkContext.getStepContext().getStepExecution().setExitStatus(new ExitStatus("YES")); // or NO
return RepeatStatus.FINISHED;
}
}).build();
}
EDIT: I thought that the decider should make the decision based on the exit status of step1, hence the previous sample. So adding a sample to show how to use a decider (after clarification):
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.job.flow.FlowExecutionStatus;
import org.springframework.batch.core.job.flow.JobExecutionDecider;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableBatchProcessing
public class MyJob {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Bean
public Step step1() {
return steps.get("step1")
.tasklet((contribution, chunkContext) -> {
System.out.println("hello");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public JobExecutionDecider decider() {
return (jobExecution, stepExecution) -> new FlowExecutionStatus("YES"); // or NO
}
#Bean
public Step step2() {
return steps.get("step2")
.tasklet((contribution, chunkContext) -> {
System.out.println("world");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Step step3() {
return steps.get("step3")
.tasklet((contribution, chunkContext) -> {
System.out.println("!!");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(decider())
.on("YES").to(step2())
.from(decider()).on("NO").to(step3())
.end()
.build();
}
public static void main(String[] args) throws Exception {
ApplicationContext context = new AnnotationConfigApplicationContext(MyJob.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(Job.class);
jobLauncher.run(job, new JobParameters());
}
}
In this example, step1 is executed first, then the decider. If the decider returns YES, step2 is executed and if returns NO, step3 is executed.
Hope this helps.

Spring Batch: AsyncItemProcessor and AsyncItemWriter

1) I have a large file (> 100k lines) that needs to be processed. I have a lot of business validation and checks against external systems for each line item. The code is being migrated from a legacy app and i just put these business logic into the AsyncitemProcessor, which also persists the data into the DB. Is this a good practise to create/save records in the ItemProcessor (in lieu of ItemWriter) ?
2) Code is ::
#Configuration
#EnableAutoConfiguration
#ComponentScan(basePackages = "com.liquidation.lpid")
#EntityScan(basePackages = "com.liquidation.lpid.entities")
#EnableTransactionManagement
public class SimpleJobConfiguration {
#Autowired
public JobRepository jobRepository;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
#Qualifier("myFtpSessionFactory")
private SessionFactory myFtpSessionFactory;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Bean
public ThreadPoolTaskExecutor lpidItemTaskExecutor() {
ThreadPoolTaskExecutor tExec = new ThreadPoolTaskExecutor();
tExec.setCorePoolSize(10);
tExec.setMaxPoolSize(10);
tExec.setAllowCoreThreadTimeOut(true);
return tExec;
}
#BeforeStep
public void beforeStep(StepExecution stepExecution){
String name = stepExecution.getStepName();
System.out.println("name: " + name);
}
#Bean
public SomeItemWriterListener someItemWriterListener(){
return new SomeItemWriterListener();
};
#Bean
#StepScope
public FlatFileItemReader<FieldSet> lpidItemReader(#Value("#{stepExecutionContext['fileResource']}") String fileResource) {
System.out.println("itemReader called !!!!!!!!!!! for customer data" + fileResource);
FlatFileItemReader<FieldSet> reader = new FlatFileItemReader<FieldSet>();
reader.setResource(new ClassPathResource("/data/stage/"+ fileResource));
reader.setLinesToSkip(1);
DefaultLineMapper<FieldSet> lineMapper = new DefaultLineMapper<FieldSet>();
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
reader.setSkippedLinesCallback(new LineCallbackHandler() {
public void handleLine(String line) {
if (line != null) {
tokenizer.setNames(line.split(","));
}
}
});
lineMapper.setLineTokenizer(tokenizer);
lineMapper.setFieldSetMapper(new PassThroughFieldSetMapper());
lineMapper.afterPropertiesSet();
reader.setLineMapper(lineMapper);
return reader;
}
#Bean
public ItemWriter<FieldSet> lpidItemWriter() {
return new LpidItemWriter();
}
#Autowired
private MultiFileResourcePartitioner multiFileResourcePartitioner;
#Bean
public Step masterStep() {
return stepBuilderFactory.get("masterStep")
.partitioner(slaveStep().getName(), multiFileResourcePartitioner)
.step(slaveStep())
.gridSize(4)
.taskExecutor(lpidItemTaskExecutor())
.build();
}
#Bean
public ItemProcessListener<FieldSet,String> processListener(){
return new LpidItemProcessListener();
}
#SuppressWarnings("unchecked")
#Bean
public Step slaveStep() {
return stepBuilderFactory.get("slaveStep")
.<FieldSet,FieldSet>chunk(5)
.faultTolerant()
.listener(new ChunkListener())
.reader(lpidItemReader(null))
.processor(asyncItemProcessor())
.writer(asyncItemWriter()).listener(someItemWriterListener()).build();
}
#Bean
public AsyncItemWriter<FieldSet> asyncItemWriter(){
AsyncItemWriter<FieldSet> asyncItemProcessor = new AsyncItemWriter<>();
asyncItemProcessor.setDelegate(lpidItemWriter());
try {
asyncItemProcessor.afterPropertiesSet();
} catch (Exception e) {
e.printStackTrace();
}
return asyncItemProcessor;
}
#Bean
public ItemProcessor<FieldSet, FieldSet> processor() {
return new lpidCheckItemProcessor();
}
#Bean
public AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor() {
AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor = new AsyncItemProcessor<FieldSet, FieldSet>();
asyncItemProcessor.setDelegate(processor());
asyncItemProcessor.setTaskExecutor(lpidItemTaskExecutor());
try {
asyncItemProcessor.afterPropertiesSet();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return asyncItemProcessor;
}
#Bean
public Job job() throws Exception {
return jobBuilderFactory.get("job").incrementer(new RunIdIncrementer()).start(masterStep()).build();
}
}
The itemwriter runs before the itemprocessor has completed. My understanding is: for every chunk, the item reader reads the data, item processor will churn through each item, and at the end of the chunk, the item writer gets called (which in my case,it does not do anything since the itemprocessor persists the data). But the itemwriter gets called before the item processor gets completed and my job never completes. What am i doing incorrectly here? (I looked at previous issues around it and the solution was to wrap the writer around the AsyncItemWriter(), which i am doing) .
Thanks
Sundar