spring batch conditional if step - spring-batch

The Spring Batch documentation does a pretty good job explaining how to correctly define a job with steps that pattern this:
call Step 1
if (some condition determined by a Decider) {
call Step 2
} ELSE {
call Step 3
}
However, I'm trying to achieve THIS in Java config:
call Step 1
if (some condition determined by a decider) {
call step 2
}
call step 3
call step 4
.. so on and so forth ..
and I think I might be misreading the API because I can't seem to get the conditional IF logic to behave as one would expect -- all that happens is when the decider evaluates to True it runs step 2 but never runs Step 3 .. and of course when the decider evaluates to false it runs only step 3. Does anybody have any examples for how to do a conditional step of this sort in Java config?

you should define your decider, for that you have two choices.
A class that implements JobExecutionDecider or to define a bean as shown in the above example :
#Bean
public JobExecutionDecider decider() {
return (JobExecution jobExecution, StepExecution stepExecution) -> {
// write code here
boolean someCondition = true;
return someCondition ? new FlowExecutionStatus("COMPLETED") : new FlowExecutionStatus("FAILED");
};
}
after that, you can define your flow using this decider like this :
#Bean
public Job job() {
return jobBuilderFactory.get("job").incrementer(new RunIdIncrementer()).start(step1()).next(decider())
.on("COMPLETED").to(step2()).from(decider()).on("FAILED").to(step3()).end().build();
}
the whole definition of your job with steps should look like this :
#Configuration
public class MJob {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Job job() {
return jobBuilderFactory.get("job").incrementer(new RunIdIncrementer()).start(step1()).next(decider())
.on("COMPLETED").to(step2()).from(decider()).on("FAILED").to(step3()).end().build();
}
#Bean
public JobExecutionDecider decider() {
return (JobExecution jobExecution, StepExecution stepExecution) -> {
// write code here
boolean someCondition = true;
return someCondition ? new FlowExecutionStatus("COMPLETED") : new FlowExecutionStatus("FAILED");
};
}
public Step step1() {
return stepBuilderFactory.get("step1").tasklet(new Step1()).build();
}
public Step step2() {
return stepBuilderFactory.get("step2").tasklet(new Step2()).build();
}
public Step step3() {
return stepBuilderFactory.get("step3").tasklet(new Step3()).build();
}
}
Is this answer your question?

Related

Spring Batch - abstract Step definition in Java configuration?

My Spring Batch Job configuration has 5 steps, all of which are identical except for the reader. Is there a way I can abstract out all of the other parts of the step into a "parent" step, so that I don't need to repeat everything? I know this can be done in XML, but I can't figure out the java equivalent.
Here's one of the steps:
public Step quarterlyStep(FileIngestErrorListener listener, ItemReader<DistributionItem> quarterlyReader) {
return stepBuilderFactory.get("quarterlyStep")
.<DistributionItem,DistributionItem>chunk(10)
.reader(quarterlyReader) // The only thing that changes among 5 different steps
.listener(listener.asReadListener())
.processor(processor())
.listener(listener.asProcessListener())
.writer(writer())
.listener(listener.asWriteListener())
.faultTolerant()
.skip(ValidationException.class)
.skip(ExcelFileParseException.class)
.build();
}
Here's the definition of one of the readers:
#Bean
#JobScope
public PoiItemReader<DistributionItem> yearEndReader(#Value("#{jobExecutionContext['filename']}") String filename) {
PoiItemReader<PortfolioFundsDistributionItem> reader = new PoiItemReader<>();
reader.setLinesToSkip(1);
reader.setRowMapper(yearEndRowMapper());
reader.setResource(new FileSystemResource(filename));
return reader;
}
You can do something like:
private StepBuilderFactory stepBuilderFactory;
private SimpleStepBuilder<Integer, Integer> createBaseStep(String stepName) {
return stepBuilderFactory.get(stepName)
.<Integer, Integer>chunk(5)
.processor(itemProcessor())
.writer(itemWriter());
}
#Bean
public Step step1(ItemReader<Integer> itemReader) {
return createBaseStep("step1")
.reader(itemReader)
.build();
}
#Bean
public Step step2(ItemReader<Integer> itemReader) {
return createBaseStep("step2")
.reader(itemReader)
.build();
}

How to quit a tasklet if error in Spring Batch?

I would like to quit a tasklet cleanly if I have an error on it and put and stop the batch without having to resort to a System.exit(1).
Here is my code:
/**
* execution de la tasklet
*
*/
#Override
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource), Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource), dateTraitement, Constantes.NAME_TRT}));
System.exit(1);
} else {
SuiviFluxBO suiviFluxBO = new SuiviFluxBO();
suiviFluxBO.setDateSysteme(Utils.getDateTodayFormatUS());
suiviFluxBO.setDateTrt(Utils.getDateFromStringFormatUS(dateTraitement));
suiviFluxBO.setLibelleTrt("Batch_Java");
suiviFluxBO.setNomficTrt(FilenameUtils.getName(resource));
suiviFluxBO.setNbrrecTrt(Utils.countNbFile(resource));
suiviFluxBO.setNomtabTrt(ConstantesNomsSql.TABLE_STCO_STAU_FIC_ADH);
suiviFluxBO.setNbrlignesTrt(0);
suiviFluxDao.insertSuiviBO(suiviFluxBO);
}
// fin de l'execution
return RepeatStatus.FINISHED;
}
The tasklet implements StepExecutionListener but how to indicate in the IF that contains the error to modify the execution status so that it is in FAILED?
Thank you for your leads.
Based on above requirement , we can build a flow using spring batch FlowBuilder object .
1 . Build a Tasklet which performs required validations and sets ExitStatus based on validation result.
#Component
public class TestTasklet implements StepExecutionListener, Tasklet {
// Any additional properties if required can be added here .
#Override
public void beforeStep(StepExecution stepExecution) {
// Any logic added here will execute before executing step
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// Any logic added here will execute after executing step
return null;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource),
Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource),
dateTraitement, Constantes.NAME_TRT}));
contribution.setExitStatus(ExitStatus.FAILED);
} else {
// any logic goes here .
contribution.setExitStatus(ExitStatus.COMPLETED);
}
return RepeatStatus.FINISHED;
}
}
2 . Below code snippet allows to configure job using flow builder :
#Configuration
public class JobConfigurations {
private StepBuilderFactory stepBuilderFactory;
private JobBuilderFactory jobBuilderFactory;
#Autowired
public JobConfigurations(StepBuilderFactory stepBuilderFactory,
JobBuilderFactory jobBuilderFactory) {
this.stepBuilderFactory = stepBuilderFactory;
this.jobBuilderFactory = jobBuilderFactory;
}
#Bean
public Job job(TestTasklet testTasklet) {
Step validationStep = stepBuilderFactory.get("validationTasklet")
.tasklet(testTasklet).build();
//create another step where you want to perform business logic
//for sake of brevity let us assume it to be businessValidationStep
//Step businessValidationStep = stepBuilderFactory.get("businessvalidationstep")
// .chunk().reader().processor().writer();
return jobBuilderFactory.get("JOB_NAME").incrementer(new RunIdIncrementer())
.start(validationStep)// start your job with validation step
.on(ExitStatus.FAILED.getExitCode()).end()// this will terminate your job cleanly
.from(validationStep)
.on(ExitStatus.COMPLETED.getExitCode())//.to("businessValidationStep")
.to(validationStep).build().build();
}
}
#lasnico37 Hope above code will solve the problem statement .

Spring Batch, JpaRepository and Rollback

I have a Spring Batch application(Spring Boot 2.3.5.RELEASE) that uses a JpaRepository to insert some custom log messages into a database as Spring Batch is processing. This is separate from the out of the box Spring Batch tables. Seems that when I throw an exception from my ItemProcessorAdapter, it is caught by the ItemProcessListener onProcessError() method. In this method I am performing a JpaRepository save() and flush(). No errors are logged, but once I leave this method the JpaRepository does a rollback.
Is this normal behavior? How can I get around it?
When using JpaRepository, is there a way to set a #Transactional(noRollbackFor = {xxxException.class})? I tried this and it seemed to have no effect.
Sample code snippet is below.
#Configuration
public class BatchJobConfiguration {
//Omitted for clarity....
#Bean
#StepScope
public CompositeItemProcessor<Decision,Decision> itemProcessor() {
CompositeItemProcessor<Decision,Decision> itemProcessor = new CompositeItemProcessor<>();
itemProcessor.setDelegates(Arrays.asList(
decisionValidatingItemProcessor(),
myItemProcessor(null)
));
return itemProcessor;
} // end itemProcessor()
#Bean
public BeanValidatingItemProcessor<Decision> decisionValidatingItemProcessor() {
BeanValidatingItemProcessor<Decision> beanValidatingItemProcessor = new BeanValidatingItemProcessor<>();
beanValidatingItemProcessor.setFilter(true);
return beanValidatingItemProcessor;
} // end decisionValidatingItemProcessor()
#Bean
public ItemProcessorAdapter<Decision,Decision> myItemProcessor(DecisionProcessingService service) {
ItemProcessorAdapter<Decision,Decision> adapter = new ItemProcessorAdapter<>();
adapter.setTargetObject(service);
adapter.setTargetMethod("processDecision");
return adapter;
}
#Bean
#StepScope
public DecisionItemProcessListener decisionItemProcessListener() {
return new DecisionItemProcessListener(mpJpaRepository);
}
}
#Service
public class DecisionProcessingService {
public Decision processDecision(Decision decision) throws BatchException {
....
throw new BatchException("An error occurred");
}
}
public class DecisionItemProcessListener implements ItemProcessListener<Decision,Decision> {
private MyJpaRepository mpJpaRepository;
public DecisionItemProcessListener(MyJpaRepository mpJpaRepository) {
this.mpJpaRepository = mpJpaRepository;
}
....
#Override
public void onProcessError(Decision decision, Exception e) {
MyEntityObject obj = MyEntityObject.builder()
.msg(e.getMessage())
.build();
mpJpaRepository.save(obj);
mpJpaRepository.flush();
// after this, the insert above is rolled back.
} // end onProcessError()
}
The callback you are using here ItemProcessListener#onProcessError is called with-in a transaction (driven by Spring Batch) that is going to be rolled-back due to the exception thrown by the item processor.
If you want to save data in that method, you need to use a new transaction (use the REQUIRES_NEW propagation).
EDIT: I shared a minimal complete example here: https://github.com/benas/spring-batch-lab/tree/master/issues/so64913980.

How to pass parameters from a job to a FlowStep in spring-batch?

In a particular spring-batch job, I am using a Flow (as it is a reusable sequence of steps) as a Step.
I have to pass a set of parameters to the Flow.
How do I do that?
My job definition is as follows:
#Component
public class MyJob {
#Bean
public Job myJob(#Qualifier("myFlowStep") Flow myFlow) {
return jobBuilderFactory.get("myJob").incrementer(new RunIdIncrementer())
.start(someFirstStep())
.next(myFlowStep(myFlow)).
.build();
}
...
#Bean
public Step myFlowStep(Flow myFlow) {
// Need to pass parameters to the flow
FlowStepBuilder flowStepBuilder = stepBuilderFactory.get("myFlowStep").flow(myFlow);
return flowStepBuilder.build();
}
...
}
You can make the flow bean definition step scoped and use late-binding of job parameters:
#Bean
#StepScope
public Flow myFlow(#Value("#{jobParameters['name']}") String name) {
// use job parameter name here
return null;
}
#Bean
public Step myFlowStep(Flow myFlow) {
// Need to pass parameters to the flow
FlowStepBuilder flowStepBuilder = stepBuilderFactory.get("myFlowStep").flow(myFlow);
return flowStepBuilder.build();
}

Spring Batch Job Execution Listener - Access details of sub-jobs defined by JobStep

I have a job(parent job) of jobs(child). In the jobexecution listener of parent job, I want to access all the details(job name, step names, read/write count etc.) of each child jobs. How can I achieve this?
Below is example code. In afterJob method of parentJobExecutionListener, I want to send an email(already have code for it), containing complete details of JOB1 and JOB2. Below code in afterJob method prints details of JOB-STEP1 and JOB-STEP2. How can I access details of child jobs(JOB1 and JOB2) and their steps?
public class ParentJobConfig {
#Bean
public Job job1() {
Job job = jobBuilderFactory.get("JOB1")
.incrementer(new RunIdIncrementer())
.start(step11())
.next(step12())
.build();
return job;
}
#Bean
public Job job2() {
Job job = jobBuilderFactory.get("JOB2")
.incrementer(new RunIdIncrementer())
.start(step21())
.next(step22())
.build();
return job;
}
#Bean
public Step jobStep1() {
Step step = stepBuilderFactory.get("JOB-STEP1")
.job(job1())
.parametersExtractor(new DefaultJobParametersExtractor())
.build();
return step;
}
#Bean
public Step jobStep2() {
Step step = stepBuilderFactory.get("JOB-STEP2")
.job(job2())
.parametersExtractor(new DefaultJobParametersExtractor())
.build();
return step;
}
#Bean
public Job parentJob() {
Job job = jobBuilderFactory.get("PARENT-JOB")
.incrementer(new RunIdIncrementer())
.start(jobStep1())
.next(jobStep2())
.listener(parentJobExecutionListener())
.build();
return job;
}
#Bean
public JobExecutionListener parentJobExecutionListener() {
return new ParentJobExecutionListener ();
}
}
public class ParentJobExecutionListener extends JobExecutionListenerSupport {
#Override
public void afterJob(JobExecution jobExecution) {
List<StepExecution> stepExecutions = (List<StepExecution>) jobExecution.getStepExecutions();
Collection<StepExecution> stepExecutions = jobExecution.getStepExecutions();
for (StepExecution stepExecution : stepExecutions) {
System.out.println("Step Name: " + stepExecution.getStepName());
System.out.println("Read Count: " + stepExecution.getReadCount());
System.out.println("Skip Count: " + stepExecution.getSkipCount());
System.out.println("Write Count: " + stepExecution.getWriteCount());
}
}
}
I don't see an obvious way to get access to the child job executions created here: https://github.com/spring-projects/spring-batch/blob/d8fc58338d3b059b67b5f777adc132d2564d7402/spring-batch-core/src/main/java/org/springframework/batch/core/step/job/JobStep.java#L119 (Only the status is reflected on the step execution).
What you can do is use a job explorer in your listener and query the repository for child job executions.