I would like to quit a tasklet cleanly if I have an error on it and put and stop the batch without having to resort to a System.exit(1).
Here is my code:
/**
* execution de la tasklet
*
*/
#Override
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource), Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource), dateTraitement, Constantes.NAME_TRT}));
System.exit(1);
} else {
SuiviFluxBO suiviFluxBO = new SuiviFluxBO();
suiviFluxBO.setDateSysteme(Utils.getDateTodayFormatUS());
suiviFluxBO.setDateTrt(Utils.getDateFromStringFormatUS(dateTraitement));
suiviFluxBO.setLibelleTrt("Batch_Java");
suiviFluxBO.setNomficTrt(FilenameUtils.getName(resource));
suiviFluxBO.setNbrrecTrt(Utils.countNbFile(resource));
suiviFluxBO.setNomtabTrt(ConstantesNomsSql.TABLE_STCO_STAU_FIC_ADH);
suiviFluxBO.setNbrlignesTrt(0);
suiviFluxDao.insertSuiviBO(suiviFluxBO);
}
// fin de l'execution
return RepeatStatus.FINISHED;
}
The tasklet implements StepExecutionListener but how to indicate in the IF that contains the error to modify the execution status so that it is in FAILED?
Thank you for your leads.
Based on above requirement , we can build a flow using spring batch FlowBuilder object .
1 . Build a Tasklet which performs required validations and sets ExitStatus based on validation result.
#Component
public class TestTasklet implements StepExecutionListener, Tasklet {
// Any additional properties if required can be added here .
#Override
public void beforeStep(StepExecution stepExecution) {
// Any logic added here will execute before executing step
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// Any logic added here will execute after executing step
return null;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource),
Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource),
dateTraitement, Constantes.NAME_TRT}));
contribution.setExitStatus(ExitStatus.FAILED);
} else {
// any logic goes here .
contribution.setExitStatus(ExitStatus.COMPLETED);
}
return RepeatStatus.FINISHED;
}
}
2 . Below code snippet allows to configure job using flow builder :
#Configuration
public class JobConfigurations {
private StepBuilderFactory stepBuilderFactory;
private JobBuilderFactory jobBuilderFactory;
#Autowired
public JobConfigurations(StepBuilderFactory stepBuilderFactory,
JobBuilderFactory jobBuilderFactory) {
this.stepBuilderFactory = stepBuilderFactory;
this.jobBuilderFactory = jobBuilderFactory;
}
#Bean
public Job job(TestTasklet testTasklet) {
Step validationStep = stepBuilderFactory.get("validationTasklet")
.tasklet(testTasklet).build();
//create another step where you want to perform business logic
//for sake of brevity let us assume it to be businessValidationStep
//Step businessValidationStep = stepBuilderFactory.get("businessvalidationstep")
// .chunk().reader().processor().writer();
return jobBuilderFactory.get("JOB_NAME").incrementer(new RunIdIncrementer())
.start(validationStep)// start your job with validation step
.on(ExitStatus.FAILED.getExitCode()).end()// this will terminate your job cleanly
.from(validationStep)
.on(ExitStatus.COMPLETED.getExitCode())//.to("businessValidationStep")
.to(validationStep).build().build();
}
}
#lasnico37 Hope above code will solve the problem statement .
Related
Working in Spring Batch (3) with Spring Boot(1.5) project. I have an end of day job "endOfDayJob" that is asynchronously execute through a web controller, in the controller i am returning the job execution id.
Below the code for configuration class. Highlight here that i am implementing BatchConfigurer interface and creating a async JobLauncer with SimpleAsyncTaskExecutor.
#Configuration
#EnableBatchProcessing
#EnableAsync
public class BatchConfiguration implements BatchConfigurer {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DataSource dataSource;
#Bean
public Job endOfDayJob() throws Exception {
SimpleJobBuilder simpleJobBuilder = jobBuilderFactory.get("endOfDayJob")
.incrementer(new RunIdIncrementer())
.start(init())
.next(updateInventory())
.next(generateSalesReport())
.next(cleanup())
.next(sendReport());
return simpleJobBuilder.build();
}
#Bean
public Step init() {
return stepBuilderFactory.get("initStep").tasklet(initTasklet()).build();
}
#Bean
public Step updateInventory() {
return stepBuilderFactory.get("updateInventoryStep").tasklet(updateInventoryTasklet()).build();
}
#Bean
public Step generateSalesReport() {
return stepBuilderFactory.get("generateSalesReportStep").tasklet(generateSalesReportTasklet()).build();
}
#Bean
public Step cleanup() {
return stepBuilderFactory.get("cleanupStep").tasklet(cleanupTasklet()).build();
}
#Bean
public Step sendReport() {
return stepBuilderFactory.get("sendReportStep").tasklet(sendReportTasklet()).build();
}
#Override
public JobRepository getJobRepository() throws Exception {
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(getTransactionManager());
factory.setIsolationLevelForCreate("ISOLATION_READ_COMMITTED");
factory.setTablePrefix("BATCH_");
return factory.getObject();
}
#Override
public PlatformTransactionManager getTransactionManager() throws Exception {
return new DataSourceTransactionManager(dataSource);
}
#Override
public JobLauncher getJobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
#Override
public JobExplorer getJobExplorer() throws Exception {
JobExplorerFactoryBean jobExplorerFactoryBean = new JobExplorerFactoryBean();
jobExplorerFactoryBean.setDataSource(dataSource);
jobExplorerFactoryBean.setTablePrefix("BATCH_");
jobExplorerFactoryBean.afterPropertiesSet();
return jobExplorerFactoryBean.getObject();
}
}
Here the code for the web controller.
#RestController
#RequestMapping("/api/job")
public class WebController {
private static final Logger logger = LoggerFactory.getLogger(WebController.class);
#Autowired
private DataSource dataSource;
#Autowired
private BatchConfiguration batchConfiguration;
#Autowired
private JobLauncher jobLauncher;
#GetMapping("/endOfDayJob")
private Long kycrBatch(#RequestParam(value = "odate", required = true) String odate) {
logger.info("ExecutingendOfDayJob with odate = {}", odate);
if (odate == null || odate.isEmpty() || odate.trim().isEmpty()) {
return -1L;
}
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder.addString("odate", odate);
long jobExecutionId = -1L;
try {
Job endOfDayJob = this.batchConfiguration.endOfDayJob();
jobParametersBuilder.addDate("runtime", new Date());
jobExecutionId = jobLauncher.run(endOfDayJob, jobParametersBuilder.toJobParameters()).getId();
} catch (Exception e) {
logger.error("Error ocurred executing endOfDayJob with message: {}", e.getMessage());
return -1L;
}
return jobExecutionId;
}
}
Then i want to add new method in the controller to to know if the job ended or not. What is a possible way to check if a job is still running or is already finished regardless of finalization status??
Then i want to add new method in the controller to to know if the job ended or not.
You can inject the JobExplorer in your controller and write something like:
public boolean isRunning(long jobExecutionId) {
JobExecution jobExecution = jobExplorer.getJobExecution(jobExecutionId);
return jobExecution.isRunning();
}
How would one detect when the Job has been signaled to stop from within the FlatFileItemReader bufferedFileReader that was created using the configured custom BufferedFileReaderFactory bean. This custom bufferedFileReader tails the file and waits for more input indefinitely so the Job STOPPING status isn't being detected as the code is blocked outside the Spring Batch framework.
I can do this with a Step1->Tasklet->Step1 flow loop that does a Thread.sleep in the Tasklet but the nature of this constantly growing file means I'll be hitting EOF every couple of seconds and generating a huge amount of StepExecution rows in the database.
public class TailingBufferedReaderFactory implements BufferedReaderFactory {
#Override
public BufferedReader create(Resource resource, String encoding) throws IOException {
return new TailingBufferedReader(new InputStreamReader(resource.getInputStream(), encoding));
}
}
public class TailingBufferedReader extends BufferedReader implements JobExecutionListener {
private JobExecution jobExecution;
public TailingBufferedReader(Reader in) {
super(in);
}
#Override
public String readLine() throws IOException {
while (!jobExecution.isStopping()) { //The elusive Job Execution status check
var line = super.readLine();
if (line == null) {
Thread.sleep(waitDurationMillis);
continue;
}
return line;
}
return null;
}
// Ideally something like this configured on the Job
#Override
public void beforeJob(JobExecution jobExecution) {
this.jobExecution = jobExecution;
}
#Override
public void afterJob(JobExecution jobExecution) {}
}
I am trying to retrieve a spring batch ExecutionContext from a SkipListener.
Here is what I have attempted (I rely on annotations instead of interfaces in order to implement my listeners):
import com.xxxx.domain.UserAccount;
import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.core.annotation.OnSkipInWrite;
import org.springframework.mail.MailSendException;
import org.springframework.stereotype.Component;
#Slf4j
#Component
public class MailSkipListener {
private StepExecution stepExecution;
#BeforeStep
public void saveStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#OnSkipInWrite
public void logSkippedEmail(UserAccount userAccount, Throwable t) {
if (t instanceof MailSendException) {
MailSendException e = (MailSendException) t;
log.warn("FailedMessages: " + e.getFailedMessages());
}
}
}
However, the logSkippedEmail method is never executed when a MailSendException is raised. When I remove the saveStepExecution method, the logSkippedEmail is again executed in case of a MailSendException.
I register my MailSkipListener as follows:
#Bean
public Step messagesDigestMailingStep(EntityManagerFactory entityManagerFactory) {
return stepBuilderFactory
.get("messagesDigestMailingStep")
.<UserAccount, UserAccount>chunk(5)
...
.writer(itemWriter)
.listener(mailSkipListener)//Here
.build();
}
What I am trying to achieve here is retrieving an ExecutionContext from my SkipListener. How can this be achieved? It seems there's no way to autowire theExecutionContext.
This is quite an old question, but I just struggled with this too.
I ended up registering the skiplistener twice in order for it to work, once as a StepExecutionListener and another as a SkipListener.
It sucks, but it seems to work:
#Bean
public Step messagesDigestMailingStep(EntityManagerFactory entityManagerFactory) {
return stepBuilderFactory
.get("messagesDigestMailingStep")
.<UserAccount, UserAccount>chunk(5)
...
.writer(itemWriter)
.listener((StepExecutionListener) mailSkipListener) // <--- 1
.listener((SkipListener) mailSkipListener) // <--- 2
.build();
}
I know this is an old Question, but I had to deal with this myself, and have put together the following implementation, wherein I made the SkipListener also implement the StepExecutionListener, and adding the same class as both the SkipListener and StepExecutionListener.
#Component
public class PersonImportListener implements SkipListener<Person, Person>, StepExecutionListener {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
private StepExecution stepExecution;
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return ExitStatus.COMPLETED;
}
#Override
public void onSkipInRead(Throwable throwable) {
logger.warn("Line skipped on read", throwable);
}
#Override
public void onSkipInWrite(Person person, Throwable throwable) {
logger.warn("Bean skipped on write", throwable);
logger.warn("Execution Context" + stepExecution);
}
#Override
public void onSkipInProcess(Person person, Throwable throwable) {
logger.warn("Bean skipped on process", throwable);
}
}
And use this class as a listener for StepExecutionListener and also SkipListener.
#Bean
public Step step1(JdbcBatchItemWriter<Person> writer) {
PersonImportListener listener = new PersonImportListener();
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.faultTolerant()
.skipLimit(10)
.skip(DataIntegrityViolationException.class)
.listener((StepExecutionListener) listener)
.listener((SkipListener) listener)
.processor(processor())
.writer(writer)
.build();
}
You can implement StepExecutionListener on your MailSkipListener to save the context in your stepExecution during the beforeStep() method :
public class MailSkipListener implements StepExecutionListener {
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
1) I have a large file (> 100k lines) that needs to be processed. I have a lot of business validation and checks against external systems for each line item. The code is being migrated from a legacy app and i just put these business logic into the AsyncitemProcessor, which also persists the data into the DB. Is this a good practise to create/save records in the ItemProcessor (in lieu of ItemWriter) ?
2) Code is ::
#Configuration
#EnableAutoConfiguration
#ComponentScan(basePackages = "com.liquidation.lpid")
#EntityScan(basePackages = "com.liquidation.lpid.entities")
#EnableTransactionManagement
public class SimpleJobConfiguration {
#Autowired
public JobRepository jobRepository;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
#Qualifier("myFtpSessionFactory")
private SessionFactory myFtpSessionFactory;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Bean
public ThreadPoolTaskExecutor lpidItemTaskExecutor() {
ThreadPoolTaskExecutor tExec = new ThreadPoolTaskExecutor();
tExec.setCorePoolSize(10);
tExec.setMaxPoolSize(10);
tExec.setAllowCoreThreadTimeOut(true);
return tExec;
}
#BeforeStep
public void beforeStep(StepExecution stepExecution){
String name = stepExecution.getStepName();
System.out.println("name: " + name);
}
#Bean
public SomeItemWriterListener someItemWriterListener(){
return new SomeItemWriterListener();
};
#Bean
#StepScope
public FlatFileItemReader<FieldSet> lpidItemReader(#Value("#{stepExecutionContext['fileResource']}") String fileResource) {
System.out.println("itemReader called !!!!!!!!!!! for customer data" + fileResource);
FlatFileItemReader<FieldSet> reader = new FlatFileItemReader<FieldSet>();
reader.setResource(new ClassPathResource("/data/stage/"+ fileResource));
reader.setLinesToSkip(1);
DefaultLineMapper<FieldSet> lineMapper = new DefaultLineMapper<FieldSet>();
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
reader.setSkippedLinesCallback(new LineCallbackHandler() {
public void handleLine(String line) {
if (line != null) {
tokenizer.setNames(line.split(","));
}
}
});
lineMapper.setLineTokenizer(tokenizer);
lineMapper.setFieldSetMapper(new PassThroughFieldSetMapper());
lineMapper.afterPropertiesSet();
reader.setLineMapper(lineMapper);
return reader;
}
#Bean
public ItemWriter<FieldSet> lpidItemWriter() {
return new LpidItemWriter();
}
#Autowired
private MultiFileResourcePartitioner multiFileResourcePartitioner;
#Bean
public Step masterStep() {
return stepBuilderFactory.get("masterStep")
.partitioner(slaveStep().getName(), multiFileResourcePartitioner)
.step(slaveStep())
.gridSize(4)
.taskExecutor(lpidItemTaskExecutor())
.build();
}
#Bean
public ItemProcessListener<FieldSet,String> processListener(){
return new LpidItemProcessListener();
}
#SuppressWarnings("unchecked")
#Bean
public Step slaveStep() {
return stepBuilderFactory.get("slaveStep")
.<FieldSet,FieldSet>chunk(5)
.faultTolerant()
.listener(new ChunkListener())
.reader(lpidItemReader(null))
.processor(asyncItemProcessor())
.writer(asyncItemWriter()).listener(someItemWriterListener()).build();
}
#Bean
public AsyncItemWriter<FieldSet> asyncItemWriter(){
AsyncItemWriter<FieldSet> asyncItemProcessor = new AsyncItemWriter<>();
asyncItemProcessor.setDelegate(lpidItemWriter());
try {
asyncItemProcessor.afterPropertiesSet();
} catch (Exception e) {
e.printStackTrace();
}
return asyncItemProcessor;
}
#Bean
public ItemProcessor<FieldSet, FieldSet> processor() {
return new lpidCheckItemProcessor();
}
#Bean
public AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor() {
AsyncItemProcessor<FieldSet, FieldSet> asyncItemProcessor = new AsyncItemProcessor<FieldSet, FieldSet>();
asyncItemProcessor.setDelegate(processor());
asyncItemProcessor.setTaskExecutor(lpidItemTaskExecutor());
try {
asyncItemProcessor.afterPropertiesSet();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return asyncItemProcessor;
}
#Bean
public Job job() throws Exception {
return jobBuilderFactory.get("job").incrementer(new RunIdIncrementer()).start(masterStep()).build();
}
}
The itemwriter runs before the itemprocessor has completed. My understanding is: for every chunk, the item reader reads the data, item processor will churn through each item, and at the end of the chunk, the item writer gets called (which in my case,it does not do anything since the itemprocessor persists the data). But the itemwriter gets called before the item processor gets completed and my job never completes. What am i doing incorrectly here? (I looked at previous issues around it and the solution was to wrap the writer around the AsyncItemWriter(), which i am doing) .
Thanks
Sundar
I would like to restart a batch after it was terminated
I stop a batch when a particular exception throws :
public class IntegrationItemProcessorExceptionHandler implements ExceptionHandler {
private static final Logger LOG = LoggerFactory.getLogger(IntegrationItemProcessorExceptionHandler.class);
#Override
public void handleException(RepeatContext context, Throwable throwable) throws Throwable {
LOG.error("handleException", throwable);
if (throwable instanceof CustomResponseException) {
context.setTerminateOnly();
}
}
}
The input is a json I read with a FlatFileItemReader
#Bean
public ItemReader<UserDto> reader() {
FlatFileItemReader<UserDto> reader = new FlatFileItemReader<MerchantDTO>();
reader.setResource(new ClassPathResource("user.json"));
reader.setRecordSeparatorPolicy(new CustomJsonRecordSeparatorPolicy());
reader.setLineMapper(new CustomLineMapper());
return reader;
}
If CustomResponseException is throws by the ItemProcessor, I stop the batch.
After, I would like to restart the batch, but at the same line I stoped it.
What I need to to to have this behavor ??
And my job config :
#Bean
public Job importUserJob(JobBuilderFactory jobs, Step s1) {
return jobs.get("integrationJob")
.incrementer(new RunIdIncrementer())
.flow(s1)
.end()
.build();
}
#Bean
public Step step1(StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("step1")
.<UserDTO, User>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.listener(new IntegrationItemProcessorListener())
.exceptionHandler(new IntegrationItemProcessorExceptionHandler())
.build();
}