Detecting when the Job is STOPPING in a FlatFileItemReader custom bufferedReader instance - spring-batch

How would one detect when the Job has been signaled to stop from within the FlatFileItemReader bufferedFileReader that was created using the configured custom BufferedFileReaderFactory bean. This custom bufferedFileReader tails the file and waits for more input indefinitely so the Job STOPPING status isn't being detected as the code is blocked outside the Spring Batch framework.
I can do this with a Step1->Tasklet->Step1 flow loop that does a Thread.sleep in the Tasklet but the nature of this constantly growing file means I'll be hitting EOF every couple of seconds and generating a huge amount of StepExecution rows in the database.
public class TailingBufferedReaderFactory implements BufferedReaderFactory {
#Override
public BufferedReader create(Resource resource, String encoding) throws IOException {
return new TailingBufferedReader(new InputStreamReader(resource.getInputStream(), encoding));
}
}
public class TailingBufferedReader extends BufferedReader implements JobExecutionListener {
private JobExecution jobExecution;
public TailingBufferedReader(Reader in) {
super(in);
}
#Override
public String readLine() throws IOException {
while (!jobExecution.isStopping()) { //The elusive Job Execution status check
var line = super.readLine();
if (line == null) {
Thread.sleep(waitDurationMillis);
continue;
}
return line;
}
return null;
}
// Ideally something like this configured on the Job
#Override
public void beforeJob(JobExecution jobExecution) {
this.jobExecution = jobExecution;
}
#Override
public void afterJob(JobExecution jobExecution) {}
}

Related

How to quit a tasklet if error in Spring Batch?

I would like to quit a tasklet cleanly if I have an error on it and put and stop the batch without having to resort to a System.exit(1).
Here is my code:
/**
* execution de la tasklet
*
*/
#Override
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource), Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource), dateTraitement, Constantes.NAME_TRT}));
System.exit(1);
} else {
SuiviFluxBO suiviFluxBO = new SuiviFluxBO();
suiviFluxBO.setDateSysteme(Utils.getDateTodayFormatUS());
suiviFluxBO.setDateTrt(Utils.getDateFromStringFormatUS(dateTraitement));
suiviFluxBO.setLibelleTrt("Batch_Java");
suiviFluxBO.setNomficTrt(FilenameUtils.getName(resource));
suiviFluxBO.setNbrrecTrt(Utils.countNbFile(resource));
suiviFluxBO.setNomtabTrt(ConstantesNomsSql.TABLE_STCO_STAU_FIC_ADH);
suiviFluxBO.setNbrlignesTrt(0);
suiviFluxDao.insertSuiviBO(suiviFluxBO);
}
// fin de l'execution
return RepeatStatus.FINISHED;
}
The tasklet implements StepExecutionListener but how to indicate in the IF that contains the error to modify the execution status so that it is in FAILED?
Thank you for your leads.
Based on above requirement , we can build a flow using spring batch FlowBuilder object .
1 . Build a Tasklet which performs required validations and sets ExitStatus based on validation result.
#Component
public class TestTasklet implements StepExecutionListener, Tasklet {
// Any additional properties if required can be added here .
#Override
public void beforeStep(StepExecution stepExecution) {
// Any logic added here will execute before executing step
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
// Any logic added here will execute after executing step
return null;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws IOException {
if (suiviFluxDao.getNbFileDateTrt(FilenameUtils.getName(resource),
Utils.getDateFromStringFormatUS(dateTraitement)) > 0) {
LOGGER.info(PropertiesUtils.getLibelleExcep(Constantes.ERREUR_NB_FILE_SELECT,
new String[]{ConstantesNomsSql.TABLE_STCO_STAU_SUIVI_FLUX, FilenameUtils.getName(resource),
dateTraitement, Constantes.NAME_TRT}));
contribution.setExitStatus(ExitStatus.FAILED);
} else {
// any logic goes here .
contribution.setExitStatus(ExitStatus.COMPLETED);
}
return RepeatStatus.FINISHED;
}
}
2 . Below code snippet allows to configure job using flow builder :
#Configuration
public class JobConfigurations {
private StepBuilderFactory stepBuilderFactory;
private JobBuilderFactory jobBuilderFactory;
#Autowired
public JobConfigurations(StepBuilderFactory stepBuilderFactory,
JobBuilderFactory jobBuilderFactory) {
this.stepBuilderFactory = stepBuilderFactory;
this.jobBuilderFactory = jobBuilderFactory;
}
#Bean
public Job job(TestTasklet testTasklet) {
Step validationStep = stepBuilderFactory.get("validationTasklet")
.tasklet(testTasklet).build();
//create another step where you want to perform business logic
//for sake of brevity let us assume it to be businessValidationStep
//Step businessValidationStep = stepBuilderFactory.get("businessvalidationstep")
// .chunk().reader().processor().writer();
return jobBuilderFactory.get("JOB_NAME").incrementer(new RunIdIncrementer())
.start(validationStep)// start your job with validation step
.on(ExitStatus.FAILED.getExitCode()).end()// this will terminate your job cleanly
.from(validationStep)
.on(ExitStatus.COMPLETED.getExitCode())//.to("businessValidationStep")
.to(validationStep).build().build();
}
}
#lasnico37 Hope above code will solve the problem statement .

store filenames in Spring Batch for send email

I’m writing an application in Spring Batch to do this:
Read the content of a folder, file by file.
Rename the files and move them to several folders.
Send two emails: one with successed name files processed and one with name files that throwed errors.
I’ve already get 1. and 2. but I need to make the 3 point. ¿How can I store the file names that have send to the writer method in an elegant way with Spring Batch?
You can use a Execution Context to store the values of file names which gets processed and also which fails with errors.
We shall have a List/ similar datastructure which has the file names after the business logic. Below is a small snippet for reference which implements StepExecutionListener,
public class FileProcessor implements ItemWriter<TestData>, StepExecutionListener {
private List<String> success = new ArrayList<>();
private List<String> failed = new ArrayList<>();
#Override
public void beforeStep(StepExecution stepExecution) {
}
#Override
public void write(List<? extends BatchTenantBackupData> items) throws Exception {
// Business logic which adds the success and failure file names to the list
after processing
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
stepExecution.getJobExecution().getExecutionContext()
.put("fileProcessedSuccessfully", success);
stepExecution.getJobExecution().getExecutionContext()
.put("fileProcessedFailure", failed);
return ExitStatus.COMPLETED;
}
}
Now we have stored the file names in the execution context which we will be able to use it in send email step.
public class sendReport implements Tasklet, StepExecutionListener {
private List<String> success = new ArrayList<>();
private List<String> failed = new ArrayList<>();
#Override
public void beforeStep(StepExecution stepExecution) {
try {
// Fetch the list of file names which we have stored in the context from previous step
success = (List<String>) stepExecution.getJobExecution().getExecutionContext()
.get("fileProcessedSuccessfully");
failed = (List<BatchJobReportContent>) stepExecution.getJobExecution()
.getExecutionContext().get("fileProcessedFailure");
} catch (Exception e) {
}
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// Business logic to send email with the file names
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
logger.debug("Email Trigger step completed successfully!");
return ExitStatus.COMPLETED;
}
}

Spring Batch How to get JobExecution Object in Process Listner

I have requirement in my project what ever exception occur in ItemProccesor need to store Exception in JobExecution context and at the end of JobExecution send mail for Exceptional records but how to get JobExecution Object in processListner?
I tried using #beforestep in processListner but JobExecution object was null is there any way to get JobExecution context in process Listner
I got solution in spring batch for above issue, need to specify jobscope in process listener and access job execution context in listner class code is mention below.
#Bean
#JobScope
public CaliberatedProcessorListener calibratedProcessorListener() {
return new CaliberatedProcessorListener();
}
public class CaliberatedProcessorListener <T, S> implements ItemProcessListener<T, S> {
#Value("#{jobExecution}")
public JobExecution jobExecution;
#Override
public void beforeProcess(T calibratedProessorInPut) {
// // do nothing
}
#Override
public void afterProcess(T calibratedProessorInput, S calibratedProessorOutPut) {
// do nothing
}
#Override
public void onProcessError(T item, Exception calibratedProcessorEx) {
FtpEmailData ftpEmailData = (FtpEmailData) jobExecution.getExecutionContext().get("calDeviceBatchInfo");
ftpEmailData.getExceptionList().add(new CalibratedDeviceException(calibratedProcessorEx.getMessage()));
}
}

Spring Batch: File not being read

I am trying to create an application that uses the spring-batch-excel extension to be able to read Excel files uploaded through a web interface by it's users in order to parse the Excel file for addresses.
When the code runs, there is no error, but all I get is the following in my log. Even though I have log/syso throughout my Processor and Writer (these are never being called, and all I can imagine is it's not properly reading the file, and returning no data to process/write). And yes, the file has data, several thousand records in fact.
Job: [FlowJob: [name=excelFileJob]] launched with the following parameters: [{file=Book1.xlsx}]
Executing step: [excelFileStep]
Job: [FlowJob: [name=excelFileJob]] completed with the following parameters: [{file=Book1.xlsx}] and the following status: [COMPLETED]
Below is my JobConfig
#Configuration
#EnableBatchProcessing
public class AddressExcelJobConfig {
#Bean
public BatchConfigurer configurer(EntityManagerFactory entityManagerFactory) {
return new CustomBatchConfigurer(entityManagerFactory);
}
#Bean
Step excelFileStep(ItemReader<AddressExcel> excelAddressReader,
ItemProcessor<AddressExcel, AddressExcel> excelAddressProcessor,
ItemWriter<AddressExcel> excelAddressWriter,
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("excelFileStep")
.<AddressExcel, AddressExcel>chunk(1)
.reader(excelAddressReader)
.processor(excelAddressProcessor)
.writer(excelAddressWriter)
.build();
}
#Bean
Job excelFileJob(JobBuilderFactory jobBuilderFactory,
#Qualifier("excelFileStep") Step excelAddressStep) {
return jobBuilderFactory.get("excelFileJob")
.incrementer(new RunIdIncrementer())
.flow(excelAddressStep)
.end()
.build();
}
}
Below is my AddressExcelReader
The late binding works fine, there is no error. I have tried loading the resource given the file name, in addition to creating a new ClassPathResource and FileSystemResource. All are giving me the same results.
#Component
#StepScope
public class AddressExcelReader implements ItemReader<AddressExcel> {
private PoiItemReader<AddressExcel> itemReader = new PoiItemReader<AddressExcel>();
#Override
public AddressExcel read()
throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
return itemReader.read();
}
public AddressExcelReader(#Value("#{jobParameters['file']}") String file, StorageService storageService) {
//Resource resource = storageService.loadAsResource(file);
//Resource testResource = new FileSystemResource("upload-dir/Book1.xlsx");
itemReader.setResource(new ClassPathResource("/upload-dir/Book1.xlsx"));
itemReader.setLinesToSkip(1);
itemReader.setStrict(true);
itemReader.setRowMapper(excelRowMapper());
}
public RowMapper<AddressExcel> excelRowMapper() {
BeanWrapperRowMapper<AddressExcel> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(AddressExcel.class);
return rowMapper;
}
}
Below is my AddressExcelProcessor
#Component
public class AddressExcelProcessor implements ItemProcessor<AddressExcel, AddressExcel> {
private static final Logger log = LoggerFactory.getLogger(AddressExcelProcessor.class);
#Override
public AddressExcel process(AddressExcel item) throws Exception {
System.out.println("Converting " + item);
log.info("Convert {}", item);
return item;
}
}
Again, this is never coming into play (no logs generated). And if it matters, this is how I'm launching my job from a FileUploadController from a #PostMapping("/") to handle the file upload, which first stores the file, then runs the job:
#PostMapping("/")
public String handleFileUpload(#RequestParam("file") MultipartFile file, RedirectAttributes redirectAttributes) {
storageService.store(file);
try {
JobParameters jobParameters = new JobParametersBuilder()
.addString("file", file.getOriginalFilename().toString()).toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException
| JobParametersInvalidException e) {
e.printStackTrace();
}
redirectAttributes.addFlashAttribute("message",
"You successfully uploaded " + file.getOriginalFilename() + "!");
return "redirect:/";
}
And last by not least
Here is my AddressExcel POJO
import lombok.Data;
#Data
public class AddressExcel {
private String address1;
private String address2;
private String city;
private String state;
private String zip;
public AddressExcel() {}
}
UPDATE (10/13/2016)
From Nghia Do's comments, I also created my own RowMapper instead of using the BeanWrapper to see if that was the issue. Still the same results.
public class AddressExcelRowMapper implements RowMapper<AddressExcel> {
#Override
public AddressExcel mapRow(RowSet rs) throws Exception {
AddressExcel temp = new AddressExcel();
temp.setAddress1(rs.getColumnValue(0));
temp.setAddress2(rs.getColumnValue(1));
temp.setCity(rs.getColumnValue(2));
temp.setState(rs.getColumnValue(3));
temp.setZip(rs.getColumnValue(4));
return temp;
}
}
All it seems I needed was to add the following to my ItemReader:
itemReader.afterPropertiesSet();
itemReader.open(new ExecutionContext());

footer call back happening after afterStep call

My spring batch writer implements footer call back and step execution listener as below:
public class MyItemWriter implements ItemWriter<MyBean>, FlatFileFooterCallback, StepExecutionListener, ItemStream
The class is having a private String property.
private String one;
My footerCallback and afterStep methods are as below:
public void writeFooter(Writer writer) throws IOException {
this.one = "this is a test";
}
public ExitStatus afterStep(StepExecution stepExecution) {
File file = new File("myFile");
try {
FileUtils.write(file, this.one);
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
As clear, I am setting the property string one in footerCallBack and then using it in afterStep method.
However, I am getting null value written in the file.
On debugging I found that afterStep method is getting called before writeFooter.
But isn't it wrong ? Shouldn't step complete only after even the footerCallBack is over ?