I have a job with Spring Batch which I read some files with BeanIO, and I would handle invalid files, so I created a SkipPolicy class.
public class FileVerificationSkipper implements SkipPolicy {
private static final FluentLogger LOGGER = LoggerService.init(FileVerificationSkipper.class);
#Override
public boolean shouldSkip(Throwable exception, int skipCount) throws SkipLimitExceededException {
if (exception instanceof FileNotFoundException) {
return false;
}
if (exception instanceof BeanReaderException && skipCount <= 10) {
LOGGER.all().logKey("Error on read file: ").value(exception).asError();
return true;
}
else {
return false;
}
}
}
On my reader step I access the name like this: #Value("#{jobParameters['input.file.name']}") String inputFile
I would like to log the filename, how could I do that?
Debugging how Spring Batch inject the parameters I found the solution.
I just need to add #StepScope in the class and create the variable where I want to inject the parameter:
#Component
#StepScope
#RequiredArgsConstructor
public class FileVerificationSkipper implements SkipPolicy {
#Value("#{jobParameters['input.file.name']}")
private String inputFile;
...
}
Related
I have a job with Spring Batch which I read some files with BeanIO, and I would handle invalid files, so I created a SkipPolicy class.
public class FileVerificationSkipper implements SkipPolicy {
private static final FluentLogger LOGGER = LoggerService.init(FileVerificationSkipper.class);
#Override
public boolean shouldSkip(Throwable exception, int skipCount) throws SkipLimitExceededException {
if (exception instanceof FileNotFoundException) {
return false;
}
if (exception instanceof BeanReaderException && skipCount <= 10) {
LOGGER.all().logKey("Error on read file: ").value(exception).asError();
return true;
}
else {
return false;
}
}
}
On my reader step I access the name like this: #Value("#{jobParameters['input.file.name']}") String inputFile
I would like to log the filename, how could I do that?
Debugging how Spring Batch inject the parameters I found the solution.
I just need to add #StepScope in the class and create the variable where I want to inject the parameter:
#Component
#StepScope
#RequiredArgsConstructor
public class FileVerificationSkipper implements SkipPolicy {
#Value("#{jobParameters['input.file.name']}")
private String inputFile;
...
}
My Springbatch Configuration class have following job :
public Job job(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory) {
Step step = stepBuilderFactory.get("sampleSetp")
.<String, String>chunk(10)
.reader(new Reader())
.writer(new Writer())
.build();
}
And My Reader class is :
public class Reader implements ItemReader<String> {
#Override
public String read() {
FlatFileItemReader<String> reader = new FlatFileItemReader<String>();
reader.setResource(someResource);
reader.setLineMapper(lineMapper());
**return reader; //This should be string, but i have FlatFileItemReader<String> type.**
}
}
I am doing some work on reader(FlatFileItemReader) and want to return to job this reader but read() return type is String.
How should i send this reader to job present in BathConfig?
NOTE: IF i make do this FlatFileItemReader processing in springConfig file and pass it to job there only then it is working.
ex:
Step step = stepBuilderFactory.get("sampleSetp")
.<String, String>chunk(10)
.reader(flatFileReader())
.write()
.build();
where flatFileReader() is :
public FlatFileItemReader<String> flatFileReader(){
...
return FlatFileItemReader<String> type
}
SpringBatch read() function in ItemReader class return type is String and Want to return FlatFileItemReader , How should i Do it?
The read method should return the actual item (a String in your case) and not another reader. If you want a custom reader that delegates to a FlatFileItemReader, you can do something like:
public class Reader implements ItemReader<String> {
private FlatFileItemReader<String> delegate;
public Reader(FlatFileItemReader<String> delegate) {
this.delegate = delegate;
}
#Override
public String read() {
return delegate.read();
}
}
I am trying to create an application that uses the spring-batch-excel extension to be able to read Excel files uploaded through a web interface by it's users in order to parse the Excel file for addresses.
When the code runs, there is no error, but all I get is the following in my log. Even though I have log/syso throughout my Processor and Writer (these are never being called, and all I can imagine is it's not properly reading the file, and returning no data to process/write). And yes, the file has data, several thousand records in fact.
Job: [FlowJob: [name=excelFileJob]] launched with the following parameters: [{file=Book1.xlsx}]
Executing step: [excelFileStep]
Job: [FlowJob: [name=excelFileJob]] completed with the following parameters: [{file=Book1.xlsx}] and the following status: [COMPLETED]
Below is my JobConfig
#Configuration
#EnableBatchProcessing
public class AddressExcelJobConfig {
#Bean
public BatchConfigurer configurer(EntityManagerFactory entityManagerFactory) {
return new CustomBatchConfigurer(entityManagerFactory);
}
#Bean
Step excelFileStep(ItemReader<AddressExcel> excelAddressReader,
ItemProcessor<AddressExcel, AddressExcel> excelAddressProcessor,
ItemWriter<AddressExcel> excelAddressWriter,
StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("excelFileStep")
.<AddressExcel, AddressExcel>chunk(1)
.reader(excelAddressReader)
.processor(excelAddressProcessor)
.writer(excelAddressWriter)
.build();
}
#Bean
Job excelFileJob(JobBuilderFactory jobBuilderFactory,
#Qualifier("excelFileStep") Step excelAddressStep) {
return jobBuilderFactory.get("excelFileJob")
.incrementer(new RunIdIncrementer())
.flow(excelAddressStep)
.end()
.build();
}
}
Below is my AddressExcelReader
The late binding works fine, there is no error. I have tried loading the resource given the file name, in addition to creating a new ClassPathResource and FileSystemResource. All are giving me the same results.
#Component
#StepScope
public class AddressExcelReader implements ItemReader<AddressExcel> {
private PoiItemReader<AddressExcel> itemReader = new PoiItemReader<AddressExcel>();
#Override
public AddressExcel read()
throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
return itemReader.read();
}
public AddressExcelReader(#Value("#{jobParameters['file']}") String file, StorageService storageService) {
//Resource resource = storageService.loadAsResource(file);
//Resource testResource = new FileSystemResource("upload-dir/Book1.xlsx");
itemReader.setResource(new ClassPathResource("/upload-dir/Book1.xlsx"));
itemReader.setLinesToSkip(1);
itemReader.setStrict(true);
itemReader.setRowMapper(excelRowMapper());
}
public RowMapper<AddressExcel> excelRowMapper() {
BeanWrapperRowMapper<AddressExcel> rowMapper = new BeanWrapperRowMapper<>();
rowMapper.setTargetType(AddressExcel.class);
return rowMapper;
}
}
Below is my AddressExcelProcessor
#Component
public class AddressExcelProcessor implements ItemProcessor<AddressExcel, AddressExcel> {
private static final Logger log = LoggerFactory.getLogger(AddressExcelProcessor.class);
#Override
public AddressExcel process(AddressExcel item) throws Exception {
System.out.println("Converting " + item);
log.info("Convert {}", item);
return item;
}
}
Again, this is never coming into play (no logs generated). And if it matters, this is how I'm launching my job from a FileUploadController from a #PostMapping("/") to handle the file upload, which first stores the file, then runs the job:
#PostMapping("/")
public String handleFileUpload(#RequestParam("file") MultipartFile file, RedirectAttributes redirectAttributes) {
storageService.store(file);
try {
JobParameters jobParameters = new JobParametersBuilder()
.addString("file", file.getOriginalFilename().toString()).toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException
| JobParametersInvalidException e) {
e.printStackTrace();
}
redirectAttributes.addFlashAttribute("message",
"You successfully uploaded " + file.getOriginalFilename() + "!");
return "redirect:/";
}
And last by not least
Here is my AddressExcel POJO
import lombok.Data;
#Data
public class AddressExcel {
private String address1;
private String address2;
private String city;
private String state;
private String zip;
public AddressExcel() {}
}
UPDATE (10/13/2016)
From Nghia Do's comments, I also created my own RowMapper instead of using the BeanWrapper to see if that was the issue. Still the same results.
public class AddressExcelRowMapper implements RowMapper<AddressExcel> {
#Override
public AddressExcel mapRow(RowSet rs) throws Exception {
AddressExcel temp = new AddressExcel();
temp.setAddress1(rs.getColumnValue(0));
temp.setAddress2(rs.getColumnValue(1));
temp.setCity(rs.getColumnValue(2));
temp.setState(rs.getColumnValue(3));
temp.setZip(rs.getColumnValue(4));
return temp;
}
}
All it seems I needed was to add the following to my ItemReader:
itemReader.afterPropertiesSet();
itemReader.open(new ExecutionContext());
I am using MultiResourceItemReader in Spring Batch for reading multiple XML files and I want to get current resource.Here is my configuration:
public class MultiFileResourcePartitioner extends MultiResourceItemReader<MyObject> {
#Override
public void update(final ExecutionContext pExecutionContext) throws ItemStreamException {
super.update(pExecutionContext);
if (getCurrentResource() != null && getCurrentResource().getFilename() != null) {
System.out.println("update:" + getCurrentResource().getFilename());
}
}
}
And my reader:
<bean id="myMultiSourceReader"
class="mypackage.MultiFileResourcePartitioner">
<property name="resources" value="file:${input.directory}/*.xml" />
</bean>
The code above read XML files correctly but the method getCurrentResources() return null.
By debugging, the batch enter to update method
Please help!
There is a specific interface for this problem called ResourceAware: it's purpouse is to inject current resource into objects read from a MultiResourceItemReader.
Check this thread for further information.
I tried it with a simple Listener for logging the current resource from a injected {#link MultiResourceItemReader}. Saves the value to the StepExecutionContext.
To get it working with a step scoped MultiResourceItemReader i access the proxy directly, see http://forum.springsource.org/showthread.php?120775-Accessing-the-currently-processing-filename, https://gist.github.com/1582202 and https://jira.springsource.org/browse/BATCH-1831.
public class GetCurrentResourceChunkListener implements ChunkListener, StepExecutionListener {
private StepExecution stepExecution;
private Object proxy;
private final List<String> fileNames = new ArrayList<>();
public void setProxy(Object mrir) {
this.proxy = mrir;
}
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return stepExecution.getExitStatus();
}
#Override
public void beforeChunk(ChunkContext cc) {
if (proxy instanceof Advised) {
try {
Advised advised = (Advised) proxy;
Object obj = advised.getTargetSource().getTarget();
MultiResourceItemReader mrirTarget = (MultiResourceItemReader) obj;
if (mrirTarget != null
&& mrirTarget.getCurrentResource() != null
&& !fileNames.contains(mrirTarget.getCurrentResource().getFilename())) {
String fileName = mrirTarget.getCurrentResource().getFilename();
fileNames.add(fileName);
String index = String.valueOf(fileNames.indexOf(fileName));
stepExecution.getExecutionContext().put("current.resource" + index, fileName);
}
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
}
#Override
public void afterChunk(ChunkContext cc) {
}
#Override
public void afterChunkError(ChunkContext cc) {
}
}
see https://github.com/langmi/spring-batch-examples-playground for a working example - look for "GetCurrentResource..."
public class CpsFileItemProcessor implements ItemProcessor<T, T> {
#Autowired
MultiResourceItemReader multiResourceItemReader;
private String fileName;
#Override
public FileDetailsEntityTemp process(T item) {
if(multiResourceItemReader.getCurrentResource()!=null){
fileName = multiResourceItemReader.getCurrentResource().getFilename();
}
item.setFileName(fileName);
return item;
}
}
My spring batch writer implements footer call back and step execution listener as below:
public class MyItemWriter implements ItemWriter<MyBean>, FlatFileFooterCallback, StepExecutionListener, ItemStream
The class is having a private String property.
private String one;
My footerCallback and afterStep methods are as below:
public void writeFooter(Writer writer) throws IOException {
this.one = "this is a test";
}
public ExitStatus afterStep(StepExecution stepExecution) {
File file = new File("myFile");
try {
FileUtils.write(file, this.one);
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
As clear, I am setting the property string one in footerCallBack and then using it in afterStep method.
However, I am getting null value written in the file.
On debugging I found that afterStep method is getting called before writeFooter.
But isn't it wrong ? Shouldn't step complete only after even the footerCallBack is over ?