i'm triyng to retry spring batch FlatFileItemReader but no success.
FlatFileItemReader<Transaction> reader = new FlatFileItemReader<>();
Resource resource = new FileSystemResource("input/1-101-D-2017-212-volume-per-transaction.csv");
try {
resource.contentLength();
} catch (IOException e) {
e.printStackTrace();
}
reader.setResource(resource);
reader.setRecordSeparatorPolicy(new BlankLineRecordSeparatorPolicy());
DefaultLineMapper<Transaction> lineMapper = new DefaultLineMapper<>();
reader.setLineMapper(lineMapper);
reader.setStrict(false);
reader.setLinesToSkip(NUMBER_OF_HEADER_LINES);
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
reader.setSkippedLinesCallback(line -> tokenizer.setNames(line.split(",")));
lineMapper.setLineTokenizer(tokenizer);
lineMapper.setFieldSetMapper(new TransactionFieldSetMapper());
reader.setLineMapper(lineMapper);
return reader;
then into my step i have
.faultTolerant()
.retryLimit(3)
.retry(FileNotFoundException.class)
can someone give my some hint how can retry it?
Hopefully it's still useful for you, please refer to this example where I put an example of the retry reader using FlatFileItemReader:
https://github.com/jeronimogalicia/batch-retry-flatfile-item-reader
Basically you have to annotate your application class with #EnableRetry and annotate your reader like this:
#Bean
#StepScope
#Retryable(include = { ItemStreamException.class }, maxAttempts = 5)
ItemReader<Player> loadRecordsReader() throws Exception {
String filePath = "src/main/resources/players.csv";
System.out.println("Loading records from " + filePath + " try " + counter);
FlatFileItemReader<Player> itemReader = new FlatFileItemReader<>();
itemReader.setResource(new FileSystemResource(filePath));
itemReader.setLinesToSkip(1);
//DelimitedLineTokenizer defaults to comma as its delimiter
DefaultLineMapper<Player> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
lineMapper.setFieldSetMapper(new PlayerFieldSetMapper());
itemReader.setLineMapper(lineMapper);
itemReader.open(new ExecutionContext());
return itemReader;
}
Related
In the spring batch project, I used JdbcCursorItemReader to read data to process them in parallel. I can run the batch locally without any problem.
I also heard that JdbcPagingItemReader is recommended for parallel processing against JdbcCursorItemReader, as cursor reader will hold the connection too long while paging reader can release connection once the page size is reached.
I then switched to JdbcPagingItemReader in step2, but out of surprise, I got the exception below when running locally.
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 -
Connection is not available, request timed out after 300001ms.
However, it seems the above exception occurs in step1 before the paging reader in step2 is executed, and that is the only change made. Please shed some light on why the exception is thrown and if it is good practice to use paging reader instead of cursor in parallel processing. Much appreciated your help!
The code snippet is pasted below:
#Bean
#StepScope
public Flow createParallelSubFlow() {
List<Flow> subFlowList = new ArrayList<>();
List<Stream> streamList;
try {
streamList = dataSourceConfig.streamMapper().
getStreamListByStatus(Constants.PENDING_STATUS_CD);
} catch (Exception e) {
}
streamList.forEach(stream -> {
long id = stream.getStreamId();
String flowName = "stream" + id + "_flow";
Flow subFlow = new FlowBuilder<Flow>(flowName)
.start(step1(id))
.next(step2(id))
.end();
subFlowList.add(subFlow);
});
return new FlowBuilder<Flow>("splitFlow").split(new SimpleAsyncTaskExecutor())
.add(subFlowList.toArray(new Flow[0])).build();
}
public Step step1(long id) {
return stepBuilderFactory.get("step1")
.<Domain, Domain>chunk(100)
.reader(reader1(id))
.writer(writer1())
.build();
}
//#StepScope
//#Bean
public Step step2(long id) {
return stepBuilderFactory.get("step2")
.<Domain, Domain>chunk(100)
.reader(cursorReader2(id))
.processor(processor2)
.writer(writer2())
.build();
}
public JdbcCursorItemReader<Domain> cursorReader2(Long id) {
return new JdbcCursorItemReaderBuilder<Domain>()
.dataSource(dataSourceConfig.dataSource())
.name("cursorReader")
.sql(Constants.QUERY_SQL)
.preparedStatementSetter(new PreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps) throws SQLException {
ps.setLong(1, id);
}})
.rowMapper(new RowMapper())
.build();
}
//Switch from cursorReader2 to pagingReader2 in step2
public JdbcPagingItemReader<Domain> pagingReader2(Long id) {
return new JdbcPagingItemReaderBuilder<Domain>()
.dataSource(dataSourceConfig.dataSource())
.name("pagingReader")
.queryProvider(queryProvider())
.parameterValues(parameterValues(id))
.rowMapper(new RowMapper())
.pageSize(100)
.build();
}
#Bean
public PagingQueryProvider queryProvider() {
SqlPagingQueryProviderFactoryBean providerFactory = new SqlPagingQueryProviderFactoryBean();
Map<String, Order> sortKeys = new HashMap<>(2);
sortKeys.put("ID", Order.ASCENDING);
providerFactory.setDataSource(dataSourceConfig.dataSource());
providerFactory.setSelectClause("SELECT Clause");
providerFactory.setFromClause("FROM Clause");
providerFactory.setWhereClause("WHERE Clause");
providerFactory.setSortKeys(sortKeys);
PagingQueryProvider pagingQueryProvider = null;
try {
pagingQueryProvider = providerFactory.getObject();
} catch (Exception e) {
logger.error("Failed to get PagingQueryProvider", e);
throw new RuntimeException("Failed to get PagingQueryProvider", e);
}
return pagingQueryProvider;
}
private Map<String, Object> parameterValues(Long id) {
Map<String, Object> parameterValues = new HashMap<>();
parameterValues.put("1", id);
return parameterValues;
}
My parameter (workingDirectory) is always null and I do not understand why?
I am using the job launcher to kick off the job.
I see the job parameter in the database but it is not getting injected into my reader.
This is my reader.
#Component("customerReader")
#StepScope
public class CustomReader implements ItemReader<Customer> {
#Value("#{jobParameters['workingDirectory']}")
protected String workingDirectory;
private BufferedReader reader;
protected ObjectMapper objectMapper = new ObjectMapper();
private int fileIdx = 1;
#Override
public Customer read() throws IOException {
if(reader == null) {
reader = new BufferedReader(new FileReader(new File(workingDirectory + "output1.json")));
}
String line = reader.readLine();
if (line == null) {
try {
fileIdx++;
reader = new BufferedReader(
new FileReader(new File(workingDirectory + "output" + fileIdx + ".json")));
}
catch (FileNotFoundException ex) {
return null;
}
line = reader.readLine();
}
return objectMapper.readValue(line, Customer.class);
}
}
I am launching by calling:
JobParameters p = new JobParametersBuilder().addString("workingDirectory", workingDirectory).toJobParameters();
JobExecution e = jobLauncher.run(
recordGenerator.getJob(file, "/Users/abc/data/"), p);
I am trying to create a Spring Batch POC with Java Configuration and PostGreSQL.
I have successfully created beans that would have otherwise been provided via the in memory DB using #EnableBatchProcessing and #EnableAutoConfiguration.
I am not able to get the beans (JobExplorer) to return a JobExecution list given a JobInstance bean created from the same JobExplorer bean.
The error I am getting is "Unable to deserialize the execution context" which seems to be coming from the method trying to deserialize the "SHORT_CONTEXT" field of the JOB_EXECUTION_CONTEXT table.
I have passed the created JobExplorer bean DefaultExecutionContextSerializer. Later passed a DefaultLobHandler with "wrapAsLob" set to True when I was still getting the error.
#Bean
public JobRegistry jobRegistry() {
JobRegistry jr = new MapJobRegistry();
return jr;
}
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor() {
JobRegistryBeanPostProcessor jrbpp = new JobRegistryBeanPostProcessor();
jrbpp.setJobRegistry(jobRegistry());
return jrbpp;
}
#Bean
public JobOperator jobOperator() {
SimpleJobOperator sjo = new SimpleJobOperator();
sjo.setJobExplorer(jobExplorer());
sjo.setJobLauncher(jobLauncher());
sjo.setJobRegistry(jobRegistry());
sjo.setJobRepository(jobRepository());
return sjo;
}
#Bean
public JobExplorer jobExplorer() {
JobExplorerFactoryBean jefb = new JobExplorerFactoryBean();
jefb.setDataSource(dataSource());
jefb.setJdbcOperations(jdbcTemplate);
jefb.setTablePrefix("batch_");
jefb.setSerializer(new DefaultExecutionContextSerializer());
DefaultLobHandler lh = new DefaultLobHandler();
lh.setWrapAsLob(true);
jefb.setLobHandler(lh);
JobExplorer je = null;
try {
je = jefb.getObject();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return je;
}
#ConfigurationProperties(prefix = "spring.datasource")
#Bean
#Primary
public DataSource dataSource() {
return DataSourceBuilder.create().build();
}
#Bean
public JobRepository jobRepository() {
JobRepositoryFactoryBean jrfb = new JobRepositoryFactoryBean();
jrfb.setDataSource(dataSource());
jrfb.setDatabaseType("POSTGRES");
jrfb.setTransactionManager(new ResourcelessTransactionManager());
jrfb.setSerializer(new DefaultExecutionContextSerializer());
jrfb.setTablePrefix("batch_");
JobRepository jr = null;
try {
jr = (JobRepository)jrfb.getObject();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return jr;
}
Below is the get method in my rest controller where I am trying handle generate a list of failed Job executions
#Autowired
JobLauncher jobLauncher;
#Autowired
JobRegistry jobRegistry;
#Autowired
JobOperator jobOperator;
#Autowired
JobExplorer jobExplorer;
#SuppressWarnings("unchecked")
#GetMapping("batch/failedJobs")
public Map<String, List<JobExecution>> getFailedJobs() {
try {
if (jobRegistry == null || jobOperator == null || jobExplorer == null) {
System.out.println("job registry, operator or explorer is null");
} else {
Map<String, List<JobExecution>> allJobInstances = new HashMap<String, List<JobExecution>>();
// Get all jobs
jobRegistry.getJobNames().stream().forEach(jobName -> {
jobExplorer.getJobInstances(jobName, 1, 1000).forEach(l -> {
System.out.println("jobName: " + jobName + " instance: " + l);
});
jobExplorer.getJobInstances(jobName, 1, 1000).stream().forEach(jobInstance -> {
List<JobExecution> execultionList = jobExplorer.getJobExecutions(jobInstance); //Failing here
if (execultionList != null) {
System.out.println("" + execultionList);
execultionList.stream().forEach(l2 -> {
System.out.println("jobName: " + jobName + " instance: " + jobInstance
+ " jobExecution: " + l2);
});
if(allJobInstances.get(jobName) == null) {
allJobInstances.put(jobName, new ArrayList<JobExecution>());
}
allJobInstances.get(jobName).addAll((Collection<? extends JobExecution>) jobExplorer.getJobExecutions(jobInstance).stream().filter(e -> e.getStatus().equals(BatchStatus.FAILED)));
}else {
System.out.println("Could not get jobExecution for jobName " + jobName + " jobInstance: " + jobInstance);
}
});
});
return allJobInstances;
}
}catch (Exception e) {
System.out.println(e.getMessage());
logger.info(e.getMessage());
}
return null;
}
I fixed a similar issue by changing to the Jackson2 serializer:
jefb.setSerializer(new Jackson2ExecutionContextStringSerializer());
You may try it.
I am generating a powerpoint file(.pptx) and i would like to return back this file when a REST call happens. But now am able to get only .File type extension.
#RequestMapping(value = "/ImageManagerPpt/{accessionId}", method = RequestMethod.GET, produces = "application/ppt")
public ResponseEntity<InputStreamResource> createPptforAccessionId(#PathVariable("accessionId") String accessionId,HttpServletResponse response) throws IOException** {
System.out.println("Creating PPT for Patient Details with id " + accessionId);
File pptFile = imageManagerService.getPptForAccessionId(accessionId);
if (pptFile == null) {
System.out.println("Patient Id with id " + accessionId + " not found");
return new ResponseEntity<InputStreamResource>(HttpStatus.NOT_FOUND);
}
InputStream stream = null;
try {
stream = new FileInputStream(pptFile);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
ClassPathResource classpathfile = new ClassPathResource("Titlelayout3.pptx");
InputStreamResource inputStreamResource = new InputStreamResource(stream);
return ResponseEntity.ok().contentLength(classpathfile.contentLength())
.contentType(MediaType.parseMediaType("application/octet-stream"))
.body(new InputStreamResource(classpathfile.getInputStream()));
}
-Bharat
Have you tried, this?
InputStream stream = new InputStream(pptFile);
org.apache.commons.io.IOUtils.copy(is, response.getOutputStream());
response.flushBuffer();
You will get file as you put into the InputStream.
I have class like "location".
"Location" class have four field.
fields are:id,city,state,country..
country is seprate class it contains 2 field , country code, country name , 2 fields must read from location class..
if i write "locationMongoRepository.save()", then it shows error as bound mismatch. please give solution for how to save in mongodb.
public void insertLocation() throws InvalidFormatException, IOException, JSONException{
FileInputStream inp;
Workbook workbook;
try {
inp = new FileInputStream( "/home/Downloads/eclipse/Workspace/Samplboot-master latest/cityListForIndia1.xlsx" );
workbook = WorkbookFactory.create( inp );
Sheet sheet = workbook.getSheetAt(0);
JSONArray json = new JSONArray();
boolean isFirstRow = true;
ArrayList<String> rowName = new ArrayList<String>();
for ( Iterator<Row> rowsIT = sheet.rowIterator(); rowsIT.hasNext(); )
{
Row row = rowsIT.next();
JSONObject jRow = new JSONObject();
if(isFirstRow)
{
for ( Iterator<Cell> cellsIT = row.cellIterator(); cellsIT.hasNext(); )
{
Cell cell = cellsIT.next();
rowName.add(cell.getStringCellValue());
}
isFirstRow = false;
}
else
{
JSONObject jRowCountry= new JSONObject();
JSONObject jRowLocation= new JSONObject();
jRowLocation.put("city", row.getCell(0));
jRowLocation.put("state", row.getCell(1));
jRowCountry.put("country",row.getCell(2) );
jRowCountry.put("countryCode", row.getCell(3) );
jRowLocation.put("country", jRowCountry);
System.out.println("Location"+jRowLocation.toString());
}
}
}
catch (InvalidFormatException e) {
// TODO Auto-generated catch block
System.out.println("Invalid Format, Only Excel files are supported");
e.printStackTrace();
} catch (IOException e) {
System.out.println("Check if the input file exists and the path is correct");
e.printStackTrace();
} catch (JSONException e) {
// TODO Auto-generated catch block
System.out.println("Unable to generate Json");
e.printStackTrace();
}
}
}
I'm using Spring Data to support working with MongoDB and it's really helpful. You should read this article to get its idea and applied to your case https://dzone.com/articles/spring-data-mongodb-hello.
P/S: In case you can't use Spring Data to work with MongoDB, please provide more detail in your code/ your exception so we can investigate it more detail.