Store huge data at Chunk Level in Spring batch - spring-batch

Question

I am new to spring and I am not having much knowledge about Spring ,please help me to solve this out .My use case is : we are using Spring batch with chunk oriented processing to process data.
At the end of each processed chunk (ie, once commit interval is met and values passed to writer), the List of values has to be stored so that once the whole tasklet is completed , the stored list of values has to be used for writing the values to a csv file. If any job failure has happened in the chunk processing then writing list of values to file should not happen.
Is there any way to store the huge data at chunk level and then finally processing those at next step/tasklet or in any other way?

Answer 1

Don't store all data in memory; is a bad pratice for a batch application.
An alternative can be create a standard read/process/write step where you write to your csv file a processed chunk.
When a job error occurs, stop job and delete your csv file (you will get the same result as not write it at all).
I think you reach your goals whitout memory issues.

Answer 2

I would suggest a different approach as from my point of view you are trying to work with spring batch in a way it was not planned to work.
Process the data chunk by chunk and write every chunk to csv using FlatFileItemWriter.
Use a file name that marks it as temp.
Wrap your step with a listener and use OnProcessError hooks.
When hitting OnProcessError log the failed item
Add a conditional flow for success and failure see here
In case of delete temp file
In case of success rename file
You may use SystemCommandTasklet or implement your own tasket for 6 and 7
Your listener will look similar to the one below
#Component
public class PromoteUpdateCountToJobContextListener implements StepListener {
#OnProcessError
public ExitStatus processError(Object item, Exception e){
String failureMessage = String.format("Failed to process due to item %s" ,
item.toString());
Logger.error(failureMessage);
return ExitStatus.FAILED;
}
}
Your Job xml will be similar to:
<batch:job>
<batch:step id="processData">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="someReader"
writer="yourFlatFileItemWriter"/>
</batch:tasklet>
<batch:next on="*" to="renameTempCsv" />
<batch:next on="FAILED" to="deleteTempCsv" />
<batch:listeners>
<batch:listener ref="lineCurserListener" />
</batch:listeners>
</batch:step>
<batch:step id="deleteTempCsv">
<batch:tasklet ref="deleteTempCsvTasklet"/>
</batch:step>
<batch:step id="renameTempCsv">
<batch:tasklet ref="renameTempCsvTasklet"/>
</batch:step>
</batch:job>