How can I read a larger dataset within other minor reads from a database using chuck based approach in Spring Batch? - spring-batch

Question

I have to make several data reads from a database including a larger dataset in between and write the content to a file after processing.
Example
| <- read account data from the database
| <- Process & Read a smaller account dataset from the database
| <- Process & Read smaller data sub-set from the database based on the above data
| <- Process & read a larger dataset from the database (chunk-based approach preferred)
| <- Process & read smaller data sub-set from the database based on the above data
| -> Process & write all the above collected/ processed data to a file
Multi-step processing is possible but it has a lot of overheads in step handling including inter-step data transfer since we have to create a single file from all the above data sets.
Caching the data set and use while processing is not possible for the larger dataset.
Simple data reads (generic) within processors for a larger dataset consume a lot of time & memory.
What would be the possible approaches to convert this to a Spring-Batch-based batch processing service?

Answer 1

I think the chunk-oriented processing model provided by Spring Batch is not suitable to your use case.
Multi-step processing is the way to go in my opinion, because this is what your requirement is about (this is almost dictated by your requirement), at least this is how I see it implemented without Spring Batch.