Custome ItemReader for reading each iteration of a for loop - spring-batch

Question

Recently started working with Spring Batch chunk based processing. I need to create batch for creating random 3 million strings, 1 million each of different type and count. Like a million strings starting with A, next 1.5 million ending with GH. Cannot do it with for loop as it will block a thread. I have to write them in db also. No idea how to make my ItemReader read each iteration.
I have understood the custom ItemReader through this
but not getting what should be the "item" here.
If i create a chunk of 1000, then how the counters will be handled, counting the chunk entry and the string generated counter.

Answer 1

The ItemReader interface is quite similar to an Iterator and like that has to maintain the iteration state internally. The Spring Batch framework even provides an IteratorItemReader.
Since Java 8 there are Streams, which I find quite versatile also for generating data and which can be converted into an Iterator.
Here is a possible solution along the lines of what you described:
#RunWith(SpringRunner.class)
#SpringBootTest(classes = Java8StreamReaderTest.TestConfig.class, properties = {"spring.batch.job.enabled=false"})
#EnableBatchProcessing
public class Java8StreamReaderTest {
#Configuration
static class TestConfig {
#Bean
JobBuilderFactory jobBuilderFactory(final JobRepository jobRepository) {
return new JobBuilderFactory(jobRepository);
}
#Bean
StepBuilderFactory stepBuilderFactory(final JobRepository jobRepository, final PlatformTransactionManager transactionManager) {
return new StepBuilderFactory(jobRepository, transactionManager);
}
#Bean
Job streamJob() {
return jobBuilderFactory(null).get("streamJob")
.start(stepBuilderFactory(null, null).get("streamStep")
.<String, String>chunk(1000)
.reader(streamReader())
.writer(listWriter())
.build()
)
.build();
}
#Bean
ListItemWriter<String> listWriter() {
return new ListItemWriter<>();
}
#Bean
ItemReader<String> streamReader() {
return new IteratorItemReader<String>(stream().iterator());
}
#Bean
Stream<String> stream() {
return Stream.of(
IntStream.range(0, 100000).boxed().map(i -> {
return new String("A"+ RandomStringUtils.random(10, true, false));
}),
IntStream.range(0, 100000).boxed().map(i -> {
return new String("B"+ RandomStringUtils.random(10, true, false));
}),
IntStream.range(0, 100000).boxed().map(i -> {
return new String(RandomStringUtils.random(10, true, false) + "GH");
})
)
.flatMap(s -> s);
}
}
#Autowired
Job streamJob;
#Autowired
JobLauncher jobLauncher;
#Autowired
ListItemWriter<String> listWriter;
#Test
public void shouldExecuteTestJob() throws Exception {
JobExecution execution = jobLauncher.run(streamJob, new JobParametersBuilder().toJobParameters());
assertEquals(ExitStatus.COMPLETED, execution.getExitStatus());
assertThat(listWriter.getWrittenItems(), hasSize(300000));
}
}