Multi-Line Records right way to handle - spring-batch

Question

Currently I follow spring-batch docs on how to handle with Multi-Line Records situation
Currently if all line in the same format I can use FlatFileItemReader and FlatFileItemWriter to write and read csv.
Something like this
#Bean
public FlatFileItemReader<AdaptNeko> reader() {
return new FlatFileItemReaderBuilder<AdaptNeko>()
.name("itemReader")
.resource(new ClassPathResource(INPUT_FILE))
.delimited()
.names(new String[]{"name", "age", "potentialLevel"})
.fieldSetMapper(fieldSet -> NormalNeko.builder()
.name(fieldSet.readString("name"))
.age(fieldSet.readInt("age"))
.potentialLevel(fieldSet.readInt("potentialLevel"))
.build())
.build();
}
Currently the format file is a bit different.
My current csv format:
START, day, play
123, 456, 899, abc, xyz
END
START, day1, play23
789, 456, 899, koq, koq
END
After doing some research I found out that currently spring-batch have docs about this case but I'm very new with spring-batch so barely understand it. And the second line which is data only don't have start attribute with default value.
In documents example it uses orderFileTokenizer() then after that fieldSetMapper back into just one class like FlatFileItemReader usage
#Bean
public FlatFileItemReader flatFileItemReader() {
FlatFileItemReader<Trade> reader = new FlatFileItemReaderBuilder<>()
.name("flatFileItemReader")
.resource(new ClassPathResource("data/iosample/input/multiLine.txt"))
.lineTokenizer(orderFileTokenizer())
.fieldSetMapper(orderFieldSetMapper())
.build();
return reader;
}
and
#Bean
public PatternMatchingCompositeLineTokenizer orderFileTokenizer() {
PatternMatchingCompositeLineTokenizer tokenizer =
new PatternMatchingCompositeLineTokenizer();
Map<String, LineTokenizer> tokenizers = new HashMap<>(4);
tokenizers.put("HEA*", headerRecordTokenizer());
tokenizers.put("FOT*", footerRecordTokenizer());
tokenizers.put("NCU*", customerLineTokenizer());
tokenizers.put("BAD*", billingAddressLineTokenizer());
tokenizer.setTokenizers(tokenizers);
return tokenizer;
}
So may I ask what is this function tokenizer meaning like headerRecordTokenizer() and mapping with Hashmap<>(4) here means 4 lines each times reading the file?
I don't have the default value in like FOT*, NCU* like the example so how do I set to the spring-batch know that I want to stop at END to begin a new reading?
I'm quite new to spirng-batch so I have bit confused right here because as the document write I see no difference to how FlatFileItemReader work when all attributes in one line is stable as my first example.

Answer 1

So may I ask what is this function tokenizer meaning like headerRecordTokenizer() and mapping with Hashmap<>(4) here means 4 lines each times reading the file?
A logical record in the input file of the example you shared spans 4 physical lines, where each line has to be tokenized differently. This is where the composite line tokenizer comes to play: it will delegate line tokenization to 4 delegate pattern-matching tokenizers, one for each line format.
There is another example which is similar to your requirement, which you can find here: https://github.com/spring-projects/spring-batch/blob/main/spring-batch-samples/src/test/java/org/springframework/batch/sample/iosample/internal/MultiLineTradeItemReader.java
This one reads items between "BEGIN" and "END", similar to what you have. The complete example can be found here: https://github.com/spring-projects/spring-batch/tree/main/spring-batch-samples#multiline