Spring Batch - Comma separated values - Save in Data Base - spring-batch

Question

I have a file which contains list of values (user IDs) separated by comma(“,”) as follows.
111, 222, 333, 444, 555, 777 …………
The file contains millions of such records and I wanted to save these values into a single column in a table in RDBMS.
I tried to use DelimitedLineTokenizer for parsing data.
The issue is that “DelimitedLineTokenizer” considers only one entry in a single line, and rest of the values are ignored.The first entry ("111") is saved and rest of the values in the same line are ignored.If there is a second line , the first element in the second line is saved and rest are ignored.
Is there a way to tokenize all the comma separated values from a single line and save all of them into DB?
The query is a s follows.
INSERT INTO users (id) VALUES (: userid).
I used the following code to parse the file and save it in DB.
public FlatFileItemReader<User> reader() {
FlatFileItemReader<User> reader = new FlatFileItemReader<User>();
DelimitedLineTokenizer reader = new DelimitedLineTokenizer(",");
reader.setNames(new String[] {“userid”});
blah…blah….blah….
reader.setLineMapper(new DefaultLineMapper<User>() {
{
setLineTokenizer(reader);
setFieldSetMapper(new BeanWrapperFieldSetMapper<User>() {
{
setTargetType(User.class);
}
});
}
});
return reader;
}
#Bean
public UserItemProcessor processor() {
return new UserItemProcessor();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener) {
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(step1()).end().build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").<User, User> chunk(5).reader(reader()).processor(processor())
.writer(writer()).build();
}

Answer 1

Basically, you have two delimiters for target object - comma & new line. So either you writer a custom reader that works on both delimiters or you need to pre process your file to bring it to standard format.
In my opinion, you are better off by pre processing your file to replace all comma with new line character.
You might retain original file as is and create pre processed data in a new temporary file.
You can either do that as a separate spring batch step ( not recommended due to file size ) or if its going to be a scheduled job then probably, in your kick off script.
Replace comma with newline in java
How to break lines at a specific character in Notepad++?
Notepad++ find and replace string with a new-line
Replace comma with new line in a text file using tr in Linux