Spring batch to read a key= value and load it into DB - spring-batch

Question

I am new to spring batch and having a feed file with key param values in .txt format. I need to load the file into Mysql DB using spring batch. Is there any way to read a text file with key value message. Two rows are separated by an empty line and the delimiter is '='.
Sample File:
Name=Jack
Id=ADC12345
ClassId=7018
Rank=-326
Name=Gile
Id=FED12345
ClassId=7018
Rank=-32
Name, ID, ClassId and Rank are the column values.

Answer 1

Here's a working solution (you just need a blank line after the last record or it won't be read) :
1) Declare your business object :
public class Student {
private String name;
private String id;
private Integer classId;
private Integer rank;
// Getter + Setters
}
2) Declare a custom itemstreamreader to which you will delegate the actual FlatFileItemReader :
public class CustomMultiLineItemReader implements ItemStreamReader<Student> {
private FlatFileItemReader<FieldSet> delegate;
#Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
delegate.open(executionContext);
}
#Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
delegate.update(executionContext);
}
#Override
public void close() throws ItemStreamException {
delegate.close();
}
// Getter + Setters
}
3) Override its read method to manually map your multiline records :
public Student read() throws Exception {
Student s = null;
for (FieldSet line = null; (line = this.delegate.read()) != null;) {
if (line.getFieldCount() == 0) {
return s; // Record must end with footer
} else {
String prefix = line.readString(0);
if (prefix.equals("Name")) {
s = new Student(); // Record must start with header
s.setName(line.readString(1));
}
else if (prefix.equals("Id")) {
s.setId(line.readString(1));
}
else if (prefix.equals("ClassId")) {
s.setClassId(line.readInt(1));
}
else if (prefix.equals("Rank")) {
s.setRank(line.readInt(1));
}
}
}
return null;
}
4) Declare the reader in the step and configure it :
<bean class="xx.xx.xx.CustomMultiLineItemReader">
<property name="delegate">
<bean class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="file:${YOUR_FILE}"></property>
<property name="linesToSkip" value="0"></property>
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.PatternMatchingCompositeLineMapper">
<property name="tokenizers">
<map>
<entry key="*">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="="></property>
</bean>
</entry>
</map>
</property>
<property name="fieldSetMappers">
<map>
<entry key="*">
<bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
</entry>
</map>
</property>
</bean>
</property>
</bean>
</property>
</bean>
I used a PatternMatchingCompositeLineMapper to associate line content (here : *) with the corresponding lineTokenizer and lineMapper (even though it's useless in this case).
Then, the PassThroughFieldSetMapper lets the reader do the mapping, and the DelimitedLineTokenizer splits the line on the "=" character.

Answer 2

there are 2 challenges with this input format
start/end for a complete item
splitting the item in key/value pairs
one solution could be to use a custom RecordSeparatorPolicy and custom LineMapper, like
import java.util.HashMap;
import java.util.Map;
import org.junit.Test;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.separator.RecordSeparatorPolicy;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.core.io.ClassPathResource;
import org.springframework.validation.BindException;
public class ReaderKeyValueTest {
#Test
public void test() throws Exception {
FlatFileItemReader<Map<String, String>> reader = new FlatFileItemReader<Map<String, String>>();
reader.setResource(new ClassPathResource("keyvalue.txt"));
// custom RecordSeparatorPolicy
reader.setRecordSeparatorPolicy(new RecordSeparatorPolicy() {
#Override
public String preProcess(final String record) {
// empty line is added to the previous 'item'
if (record.isEmpty()) {
return record;
} else {
// line with content means it is part of an 'item', lets enhance it with adding a separator
return record + ",";
}
}
#Override
public String postProcess(final String record) {
return record;
}
#Override
public boolean isEndOfRecord(final String record) {
// the end of a record is marked with the last key/value pair for "Rank"
if (record.contains("Rank=")) {
return true;
} else {
return false;
}
}
});
DefaultLineMapper<Map<String, String>> lineMapper = new DefaultLineMapper<Map<String, String>>();
// the key/value pairs are separated with ',', so we can use the standard DelimitedLineTokenizer here
lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
lineMapper.setFieldSetMapper(new FieldSetMapper<Map<String, String>>() {
#Override
public Map<String, String> mapFieldSet(final FieldSet fieldSet) throws BindException {
Map<String, String> item = new HashMap<String, String>();
// split each "Key=Value" and add to the Map
for (int i = 0; i < fieldSet.getValues().length; i++) {
String[] entry = fieldSet.getValues()[i].split("=");
item.put(entry[0], entry[1]);
}
return item;
}
});
reader.setLineMapper(lineMapper);
reader.open(new ExecutionContext());
Map<String, String> item;
while ((item = reader.read()) != null) {
System.out.println(item.toString());
}
reader.read();
reader.close();
}
}
the sysout produces
{ClassId=7018, Id=ADC12345, Name=Jack, Rank=-326}
{ClassId=7018, Id=FED12345, Name=Gile, Rank=-32}