ETL implementation using Spring Batch - spring-batch

Question

I need to implement an ETL application for one of the projects am working on.
It has following steps:
Need to read from a table to retrieve some values that will be
passed in as Job parameters.
The returned object of the step 1 will be further used to retrieve
some data from a second table.
Then has to read from a flat file that will be used along with the
values from step 2. Apply the business logic. Then write to a table.
We are using Spring Data JPA, Spring integration.
The challenge I am facing is to read the values from a table to retrieve the parameters for the job then launch the job.
And then the output of step 2 has to be sent along with the File information for further processing.
I know how to implement the above steps independently but struggling to tie them from end to end.
Sharing any ideas to design the above would be great. Thanks in advance.

Answer 1

I'll try to give you some ideas for your differents points.
1 - Read table values and pass them as Job Parameters
I see 2 solutions here :
You could do a "manual" query (ie. without springbatch), and then do your business logic to pass the results as JobParameters (you just need a JobLauncher or a CommandLineJobRunner, see Springbatch Documentation §4.4) :
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean(jobName);
// Do your business logic and your database query here.
// Create your parameters
JobParameter parameter = new JobParameter(resultOfQuery);
// Add them to a map
Map<String, JobParameter> parameters = new HashMap<String, JobParameter>();
parameters.add("yourParameter", parameter);
// Pass them to the job
JobParameters jobParameters = new JobParameters(parameters);
JobExecution execution = jobLauncher.run(job, parameters);
The other solution would be to add a JobExecutionListener and override the method beforeJob to do your query and then save the results in the executionContext (which you can then access with : #{jobExecutionContext[name]}).
#Override
public void beforeJob(JobExecution jobExecution) {
// Do your business logic and your database query here.
jobExecution.getExecutionContext().put(key, value);
}
In each case, you can use a SpringBatch ItemReader to do your query. You can, for example, declare an item reader as a field for your listener (don't forget the setter) and configure it as such :
<batch:listener>
<bean class="xx.xx.xx.YourListener">
<property name="reader">
<bean class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"></property>
<property name="sql" value="${yourSQL}"></property>
<property name="rowMapper">
<bean class="xx.xx.xx.YourRowMapper"></bean>
</property>
</bean>
</property>
</bean>
</batch:listener>
2 - Read a table depending on results from previous step
Once more, you can use the JobExecutionContext to store and retreive data between steps. You can then implement a StepExecutionListener to override the method beforeStep and access StepExecution which will lead you to JobExecution.
3 - Send result from table reading along results of file reading
There is no "default" CompositeItemReader which would let you read from 2 sources at the same time, but I don't think that's what you actually want to do.
For your case, I would declare the "table reader" as the reader in a <batch:chunk> and then declare a custom ItemProcessor which would have another ItemReader field. This reader would be your FlatFileItemReader. You can then manually start the read and apply your business logic in the process method.