How to pass data between Spring Batch jobs - spring-batch

Question

I'm familiar with how to pass data between steps in a Spring Batch job. But what happens when you job is composed of many smaller jobs? In the example below, I would like to set some data in the JobExecutionContext at the end of the first job, siNotificationJob. Then, that data could be read from the JobExecutionContext of StepExecutionContext in the next job, ciNotificationJob. Do I need to promote this data somehow? I can't seem to see the results in the Job Parameter Extractor defined in step 'ciNotificationJob' that I use to configure my job parameters.
Thoughts?
Andrew
<job id="notificationJob" xmlns="http://www.springframework.org/schema/batch">
<batch:step id="pn_step_0" next="pn-step-1">
<batch:job ref="siNotificationJob" job-launcher="jobLauncher"
job-parameters-extractor="jobParamsExtractor"/>
</batch:step>
<batch:step id="pn-step-1" next="pn-step-2">
<batch:job ref="ciNotificationJob" job-launcher="jobLauncher"
job-parameters-extractor="jobParamsExtractor"/>
</batch:step>
</job>

Answer 1

I was able to resolve this. I'll show you through example how I solved it. It was complicated but I think the end result is fairly easy to understand.
I have one overall job called 'notificationJob'. It has three steps that calls 3 different jobs (not steps). Each of these jobs can run independently, or be called from within the top level 'notificationJob'. Also, each sub-job has many steps. I'm not going to show all those steps here, but just wanted to highlight that these are complete jobs themselves with more multliple steps.
<job id="notificationJob" xmlns="http://www.springframework.org/schema/batch">
<batch:listeners>
<batch:listener ref="pn_job-parent-listener" />
</batch:listeners>
<batch:step id="pn_step-0" next="pn-step-1">
<batch:job ref="siNotificationJob" job-launcher="jobLauncher"
job-parameters-extractor="futureSiParamsExtractor"/>
</batch:step>
<batch:step id="pn-step-1" next="pn-step-2">
<batch:job ref="ciNotificationJob" job-launcher="jobLauncher"
job-parameters-extractor="futureCiParamsExtractor"/>
</batch:step>
<batch:step id="pn-step-2">
<batch:job ref="combineResultsJob" job-launcher="jobLauncher"
job-parameters-extractor="jobParamsExtractor"/>
</batch:step>
</job>
The key is being able to extract the results from one job and read them in the next job. Now, you could do this multiple ways. One way would be to output the result from one job into a DB or text file and then have the next job read from that file/table. Since I wasn't dealing with that much data, I passed the information around in memory. So, you'll notice the job-parameter-extractors. You can either rely on a built-in implementation of a paramter extractor, or you can implement your own. I actually use both. All they do is extract the value from the StepExecution and then we'll need to promote/move them to the next sub-job.
<bean id="jobParamsExtractor" class="org.springframework.batch.core.step.job.DefaultJobParametersExtractor">
<property name="keys">
<list>
<value>OUTPUT</value>
</list>
</property>
</bean>
<bean id="futureSiParamsExtractor" class="jobs.SlideDatesParamExtractor">
<property name="mode" value="FORWARD" />
<property name="addedParams">
<map><entry>
<key><value>outputName</value></key>
<value>FUTURE_SI_JOB_RESULTS</value>
</entry></map>
</property>
</bean>
<bean id="futureCiParamsExtractor" class="jobs.SlideDatesParamExtractor">
<property name="mode" value="FORWARD" />
<property name="addedParams">
<map><entry>
<key><value>outputName</value></key>
<value>FUTURE_CI_JOB_RESULTS</value>
</entry></map>
</property>
</bean>
Finally, you'll notice that there is a parent job listener. This is the magic that transfer the state from one job and makes it available to the next. Here is my implementation of the class that does that.
<bean id="pn_job-state-listener" class="jobs.JobStateListener">
<property name="parentJobListener" ref="pn_job-parent-listener" />
</bean>
<bean id="pn_job-parent-listener" class="cjobs.ParentJobListener">
</bean>
package jobs.permnotification;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
public class ParentJobListener implements JobExecutionListener
{
private JobExecution parentExecution;
#Override
public void beforeJob(JobExecution jobExecution)
{
this.parentExecution = jobExecution;
}
#Override
public void afterJob(JobExecution jobExecution)
{
// TODO Auto-generated method stub
}
public void setParentExecution(JobExecution parentExecution)
{
this.parentExecution = parentExecution;
}
public JobExecution getParentExecution()
{
return parentExecution;
}
}
package jobs.permnotification;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
public class JobStateListener implements JobExecutionListener
{
private ParentJobListener parentJobListener;
#Override
public void beforeJob(JobExecution jobExecution)
{
if(parentJobListener == null || parentJobListener.getParentExecution() == null) return;
passStateFromParentToJob(StepKey.FUTURE_SI_JOB_RESULTS.toString(), jobExecution);
passStateFromParentToJob(StepKey.FUTURE_CI_JOB_RESULTS.toString(), jobExecution);
passStateFromParentToJob(StepKey.OUTPUT.toString(), jobExecution);
}
#Override
public void afterJob(JobExecution jobExecution)
{
if(parentJobListener == null || parentJobListener.getParentExecution() == null) return;
//take state from child step and move it into the parent execution context
passStateFromJobToParent(StepKey.FUTURE_SI_JOB_RESULTS.toString(), jobExecution);
passStateFromJobToParent(StepKey.FUTURE_CI_JOB_RESULTS.toString(), jobExecution);
passStateFromJobToParent(StepKey.OUTPUT.toString(), jobExecution);
}
private void passStateFromJobToParent(String key, JobExecution jobExecution)
{
Object obj = jobExecution.getExecutionContext().get(key);
if(obj != null)
parentJobListener.getParentExecution().getExecutionContext().put(key, obj);
}
private void passStateFromParentToJob(String key, JobExecution jobExecution)
{
Object obj = parentJobListener.getParentExecution().getExecutionContext().get(key);
if(obj != null)
jobExecution.getExecutionContext().put(key, obj);
}
public void setParentJobListener(ParentJobListener parentJobListener)
{
this.parentJobListener = parentJobListener;
}
public ParentJobListener getParentJobListener()
{
return parentJobListener;
}
}

Answer 2

this is sort of a hack.... recommend you use spring integration instead..but see if this applies for your situation.
if you have the spring batch meta data tables set up, you can probably get at the data that you generate within each job if you query the tables for your latest job run. All your data in the job execution context is stored and can be queried.
spring batch meta tables