#Bean
public Step step1(){
return stepBuilderFactory.get("checkInfo").<A, B>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
I have created this step called "checkInfo", and i have other steps with other name. In my database I have a table "STEPS" with the name of the step and if it is enabled or disabled.
So i have to chain only the steps enabled to my JOB.
#PostConstruct
public void getActiveSteps(){
stepsList = stepManagementRepository.findAllByActive(true);
for (StepManagement s : stepsList){
System.out.println(s.getDescription());
}
I get all of the active ones in this function. The problem is, how can I get the step i want by the name saved on my DB? (So then i can use the .next() in the job, only if the step is active)
#Bean
public Job runJob(){
SimpleJobBuilder jobBuilder = jobBuilderFactory.get("mainCalculationJob")
.incrementer(new RunIdIncrementer())
.start(step1());
return jobBuilder.build();
}
I solved getting bean by name:
#Bean(name = "checkInfo")
public Step step1(){
return stepBuilderFactory.get("checkInfo").<A, B>chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
#Bean
public Job runJob(){
SimpleJobBuilder jobBuilder = jobBuilderFactory.get("mainCalculationJob")
.incrementer(new RunIdIncrementer())
.start((Step) context.getBean("checkInfo"));
return jobBuilder.build();
}
Related
My Spring Batch Job configuration has 5 steps, all of which are identical except for the reader. Is there a way I can abstract out all of the other parts of the step into a "parent" step, so that I don't need to repeat everything? I know this can be done in XML, but I can't figure out the java equivalent.
Here's one of the steps:
public Step quarterlyStep(FileIngestErrorListener listener, ItemReader<DistributionItem> quarterlyReader) {
return stepBuilderFactory.get("quarterlyStep")
.<DistributionItem,DistributionItem>chunk(10)
.reader(quarterlyReader) // The only thing that changes among 5 different steps
.listener(listener.asReadListener())
.processor(processor())
.listener(listener.asProcessListener())
.writer(writer())
.listener(listener.asWriteListener())
.faultTolerant()
.skip(ValidationException.class)
.skip(ExcelFileParseException.class)
.build();
}
Here's the definition of one of the readers:
#Bean
#JobScope
public PoiItemReader<DistributionItem> yearEndReader(#Value("#{jobExecutionContext['filename']}") String filename) {
PoiItemReader<PortfolioFundsDistributionItem> reader = new PoiItemReader<>();
reader.setLinesToSkip(1);
reader.setRowMapper(yearEndRowMapper());
reader.setResource(new FileSystemResource(filename));
return reader;
}
You can do something like:
private StepBuilderFactory stepBuilderFactory;
private SimpleStepBuilder<Integer, Integer> createBaseStep(String stepName) {
return stepBuilderFactory.get(stepName)
.<Integer, Integer>chunk(5)
.processor(itemProcessor())
.writer(itemWriter());
}
#Bean
public Step step1(ItemReader<Integer> itemReader) {
return createBaseStep("step1")
.reader(itemReader)
.build();
}
#Bean
public Step step2(ItemReader<Integer> itemReader) {
return createBaseStep("step2")
.reader(itemReader)
.build();
}
This section reads in the file from our server, processes it, writes it out and archives it.
#Bean
public Step step1() {
log.info("Made if to step1");
System.out.println("Made it to Step 1");
return this.stepBuilderFactory.get("step1")
.<PaymentTransaction, PaymentTransaction>chunk(10)
.reader(paymentTransactionItemReader())
.writer(paymentTransactionItemWriter())
.build();
}
#Bean
public JobExecutionDecider decider() {
System.out.println("Made it to the decider");
return (jobExecution, stepExecution) -> new FlowExecutionStatus("Success"); }
#Bean
public FlowJobBuilder job() {
return jobBuilderFactory.get("BenefitIssuance")
.start(step1())
.next(decider())
.on("Success")
.end()
.build();
}
However when it reaches the build() step at the end, it loops back to the reader
As mentioned in the comments, I don't see why the job() method returns a FlowJobBuilder and not a Job. The following job definition does not loop back on the same step:
#Bean
public Job job() {
return jobs.get("job")
.start(step1())
.next(decider())
.on("Success")
.end()
.build()
.build();
}
I am trying to use a custom reader, processor and writer in each step:
public Step step1(StepBuilderFactory factory,
ItemReader reader,
ExpireAssessmentWriter writer,
AssessmentItemProcessor processor,
PlatformTransactionManager platformTransactionManager){
return stepBuilderFactory.get("step1")
.transactionManager(platformTransactionManager)
.<Assessment,Assessment>chunk(10)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
//update aggregate balance table
#Bean
public Step step2(StepBuilderFactory factory,
ItemReader reader,
BalanceItemWriter writer,
BalanceProcessor processor,
PlatformTransactionManager platformTransactionManager){
return stepBuilderFactory.get("step2")
.transactionManager(platformTransactionManager)
.<Assessment,Assessment>chunk(10)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
#Bean
public Step step3(StepBuilderFactory factory,
ItemReader<Assessment> reader,
CustomWriter3 writer,
CustomItemProcessor3 processor,
PlatformTransactionManager platformTransactionManager){
return stepBuilderFactory.get("step3")
.transactionManager(platformTransactionManager)
.<Assessment,Assessment>chunk(10)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
The first steps works fine but thats only when I leave this reader in the same class:
private static final String READER_QUERY = "SELECT * FROM TABLE1 WHERE COLUMN='TEST'";
#Bean
public JdbcCursorItemReader<Assessment> reader(DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<Assessment>()
.dataSource(dataSource)
.name("AssessmentUtilityReader")
.sql(READER_QUERY)
.rowMapper(new AssessmentMapper())
.build();
}
How can I create a custom reader for each of these steps that will read it's own query?
Can I create a custom reader that extends JdbcCursorItemReader
and returns this same snippet of code? :
#Bean
public JdbcCursorItemReader<Assessment> reader(DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<Assessment>()
.dataSource(dataSource)
.name("AssessmentUtilityReader")
.sql(READER_QUERY)
.rowMapper(new AssessmentMapper())
.build();
}
```
Since the item type is the same for all steps, you can create a method that accepts a query and returns an item reader:
public JdbcCursorItemReader<Assessment> getReader(DataSource dataSource, String query) {
return new JdbcCursorItemReaderBuilder<Assessment>()
.dataSource(dataSource)
.name("AssessmentUtilityReader") // can be passed as a parameter as well
.sql(query)
.rowMapper(new AssessmentMapper())
.build();
}
Then call this method in each step definition and pass the required query for each step.
To Turn your reader in a custom component, which can be autowired, add the following class:
#Component
public class AssessmentUtilityReader extends JdbcCursorItemReader<Assessment> {
public AssessmentUtilityReader(final DataSource dataSource) {
setName(getClass().getSimpleName());
setDataSource(dataSource);
setRowMapper(new AssessmentMapper());
// language=SQL
setSql(
"""
SELECT *
FROM TABLE1
WHERE COLUMN = 'TEST'
""");
}
}
Hint: The Comment (// language=SQL) is an hint for IntelliJ to use SQL highlighting in the following lines. It's optional.
Simply autowire in steps definition:
#Bean
public Step step3(StepBuilderFactory factory,
AssessmentUtilityReader<Assessment> assessmentUtilityReader,
CustomWriter3 writer,
CustomItemProcessor3 processor,
PlatformTransactionManager platformTransactionManager){
return stepBuilderFactory.get("step3")
.transactionManager(platformTransactionManager)
.<Assessment,Assessment>chunk(10)
.reader(assessmentUtilityReader)
.processor(processor)
.writer(writer)
.build();
}
I have a job(parent job) of jobs(child). In the jobexecution listener of parent job, I want to access all the details(job name, step names, read/write count etc.) of each child jobs. How can I achieve this?
Below is example code. In afterJob method of parentJobExecutionListener, I want to send an email(already have code for it), containing complete details of JOB1 and JOB2. Below code in afterJob method prints details of JOB-STEP1 and JOB-STEP2. How can I access details of child jobs(JOB1 and JOB2) and their steps?
public class ParentJobConfig {
#Bean
public Job job1() {
Job job = jobBuilderFactory.get("JOB1")
.incrementer(new RunIdIncrementer())
.start(step11())
.next(step12())
.build();
return job;
}
#Bean
public Job job2() {
Job job = jobBuilderFactory.get("JOB2")
.incrementer(new RunIdIncrementer())
.start(step21())
.next(step22())
.build();
return job;
}
#Bean
public Step jobStep1() {
Step step = stepBuilderFactory.get("JOB-STEP1")
.job(job1())
.parametersExtractor(new DefaultJobParametersExtractor())
.build();
return step;
}
#Bean
public Step jobStep2() {
Step step = stepBuilderFactory.get("JOB-STEP2")
.job(job2())
.parametersExtractor(new DefaultJobParametersExtractor())
.build();
return step;
}
#Bean
public Job parentJob() {
Job job = jobBuilderFactory.get("PARENT-JOB")
.incrementer(new RunIdIncrementer())
.start(jobStep1())
.next(jobStep2())
.listener(parentJobExecutionListener())
.build();
return job;
}
#Bean
public JobExecutionListener parentJobExecutionListener() {
return new ParentJobExecutionListener ();
}
}
public class ParentJobExecutionListener extends JobExecutionListenerSupport {
#Override
public void afterJob(JobExecution jobExecution) {
List<StepExecution> stepExecutions = (List<StepExecution>) jobExecution.getStepExecutions();
Collection<StepExecution> stepExecutions = jobExecution.getStepExecutions();
for (StepExecution stepExecution : stepExecutions) {
System.out.println("Step Name: " + stepExecution.getStepName());
System.out.println("Read Count: " + stepExecution.getReadCount());
System.out.println("Skip Count: " + stepExecution.getSkipCount());
System.out.println("Write Count: " + stepExecution.getWriteCount());
}
}
}
I don't see an obvious way to get access to the child job executions created here: https://github.com/spring-projects/spring-batch/blob/d8fc58338d3b059b67b5f777adc132d2564d7402/spring-batch-core/src/main/java/org/springframework/batch/core/step/job/JobStep.java#L119 (Only the status is reflected on the step execution).
What you can do is use a job explorer in your listener and query the repository for child job executions.
I am using JdbcPagingItemReader as below,
#Bean
public ItemReader<RemittanceVO> reader() {
JdbcPagingItemReader<RemittanceVO> reader = new JdbcPagingItemReader<RemittanceVO>();
reader.setDataSource(dataSource);
reader.setRowMapper(new RemittanceRowMapper());
reader.setQueryProvider(queryProvider);
reader.setPageSize(100);
return reader;
}
#Bean
public PagingQueryProvider queryProvider() throws Exception{
SqlPagingQueryProviderFactoryBean queryProviderBean= new SqlPagingQueryProviderFactoryBean();
queryProviderBean.setDataSource(dataSource);
queryProviderBean.setSelectClause(Constants.REMITTANCES_SELECT_CLAUSE);
queryProviderBean.setFromClause(Constants.REMITTANCES_FROM_CLAUSE);
queryProviderBean.setWhereClause(Constants.REMITTANCES_WHERE_CLAUSE);
queryProviderBean.setSortKey(Constants.REMITTANCES_SORT_KEY);
PagingQueryProvider queryProvider = queryProviderBean.getObject();
return queryProvider;
}
As of now, I launch job as below ( as I am very new to Spring batch )
JobLauncher jobLauncher = (JobLauncher) ctx.getBean("jobLauncher");
Job job = (Job) ctx.getBean("runRCMatcher");
try {
JobExecution execution = jobLauncher.run(job, new JobParameters());
}catch (Exception e) {
e.printStackTrace();
}
I am running this app as SpringBoot app. It fetches first 100 records successfully and hands over to processor and then next query fails. Query fails because sort key value has not been placed in it. This is there in query , AND ((REMIT_ID > ?)) ORDER BY REMIT_ID ASC FETCH FIRST 100 ROWS ONLY;
Where am I wrong?
My DB is DB2 so I guess it should be using - Db2PagingQueryProvider
Step & Job are defined as ,
#Bean
public Step step1(StepBuilderFactory stepBuilderFactory,
ItemReader<RemittanceVO> reader, ItemWriter<RemittanceClaimVO> writer,
ItemProcessor<RemittanceVO, RemittanceClaimVO> processor) {
return stepBuilderFactory.get("step1")
.<RemittanceVO, RemittanceClaimVO> chunk(100).reader(reader)
.processor(processor).writer(writer).build();
}
#Bean
public Job runRCMatcher(JobBuilderFactory jobs, Step s1) {
return jobs.get("RCMatcher")
.incrementer(new RunIdIncrementer())
.flow(s1)
.end()
.build();
}
Sort key specified is table column name - Constants.REMITTANCES_SORT_KEY and that is a primary key of table and of type BIGINT