Can multi job Spring Batch app load minimum set of beans? [duplicate] - spring-batch

I have a Spring Batch project with multiple jobs (job A, job B, job C,...). When I run a particular job A, I got the log of the job A shows that all of the beans of job B, C,... are created too. Is there any way to avoid the creation of the other beans when job A is launched.
I have tried to use #Lazy annotation but it 's seem not working.
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
#Qualifier("springDataSource")
public DataSource springDataSource;
#Autowired
#Qualifier("batchJobDataSource")
public DataSource batchJobDataSource;
}
#Configuration
#PropertySource("classpath:partner.properties")
public class B extends BatchConfiguration {
#Value("${partnerId}")
private String partnerId;
#Lazy
#Bean
public Job ProcessB(JobCompletionNotificationListener listener) {
return jobBuilderFactory
.get("ProcessB")
.incrementer(new RunIdIncrementer())
.listener(listener)
.start(ProcessStepB())
.build();
}
#Lazy
#Bean
public Step (ProcessStepB() {
return stepBuilderFactory
.get("(ProcessStepB")
.<PartnerDTO, PartnerDTO> chunk(1)
.reader(getPartner())
.processor(process())
.writer(save())
.build();
}
#Lazy
#Bean(destroyMethod = "")
public Reader getPartner() {
return new Reader(batchJobDataSource,partnerId);
}
#Lazy
#Bean
public Processor process() {
return new Processor();
}
#Lazy
#Bean
HistoryWriter historyWriter() {
return new HistoryWriter(batchJobDataSource);
}
#Lazy
#Bean
UpdateWriter updateWriter() {
return new UpdateWriter(batchJobDataSource);
}
#Lazy
#Bean
public CompositeItemWriter<PartnerDTO> saveTransaction() {
List<ItemWriter<? super PartnerDTO>> delegates = new ArrayList<>();
delegates.add(updateWriter());
delegates.add(historyWriter());
CompositeItemWriter<PartnerDTO> itemWriter = new CompositeItemWriter<>();
itemWriter.setDelegates(delegates);
return itemWriter;
}
}
I have also put the #Lazy over the #Configuration but it does work too.

That should not be an issue. But here are a few ideas to try:
Use Spring profiles to isolate job beans
If you use Spring Boot 2.2+, try to activate the lazy bean initialization mode
Package each job in its own jar. This is the best option IMO.

Related

Multithreading with StoredProcedureItemReader

Is it possible to have multiple threads with StoredProcedureItemReader? I have done multithreading with PageReader but not sure if it will work with StoredProcedureItemReader.
Below is the job configuration for using a StoredProcedureReader. I have wrapped the reader in Thread safe reader. I want to use ThreadPoolTaskExecutor but not able to figure out how I can do partition for each thread with the Stored procedure.
***#Configuration
public class SpPocJobConfigurationMT {
private DataSource dataSource;
/**
* The Job builder factory.
*/
private JobBuilderFactory jobBuilderFactory;
/**
* The Jdbc template.
*/
#Autowired
JdbcTemplate jdbcTemplate;
/**
* The Step builder factory.
*/
private StepBuilderFactory stepBuilderFactory;
#Autowired
private BillingRecordAuditRepository billingRecordAuditRepository;
#Autowired
private StagingMortgageDataTxnRepository stagingMortgageDataTxnRepository;
private SystemRepository systemRepository;
#Autowired
public SpPocJobConfigurationMT(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory, SystemRepository systemRepository, DataSource dataSource) {
Assert.notNull(systemRepository, "SystemRepository cannot be null");
Assert.notNull(jobBuilderFactory, "JobBuilderFactory cannot be null");
Assert.notNull(stepBuilderFactory, "StepBuilderFactory cannot be null");
Assert.notNull(dataSource, "DataSource cannot be null");
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
this.systemRepository = systemRepository;
this.dataSource = dataSource;
}
#Bean
#Transactional
#Description(value = "")
public Job SpPocJobMT() throws Exception {
return jobBuilderFactory.get("spPocJobMT")
.start(spPocStepMT())
.build();
}
#Bean
public Step spPocStepMT() throws Exception {
return stepBuilderFactory.get("spPocStepMT")
.allowStartIfComplete(false)
.<StagingDataDto,StagingDataDto> chunk(20)
.reader(sybcSpReaderMT())
.processor(spPocProcessorMT())
.writer(spPocWriterMT())
// .taskExecutor(new ThreadPoolTaskExecutor ())
// .taskExecutor(new SimpleAsyncTaskExecutor())
.build();
}
#Bean
public SpPocWriter spPocWriterMT() {
return new SpPocWriter(this.billingRepository, this.stagingTxnRepository);
}
#Bean
public SpPocProcessor spPocProcessorMT() {
return new SpPocProcessor();
}
#Bean
#StepScope
public SynchronizedItemStreamReader sybcSpReaderMT() {
StoredProcedureItemReader reader = new StoredProcedureItemReader();
SqlParameter[] parameters = {new SqlParameter("#p_id", OracleTypes.NUMBER)
, new SqlOutParameter("#p_out_c1", OracleTypes.CURSOR)
, new SqlOutParameter("#p_out_c2", OracleTypes.CURSOR)
};
reader.setDataSource(dataSource);
reader.setProcedureName("SP_POC_FINAL");
reader.setRowMapper(new SPRowMapper());
reader.setRefCursorPosition(3);
reader.setPreparedStatementSetter(new MyItemPreparedStatementSetter());
reader.setParameters(parameters);
reader.setSaveState(false);
reader.setVerifyCursorPosition(false);
SynchronizedItemStreamReader synchronizedItemStreamReader = new SynchronizedItemStreamReader();
synchronizedItemStreamReader.setDelegate(reader);
return synchronizedItemStreamReader;
}
public class MyItemPreparedStatementSetter implements PreparedStatementSetter {
#Override
public void setValues(PreparedStatement ps) throws SQLException {
ps.setInt(1, 1);
((CallableStatement) ps).registerOutParameter(2, OracleTypes.CURSOR);
((CallableStatement) ps).registerOutParameter(3, OracleTypes.CURSOR);
}
}
}***
The StoredProcedureItemReader extends AbstractItemCountingItemStreamItemReader which is not thread-safe, please check its javadoc.
If you want to use the StoredProcedureItemReader in a multi-threaded step, you need to wrap it in a SynchronizedItemStreamReader or make it step-scoped.

Spring boot reading CSV file using "FlatFileItemReader" in Service method

I am going to develop an application where I am trying to read CSV file using Spring FlatFileItemReader object. I like to use a Service in where a method will be called to execute reading process. But I am not directly using configuration bean object of FlatFileItemReader and others. An example of my prototype bellow.
public void executeCsvReading(){
FlatFileItemReader<SomeModel> reader = new FlatFileItemReader<SomeModel>();
reader.setResource(new ClassPathResource("someFile.csv"));
reader.setLineMapper(new DefaultLineMapper<SomeModel>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[] { "somefield1", "somefield2" });
}
});
setFieldSetMapper(new BeanWrapperFieldSetMapper<SomeModel>() {
{
setTargetType(SomeModel.class);
}
});
}
});
// But how do I start this Job?
Job job = jobBuilderFactory
.get("readCSVFilesJob")
.incrementer(new RunIdIncrementer())
.start(step)
.build();
Step step = stepBuilderFactory.get("step1").<SomeModel, SomeModel>chunk(5)
.reader(reader)
.writer(new WriteItemsOn()) // Call WriteItemsOn class constructor
.build();
}
public class WriteItemsOn implements ItemWriter<User> {
#Override
public void write(List<? extends SomeModel> items) throws Exception {
// TODO Auto-generated method stub
for (SomeModel item : items) {
System.out.println("Item is " + item.someMethod()");
}
}
}
// But how do I start this Job?
To start the job, you need to use a JobLauncher. For example:
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
//jobLauncher.setJobRepository(yourJobRepository);
//jobLauncher.setTaskExecutor(yourTaskExecutor);
jobLauncher.afterPropertiesSet();
jobLauncher.run(job, new JobParameters());
However, with this approach, you will need to configure infrastructure beans required by Spring Batch (JobRepository, JobLauncher, etc) yourself.
I would recommend to use a typical Spring Batch job configuration and run your job from the method executeCsvReading. Here is an example:
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableBatchProcessing
public class MyJob {
private final JobBuilderFactory jobBuilderFactory;
private final StepBuilderFactory stepBuilderFactory;
public MyJob(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory) {
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
}
#Bean
public Step step() {
return stepBuilderFactory.get("step")
.tasklet((contribution, chunkContext) -> {
System.out.println("hello world");
return RepeatStatus.FINISHED;
})
.build();
}
#Bean
public Job job() {
return jobBuilderFactory.get("job")
.start(step())
.build();
}
}
With this job configuration in place, you can load the Spring application context and launch your job as follows:
public void executeCsvReading() {
ApplicationContext context = new AnnotationConfigApplicationContext(MyJob.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(Job.class);
jobLauncher.run(job, new JobParameters());
}
Note that loading the Spring application context can be done outside the method executeCsvReading so that it is not loaded each time this method is called. With this approach, you don't have to configure infrastructure beans required by Spring Batch yourself, they will be automatically created and added to the application context. It is of course possible to override them if needed.
Heads up: If you put the MyJob configuration class in the package of your Spring Boot app, Spring Boot will by default execute the job at startup. You can disable this behaviour by adding spring.batch.job.enabled=false to your application properties.
Hope this helps.

Spring Batch which configuration is going to run?

In my simple Spring Batch application, I have a configuration...
#Configuration
public class StepTransitionConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Step step1() {
...
}
#Bean
public Step step2() {
...
}
#Bean
public Job transitionJobSimpleNext() {
return jobBuilderFactory.get("transitionJobNext")
.start(step1())
.next(step2())
.build();
}
The only other class is
#SpringBootApplication
#EnableBatchProcessing
public class TransitionsApplication {
public static void main(String[] args) {
SpringApplication.run(TransitionsApplication.class, args);
}
}
So I started thinking how does that Application class know to use that particular configuration for the Batch. So I added another Configuration...
#Configuration
public class AnotherTransitionConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Step step4() {
...
}
#Bean
public Step step5() {
...
}
#Bean
public Job transitionJobSimpleNext() {
return jobBuilderFactory.get("transitionJobNext")
.start(step4())
.next(step5())
.build();
}
Now when I run the application, it always uses the second configuration even though both are on the classpath. Why is this?

I am getting error Table 'test.batch_job_instance' doesn't exist

I am new to Spring Batch. I have configured my job with inmemoryrepository. But still, it seems it is using DB to persist job Metadata.
My spring batch Configuration is :
#Configuration
public class BatchConfiguration {
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private JobBuilderFactory jobBuilder;
#Bean
public JobLauncher jobLauncher() throws Exception {
SimpleJobLauncher job =new SimpleJobLauncher();
job.setJobRepository(getJobRepo());
job.afterPropertiesSet();
return job;
}
#Bean
public PlatformTransactionManager getTransactionManager() {
return new ResourcelessTransactionManager();
}
#Bean
public JobRepository getJobRepo() throws Exception {
return new MapJobRepositoryFactoryBean(getTransactionManager()).getObject();
}
#Bean
public Step step1(JdbcBatchItemWriter<Person> writer) throws Exception {
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer).repository(getJobRepo())
.build();
}
#Bean
public Job job( #Qualifier("step1") Step step1) throws Exception {
return jobBuilder.get("myJob").start(step1).repository(getJobRepo()).build();
}
}
How to resolve above issue?
If you are using Sprint boot
a simple property in your application.properties will solve the issue
spring.batch.initialize-schema=ALWAYS
For a non-Spring Boot setup:This error shows up when a datasource bean is declared in the batch configuration. To workaround the problem I added an embedded datasource, since I didn't want to create those tables in the application database:
#Bean
public DataSource mysqlDataSource() {
// create your application datasource here
}
#Bean
#Primary
public DataSource batchEmbeddedDatasource() {
// in memory datasource required by spring batch
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
return builder.setType(EmbeddedDatabaseType.H2)
.addScript("classpath:schema-drop-h2.sql")
.addScript("classpath:schema-h2.sql")
.build();
}
The initialization scripts can be found inside the spring-batch-core-xxx.jar under org.springframework.batch.core package.Note I used an in-memory database but the solution is valid also for other database systems.
Those who face the same problem with MySql database in CentOS(Most Unix based systems).
Table names are case-sensitive in Linux. Setting lower_case_table_names=1 has solved the problem.
Find official document here
For those using versions greater then spring-boot 2.5 this worked inside of application.properties
spring.batch.jdbc.initialize-schema = ALWAYS
This solved my case:
spring.batch.jdbc.initialize-schema=ALWAYS

Spring Batch Integration using Java DSL / launching jobs

I've a working spring boot/batch projet containing 2 jobs.
I'm now trying to add Integration to poll files from a remote SFTP using only java configuration / java DSL, and then launch a job.
The file polling is working but I've no idea on how to launch a Job in my flow, despite reading these links :
Spring Batch Integration config using Java DSL
and
Spring Batch Integration job-launching-gateway
some code snippets:
#Bean
public SessionFactory SftpSessionFactory()
{
DefaultSftpSessionFactory sftpSessionFactory = new DefaultSftpSessionFactory();
sftpSessionFactory.setHost("myip");
sftpSessionFactory.setPort(22);
sftpSessionFactory.setUser("user");
sftpSessionFactory.setPrivateKey(new FileSystemResource("path to my key"));
return sftpSessionFactory;
}
#Bean
public IntegrationFlow ftpInboundFlow() {
return IntegrationFlows
.from(Sftp.inboundAdapter(SftpSessionFactory())
.deleteRemoteFiles(Boolean.FALSE)
.preserveTimestamp(Boolean.TRUE)
.autoCreateLocalDirectory(Boolean.TRUE)
.remoteDirectory("remote dir")
.regexFilter(".*\\.txt$")
.localDirectory(new File("C:/sftp/")),
e -> e.id("sftpInboundAdapter").poller(Pollers.fixedRate(600000)))
.handle("FileMessageToJobRequest","toRequest")
// what to put next to process the jobRequest ?
For .handle("FileMessageToJobRequest","toRequest") I use the one described here http://docs.spring.io/spring-batch/trunk/reference/html/springBatchIntegration.html
I would appreciate any help on that, many thanks.
EDIT after Gary comment
I've added, it doesn't compile -of course- because I don't understand how the request is propagated :
.handle("FileMessageToJobRequest","toRequest")
.handle(jobLaunchingGw())
.get();
}
#Bean
public MessageHandler jobLaunchingGw() {
return new JobLaunchingGateway(jobLauncher());
}
#Autowired
private JobLauncher jobLauncher;
#Bean
public JobExecution jobLauncher(JobLaunchRequest req) throws JobExecutionException {
JobExecution execution = jobLauncher.run(req.getJob(), req.getJobParameters());
return execution;
}
I've found a way to launch a job using a #ServiceActivator and adding this to my flow but I'm not sure it's good practice :
.handle("lauchBatchService", "launch")
#Component("lauchBatchService")
public class LaunchBatchService {
private static Logger log = LoggerFactory.getLogger(LaunchBatchService.class);
#Autowired
private JobLauncher jobLauncher;
#ServiceActivator
public JobExecution launch(JobLaunchRequest req) throws JobExecutionException {
JobExecution execution = jobLauncher.run(req.getJob(), req.getJobParameters());
return execution;
}
}
.handle(jobLaunchingGw())
// handle result
...
#Bean
public MessageHandler jobLaunchingGw() {
return new JobLaunchingGateway(jobLauncher());
}
where jobLauncher() is the JobLauncher bean.
EDIT
Your service activator is doing about the same as the JLG; it uses this code.
Your jobLauncher #Bean is wrong.
#Beans are definitions; they don't do runtime stuff like this
#Bean
public JobExecution jobLauncher(JobLaunchRequest req) throws JobExecutionException {
JobExecution execution = jobLauncher.run(req.getJob(), req.getJobParameters());
return execution;
}
Since you are already autowiring a JobLauncher, just use that.
#Autowired
private JobLauncher jobLauncher;
#Bean
public MessageHandler jobLaunchingGw() {
return new JobLaunchingGateway(jobLauncher);
}