Standalone SpringBatch reading from DB - spring-batch

Question

Im very new to Spring batch, trying to make a standalone spring batch (Without springboot) that reading from MySql DB and write the results in CSV.
I tried googling and nothing worked out. I also can't get to understand the itemWriter/Itemreader creating are they just methods or classes inherit Writer/Reader? the read/write method what are their inputs/outputs ? I can't get any clear rules about this.
The springBatch can never be implanted without SpringBoot? if no so how?
Why I want standalone Springbatch cause I just want a main method to trigger the job. And I don't want a web app to call a certain endpoint to trigger the job.
The injection works without Springboot or its mandatory for the injection to work?
Also The jobLuncher is it injected or configured by xml files? cause sometimes I got error of jobLuancher is null after injecting it.
MY configuration file is :
import com.techtalk.debu.batch.entity.Employee;
import org.springframework.batch.core.*;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.database.JdbcCursorItemReader;
import org.springframework.batch.item.database.builder.JdbcCursorItemReaderBuilder;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.builder.FlatFileItemWriterBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.core.io.FileSystemResource;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType;
import org.springframework.jdbc.support.JdbcTransactionManager;
import javax.sql.DataSource;
#EnableBatchProcessing(dataSourceRef = "batchDataSource", transactionManagerRef = "batchTransactionManager")
public class LoadController {
#Bean
public DataSource batchDataSource() {
return new EmbeddedDatabaseBuilder().setType(EmbeddedDatabaseType.H2)
.addScript("/org/springframework/batch/core/schema-h2.sql")
.generateUniqueName(false).build();
}
#Bean
public DataSource dataSource() {
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url("jdbc:mysql://localhost:3311/new");
dataSourceBuilder.username("sa");
dataSourceBuilder.password("12345678908");
return dataSourceBuilder.build();
}
#Bean
public JdbcTransactionManager batchTransactionManager(DataSource dataSource) {
return new JdbcTransactionManager(dataSource);
}
public static void main(String e []) throws Exception {
JobParameters jobParameters = new JobParametersBuilder().addLong("time", System.currentTimeMillis()).toJobParameters();
ApplicationContext context = new AnnotationConfigApplicationContext(LoadController.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
JobExecution jobExecution = jobLauncher.run(context.getBean(Job.class), jobParameters);
System.out.println("Job Exit Status : " + jobExecution.getStatus());
}
#Bean
public Step step(JobRepository jobRepository, JdbcTransactionManager transactionManager) {
return new StepBuilder("step", jobRepository)
.<Employee, Employee>chunk(5, transactionManager)
.reader(itemReader())
.writer(itemWriter())
.processor(new CSVItemProcessor())
.build();
}
#Bean
public JdbcTransactionManager transactionManager(DataSource dataSource) {
return new JdbcTransactionManager(dataSource);
}
#Bean
public Job job(JobRepository jobRepository, JdbcTransactionManager transactionManager) {
return new JobBuilder("job")
.start(step(jobRepository, transactionManager))
.build();
}
#Bean
public JdbcCursorItemReader<Employee> itemReader() {
String sql = "select * from person";
return new JdbcCursorItemReaderBuilder<Employee>()
.name("personItemReader")
.dataSource(dataSource())
.sql(sql)
.beanRowMapper(Employee.class)
.build();
}
#Bean
public FlatFileItemWriter<Employee> itemWriter() {
return new FlatFileItemWriterBuilder<Employee>()
.resource(new FileSystemResource("persons.csv"))
.name("personItemWriter")
.delimited()
.names("id", "name")
.build();
}
}
and my pom file is:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.0.2</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.techtalk.debu</groupId>
<artifactId>Spring-Batch-CSV-to-DB-Project</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>spring-batch-CSV-To_DB-Write-Example-For-Beginners</name>
<description>Demo project for Beginners to understand Spring Boot</description>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>3.0.2</version>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>5.0.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.21</version>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-infrastructure</artifactId>
<version>5.0.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
when I run my code I got this error:
Failed to instantiate [org.springframework.batch.core.Job]: Factory
method 'job' threw exception with message:
java.lang.IllegalStateException: JobRepository must be set
Also I have some questions:
reader()/writer() this function should return what? what if I want the reader to read from another DataSource what should I do?
The data source defining in my application.properties by default is used to save job meta data or to apply the spring batch work?

Answer 1

You can run a standalone Spring Batch application without Spring Boot. The reference documentation contains a code sample for that. The javadoc of EnableBatchProcessing contains an example as well. You also have the samples which are not based on Spring Boot.
Here is a quick example for a non Boot standalone Spring Batch app with a main method:
package org.springframework.batch.sample;
import javax.sql.DataSource;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.database.JdbcCursorItemReader;
import org.springframework.batch.item.database.builder.JdbcCursorItemReaderBuilder;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.builder.FlatFileItemWriterBuilder;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabase;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseBuilder;
import org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType;
import org.springframework.jdbc.support.JdbcTransactionManager;
#Configuration
#EnableBatchProcessing
public class MyJobConfiguration {
#Bean
public JdbcCursorItemReader<Person> itemReader() {
String sql = "select * from person";
return new JdbcCursorItemReaderBuilder<Person>()
.name("personItemReader")
.dataSource(dataSource())
.sql(sql)
.beanRowMapper(Person.class)
.build();
}
#Bean
public FlatFileItemWriter<Person> itemWriter() {
return new FlatFileItemWriterBuilder<Person>()
.resource(new FileSystemResource("persons.csv"))
.name("personItemWriter")
.delimited()
.names("id", "name")
.build();
}
#Bean
public Job job(JobRepository jobRepository, JdbcTransactionManager transactionManager) {
return new JobBuilder("job", jobRepository)
.start(new StepBuilder("step", jobRepository)
.<Person, Person>chunk(5, transactionManager)
.reader(itemReader())
.writer(itemWriter())
.build())
.build();
}
#Bean
public DataSource dataSource() {
EmbeddedDatabase embeddedDatabase = new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.HSQL)
.addScript("/org/springframework/batch/core/schema-hsqldb.sql")
.build();
JdbcTemplate jdbcTemplate = new JdbcTemplate(embeddedDatabase);
jdbcTemplate.execute("create table person (id int primary key, name varchar(20));");
for (int i = 1; i <= 10; i++) {
jdbcTemplate.execute(String.format("insert into person values (%s, 'foo%s');", i, i));
}
return embeddedDatabase;
}
#Bean
public JdbcTransactionManager transactionManager(DataSource dataSource) {
return new JdbcTransactionManager(dataSource);
}
static class Person {
private int id;
private String name;
public Person() {
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String toString() {
return "Person{id=" + id + ", name='" + name + '\'' + '}';
}
}
public static void main(String[] args) throws Exception {
ApplicationContext context = new AnnotationConfigApplicationContext(MyJobConfiguration.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(Job.class);
jobLauncher.run(job, new JobParameters());
}
}
You just need to make sure to use Spring Batch 5.0.0 and have Hsqldb in the classpath.
EDIT: based on question update
rying to make a standalone spring batch (Without springboot) that reading from MySql DB and write the results in CSV.
I updated the sample to something like what you are trying to do. You just need to update the datasource to point to your MySQL database.
reader()/writer() this function should return what?
Those should return a type implementing ItemReader and ItemWriter repectively.
what if I want the reader to read from another DataSource what should I do?
In this case, you can configure your reader to read from that datasource, as shown in the example above. The datasource could be different from the one storing the batch meta-data.
CSVItemProcessor is a class implementing ItemProcessor <Employee, Employee> and having the process method but the reader not accepted this code
Your CSVItemProcessor should implement ItemProcessor<Employee, Employee> and your step should be declared as follows:
#Bean
public Step step() {
return new StepBuilder("MY Step")
.<Employee, Employee>chunk(10) // note the explicit input and output type definition here, which is missing in your code
.reader(reader())
.processor(new CSVItemProcessor())
.writer(writer())
.build();
}
Moreover, you need to pass the job repository and transaction manager to the step as shown in the complete example above, something like:
#Bean
public Step step(JobRepository jobRepository, JdbcTransactionManager transactionManager) {
return new StepBuilder("MY Step", jobRepository)
.<Employee, Employee>chunk(10, transactionManager) // note the explicit input and output type definition here, which is missing in your code
.reader(reader())
.processor(new CSVItemProcessor())
.writer(writer())
.build();
}