Spring Batch - Call as webservice - spring-batch

I have a Spring Batch job. I am new to Spring Batch and have always been called via the CommandLineJobRunner.
This is what my call looks like:
org.springframework.batch.core.launch.support.CommandLineJobRunner spring-batch-myProject.xml SpringJobBean.MyProjectImportDataJob
Now I have to call my batch job from within a webservice (Spring MVC). In my endpoint this is the invoke call. I need to call the batch job in the if statement. How would I do this? I read about JobLauncher...but not sure how to tell it what to launch?
protected Object invokeInternal(Object aObj) throws Exception {
RunDataProcessingImportRequest request = (RunDataProcessingImportRequest) aObj;
RunDataProcessingImportResponse response = new RunDataProcessingImportResponse();
if (request.getDataProcessingType().equals(PROJECT_TYPE)){
response.setResultCd(1);
} else {
response.setResultCd(0);
response.setErrorCode(1l);
response.setErrorMessage("Incorrect process type");
}
return response;
}

The answer to this really depends on the version of Spring Batch you're using.
If you're using 2.0.x or older you can use Spring Batch Admin to provide REST endpoints for starting/stopping/etc jobs. All you need to do is add the jars to your app and provide a small amount of configuration.
If you're using 2.2.x or newer and are allowed to use the snapshot versions of Spring Batch Admin, the same applies as mentioned above.
If you're not interested in using Spring Batch Admin, you'll need to write your own endpoint and launch the job from there. However, it should be fairly trivial (I haven't tested the code below):
#Controller
public class JobLaunchingController {
#Autowire
JobLauncher jobLauncher;
#Autowire
JobRegistry jobRegistry;
#RequestMapping("/launch")
public #ResponseBody JobExecution launch(
#RequestParam(value="name", required=true) String name,
#RequestParam(value="params", required=false) String params) {
Job job = jobRegistry.getJob(name);
JobParametersBuilder paramsBuilder = new JobParametersBuilder();
if(params != null) {
// parse job parameters
}
return jobLauncher.run(job, paramsBuilder.toJobParameters());
}
}
The above code assumes you have more than one job to provide the ability to execute. If not, you could just #Autowire the Job itself into your controller if you wanted (instead of the JobRegistry).
You can read more about REST services in Spring here: https://spring.io/guides/gs/rest-service/
You can read more about the JobLauncher here: http://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/core/launch/JobLauncher.html
Finally, you can read more about the JobRegistry in section 4.6.2 here: http://docs.spring.io/spring-batch/reference/html/configureJob.html

Related

Role of PlatformTransactionManager with Spring batch 5

The documentation is not very clear about the role of PlatformTransactionManager in steps configuration.
First, stepBuilder.tasklet and stepBuilder.chunk requires a PlatformTransactionManager as second parameter while the migration guide says it is now required to manually configure the transaction manager on any tasklet step definition (...) This is only required for tasklet steps, other step types do not require a transaction manager by design..
More over, in the documentation the transactionManager is injected via a method parameter:
/**
* Note the TransactionManager is typically autowired in and not needed to be explicitly
* configured
*/
But the transactionManager created by Spring Boot is linked to the DataSource created by Spring Boot based on spring.datasource.url. So with autoconfiguration, the following beans works together: dataSource, platformTransactionManager, jobRepository. It makes sense for job and step executions metadata management.
But unless readers, writers and tasklet works with this default DataSource used by JobOperator, the auto configured transactionManager must not be used for the steps configuration. Am I right ?
Tasklets or a chunk oriented steps will often need another PlatformTransactionManager:
if a step writes data in a specific db it needs a specific DataSource (not necessarily declared as bean otherwise the JobRepository will use it) and a specific PlatformTransactionManager linked to this DataSource
if a step writes data in a file or send message to a MOM, the ResourcelessTransactionManager is more appropriate. This useful implementation is not mentioned in the documentation.
As far as I understand, the implementation of PlatformTransactionManager for a step depends on where the data are written and has nothing to do with the transactionManager bean used by the JobOperator Am I right ?
Example:
var builder = new StepBuilder("step-1", jobRepository);
PlatformTransactionManager txManager = new ResourcelessTransactionManager();
return builder.<Input, Output> chunk(10, txManager)
.reader(reader())
.processor(processor())
.writer(writer()/*a FlatFileItemWriter*/)
.build();
or
#SpringBootApplication
#EnableBatchProcessing
public class MyJobConfiguration {
private DataSource dsForStep1Writer;
public MyJobConfiguration(#Value("${ds.for.step1.writer.url"} String url) {
this.dsForStep1Writer = new DriverManagerDataSource(url);
}
// reader() method, processor() method
JdbcBatchItemWriter<Output> writer() {
return new JdbcBatchItemWriterBuilder<Output>()
.dataSource(this.dsForStep1Writer)
.sql("...")
.itemPreparedStatementSetter((item, ps)->{/*code*/})
.build();
}
#Bean
Step step1(JobRepository jobRepository) {
var builder = new StepBuilder("step-1", jobRepository);
var txManager = new JdbcTransactionManager(this.dsForStep1Writer);
return builder.<Input, Output> chunk(10, txManager)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
// other methods
}
Is that correct ?
Role of PlatformTransactionManager with Spring batch 5
The role of the transaction manager did not change between v4 and v5. I wrote an answer about this a couple of years ago for v4, so I will update it for v5 here:
In Spring Batch, there are two places where a transaction manager is used:
In the proxies created around the JobRepository/JobExplorer to create transactional methods when interacting with the job repository/explorer
In each step definition to drive the step's transaction
Typically, the same transaction manager is used in both places, but this is not a requirement. It is perfectly fine to use a ResourcelessTransactionManager with the job repository to not store any meta-data and a JpaTransactionManager in the step to persist data in a database.
Now in v5, #EnableBatchProcessing does not register a transaction manager bean in the application context anymore. You either need to manually configure one in the application context, or use the one auto-configured by Spring Boot (if you are a Spring Boot user).
What #EnableBatchProcessing will do though is look for a bean named transactionManager in the application context and set it on the auto-configured JobRepository and JobExplorer beans (this is configurable with the transactionManagerRef attribute). Again, this transaction manager bean could be manually configured or auto-configured by Boot.
Once that in place, it is up to you to set that transaction manager on your steps or not.

Why isn't JobScope and StepScope available from an MVC thread?

I'm getting Error creating bean with name 'scopedTarget.scopedTarget.processVlsCasesJob': Scope 'job' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton from a job factory class. The factory is where the job and step beans are created in the correct job/step scopes from a bean invoked during main application start up.
#Component("processVlsCasesJobFactory")
public class ProcessVlsCasesJobFactoryImpl
extends BatchJobFactoryAncestorImpl
implements ProcessVlsCasesJobFactory {
...
#Bean
#Scope(scopeName = "job", proxyMode = ScopedProxyMode.INTERFACES)
public ProcessVlsCasesJob processVlsCasesJob() {
return new ProcessVlsCasesJobImpl();
}
...
#Bean
#Scope(scopeName = "step", proxyMode = ScopedProxyMode.INTERFACES)
public ProcessVlsCasesProcessCases processVlsCasesProcessCases() {
return new ProcessVlsCasesProcessCasesImpl();
}
...
// other bean methods creating the step objects
Any attempt to allow Spring to auto-register any bean in the Job/Steps scope fails with that type of error. If those scopes are only available when (I guess) a job is running, how do I "create" the bean in the scope from the thread of the main MVC application running in Tomcat?
Why isn't JobScope and StepScope available from an MVC thread?
Those are custom scopes specific to Spring Batch, they are not part of Spring MVC. You need to specifically register them (or use #EnableBatchProcessing to have them automatically registered)
how do I "create" the bean in the scope from the thread of the main MVC application running in Tomcat?
The main thread (processing the web request) should call a JobLauncher configured with an asynchronous TaskExecutor so that the batch job is executed in a separate thread. Please see the Running Jobs from within a Web Container section which provides more details and a code example of how to do that.
I finally found the answer: #EnableBatchProcessing doesn't work within an MVC application context. In the #Configuration bean I created to configure SB (with DB2) and set up all the SB beans (like jobLauncher), I added:
jobScope = new JobScope();
jobScope.setAutoProxy(Boolean.FALSE);
jobScope.setName(JobScoped.SCOPE_NAME);
((ConfigurableBeanFactory)applicationContext.getAutowireCapableBeanFactory())
.registerScope(JobScoped.SCOPE_NAME, jobScope);
stepScope = new StepScope();
stepScope.setAutoProxy(Boolean.FALSE);
stepScope.setName(StepScoped.SCOPE_NAME);
((ConfigurableBeanFactory)applicationContext.getAutowireCapableBeanFactory())
.registerScope(StepScoped.SCOPE_NAME, stepScope);
Then the two scopes were finally available at run time and the job/step scoped beans were registered at deployment and ran properly.
Was #EBP added as part of Spring Boot? Is it only supposed to be used via a command line tool?

Spring Boot with application managed persistence context

I am trying to migrate an application from EJB3 + JTA + JPA (EclipseLink). Currently, this application makes use of application managed persistent context due to an unknown number of databases on design time.
The application managed persistent context allows us to control how to create EntityManager (e.g. supply different datasources JNDI to create proper EntityManager for specific DB on runtime).
E.g.
Map properties = new HashMap();
properties.put(PersistenceUnitProperties.TRANSACTION_TYPE, "JTA");
//the datasource JNDI is by configuration and without prior knowledge about the number of databases
//currently, DB JNDI are stored in a externalized file
//the datasource is setup by operation team
properties.put(PersistenceUnitProperties.JTA_DATASOURCE, "datasource-jndi");
properties.put(PersistenceUnitProperties.CACHE_SHARED_DEFAULT, "false");
properties.put(PersistenceUnitProperties.SESSION_NAME, "xxx");
//create the proper EntityManager for connect to database decided on runtime
EntityManager em = Persistence.createEntityManagerFactory("PU1", properties).createEntityManager();
//query or update DB
em.persist(entity);
em.createQuery(...).executeUpdate();
When deployed in a EJB container (e.g. WebLogic), with proper TransactionAttribute (e.g. TransactionAttributeType.REQUIRED), the container will take care of the transaction start/end/rollback.
Now, I am trying to migrate this application to Spring Boot.
The problem I encounter is that there is no transaction started even after I annotate the method with #Transactional(propagation = Propagation.REQUIRED).
The Spring application is packed as an executable JAR file and run with embadded Tomcat.
When I try to execute those update APIs, e.g. EntityManager.persist(..), EclipseLink always complains about:
javax.persistence.TransactionRequiredException: 'No transaction is currently active'
Sample code below:
//for data persistence
#Service
class DynamicServiceImpl implements DynamicService {
//attempt to start a transaction
#Transactional(propagation = Propagation.REQUIRED)
public void saveData(DbJndi, EntityA){
//this return false that no transaction started
TransactionSynchronizationManager.isActualTransactionActive();
//create an EntityManager based on the input DbJndi to dynamically
//determine which DB to save the data
EntityManager em = createEm(DbJndi);
//save the data
em.persist(EntityA);
}
}
//restful service
#RestController
class RestController{
#Autowired
DynamicService service;
#RequestMapping( value = "/saveRecord", method = RequestMethod.POST)
public #ResponseBody String saveRecord(){
//save data
service.saveData(...)
}
}
//startup application
#SpringBootApplication
class TestApp {
public static void main(String[] args) {
SpringApplication.run(TestApp.class, args);
}
}
persistence.xml
-------------------------------------------
&ltpersistence-unit name="PU1" transaction-type="JTA">
&ltproperties>
&lt!-- comment for spring to handle transaction??? -->
&lt!--property name="eclipselink.target-server" value="WebLogic_10"/ -->
&lt/properties>
&lt/persistence-unit>
-------------------------------------------
application.properties (just 3 lines of config)
-------------------------------------------
spring.jta.enabled=true
spring.jta.log-dir=spring-test # Transaction logs directory.
spring.jta.transaction-manager-id=spring-test
-------------------------------------------
My usage pattern does not follow most typical use cases (e.g. with known number of DBs - Spring + JPA + multiple persistence units: Injecting EntityManager).
Can anybody give me advice on how to solve this issue?
Is there anybody who has ever hit this situation that the DBs are not known in design time?
Thank you.
I finally got it work with:
Enable tomcat JNDI and create the datasource JNDI to each DS programmatically
Add transaction stuff
com.atomikos:transactions-eclipselink:3.9.3 (my project uses eclipselink instead of hibernate)
org.springframework.boot:spring-boot-starter-jta-atomikos
org.springframework:spring-tx
You have pretty much answered the question yourself: "When deployed in a EJB container (e.g. WebLogic), with proper TransactionAttribute (e.g. TransactionAttributeType.REQUIRED), the container will take care of the transaction start/end/rollback".
WebLogic is compliant with the Java Enterprise Edition specification which is probably why it worked before, but now you are using Tomcat (in embedded mode) which are NOT.
So you simply cannot do what you are trying to do.
This statement in your persistence.xml file:
<persistence-unit name="PU1" transaction-type="JTA">
requires an Enterprise Server (WebLogic, Glassfish, JBoss etc.)
With Tomcat you can only do this:
<persistence-unit name="PU1" transaction-type="RESOURCE_LOCAL">
And you need to handle transactions by your self:
myEntityManager.getTransaction.begin();
... //Do your transaction stuff
myEntityManager.getTransaction().commit();

Spring Batch - How to keep job running when exception occur?

I try to write example app using spring batch:
#Bean
public Step testStep() {
return steps.get("testStep").<String,String>chunk(1)
.reader(testReader())
.processor(testProcessor())
.writer(testWriter())
.listener(testListener())
.build();
}
When I throw exception in reader or processor or writer, the job stopped with status FAILED. How can I make job ignore exception and keep running.
I'm not use any xml config, just annotation and class. Please give me a hint or link.
Thanks for any support!
Edit: Can we add skip dynamically, like I post in answer below.

Is it possible to write JUnit tests that are agnostic to your JAX-RS implementation?

I wrote a REST web service using JAX-RS that knows nothing about the specific JAX-RS implementation I chose. I happen to be using TomEE which means my JAX-RS implementation is ApacheCXF.
I'd like to write unit tests for the web service that also know nothing about the JAX-RS implementation. Is this possible? So far every example I've found involves using classes from a specific JAX-RS implementation (JAXRSClientFactory for ApacheCXF, Jersey Test Framework, etc).
I've started experimenting with tomee-embedded and am able to test my EJB's but it doesn't seem to startup the REST services.
My solution was to use Arquillian paired with an Embedded TomEE. Arquillian provides a ton of functionality but I'm only using it to start/stop the Embedded TomEE. Therefore, all I needed to do was add this to my pom.xml:
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>arquillian-tomee-embedded</artifactId>
<version>${tomee.version}</version>
<scope>test</scope>
</dependency>
Then I could write a JUnit test with a little extra Arquillian stuff and plain JAX-RS:
#RunWith(Arquillian.class)
public class MyServiceIT {
#ArquillianResource
private URL webappUrl;
#Deployment()
public static WebArchive createDeployment() {
return ShrinkWrap.create(WebArchive.class)
.addClasses(MyService.class)
.addAsWebInfResource("META-INF/persistence.xml") //Refers to src/main/resources/META-INF/persistence.xml
.addAsWebInfResource("test-resources.xml", "resources.xml") //Refers to src/test/resources/test-resources.xml
.addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml");
}
#Test
public void randomTest() throws URISyntaxException {
//Get data from the web service.
Client client = ClientBuilder.newClient();
WebTarget webTarget = client.target(webappUrl.toURI().resolve("myentity"));
Response response = webTarget.request(MediaType.APPLICATION_JSON).get();
int status = response.getStatus();
List<MyEntity> myEntities = response.readEntity(new GenericType<List<MyEntity>>() {});
//Perform some tests on the data
}
}