Spring batch, manage rollback for transactional items done in item processor - spring-batch

This involves using a JSR job, so I do not have access to use the tasklet definition in the job specification. The direction I've been given is to support only JSR 352 features as far as job specification goes.
Here is an example of a job:
<job id="xmlWriter" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
<step id="firstStep">
<chunk item-count="2">
<reader ref="DelimitedFlatFileReader">
<properties>
<!-- Reads in from file Test.csv -->
<property name="fileNameAndPath" value="#{jobParameters['readerFileNameAndPath']}" />
<property name="fieldNames" value="firstName, lastName, city" />
<property name="fullyQualifiedTargetClass" value="com.test.test.DatabaseMember" />
</properties>
</reader>
<processor ref="com.test.test.ItemProcessor" />
<writer ref="FlatFileWriter" >
<properties>
<property name="appendOn" value="true" />
<!-- Read to file demoXMLReaderFlatFileWriterOutput.txt -->
<property name="fileNameAndPath" value="#{jobParameters['writerFileNameAndPath']}" />
<property name="fullyQualifiedTargetClass" value="com.test.test.DatabaseMember" />
</properties>
</writer>
</chunk>
</step>
In the ItemProcessor I am interested in having some type of callback/listener that could trigger roll back for transactional items that occur inside it should the chunck for example fail during the write step.
So a good example would be I have some sort of db or jms operation that occurs in the ItemProcessor. If during the writer portion that chunck fails I would want to roll back all of those operations that occurred during the item processor that were part of this iteration (or several with item count > 1).
I have looked into the various listeners available based on the jsr spec and nothing seems to allow for this type of implementation. Also spring does not seem to wrap the processor in a transaction like it does when it is a true spring batch. Testing I did in creating db connections and testing them to see if they were transactional aware always returned false.
Is what I'm looking for even possible outside a entirely custom implementation and if so any guidance in how to accomplish it.

Related

Is JMS message doesnt rollback to queue in exceptional case spring batch JMSItemReader implementation

In my development work i have to use Spring batch JmsItemReader to read messages from Hornetq. I am trying the test scenario for rollback the JMS messages to queue in case errors occurs in spring batch process. But it doesnt work for me.
For Ex:
Spring step execution table showing Rollback count is 1. But actually i doesnt rollback the message into queue.
I used following configuration in my xml.
<batch:job id="submitOrderJmsToWebServiceJob">
<batch:step id="submitOrderJmsToLocateStep">
<batch:tasklet transaction-manager="myTxManager">
<batch:chunk reader="jmsMessageReader" reader-transactional-queue="true" processor="jmsMessageProcessor"
writer="jmsMessageToWebSeviceWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="jmsMessageReader" class="org.springframework.batch.item.jms.JmsItemReader">
<property name="jmsTemplate" ref="orderJmsTemplate" />
</bean>
<bean id="myTxManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
I solved this issue using following property in spring JMS Template bean.
<property name="sessionTransacted" value="true" />
Now jms message could be rollback.
Note: I have removed older configuration like transaction-manager attribute in tasklet tag and is transactional queue=true in chunk.

Write to File and Audit Table

I am trying to use Springbatch for a batch job that has to write to a file (Step 1) and also write the data (that was sent in the file) into an Audit Table.(step 2)
Is there any Better way other than giving the output file from Step 1 as input to Step 2 to write to DB?
The data cannot be passed in Exceution Context because it is huge.
Thanks.
Yes, using a CompositeItemWriter.
From its javadoc
Calls a collection of ItemWriters in fixed-order sequence.
Just create a CompositeItemWriter passing delegates (yours real writer: one for file and one for audit table) and register them as stream to allow restartability.
<bean id="fileWriter" class="path.to.filewriterClass" />
<bean id="auditTableWriter" class="path.to.auditawriterClass" />
<bean id="compositeWriter" class="path.to.CompositeItemWriter>
<property name="delegates">
<list>
<bean ref="fileWriter" />
<bean ref="auditTableWriter />
</property>
</bean>
<step name="step1">
<tasklet>
<chunk reader="path.to.readerClass" writer="compositeWriter" />
<streams>
<stream ref="fileWriter" />
<stream ref="auditTableWriter" />
</streams>
</tasklet>
</step>
See official doc for extra information and google around for a ton of examples!

apache restlet connector overload

I use restlet in camel route in from("restlet:http/myLink") clause. When user's requests more then ten per second, I begin recieve errors processing request like a "org.restlet.engine.connector.Controller run
INFO: Connector overload detected. Stop accepting new work"
I think, that error is caused by number of threads,request query's size or number,or something like that. I try set to maxThreads param different values in spring config
<bean id="restlet" class="org.apache.camel.component.restlet.RestletComponent">
<property name="maxThreads" value="15"/>
</bean>
but I am not succeed. In documentation http://camel.apache.org/restlet.html I ddin't find ant param for setting size\number of request queue. I need help :(
P.S. camel-restlet version is 2.12.2
Update
I try to set big numbers to maxThreads,maxConnectionsPerHost,maxTotalConnections, but it's useless. If inject org.restlet.Component to camel's config like that:
<bean id="restletComponent" class="org.restlet.Component" />
<bean id="restlet" class="org.apache.camel.component.restlet.RestletComponent">
<constructor-arg index="0">
<ref bean="restletComponent" />
</constructor-arg>
<property name="maxThreads" value="255"/>
<property name="maxConnectionsPerHost" value="1000"/>
<property name="maxTotalConnections" value="1000" />
</bean>
How I can override properties, that use BaseHelper params?
After go through the options of lowThread as well.
But I found current released camel doesn't support it.

Implementing ItemProcessListener for a chain of ItemProcessors

I am using Spring batch.
I have 2 Item Procesors which are used for doing the processing logic.
I have configured the listener for both ItemReader and ItemWriter.
I have used Spring Batch CompositeItemProcessor (org.springframework.batch.item.support.CompositeItemProcessor).
My Job configuration is as follows :-
<job id="SoDJob" xmlns="http://www.springframework.org/schema/batch">
<step id="step1">
<tasklet>
<chunk reader="itemReader" processor="SoDConflictProcessor" writer="SoDConflictExcelWriter"
commit-interval="1" />
<listeners>
<listener ref="sodJobListener" />
<listener ref="SoDItemReaderListener" />
<listener ref="SoDItemWriterListener" />
</listeners>
</tasklet>
</step>
</job>
My Processors are configured as :-
<bean id="SoDConflictProcessor"
class="org.springframework.batch.item.support.CompositeItemProcessor">
<property name="delegates">
<list>
<ref bean="SoDDataProcessor" />
<ref bean="SoDLogicProcessor" />
</list>
</property>
</bean>
How Can i write individual ItemProcessListener for both ItemProcessor (SoDDataProcessor and SoDLogicProcessor).
FYI :
1) SoDDataProcessor - implements ItemProcessor<User, HashSet<String>>.
2) SoDLogicProcessor - implements ItemProcessor<HashSet<String>, HashSet<Object>>
Hope this clears my question.
There is a CompositeItemProcessListener but this doesn't fit your requirement because - I think - your request is to have SoDDataProcessorListener called when SoDDataProcessor process an item and SoDLogicProcessorListener called when the item transformed from previous step pass into SoDLogicProcessor.
This is not possible for two reasons:
ItemProcessorListener is called around your CompositeItemProcessor and not called around its delegated ItemProcessors
CompositeItemProcessListener will run into a ClassCastException because is signature must be CompositeItemProcessListener<User,HashSet<Object>> and when receiving HashSet<String> will thrown an exception
You have to resolve in another way, this depends by your needs; I'll give you just an idea for your question, but you can also reconsider your design to achieve the same result
My idea:
Store intermediate transformed data in StepExecutionContext, write your own ItemProcessorListener that delegate each intermediate result to SoDDataProcessorListener, SoDLogicProcessorListener and others if you (will) have more and dispatch data to correct listener; in this way you can also write specific classes to mantain listeners separated.
Hope can help.

Spring Integration 2 with Quartz scheduler

I am new to Spring Integration.
I've configured a Spring file inbound-channel-adapter, e.g.
<file:inbound-channel-adapter channel="channel1" directory="${location}" prevent-duplicates="true" filename-pattern="*.csv">
<si:poller>
<si:interval-trigger interval="1000"/>
</si:poller>
</file:inbound-channel-adapter>
<si:service-activator input-channel="channel1" output-channel="channel2" ref="filenameGenerator" method="generate"/>
Now this is working fine.
But this needs to be deployed in a clustered environment. I want to make sure that multiple instances in the cluster do not attempt to read the same file. So will this work in such environment?
If no, can I use Quartz scheduler like this:
<file:inbound-channel-adapter channel="channel1" directory="${location}" prevent-duplicates="true" filename-pattern="*.csv">
<si:poller task-executor="taskExecutor" fixed-rate="1000"/>
</file:inbound-channel-adapter>
<si:service-activator input-channel="channel1" output-channel="channel2" ref="filenameGenerator" method="generate"/>
<bean id="taskExecutor" class="org.springframework.scheduling.quartz.SimpleThreadPoolTaskExecutor">
<property name="threadCount" value="20"/>
<property name="threadNamePrefix" value="consumer"/>
</bean>
Will this work and solve my problem??
Or do I have to use Transaction?
I hope the question is clear.
Thanks,
Adi
When multiple processes are reading from the same directory it can be
desirable to lock files to prevent them from being picked up
concurrently. To do this you can use a FileLocker
Check out the documentation around file lockers here. It seems that you can do soemthing like this:
<file:inbound-channel-adapter ... >
<file:nio-locker/>
</file:inbound-channel-adapter>
When multiple processes are reading from the same directory it can be
desirable to lock files to prevent them from being picked up
concurrently. To do this you can use a FileLocker
To ensure that a quartz-scheduled job executes once and only once within a cluster, configure a persistent, clustered quartz job schedule. Here's a sample config, for Quartz 1.6.6:
<bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<!-- Set whether any jobs defined on this SchedulerFactoryBean should
overwrite existing job definitions.
-->
<property name="overwriteExistingJobs" value="true" />
<property name="dataSource" ref="myTransactionalDataSource" />
<!-- nonTransactionalDataSource is only necessary with clustered Quartz with an XA DataSource.
-->
<property name="nonTransactionalDataSource" ref="myNonTransactionalDataSource" />
<property name="quartzProperties">
<props>
<prop key="org.quartz.jobStore.selectWithLockSQL">SELECT * FROM {0}LOCKS WITH(UPDLOCK,HOLDLOCK) WHERE LOCK_NAME = ?</prop>
<!--
Run in cluster. Quartz ensures persisted jobs are executed once within the
cluster
-->
<prop key="org.quartz.jobStore.isClustered">true</prop>
<!-- Each node in the cluster must have a unique instance id.
-->
<prop key="org.quartz.scheduler.instanceId">AUTO</prop>
<!-- Default clusterCheckinInterval is 15000
-->
<!-- <prop key="org.quartz.jobStore.clusterCheckinInterval">20000</prop>
-->
</props>
</property>
<property name="transactionManager" ref="transactionManager" />
- <!--
In Quartz 1.6.6, Quartz's ThreadPool interface is used when firing job triggers,
in org.quartz.core.QuartzSchedulerThread.
Quartz 1.x still starts some unmanaged threads, notably org.quartz.impl.jdbcjobstore.JobStoreSupport's
ClusterManager which is used when clustered=true. Quartz 2.0 should correct this problem.
-->
<property name="taskExecutor" ref="myTaskExecutor" />
</bean>