Tasklet transaction-manager and chunk transaction - spring-batch

I specified a tasklet with a chunk orientated procssing.
<batch:step id="midxStep" parent="stepParent">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk
reader="processDbItemReaderJdbc"
processor="midxItemProcessor"
writer="midxCompositeItemWriter"
processor-transactional="false"
reader-transactional-queue="false"
skip-limit="${cmab.batch.skip.limit}"
commit-interval="#{jobParameters['toProcess']==T(de.axa.batch.ecmcm.cmab.util.CmabConstants).TYPE_POSTAUSGANG ? '${consumer.global.pa.midx.commitSize}' : '${consumer.global.pe.midx.commitSize}' }"
cache-capacity="20">
<batch:skippable-exception-classes>
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabProcessMidxException" />
<batch:exclude class="java.lang.IllegalArgumentException" />
</batch:skippable-exception-classes>
<batch:retryable-exception-classes>
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabTechnicalMidxException" />
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabTechnicalException" />
</batch:retryable-exception-classes>
<batch:retry-listeners>
<batch:listener ref="logRetryListener"/>
</batch:retry-listeners>
<batch:listeners>
<batch:listener>
<bean id="midxProcessSkipListener" class="de.axa.batch.ecmcm.cmab.core.batch.listener.CmabDbSkipListener" scope="step">
<constructor-arg index="0" value="#{jobParameters['errorStatus']}" type="java.lang.String"/>
</bean>
</batch:listener>
</batch:listeners>
</batch:chunk>
<batch:transaction-attributes isolation="SERIALIZABLE" propagation="MANDATORY" timeout="${cmab.jta.usertransaction.timeout}"/>
<batch:listeners>
<batch:listener ref="midxStepListener"/>
<batch:listener>
<bean id="cmabChunkListener" class="de.axa.batch.ecmcm.cmab.core.batch.listener.CmabChunkListener" scope="step"/>
</batch:listener>
</batch:listeners>
</batch:tasklet>
</batch:step>
The tasklet runs with a JtaTransaction manger (Atomikos, name="transactionManager").
Now my question:
Is this transaction manager "delegate" to the chunk-process?
Why I'm asking this? If I set the transaction-attributes (see chunk) to propagation level "MANDATORY" the chunk process aborted with the error that no transaction is available.
Therefore it left me confused because I thought that the tasklet transaction specification implies that the chunk running within this tasklet transaction, too.
Furthermore I intended to run the application within a cloud system with more than one pod. The processDbIemReaderJdbs fetches via a StoredProcedureItemReader Items with a "FOR UPDATE SKIP LOCKED" from a PostgresDB.
So my intention is to run the hole chunk, means also the reader, within one transaction in order to block the reader resultSet to other POD-Processes.

The transaction attributes are for the transaction that Spring Batch will create to run your step with the transaction manager that you set on the step. Those are the attributes of the transaction itself, not the transaction manager (that does not make sense).
All batch artifacts are executed within the scope of that same transaction, including the reader and the writer. The only exception to that is the JdbcCursorItemReader, which by default does not participate in the transaction, unless useSharedExtendedConnection is set.

Related

Spring Batch: Do I need a Transactional annotation?

I have a Spring Batch app that I've configured with a SkipPolicy so the whole batch won't fail if one record can't be inserted. But the behavior doesn't seem to be that way. When an insert into the database fails, Postgresql says the transaction is "bad", and all commands will aborted. So all the following inserts into the db fail too. Spring is supposed to retry the transaction one record at a time, I thought?
So we're wondering if the problem is because we mark the service method with #Transactional? Should I just let Spring Batch control the transactions? Here's my job configuration:
<bean id="stepScope" class="org.springframework.batch.core.scope.StepScope">
<property name="autoProxy" value="true"/>
</bean>
<bean id="skipPolicy" class="com.company.batch.common.job.listener.BatchSkipPolicy"/>
<bean id="chunkListener" class="com.company.batch.common.job.listener.ChunkExecutionListener"/>
<batch:job id="capBkdnJob">
<batch:step id="capStep">
<tasklet throttle-limit="20">
<chunk reader="CapReader" processor="CapProcessor" writer="CapWriter" commit-interval="50"
skip-policy="skipPolicy" skip-limit="10">
<batch:skippable-exception-classes>
<batch:include class="com.company.common.exception.ERDException"/>
</batch:skippable-exception-classes>
</chunk>
<batch:no-rollback-exception-classes>
<batch:include class="com.company.common.exception.ERDException"/>
</batch:no-rollback-exception-classes>
<batch:listeners>
<batch:listener ref="chunkListener"/>
</batch:listeners>
</tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="batchWorkerJobExecutionListener"/>
</batch:listeners>
</batch:job>
Short answer is: no
Spring Batch will use the transaction manager defined as part of your JobRepository by default. This will allow it to roll back the whole chunk when an error has been encountered and then retry each item individually in its own transaction.

Spring: How to restart transaction in a for-loop?

I have a Spring Batch app that, during the write step, loops through records to be inserted into a Postgres database. Every now and again we get a DuplicateKeyException in the loop, but don't want the whole job to fail. We log that record and want to continue inserting the following records.
But upon getting an exception, the transaction becomes "bad" and Postgres won't accepted any more commands, as described in this excellent post. So my question is, what's the best way to restart the transaction? Again, I'm not retrying the record that failed - I just want to continue in my loop with the next record.
This is part of my job config xml:
<batch:job id="portHoldingsJob">
<batch:step id="holdingsStep">
<tasklet throttle-limit="10">
<chunk reader="PortHoldingsReader" processor="PortHoldingsProcessor" writer="PortHoldingsWriter" commit-interval="1" />
</tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="JobExecutionListener"/>
</batch:listeners>
</batch:job>
Thanks for any input!
Not sure if you are using the Spring transaction annotations to manage the transactions or not ... if so perhaps you can try.
#Transactional(noRollbackFor = DuplicateKeyException.class)
Hope that helps.
No rollback exceptions in Spring Batch are apparently designated like
<batch:tasklet>
<batch:chunk ... />
<batch:no-rollback-exception-classes>
<batch:include class="MyRuntimeException"/>
</batch:no-rollback-exception-classes>
</batch:tasklet>

Is JMS message doesnt rollback to queue in exceptional case spring batch JMSItemReader implementation

In my development work i have to use Spring batch JmsItemReader to read messages from Hornetq. I am trying the test scenario for rollback the JMS messages to queue in case errors occurs in spring batch process. But it doesnt work for me.
For Ex:
Spring step execution table showing Rollback count is 1. But actually i doesnt rollback the message into queue.
I used following configuration in my xml.
<batch:job id="submitOrderJmsToWebServiceJob">
<batch:step id="submitOrderJmsToLocateStep">
<batch:tasklet transaction-manager="myTxManager">
<batch:chunk reader="jmsMessageReader" reader-transactional-queue="true" processor="jmsMessageProcessor"
writer="jmsMessageToWebSeviceWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="jmsMessageReader" class="org.springframework.batch.item.jms.JmsItemReader">
<property name="jmsTemplate" ref="orderJmsTemplate" />
</bean>
<bean id="myTxManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
I solved this issue using following property in spring JMS Template bean.
<property name="sessionTransacted" value="true" />
Now jms message could be rollback.
Note: I have removed older configuration like transaction-manager attribute in tasklet tag and is transactional queue=true in chunk.

Can you configure an ItemReader for a Partitioner in Spring Batch?

I have the following requirement: a CSV input file contains lines where one of the fields is an ID. There can be several lines with the same ID. The lines should be processed grouped by ID (meaning, if one line fails validation, then all lines with that same ID should fail to process). The groups of lines can be processed in parallel.
I have an implementation that works fine, but it is reading the CSV input file using my own code in a Partitioner implementation. It would be nicer if I could use an out-of-the-box implementation for that (e.g. FlatFileItemReader) and just configure that just like you would for a Chunk step.
To clarify, my job config is like this:
<batch:job id="job">
<batch:step id="partitionStep">
<batch:partition step="chunkStep" partitioner="partitioner">
<batch:handler grid-size="10" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
<batch:step id="chunkStep">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="reader" processor="processor" writer="writer" chunk-completion-policy="completionPolicy">
.. skip and retry policies omitted for brevity
</batch:chunk>
</batch:tasklet>
</batch:step>
<bean id="partitioner" class="com.acme.InputFilePartitioner" scope="step">
<property name="inputFileName" value="src/main/resources/input/example.csv" />
</bean>
<bean id="reader" class="org.springframework.batch.item.support.ListItemReader" scope="step">
<constructor-arg value="#{stepExecutionContext['key']}"/>
</bean>
where the Partitioner implementation reads the input file, "manually" parses the lines to get the ID field, groups them by that ID and puts them in Lists, and create ExecutionContexts that each get one of those Lists.
It would be great if I could replace that "manual" code in the Partitioner by a configuration of FlatFileItemReader with an ObjectMapper. (I hope I express myself clearly).
Is it possible ?

Query related to job queuing in Spring batch admin project

I am working on a project based on spring batch admin. I use spring-integration's
<int-jms:message-driven-channel-adapter/>
which picks the message from the queue and pushes them into a channel which invokes the service activator. the service activator then invokes the batch job.
spring-batch-admin internally uses a taskExecutor with pool size as 6(available in spring-batch-admin-manager-1.2.2-release.jar). This task executor has a rejectionPolicy configured as ABORT i.e. if the request for jobs are more than 6, abort other job requests. But when i run the project with over 100 requests, i see them with status as STARTING in the spring batch admin console although, only 6 job requests at a time gets processed.
I am not understanding where are the remaining job requests getting queued. Would appreciate if someone could explain me this or give some pointers.
Configurations:
<int-jms:message-driven-channel-adapter id="jmsIn"
connection-factory="connectionFactory"
destination-name="${JMS.SERVER.QUEUE}" channel="jmsInChannel"
extract-payload="false" send-timeout="20000"/>
<integration:service-activator id="serviceAct" input-channel="jmsInChannel" output-channel="fileNamesChannel"
ref="handler" method="process" />
<bean id="handler" class="com.mycompany.integration.AnalysisMessageProcessor">
<property name="jobHashTable" ref="jobsMapping" />
</bean>
<batch:job id="fullRebalanceJob" incrementer="jobIdIncrementer">
<batch:step id="stepGeneral">
<batch:tasklet>
<bean class="com.mycompany.batch.tasklet.DoGeneralTasklet" scope="step">
<property name="resultId" value="#{jobParameters[resultId]}" />
</bean>
</batch:tasklet>
<batch:next on="REC-SELLS" to="stepRecordSells"/>
<batch:fail on="FAILED" />
<batch:listeners>
<batch:listener ref="stepListener" />
</batch:listeners>
</batch:step>
<batch:step id="stepDoNext">
<batch:tasklet ref="dcnNext" />
</batch:step>
</batch:job>
Thanks in advance. Let me know if more details are required.