I have a Spring Batch app that I've configured with a SkipPolicy so the whole batch won't fail if one record can't be inserted. But the behavior doesn't seem to be that way. When an insert into the database fails, Postgresql says the transaction is "bad", and all commands will aborted. So all the following inserts into the db fail too. Spring is supposed to retry the transaction one record at a time, I thought?
So we're wondering if the problem is because we mark the service method with #Transactional? Should I just let Spring Batch control the transactions? Here's my job configuration:
<bean id="stepScope" class="org.springframework.batch.core.scope.StepScope">
<property name="autoProxy" value="true"/>
</bean>
<bean id="skipPolicy" class="com.company.batch.common.job.listener.BatchSkipPolicy"/>
<bean id="chunkListener" class="com.company.batch.common.job.listener.ChunkExecutionListener"/>
<batch:job id="capBkdnJob">
<batch:step id="capStep">
<tasklet throttle-limit="20">
<chunk reader="CapReader" processor="CapProcessor" writer="CapWriter" commit-interval="50"
skip-policy="skipPolicy" skip-limit="10">
<batch:skippable-exception-classes>
<batch:include class="com.company.common.exception.ERDException"/>
</batch:skippable-exception-classes>
</chunk>
<batch:no-rollback-exception-classes>
<batch:include class="com.company.common.exception.ERDException"/>
</batch:no-rollback-exception-classes>
<batch:listeners>
<batch:listener ref="chunkListener"/>
</batch:listeners>
</tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="batchWorkerJobExecutionListener"/>
</batch:listeners>
</batch:job>
Short answer is: no
Spring Batch will use the transaction manager defined as part of your JobRepository by default. This will allow it to roll back the whole chunk when an error has been encountered and then retry each item individually in its own transaction.
Related
I specified a tasklet with a chunk orientated procssing.
<batch:step id="midxStep" parent="stepParent">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk
reader="processDbItemReaderJdbc"
processor="midxItemProcessor"
writer="midxCompositeItemWriter"
processor-transactional="false"
reader-transactional-queue="false"
skip-limit="${cmab.batch.skip.limit}"
commit-interval="#{jobParameters['toProcess']==T(de.axa.batch.ecmcm.cmab.util.CmabConstants).TYPE_POSTAUSGANG ? '${consumer.global.pa.midx.commitSize}' : '${consumer.global.pe.midx.commitSize}' }"
cache-capacity="20">
<batch:skippable-exception-classes>
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabProcessMidxException" />
<batch:exclude class="java.lang.IllegalArgumentException" />
</batch:skippable-exception-classes>
<batch:retryable-exception-classes>
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabTechnicalMidxException" />
<batch:include class="de.axa.batch.ecmcm.cmab.util.CmabTechnicalException" />
</batch:retryable-exception-classes>
<batch:retry-listeners>
<batch:listener ref="logRetryListener"/>
</batch:retry-listeners>
<batch:listeners>
<batch:listener>
<bean id="midxProcessSkipListener" class="de.axa.batch.ecmcm.cmab.core.batch.listener.CmabDbSkipListener" scope="step">
<constructor-arg index="0" value="#{jobParameters['errorStatus']}" type="java.lang.String"/>
</bean>
</batch:listener>
</batch:listeners>
</batch:chunk>
<batch:transaction-attributes isolation="SERIALIZABLE" propagation="MANDATORY" timeout="${cmab.jta.usertransaction.timeout}"/>
<batch:listeners>
<batch:listener ref="midxStepListener"/>
<batch:listener>
<bean id="cmabChunkListener" class="de.axa.batch.ecmcm.cmab.core.batch.listener.CmabChunkListener" scope="step"/>
</batch:listener>
</batch:listeners>
</batch:tasklet>
</batch:step>
The tasklet runs with a JtaTransaction manger (Atomikos, name="transactionManager").
Now my question:
Is this transaction manager "delegate" to the chunk-process?
Why I'm asking this? If I set the transaction-attributes (see chunk) to propagation level "MANDATORY" the chunk process aborted with the error that no transaction is available.
Therefore it left me confused because I thought that the tasklet transaction specification implies that the chunk running within this tasklet transaction, too.
Furthermore I intended to run the application within a cloud system with more than one pod. The processDbIemReaderJdbs fetches via a StoredProcedureItemReader Items with a "FOR UPDATE SKIP LOCKED" from a PostgresDB.
So my intention is to run the hole chunk, means also the reader, within one transaction in order to block the reader resultSet to other POD-Processes.
The transaction attributes are for the transaction that Spring Batch will create to run your step with the transaction manager that you set on the step. Those are the attributes of the transaction itself, not the transaction manager (that does not make sense).
All batch artifacts are executed within the scope of that same transaction, including the reader and the writer. The only exception to that is the JdbcCursorItemReader, which by default does not participate in the transaction, unless useSharedExtendedConnection is set.
In my development work i have to use Spring batch JmsItemReader to read messages from Hornetq. I am trying the test scenario for rollback the JMS messages to queue in case errors occurs in spring batch process. But it doesnt work for me.
For Ex:
Spring step execution table showing Rollback count is 1. But actually i doesnt rollback the message into queue.
I used following configuration in my xml.
<batch:job id="submitOrderJmsToWebServiceJob">
<batch:step id="submitOrderJmsToLocateStep">
<batch:tasklet transaction-manager="myTxManager">
<batch:chunk reader="jmsMessageReader" reader-transactional-queue="true" processor="jmsMessageProcessor"
writer="jmsMessageToWebSeviceWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="jmsMessageReader" class="org.springframework.batch.item.jms.JmsItemReader">
<property name="jmsTemplate" ref="orderJmsTemplate" />
</bean>
<bean id="myTxManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
I solved this issue using following property in spring JMS Template bean.
<property name="sessionTransacted" value="true" />
Now jms message could be rollback.
Note: I have removed older configuration like transaction-manager attribute in tasklet tag and is transactional queue=true in chunk.
I was trying to copy a data from one data source to another, used to Ibaitsbatchitemwriter class to do so. Record was got inserted into target database but end of the batch getting null pointer exception as below,
java.lang.NullPointerException
at org.springframework.batch.item.database.IbatisBatchItemWriter.write(IbatisBatchItemWriter.java:142)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:175)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:151)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:274)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
but after adding the property assertupdates = fales i was not getting the error and then data is also got copied. but i was not convinced with the null pointer error, looks like am missing something in my config or so.
i use spring infra 2.2.4 and ibatis version 2.3.0.
<bean id="targetWriterDepAcct03"
class="org.springframework.batch.item.database.IbatisBatchItemWriter">
<property name="sqlMapClient" ref="targetDatabaseMap" />
<property name="statementId" value="DepositAccountSqlMap.updtDepositAccount" />
<property name="assertUpdates" value="false" />
</bean>
<batch:job id="baseJob" abstract="true" restartable="true"
job-repository="jobRepository" />
<batch:job id="TboltSyncBatchJob">
<batch:step id="CheckForConfigFileStep">
<batch:tasklet ref="CheckForConfigFile" />
<batch:next on="COMPLETED" to="SyncDataDepAcct03" />
<batch:end on="FAILED" />
</batch:step>
<batch:step id="SyncDataDepAcct03">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="sourceReaderForDepAcct03" writer="targetWriterDepAcct03"
commit-interval="1000" />
</batch:tasklet>
</batch:step>
any thoughts?
That NPE is due to the fact that zero results were returned by the SqlMapClient. If you don't need the number of records to be checked, you can turn that off. If you need them checked, you'll want to look into why that query isn't returning any results. You can see the code for the IbatisBatchItemWriter here: IbatisBatchItemWriter.
I am working on a project based on spring batch admin. I use spring-integration's
<int-jms:message-driven-channel-adapter/>
which picks the message from the queue and pushes them into a channel which invokes the service activator. the service activator then invokes the batch job.
spring-batch-admin internally uses a taskExecutor with pool size as 6(available in spring-batch-admin-manager-1.2.2-release.jar). This task executor has a rejectionPolicy configured as ABORT i.e. if the request for jobs are more than 6, abort other job requests. But when i run the project with over 100 requests, i see them with status as STARTING in the spring batch admin console although, only 6 job requests at a time gets processed.
I am not understanding where are the remaining job requests getting queued. Would appreciate if someone could explain me this or give some pointers.
Configurations:
<int-jms:message-driven-channel-adapter id="jmsIn"
connection-factory="connectionFactory"
destination-name="${JMS.SERVER.QUEUE}" channel="jmsInChannel"
extract-payload="false" send-timeout="20000"/>
<integration:service-activator id="serviceAct" input-channel="jmsInChannel" output-channel="fileNamesChannel"
ref="handler" method="process" />
<bean id="handler" class="com.mycompany.integration.AnalysisMessageProcessor">
<property name="jobHashTable" ref="jobsMapping" />
</bean>
<batch:job id="fullRebalanceJob" incrementer="jobIdIncrementer">
<batch:step id="stepGeneral">
<batch:tasklet>
<bean class="com.mycompany.batch.tasklet.DoGeneralTasklet" scope="step">
<property name="resultId" value="#{jobParameters[resultId]}" />
</bean>
</batch:tasklet>
<batch:next on="REC-SELLS" to="stepRecordSells"/>
<batch:fail on="FAILED" />
<batch:listeners>
<batch:listener ref="stepListener" />
</batch:listeners>
</batch:step>
<batch:step id="stepDoNext">
<batch:tasklet ref="dcnNext" />
</batch:step>
</batch:job>
Thanks in advance. Let me know if more details are required.
I'm setting up a batch job that moves data between three databases. I am planning on using the out of the box spring batch classes to handle the query from the first database, but i want to include details of the current job/step in the extract. The example spring config might look like this
<bean id="jdbcPagingItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader"> <property name="dataSource" ref="dataSource"/>
<property name="pageSize" value="1000"/>
<property name="fetchSize" value="100"/>
<property name="queryProvider">
<bean class="org.springframework.batch.item.database.support.HsqlPagingQueryProvider">
<property name="selectClause" value="select id, bar"/>
<property name="fromClause" value="foo"/>
<property name="sortKeys">
Is there a way via groovy or SpEL to access the current JobExecution? I had found this thread on access-spring-batch-job-definition but is assumes custom code.
Your configuration is cut off at the sortKeys entry so I'm not 100% sure what you are attempting to accomplish. That being said, using step scope, you can inject the StepExecution which has a reference to the JobExecution. Getting the JobExecution would look something like this:
<bean id="jdbcPagingItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader">
…
<property id="jobExecution" value="#{stepExecution.jobExecution}"/>
</bean>
The write count exists on the step context (StepExecution.getWriteCount()) so first you need to promote it to the JobExecutionContext.
I suggest using a Step listener with #AfterStep annotation for this
#Component
public class PromoteWriteCountToJobContextListener implements StepListener {
#AfterStep
public ExitStatus afterStep(StepExecution stepExecution){
int writeCount = stepExecution.getWriteCount();
stepExecution.getJobExecution().getExecutionContext()
.put(stepExecution.getStepName()+".writeCount", writeCount);
return stepExecution.getExitStatus();
}
}
Every step that perform the inserts will be added with this listner
<batch:step id="readFromDB1WrietToDB2" next="readFroDB2WrietToDB3">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="reader" writer="writter" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="promoteWriteCountToJobContextListener"/>
</batch:listeners>
</batch:step>
<batch:step id="readFromDB2WrietToDB3" next="summerizeWrites">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="reader" writer="writter" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="promoteWriteCountToJobContextListener"/>
</batch:listeners>
</batch:step>
You can use log4j to write the results to log within the Step Listener, or you can use the saved values in a future step. You need to use SpEL expression to read it, to Use SpEL expression you need to set the Writter/Tasklet at scope="step"
<batch:step id="summerizeWrites">
<batch:tasklet id="summerizeCopyWritesTaskelt"">
<bean class="Tasklet" scope="step">
<property name="writeCountsList">
<list>
<value>#{jobExecutionContext['readFromDB1WrietToDB2.writeCount']}</value>
<value>#{jobExecutionContext['readFromDB2WrietToDB3.writeCount']}</value>
</list>
</property>
</bean>
</batch:tasklet>