I am designing a Spring Batch, which reads multiple csv files. I have used partitioning to read each file in chunk and process it to decrypt a certain column in the csv. Before decrypting if i encounter any validation error , i throw custom exception.
Now what i want is if the processing finds any validation error in the first line, the other lines should not be processed, and the job should end.
How can i achieve this? I tried to implement ProcessorListener too but it has no StepExecution object so that i can call SetTerminateOnly() or ExitStatus=Failed
Also note that i have multiple thread accessing the file in different lines.I want to kill all threads in the event of the first encountered error.
Thanks in advance
So, I identified that running multiple asynchronous concurrent threads (Spring Batch partitioning) was the real issue. Though one of the thread threw an Exception, the other threads were parallely running, and finished executing till the end.
Ath the end, the Job FAILED overall and there was no output processed, but it consumed time to process rest of the data.
Well,the solution to it is as simple as it gets. We just need stop the Job while encountering an error during processing.
The Custom Processor
public class MultiThreadedFlatFileItemProcessor implements ItemProcessor<BinFileVO, BinFileVO>,JobExecutionListener{
private JobExecution jobExecution;
private RSADecrypter decrypter;
public RSADecrypter getDecrypter() {
return decrypter;
}
public void setDecrypter(RSADecrypter decrypter) {
this.decrypter = decrypter;
}
#Override
/**
This method is used process the encrypted data
#param item
* */
public BinFileVO process(BinFileVO item) throws JobException {
if(null!=item.getEncryptedText() && !item.getEncryptedText().isEmpty()){
String decrypted = decrypter.getDecryptedText(item.getEncryptedText());
if(null!=decrypted && !decrypted.isEmpty()){
if(decrypted.matches("[0-9]+")){
if(decrypted.length() >= 12 && decrypted.length() <= 19){
item.setEncryptedText(decrypted);
}else{
this.jobExecution.stop();
throw new JobException(PropertyLoader.getValue(ApplicationConstants.DECRYPTED_CARD_NO_LENGTH_INVALID),item.getLineNumber());
}
}
}else{
this.jobExecution.stop();
throw new JobException(PropertyLoader.getValue(ApplicationConstants.EMPTY_ENCRYPTED_DATA),item.getLineNumber());
}
return item;
}
#Override
public void beforeJob(JobExecution jobExecution) {
this.jobExecution=jobExecution;
}
#Override
public void afterJob(JobExecution jobExecution) {
}
}
The Job xml config
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
.....>
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- Job Details -->
<job id="simpleMultiThreadsReaderJob" xmlns="http://www.springframework.org/schema/batch">
<step id="step" >
<partition step="step1" partitioner="partitioner">
<handler grid-size="5" task-executor="taskExecutor"/>
</partition>
</step>
<listeners>
<listener ref="decryptingItemProcessor"/>
</listeners>
</job>
<step id="step1" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="itemReader" writer="itemWriter" processor="decryptingItemProcessor" commit-interval="500"/>
<listeners>
<listener ref="customItemProcessorListener" />
</listeners>
</tasklet>
</step>
<!-- Processor Details -->
<bean id="decryptingItemProcessor" class="com.test.batch.io.MultiThreadedFlatFileItemProcessor">
<property name="decrypter" ref="rsaDecrypter" />
</bean>
<!-- RSA Decrypter class -->
<bean id="rsaDecrypter" class="test.batch.secure.rsa.client.RSADecrypter"/>
<!-- Partitioner Details -->
<bean class="org.springframework.batch.core.scope.StepScope" />
<bean id="partitioner" class="com.test.batch.partition.FlatFilePartitioner" scope="step">
<property name="resource" ref="inputFile"/>
</bean>
<bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="10"/>
</bean>
<!-- Step will need a transaction manager -->
<bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
........
.................
</beans>
Here are the logs
2016-09-01 06:32:40 INFO SimpleJobRepository:273 - Parent JobExecution is stopped, so passing message on to StepExecution
2016-09-01 06:32:43 INFO ThreadStepInterruptionPolicy:60 - Step interrupted through StepExecution
2016-09-01 06:32:43 INFO AbstractStep:216 - Encountered interruption executing step: Job interrupted status detected.
; org.springframework.batch.core.JobInterruptedException
2016-09-01 06:32:45 ERROR CustomJobListener:163 - exception :At line No. 1 : The decrypted card number is less than 12 or greater than 19 in length
2016-09-01 06:32:45 ERROR CustomJobListener:163 - exception :Job interrupted status detected.
2016-09-01 06:32:45 INFO SimpleJobLauncher:135 - Job: [FlowJob: [name=simpleMultiThreadsReaderJob]] completed with the following parameters: [{outputFile=/usr/local/pos/bulktokenization/csv/outputs/cc_output_EDWError_08162016.csv, partitionFile=/usr/local/pos/bulktokenization/csv/partitions/, inputFile=C:\usr\local\pos\bulktokenization\csv\inputs\cc_input_EDWError_08162016.csv, fileName=cc_input_EDWError_08162016}] and the following status: [FAILED]
2016-09-01 06:32:45 INFO BatchLauncher:122 - Exit Status : FAILED
2016-09-01 06:32:45 INFO BatchLauncher:123 - Time Taken : 8969
If we throw Custom Exception in Processor, Spring Batch will terminate and mark the job failed unless you setup 'skipable' exception. You have not mentioned where you perform validate step, are you doing in Processor or Reader? Let me know because it is where Spring Batch decides.
In my project, if I want to stop the job and throw Custom Exception, we put validation logic in a Tasklet or Processor and throw exception as below
private AccountInfoEntity getAccountInfo(Long partnerId) {
if(partnerId != null){
.....
return ....;
} else {
throw new ReportsException("XXXXX");
}
}
I'm currently implementing a Spring-batch that reads and writes to files BUT also needs to do CRUD operations on a database.
I've tried to simply define an Entity manager in my xml configuration, and use it in my DAO class. However, right after the init, the EntityManager becomes null.
Can anyone help me with this ? (a solution or a link via something usable would be perfect).
My batchContext.xml
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${batch.datasource.driverClassName}"/>
<property name="url" value="${batch.datasource.url}"/>
<property name="username" value="${batch.datasource.username}"/>
<property name="password" value="${batch.datasource.password}"/>
<property name="testOnBorrow" value="true"/>
<property name="testOnReturn" value="true"/>
<property name="testWhileIdle" value="true"/>
<property name="timeBetweenEvictionRunsMillis" value="1800000"/>
<property name="numTestsPerEvictionRun" value="3"/>
<property name="minEvictableIdleTimeMillis" value="1800000"/>
</bean>
<bean id="entityManagerFactory" name="entTransactionMgr" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<!-- <property name="persistenceXmlLocation" value="classpath:/META-INF/spring/persistence.xml" /> -->
<property name="persistenceUnitName" value="persistenceUnit"/>
<property name="packagesToScan" value="${jpa.scan.packages}"/>
<property name="dataSource" ref="dataSource"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
</property>
<property name="loadTimeWeaver">
<bean class="org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver" />
</property>
<!-- Custom jpaDialect pour le deuxieme batch:job-repository-->
<property name="jpaDialect">
<bean class="fr.mma.soecm.batchfacade.util.CustomHibernateJpaDialect" />
</property>
<property name="jpaProperties">
<props>
<!-- multiple props here-->
</props>
</property>
</bean>
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>
<!-- mode="aspectj" -->
<tx:annotation-driven transaction-manager="transactionManager"/>
<batch:job-repository id="jobRepository" data-source="dataSource" transaction-manager="transactionManager"/>
<!-- Jobs held separatelly -->
<import resource="batchContext-job.xml"/>
My DAO
#Repository("batchJobDao")
#Transactional
public class BatchJobDaoImpl implements BatchJobDao{
#PersistenceContext(unitName="persistenceUnit")
#Autowired
private EntityManager entityManager;
// #PersistenceContext(unitName="persistenceUnit", type=PersistenceContextType.EXTENDED)
// public void setEntityManager(EntityManager entityManager) {
// System.out.println("Setting Entity Manager :" + entityManager);
// this. entityManager = entityManager;
// }
#PostConstruct
public void init(){
if (entityManager == null){
System.out.println(" Entity Manager is null");
} else {
System.out.println(" Entity Manager is not null");
getAllJobExecutions();
}
}
private static final String RECHERCHER_JOB_EXECUTION = "Select bje from BatchJobExecution bje";
#SuppressWarnings("unchecked")
#Override
#Transactional("transactionManager")
public List<BatchJobExecution> getAllJobExecutions(){
// EntityManagerFactory emf=Persistence.createEntityManagerFactory("entTransactionMgr");
// EntityManager em=emf.createEntityManager();
Query query = entityManager.createQuery(RECHERCHER_JOB_EXECUTION, BatchJobExecution.class);
List<BatchJobExecution> executions = (List<BatchJobExecution>) query.getResultList();
System.out.println("EXES : " + executions);
return executions;
}
}
There is some code commented out because I've tried multiple aproaches (changing the persistence context type, recovering the entity manager "manually", having a persistence.xml file for the entityManager) without succes.
My output when running the job is (without all the extra lines..):
Entity Manager is not null
EXES : [fr.mma.soecm.batchfacade.domain.BatchJobExecution#10e6c33,...]
[BatchService] - Synchronous job launch
[AbstractStep] - Encountered an error executing the step
Caused by: java.lang.NullPointerException
And the null pointer, on debug, is throws by the EntityManager being null when I call the "createQuery" in my DAO.
Thanks for your help.
I'll keep searching on my end.. God Speed!
As mentioned in the comment above, my aparent problem was due to the fact that I was trying to call the Service or the DAO in the Constructor of the Step.
After moving the Service call in the "doRead()" method, I could perform all CRUD operations with the EntityManager.
Please let me know if anyone has questions about this/and how to make it work otherwise, as I've not found any explanation on the internet, since I've began searching last week.
When I add a DAO call in the MdwValidatingItemProcessor below, I get a serious hit in performance. Prior to adding the additional DAO call, 2500 records where being read/processed/written in 40ish seconds. When I add the one additional DAO call (lobPolicyMapper.getPolicyOrigin(item.getLob(),item.getRgn_cd())
), it becomes 120 seconds. And the lob_policy table only has one record in it. I plan to have many such DAO calls in my processor to do additional validation of the item. Can I reuse the DAO's connection so for each processor that is validating an item, the connection doesn't have to be opened and closed constantly? The AsyncItemProcessor delegates to the MdwValidatingItemProcessor:
public class MdwValidatingItemProcessor implements
ItemProcessor<MecMdw, MecMdw> {
#Autowired
DataSourceTransactionManager txManager;
#Autowired
LobPolicyMapper lobPolicyMapper;
#Autowired
MecUtils mecUtils;
#Autowired
LobShopMapper lobShopMapper;
#Autowired
KaiserIssuerMapper kaiserIssuerMapper;
private Validator validator;
public void setValidator(Validator validator) {
this.validator = validator;
}
private List<Map> validationReasons;
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
validationReasons = (List<Map>) stepExecution.getJobExecution().getExecutionContext().get("validationReasons");
}
public MecMdw process(MecMdw item) throws Exception {
DefaultTransactionDefinition def = new DefaultTransactionDefinition();
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);
TransactionStatus status = txManager.getTransaction(def);
BindingResult results = BindAndValidate(item);
try {
if (results.hasErrors()) {
buildValidationException(results, item);
return setAsKickOut(item);
}
mecUtils.checkForPaddingAndValidateSSN(item);
//if both SSN and DOB are valid, set Rep_DOB_or_SSN to be SSN
if (item.getInvalid_ssn().equals(MecConstants.Y_FLAG) && mecUtils.isDOBValid(item)) {
item.setRep_doborssn("SSN");
}
else if(mecUtils.isDOBValid(item)){
item.setRep_doborssn("DOB");
}
else{
List<String> listOfErrors = new ArrayList<String>();
listOfErrors.add("BothSSNAndDOBInvalid");
item.setValidationErrors(listOfErrors);
return setAsKickOut(item);
}
// Policy Origin code not found in MEC database based on Region /
// LOB.
LobPolicy lobPolicy = lobPolicyMapper.getPolicyOrigin(
item.getLob(), item.getRgn_cd());
if (lobPolicy == null || ("").equals(lobPolicy.getLob())
|| lobPolicy.getLob() == null) {
return setAsKickOut(item);
}
//set the Rep_PolicyOgnCd
item.setRep_policy_ogn_cd(lobPolicy.getPolicyOrigin());
//If origin of policy = SHOP, look for Shop Identifier.(mec_lob_shop table)
if(("SHOP").equals(lobPolicy.getPolicyOrigin())){
if(lobShopMapper.getValidationShopIdentifier(item) == null){
return setAsKickOut(item);
}
}
//Validation Shop Identifier not found in MEC database based on Region/LOB/PID
if(lobShopMapper.getValidationShopIdentifier(item) == null){
return setAsKickOut(item);
}
//Kaiser Issuer EIN not found in MEC database based on Region/LOB.
Integer kaiserIssuer = kaiserIssuerMapper.getKaiserIssuerIdWithLobAndRegion(item);
if(kaiserIssuer == 0){
return setAsKickOut(item);
}
else{
item.setRep_kaiser_issuer_id(kaiserIssuer.toString() );
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
txManager.commit(status);
}
return item;
}
private MecMdw setAsKickOut(MecMdw item) {
item.setKick_out_fl('Y');
return item;
}
private BindingResult BindAndValidate(MecMdw item) {
DataBinder binder = new DataBinder(item);
binder.setValidator(validator);
binder.validate();
return binder.getBindingResult();
}
private void buildValidationException(BindingResult results, MecMdw item) {
List<String> listOfErrors = new ArrayList<String>();
for (ObjectError error : results.getAllErrors()) {
String[] codes = error.getCodes();
listOfErrors.add(codes[1]);
}
item.setValidationErrors(listOfErrors);
// throw new ValidationException(msg.toString());
}
I have a batch job using AsyncItemProcessor and AsyncItemWriter as follows:
<job id="mecmdwvalidatorJob" xmlns="http://www.springframework.org/schema/batch">
<step id="mdwvalidatorStep1">
<tasklet>
<chunk reader="pageItemReader" processor="asyncItemProcessor"
writer="asynchItemWriter" commit-interval="1000" skip-limit="2147483647">
<skippable-exception-classes> <!-- TODO -->
<include class="java.lang.Exception" />
</skippable-exception-classes>
</chunk>
</tasklet>
</step>
</job>
<bean id="pageItemReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader">
<property name="dataSource" ref="dataSource" />
<property name="queryProvider">
<bean
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause"
value="select MDW_ID,FK_LOG_FILE_ID,TAX_YEAR,SUBS_TYPE_CD,SUB_FIRST_NM,SUB_MIDDLE_NM,SUB_LAST_NM,SUB_SUFFIX,SUB_DOB,SUB_ADDR1,SUB_ADDR2,SUB_CITY,SUB_STATE,SUB_PROVINCE,SUB_ZIP,SUB_ZIP4,SUB_COUNTRY_CD,SUB_COUNTRY,SUB_F_POSTAL_CD,LOB,SUB_SSN,GRP_EMP_NAME1,GRP_EMP_NAME2,GRP_EIN,GRP_ADDR1,GRP_ADDR2,GRP_CITY,GRP_STATE,GRP_PROVINCE,GRP_ZIP,GRP_ZIP4,GRP_COUNTRY_CD,GRP_COUNTRY,GRP_F_POSTAL_CD,ISSUER_NAME1,ISSUER_NAME2,ISSUER_PHONE,ISSUER_ADDR1,ISSUER_ADDR2,ISSUER_CITY,ISSUER_PROVINCE,ISSUER_ZIP,ISSUER_ZIP4,ISSUER_COUNTRY_CD,ISSUER_COUNTRY,ISSUER_F_POSTAL_CD,MEM_FIRST_NM,MEM_MIDDLE_NM,MEM_LAST_NM,MEM_SUFFIX,MEM_SSN,MEM_DOB,MEM_START_DATE,MEM_END_DATE,REGION_CD,SUB_MRN,SUB_MRN_PREFIX,MEM_MRN,MRN_PREFIX,PID,SUB_GRP_ID,SUB_GRP_NAME,INVALID_ADDR_FL" />
<property name="fromClause"
value="from MEC_MDW JOIN MEC_FILE_LOG on MEC_FILE_LOG.LOG_FILE_ID=MEC_MDW.FK_LOG_FILE_ID " />
<property name="whereClause" value="where MEC_FILE_LOG.STATUS=:status" />
<property name="sortKey" value="MDW_ID" />
</bean>
</property>
<property name="parameterValues">
<map>
<entry key="status" value="READY TO VALIDATE" />
</map>
</property>
<property name="pageSize" value="1000" />
<property name="rowMapper" ref="mdwRowMapper" />
</bean>
<bean id="mdwRowMapper" class="org.my.rowmapper.MdwRowMapper" />
<bean id="asyncItemProcessor"
class="org.springframework.batch.integration.async.AsyncItemProcessor">
<property name="delegate">
<bean
class="org.my.itemprocessor.MdwValidatingItemProcessor">
<property name="validator">
<bean
class="org.springframework.validation.beanvalidation.LocalValidatorFactoryBean" />
</property>
</bean>
</property>
<property name="taskExecutor" ref="taskExecutor" />
</bean>
<task:executor id="taskExecutor" pool-size="10" />
<bean id="asynchItemWriter"
class="org.springframework.batch.integration.async.AsyncItemWriter">
<property name="delegate" ref="customerCompositeWriter">
</property>
</bean>
<bean id="customerCompositeWriter"
class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="itemWriter1" />
<ref bean="itemWriter2" />
</list>
</property>
</bean>
<bean id="itemWriter1" class="org.my.writer.MdwWriter" />
<bean id="itemWriter2" class="org.my.writer.KickoutWriter" />
The transaction manager and MyBatix SqlSessionTemplate:
<!-- transaction manager, use JtaTransactionManager for global tx -->
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<!-- define SqlSessionFactory as BATCH execution -->
<bean id="sqlSession" class="org.mybatis.spring.SqlSessionTemplate">
<constructor-arg index="0" ref="sqlSessionFactory" />
<constructor-arg index="1" value="BATCH" />
</bean>
<!-- stored job-meta in database -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="batchTransactionManager" />
<property name="databaseType" value="sqlserver" />
</bean>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
The DAO making the call:
public interface LobPolicyMapper {
public abstract LobPolicy getPolicyOrigin(#Param("lob") String lob, #Param("regionCd") String regionCd);
UPDATE I added ehcache to the the MyBatis XML. This will help with the repetitive queries. But again, I am looking for a way to share this same DAO's connection across the AsyncItemProcessors:
<mapper namespace="org.mybatis.LobPolicyMapper">
<cache type="org.mybatis.caches.ehcache.EhcacheCache"/>
<select id="getPolicyOrigin" parameterType="hashmap" resultType='org.my.domain.LobPolicy'>
SELECT
l.lob_id as lobId,
l.lob as lob,
l.policy_origin as policyOrigin,
l.region_cd as regionCd
from mec_lob_policy l
where lob= #{lob} and region_cd=#{regionCd}
</select>
I added ehcache to the project
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-ehcache</artifactId>
<version>1.0.0</version>
</dependency>
I think I need a database pool. Here is what I am seeing in my logs on exiting the processor:
2015-08-27 17:09:54,194 [taskExecutor-3] DEBUG org.mybatis.spring.SqlSessionUtils - Releasing transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession#77b027f]
2015-08-27 17:10:06,674 [taskExecutor-3] DEBUG org.mybatis.spring.SqlSessionUtils - Transaction synchronization committing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession#77b027f]
2015-08-27 17:10:06,675 [taskExecutor-3] DEBUG org.mybatis.spring.SqlSessionUtils - Transaction synchronization deregistering SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession#77b027f]
2015-08-27 17:10:06,675 [taskExecutor-3] DEBUG org.mybatis.spring.SqlSessionUtils - Transaction synchronization closing SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession#77b027f]
2015-08-27 17:10:06,675 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Initiating transaction commit
2015-08-27 17:10:06,675 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Committing JDBC transaction on Connection [ConnectionID:14 ClientConnectionId: 1b95dc0e-83c0-487b-af59-f5be52931818]
2015-08-27 17:10:06,805 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Releasing JDBC Connection [ConnectionID:14 ClientConnectionId: 1b95dc0e-83c0-487b-af59-f5be52931818] after transaction
2015-08-27 17:10:06,805 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
Here is what I am seeing in my logs on entering the processor. New transaction is created and a new database connection is fetched:
2015-08-27 17:10:06,805 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
2015-08-27 17:10:06,805 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DriverManagerDataSource - Creating new JDBC DriverManager Connection to [jdbc:sqlserver://blah]
2015-08-27 17:10:07,115 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [ConnectionID:23 ClientConnectionId: d1f016e6-3e9d-4b0e-a34d-14298c292a65] for JDBC transaction
2015-08-27 17:10:07,115 [taskExecutor-3] DEBUG org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [ConnectionID:23 ClientConnectionId: d1f016e6-3e9d-4b0e-a34d-14298c292a65] to manual commit
I'm trying to read some records from a database using loops then do some calculations on the records (updating a field called total).
But i'm new to spring batch so please can anyone provide me with some tips.
this sounds like something the chunk pattern would address. you can use re-use existing Spring Batch components to read from the database, compose your own processor, then pass back to a Spring Batch component to store.
say the use case is like this;
- read a record
- record.setTotalColumn(record.getColumn2() + record.getColumn3())
- update
this configuration might look like this
<batch:job id="recordProcessor">
<batch:step id="recordProcessor.step1">
<batch:tasklet>
<batch:chunk reader="jdbcReader" processor="calculator" writer="jdbcWriter" commit-interval="10"/>
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="jdbcReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource"/>
<property name="sql" value="select idColumn,column1,column2,totalColumn from my_table"/>
<property name="rowMapper" ref="myRowMapper"/>
</bean>
<bean id="jdbcWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="dataSource"/>
<property name="sql" value="update my_table set totalColumn = :value where idColumn = :id"/>
</bean>
<bean id="calculator" class="de.incompleteco.spring.batch.step.item.CalculationProcessor"/>
this means that the only thing you have to 'write' from scratch would be the calculator processor which could e something like this;
package de.incompleteco.spring.batch.step.item;
import org.springframework.batch.item.ItemProcessor;
public class CalculationProcessor implements ItemProcessor<MyObject, MyObject> {
#Override
public MyObject process(MyObject item) throws Exception {
//do the math
item.setTotalColumn(item.getColumn1() + item.getColumn2());
//return
return item;
}
}
I'm getting a LazyInitializationException trying to test my DAO methods using the tool stack defined in the title. My understanding is that my test must be running outside the hibernate session, or it has been closed before I try to read children objects from my DAO. From reading the documentation, I understood that using the #TransactionConfiguration tag would allow me to define the transaction manager in which to run the tests.
I've read the documentation multiple times and a zillion forum posts. Still slamming my head into my keyboard... What am I missing? Thanks for your help!
my unit test class:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = {
"classpath:/WEB-INF/applicationContext-db.xml",
"classpath:/WEB-INF/applicationContext-hibernate.xml",
"classpath:/WEB-INF/applicationContext.xml" })
#TestExecutionListeners({DependencyInjectionTestExecutionListener.class, CleanInsertTestExecutionListener.class})
#DataSetLocation("test/java/com/yada/yada/dao/dbunit-general.xml")
#TransactionConfiguration(transactionManager="transactionManager", defaultRollback = true)
#Transactional
public class RealmDAOJU4Test {
#Autowired
private DbUnitInitializer dbUnitInitializer;
#Autowired
private RealmDAO realmDAO;
#Test
public void testGetById() {
Integer id = 2204;
Realm realm = realmDAO.get(id);
assertEquals(realm.getName().compareToIgnoreCase(
"South Technical Realm"), 0);
assertEquals(8, realm.getRealmRelationships().size());
}
}
my applicationContext-hibernate.xml:
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"></property>
<property name="useTransactionAwareDataSource" value="true" />
... other properties removed ...
</bean>
<bean id="transactionManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory">
<ref bean="sessionFactory" />
</property>
</bean>
my dao definition in applicationContext.xml
<bean id="realmDAOTarget" class="com.yada.yada.dao.hibernate.RealmDAOImpl">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<bean id="realmDAO" class="org.springframework.aop.framework.ProxyFactoryBean">
<property name="proxyInterfaces">
<value>com.yada.yada.dao.RealmDAO</value>
</property>
<property name="interceptorNames">
<list>
<value>hibernateInterceptor</value>
<value>realmDAOTarget</value>
</list>
</property>
</bean>
well, for anyone following along at home, here's what I missed:
TransactionalTestExecutionListener
it is required in the #TestExecutionListeners list for the #Transactional annotation to have any effect.