RetryLogic using AOP is not working for MongoItemWritrer , but the same is working for my custom Readers and writers - spring-batch

RetryLogic using AOP is not working for MongoItemWritrer , but the same is working for my custom Readers and writers
Is there anything that I'm doing wrong here.
Below is the code snipopet.
<bean id="retryAdvice"
class="org.springframework.retry.interceptor.RetryOperationsInterceptor">
<property name="retryOperations" ref="taskBatchRetryTemplate" />
</bean>
<bean id="taskBatchRetryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy" ref="genericRetryPolicy" />
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="${mongoloader.backOffPeriod.initialInterval}"/>
<property name="maxInterval" value="${mongoloader.backOffPeriod.maxInterval}"/>
<property name="multiplier" value="${mongoloader.backOffPeriod.multiplier}"/>
</bean>
</property>
<property name="listeners">
<bean class="com.company.ens.myload.job.StepRetryListener"/>
</property>
</bean>
<bean id="genericRetryPolicy" class="org.springframework.retry.policy.SimpleRetryPolicy" >
<constructor-arg index="0" value="${mongoloader.retry.limit}"/>
<constructor-arg index="1">
<map>
<entry key="org.springframework.data.mongodb.CannotGetMongoDbConnectionException" value="true"/>
<entry key="org.springframework.jdbc.CannotGetJdbcConnectionException" value="true"/>
<!-- Just included the below exception for testing purpose, needs to be removed -->
<entry key="java.io.FileNotFoundException" value="true"/>
<entry key="org.springframework.dao.DuplicateKeyException" value="true"/>
</map>
</constructor-arg>
</bean>
//including only my step configution here
<batch:step id="Step4a-MainFlow_TranslateRawFedObjectsToFilingModel"
allow-start-if-complete="false">
<batch:tasklet>
<batch:chunk reader="rawFedObjectMongoReader"
processor="translatingProcessor" writer="SctModelMongoWriter"
commit-interval="100"/>
</batch:tasklet>
<batch:listeners>
<batch:listener ref="step4aListener"/>
</batch:listeners>
</batch:step>
Did not find any posts related to my question. Any help would be appreciated.

Related

Mapping JPA entity to more than one entityManagers with SpringBatch program

I have developed SpringBatch application and deployed as Web Application in Websphere Liberty profile container. The batch program is designed to read records from a table and invokes HTTP service. Based on the service response a column named status is updated as RECORD_SENT/COMPLETE/ERROR type.
Objective is to reuse the same program for multiple datasources. The data source is passed in job parameter using client type. The datasources are in different schemas but having same datamodel.
Question: How does the transaction manager can be applied at run time inside Job Step or Tasklet?. Seeking help in this regard.
Configuration:
<bean id="entityManagerFactory1"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource1" />
<property name="persistenceUnitName" value="user" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="false" />
</bean>
</property>
<property name="jpaDialect">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect" />
</property>
</bean>
<bean id="entityManagerFactory2"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource2" />
<property name="persistenceUnitName" value="user" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="false" />
</bean>
</property>
<property name="jpaDialect">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect" />
</property>
</bean>
<bean id="entityManagerSelector" class="*com.spring.jpa.test.EntitymanagerSelector">
<property name="entityManagerFactory1" ref="entityManagerFactory1"></property>
<property name="entityManagerFactory2" ref="entityManagerFactory2"></property>
</bean>
job.xml snippet
<bean id="itemReader" class="org.springframework.batch.item.database.JpaPagingItemReader" scope="step">
<property name="entityManagerFactory" value="#{entityManagerSelector.getEntitymanagerForClient({jobParameters['client']})}" />
<property name="queryString" value="select u from User u where u.age > #{jobParameters['age']}" />
</bean>
Setting the job parameters during runtime to identify the client
JobParameters param = new JobParametersBuilder()
.addString("age", "20").addString("client", "client2")
.toJobParameters();
JobExecution execution = jobLauncher.run(job, param);
It will not be possible for you to set the transaction-manager of the Step/tasklet during runtime. You will be better off creating a separate Job's for each client and using their own transaction manager in the tasklet.
<bean id="transactionManager1" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory1" />
</bean>
<bean id="transactionManager2" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory2" />
</bean>
Now use these transaction manager when creating the batch job's
<job id="testJob1" xmlns="http://www.springframework.org/schema/batch">
<step id="client1step1">
<tasklet transaction-manager="transactionManager1">
<chunk reader="itemReader" writer="itemWriter" commit-interval="1" />
</tasklet>
</step>
</job>
<job id="testJob2" xmlns="http://www.springframework.org/schema/batch">
<step id="client2step2">
<tasklet transaction-manager="transactionManager2">
<chunk reader="itemReader" writer="itemWriter" commit-interval="1" />
</tasklet>
</step>
</job>
Let me know if this works out.

Use named parameters in queries Spring batch

I have a spring batch job in which a step is as follows:
<bean id="abstractReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" abstract="true">
<property name="fetchSize" value="1000"/>
<property name="verifyCursorPosition" value="true"/>
<property name="rowMapper">
<bean class="org.springframework.jdbc.core.ColumnMapRowMapper"/>
</property>
</bean>
<bean id="masterReader" parent="abstractReader" abstract="true">
<property name="fetchSize" value="1000"/>
<property name="dataSource" ref="masterDataSource"/>
</bean>
<bean id="abstractWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter" abstract="true">
<property name="assertUpdates" value="false"/>
<property name="itemPreparedStatementSetter">
<bean class="org.springframework.batch.item.database.support.ColumnMapItemPreparedStatementSetter"/>
</property>
</bean>
<bean id="masterWriter" parent="abstractWriter" abstract="true">
<property name="dataSource" ref="masterDataSource"/>
</bean>
<bean id="tempWriter" parent="masterWriter" scope="step">
<property name="sql" value="${insert_query}"/>
</bean>
<bean id="tempReader" parent="masterReader" scope="step">
<property name="sql" value="${select_query}"/>
</bean>
<batch:step id="tempStep">
<batch:tasklet>
<batch:chunk commit-interval="100"
reader="tempReader"
writer="tempWriter"/>
</batch:tasklet>
</batch:step>
Is there a way to bring named parameter support in the queries? Currently JdbcCursorItemReader is using PreparedStatement. (Too many ? in queries now)
There isn't a way with the JdbcCursorItemReader however you can do it with the JdbcPagingItemReader. You can read more about that reader in the documentation here: https://docs.spring.io/spring-batch/apidocs/org/springframework/batch/item/database/JdbcPagingItemReader.html

Spring Batch MultiResourceItemWriter doesn't correctly writes data in files

This is my SpringBatch maven depedency:
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-core</artifactId>
<version>2.2.0.RELEASE</version>
</dependency>
Below is my job.xml file
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
<import resource="../config/launch-context.xml" />
<bean id="inputFileForMultiResource" class="org.springframework.core.io.FileSystemResource" scope="step">
<constructor-arg value="src/main/resources/files/customerInputValidation.txt"/>
</bean>
<bean id="outputFileForMultiResource" class="org.springframework.core.io.FileSystemResource" scope="step">
<constructor-arg value="src/main/resources/files/xml/customerOutput.xml"/>
</bean>
<bean id="readerForMultiResource" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" ref="inputFileForMultiResource" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="firstName,middleInitial,lastName,address,city,state,zip" />
<property name="delimiter" value="," />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="customer5TO" />
</bean>
</property>
</bean>
</property>
</bean>
<bean id="customer5TO" class="net.gsd.group.spring.batch.tutorial.to.Customer5TO"></bean>
<bean id="xmlOutputWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
<property name="resource" ref="outputFileForMultiResource" />
<property name="marshaller" ref="customerMarshallerJMSJob" />
<property name="rootTagName" value="customers" />
</bean>
<bean id="customerMarshallerJMSJob" class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases">
<map>
<entry key="customer" value="net.gsd.group.spring.batch.tutorial.to.Customer5TO"></entry>
</map>
</property>
</bean>
<bean id="multiResourceItemWriter" class="org.springframework.batch.item.file.MultiResourceItemWriter">
<property name="resource" ref="customerOutputXmlFile"/>
<property name="delegate" ref="xmlOutputWriter"/>
<property name="itemCountLimitPerResource" value="3"/>
<property name="resourceSuffixCreator" ref="suffix"/>
</bean>
<bean id="suffix" class="net.gsd.group.spring.batch.tutorial.util.CustomerOutputFileSuffixCreator"></bean>
<batch:step id="multiResourceWriterParentStep">
<batch:tasklet>
<batch:chunk
reader="readerForMultiResource"
writer="multiResourceItemWriter"
commit-interval="3">
</batch:chunk>
</batch:tasklet>
</batch:step>
<batch:job id="multiResourceWriterJob">
<batch:step id="multiResourceStep" parent="multiResourceWriterParentStep"/>
</batch:job>
</beans>
Basically I have one input file and more output files. I read the data from the input file (which contains 10 rows) and I want to write it by chunks of 3 into more than one output file (in my case there will be 4 files 3/3/3/1). For each chunk there will be one file.
The job works correctly but the content of the output files is wrong.
Each of the files contain the last element that has bean read in the current chunk.
Let's say the first chunk reads A, B and C. When this chunk is written to the file only the C is written and is written 3 times.
From my tests it works correctly only when the chunk commit-interval is 1.
Is this the correct behaviour?
What am I doing wrong?
Thank you in advance
Your customer5TO bean should have scope prototype. Currently it is a singleton, so the bean properties will always be overwritten with the last item.

I get "Writer must be open before it can be written to" while using ClassifierCompositeItemWriter

The question states my problem. Can you not use FlatFileItemWriters (FFIW) as the writers on anything but simple chunk processing with a single writer? I'm new.
I've attempted to inject an FFIW into ItemProcessors and gotten the same thing. Perhaps I need to write my own custom writers. I was trying to leverage the FFIW to do the work, because all I need is to sift the one input file and populate three outfiles. My routerDelegate works fine, no problems there. Just fails on the write because the file is not open, and I can't see how to manually open it (which I think is the wrong approach, even if I could).
Thanks...
here's my code:
<batch:step id="processCustPermits" next="somethingElse">
<batch:description>Sift permits></batch:description>
<batch:tasklet>
<batch:chunk reader="custPermitReader" writer="custPermitCompositeWriter"
commit-interval="1" />
</batch:tasklet>
</batch:step>
<bean id="custPermitCompositeWriter"
class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
<property name="classifier">
<bean
class="org.springframework.batch.classify.BackToBackPatternClassifier">
<property name="routerDelegate" ref="permitRouterClassifier" />
<property name="matcherMap">
<map>
<entry key="hierarchy" value-ref="custPermitWriter" />
<entry key="omit" value-ref="custPermitOmithWriter" />
<entry key="trash" value-ref="custPermitTrashWriter" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="custPermitWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.heirarchy}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="false" />
</bean>
<bean id="custPermitOmithWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.omits}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="true" />
</bean>
<bean id="custPermitTrashWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.trash}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="true" />
</bean>
Sometimes you just have to read real closely. I added the Streams element to my chunk element and voila!
<batch:step id="processCustPermits" next="somethingElse">
<batch:description>Sort out unwanted permits></batch:description>
<batch:tasklet>
<batch:chunk reader="custPermitReader" writer="custPermitCompositeWriter"
commit-interval="1">
<batch:streams>
<batch:stream ref="custPermitWriter" />
<batch:stream ref="custPermitOmithWriter" />
<batch:stream ref="custPermitTrashWriter" />
</batch:streams>
</batch:chunk>
</batch:tasklet>
</batch:step>
For those who prefer a Java configuration to XML configuration, it is done as follows:
#Bean
public Step processCustPermits(StepBuilderFactory stepBuilderFactory,
#Qualifier("custPermitReader") ItemReader<Wscpos> custPermitReader,
#Qualifier("custPermitCompositeWriter") ItemWriter<Wscpos> custPermitCompositeWriter,
#Qualifier("custPermitWriter") FlatFileItemWriter<Wscpos> custPermitWriter,
#Qualifier("custPermitOmithWriter") FlatFileItemWriter<Wscpos> custPermitOmithWriter,
#Qualifier("custPermitTrashWriter") FlatFileItemWriter<Wscpos> custPermitTrashWriter)
{
return stepBuilderFactory.get("processCustPermits")
.<Wscpos, Wscpos> chunk(1)
.reader(custPermitReader)
.writer(custPermitCompositeWriter)
.stream(writerCustodyMismatch)
.stream(writerNoMatch)
.stream(custPermitTrashWriter)
.build();
}

File to file basics

Being new to Spring Batch, I wanted to begin with something simple....Reading a csv file and writing the same objects(records) to another one. Simple, isn't it ? But I was unable to find a working sample. After some time of research, I found some things which almost work....The file I want to write to is allways empty. Is it because I use a ressourcelesstransactionmanager ? Do I need to declare some optional property somewhere to flush the thing on my hard drive ?
By the way, I find the documentation on the subject very light and confusing, for a beginner. Maybe it's because one must earn spring Batch...
Here's the evil but very simple code that's driving me crazy.
TIA.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.1.xsd">
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.SimpleJobRepository">
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapJobInstanceDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapJobExecutionDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapStepExecutionDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapExecutionContextDao"/>
</constructor-arg>
</bean>
<bean id="jobRepository-transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
p:jobRepository-ref="jobRepository"/>
<!-- Les bean projet -->
<bean id="csvReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
p:resource="file:c:\AppPerso\csvManager\client.csv">
<property name="lineMapper">
<bean
class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer"
p:delimiter=";"
p:names="client_id;client_status;client_name;client_surname;client_dnais;client_addr1;client_addr2;client_addr3;client_addr4"/>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"
p:targetType="com.bigmac.spring.batch.csv.Client"/>
</property>
</bean>
</property>
</bean>
<bean id="csvWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:c:\AppPerso\csvManager\client-reformat.csv"/>
<property name="shouldDeleteIfExists" value="true"/>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";"/>
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="client_id;client_status;client_name;client_surname;client_dnais;client_addr1;client_addr2;client_addr3;client_addr4"/>
</bean>
</property>
</bean>
</property>
</bean>
<batch:job id="CSVManager" job-repository="jobRepository">
<batch:step id="step1">
<batch:tasklet
transaction-manager="jobRepository-transactionManager">
<batch:chunk
reader="csvReader"
writer="csvWriter"
commit-interval="10"/>
</batch:tasklet>
</batch:step>
</batch:job>
</beans>
Sorry guys.
My mistake. I was getting confused with delimiters...The lack of properly configured log4j wasn't helping either.
I put the working configuration here.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch-2.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx-2.1.xsd">
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.SimpleJobRepository">
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapJobInstanceDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapJobExecutionDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapStepExecutionDao"/>
</constructor-arg>
<constructor-arg>
<bean
class="org.springframework.batch.core.repository.dao.MapExecutionContextDao"/>
</constructor-arg>
</bean>
<bean id="jobRepository-transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher"
p:jobRepository-ref="jobRepository"/>
<!-- Les bean projet -->
<bean id="csvReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
p:resource="file:c:\AppPerso\springBatch\csvManager\client.csv">
<property name="lineMapper">
<bean
class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer"
p:delimiter=";"
p:names="client_id,client_status,client_name,client_surname,client_dnais,client_addr1,client_addr2,client_addr3,client_addr4"/>
</property>
<property name="fieldSetMapper">
<bean
class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper"
p:targetType="com.bigmac.spring.batch.csv.Client"/>
</property>
</bean>
</property>
</bean>
<bean id="csvWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:c:\AppPerso\springBatch\csvManager\client-reformat.csv"/>
<property name="shouldDeleteIfExists" value="true"/>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";"/>
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="client_id,client_status,client_name,client_surname,client_dnais,client_addr1,client_addr2,client_addr3,client_addr4"/>
</bean>
</property>
</bean>
</property>
</bean>
<batch:job id="CSVManager" job-repository="jobRepository">
<batch:step id="step1">
<batch:tasklet
transaction-manager="jobRepository-transactionManager">
<batch:chunk
reader="csvReader"
writer="csvWriter"
commit-interval="10"/>
</batch:tasklet>
</batch:step>
</batch:job>
</beans>