Using optional fields with StaxEventItemReader - spring-batch

I have a Spring Batch application and I'm using the StaxEventItemReader as my ItemReader. By default XStream requires us to declare a property for each possible XML tag or else it throws an UnknownFieldException exception. There are ways to code around this with Java but with Spring Batch, the InputReader doesn't seem to have a way to modify it. Is there a way to flag fields as optional in the xml?
My bean is configured basically like this
<job id="synchronizecustomerData" xmlns="http://www.springframework.org/schema/batch">
<step id="readWritecustomers">
<tasklet>
<chunk reader="customerReader"
processor="customerProcessor"
writer="customerSyncWriter"
commit-interval="1"
skip-policy="alwaysSkip" >
</chunk>
</tasklet>
</step>
</job>
<bean id="customerReader" class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="customer" />
<property name="resource" ref="inputResource" />
<property name="unmarshaller" ref="customerMarshaller" />
</bean>
<bean id="inputResource" class="org.springframework.core.io.FileSystemResource">
<constructor-arg value="c:/sf/data.xml" />
</bean>
<bean id="customerMarshaller" class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases">
<util:map id="aliases">
<entry key="customer" value="com.company.batchmaster.sf.beans.customer" />
<entry key="name" value="java.lang.String" />
</util:map>
</property>
</bean>
<bean id="customerProcessor" class="org.springframework.batch.item.support.CompositeItemProcessor">
<property name="delegates">
<list>
<ref bean="customerTransformer" />
</list>
</property>
</bean>
<bean id="customerTransformer" class="com.company.batchmaster.sf.chunk.customerTransformer" />
<bean id="customerSyncWriter" class="com.company.batchmaster.sf.chunk.customerSyncWriter" />
My import file looks like this, just getting it up and running
<?xml version="1.0" encoding="UTF-8"?>
<records>
<customer xmlns="http://springframework.org/batch/sample/io/oxm/domain">
<name>ABC Dealer</name>
<types>CR</types>
</customer>
</records>
Thanks for any help.

I am assuming Customer class has properties name and type.
Annotate it as XmlAttribute.defaultValue() as described in Jaxb guide
No need to this (<entry key="name" value="java.lang.String" />) alias because you are unmarshalling a complete Customer object from a node as specified with <property name="fragmentRootElementName" value="customer" />

Related

Spring Batch - create a new file each time instead of overriding it for transferring data from CSV to XML

I am new to Spring Batch. I was trying to shift data from CSV file to XML file & able to shift it successfully. But when each time I run the code my XML (output file) getting override which I dont want, instead I want to create new output file (old output files should be there, require for data tracking purpose) for each run. How can I do that ?
Here is the my code: What I need to change in below file? Let me know if you need more file code from my side.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- ============= ItemReader reads a complete line one by one from input file ============ -->
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<!-- Get the Resource file -->
<property name="resource" value="classpath:ExamResult.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<!-- Mapper which maps each individual items in a record to properties in POJO -->
<bean class="com.websystique.springbatch.mapper.ExamResultFieldSetMapper" />
</property>
<property name="lineTokenizer">
<!-- A tokenizer class to be used when items in input record are separated by specific characters -->
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="|" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- ======== XML ItemWriter which writes the data in XML format =========== -->
<bean id="xmlItemWriter" class="org.springframework.batch.item.xml.StaxEventItemWriter">
<property name="resource" value="file:xml/ExamResult.xml" />
<property name="rootTagName" value="UniversityExamResultList" />
<property name="marshaller">
<bean class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<property name="classesToBeBound">
<list>
<value>com.websystique.springbatch.model.ExamResult</value>
</list>
</property>
</bean>
</property>
</bean>
<!-- Optional ItemProcessor to perform business logic/filtering on the input records -->
<bean id="itemProcessor" class="com.websystique.springbatch.processor.ExamResultItemProcessor" />
<!-- Optional JobExecutionListener to perform business logic before and after the job -->
<bean id="jobListener" class="com.websystique.springbatch.listener.ExamResultJobListener" />
<!-- Step will need a transaction manager -->
<bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<!-- ==================== Actual Job =================== -->
<batch:job id="examResultJob">
<batch:step id="step1">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="flatFileItemReader" writer="xmlItemWriter" processor="itemProcessor" commit-interval="10" />
</batch:tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="jobListener" />
</batch:listeners>
</batch:job>
</beans>
Try using the Spring Expression Language (SpEL) to add a date and time to the end of the output file name. Something like:
<property name="resource"
value="file:xml/ExamResult-#{new java.text.SimpleDateFormat("Mddyyyyhhmmss").format(new java.util.GregorianCalendar().getTime())}.xml" />

Mapping JPA entity to more than one entityManagers with SpringBatch program

I have developed SpringBatch application and deployed as Web Application in Websphere Liberty profile container. The batch program is designed to read records from a table and invokes HTTP service. Based on the service response a column named status is updated as RECORD_SENT/COMPLETE/ERROR type.
Objective is to reuse the same program for multiple datasources. The data source is passed in job parameter using client type. The datasources are in different schemas but having same datamodel.
Question: How does the transaction manager can be applied at run time inside Job Step or Tasklet?. Seeking help in this regard.
Configuration:
<bean id="entityManagerFactory1"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource1" />
<property name="persistenceUnitName" value="user" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="false" />
</bean>
</property>
<property name="jpaDialect">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect" />
</property>
</bean>
<bean id="entityManagerFactory2"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource2" />
<property name="persistenceUnitName" value="user" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="false" />
</bean>
</property>
<property name="jpaDialect">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect" />
</property>
</bean>
<bean id="entityManagerSelector" class="*com.spring.jpa.test.EntitymanagerSelector">
<property name="entityManagerFactory1" ref="entityManagerFactory1"></property>
<property name="entityManagerFactory2" ref="entityManagerFactory2"></property>
</bean>
job.xml snippet
<bean id="itemReader" class="org.springframework.batch.item.database.JpaPagingItemReader" scope="step">
<property name="entityManagerFactory" value="#{entityManagerSelector.getEntitymanagerForClient({jobParameters['client']})}" />
<property name="queryString" value="select u from User u where u.age > #{jobParameters['age']}" />
</bean>
Setting the job parameters during runtime to identify the client
JobParameters param = new JobParametersBuilder()
.addString("age", "20").addString("client", "client2")
.toJobParameters();
JobExecution execution = jobLauncher.run(job, param);
It will not be possible for you to set the transaction-manager of the Step/tasklet during runtime. You will be better off creating a separate Job's for each client and using their own transaction manager in the tasklet.
<bean id="transactionManager1" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory1" />
</bean>
<bean id="transactionManager2" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory2" />
</bean>
Now use these transaction manager when creating the batch job's
<job id="testJob1" xmlns="http://www.springframework.org/schema/batch">
<step id="client1step1">
<tasklet transaction-manager="transactionManager1">
<chunk reader="itemReader" writer="itemWriter" commit-interval="1" />
</tasklet>
</step>
</job>
<job id="testJob2" xmlns="http://www.springframework.org/schema/batch">
<step id="client2step2">
<tasklet transaction-manager="transactionManager2">
<chunk reader="itemReader" writer="itemWriter" commit-interval="1" />
</tasklet>
</step>
</job>
Let me know if this works out.

Spring Batch issue w/ CompositeItemWriter and ClassifierCompositeItemWriter

I am using spring batch in order to read data from the database (using partitioning) and writing the same to a set of files based upon entry keys - 1,2,3,4.
I have created a CompositeItemWriter which is a composition of two ClassifierCompositeItemWriter(s). Even though I have registered the individual writers as stream, I still get the following exception:
org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to
I even tried registering ItemWriter1 and ItemWriter2 as streams, but, that gives me a different error:
Caused by: java.lang.IllegalStateException: Cannot convert value of type [com.sun.proxy.$Proxy13 implementing org.springframework.batch.item.ItemWriter,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised] to required type [org.springframework.batch.item.ItemStream] for property 'streams[0]': no matching editors or conversion strategy found
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:264)
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:128)
at org.springframework.beans.TypeConverterDelegate.convertToTypedArray(TypeConverterDelegate.java:463)
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:195)
at org.springframework.beans.BeanWrapperImpl.convertIfNecessary(BeanWrapperImpl.java:448)
... 74 more
I have even implemented the ItemStream in the writers, but, it does not work yet.
public class WriterA1 implements ItemWriter<List<Object>>, ItemStream {
...
}
The following is the xml configuration:
...
<job id="abcJob" xmlns="http://www.springframework.org/schema/batch"
restartable="true">
<step id="masterStep">
<partition step="slaveStep" partitioner="abcPartitioner">
<handler grid-size="${grid-size}" task-executor="abcTaskExecutor" />
</partition>
</step>
</job>
<step id="slaveStep" xmlns="http://www.springframework.org/schema/batch">
<tasklet transaction-manager="transactionManager">
<chunk reader="abcReader" writer="abcWriter"
processor="abcProcessor" commit-interval="${a}" skip-limit="${b}" retry-limit="${c}" >
<streams>
<!--
<stream ref="ItemWriter1"/>
<stream ref="ItemWriter2"/>
-->
<stream ref="WriterA1"/>
<stream ref="WriterB2"/>
<stream ref="WriterC3"/>
<stream ref="WriterD4"/>
<stream ref="WriterA5"/>
<stream ref="WriterB6"/>
<stream ref="WriterC7"/>
<stream ref="WriterD8"/>
</streams>
</chunk>
<listeners>
...
</listeners>
</tasklet>
</step>
<bean id="abcWriter" class="org.springframework.batch.item.support.CompositeItemWriter" scope="step">
<property name="delegates">
<list>
<ref bean="ItemWriter1" />
<ref bean="ItemWriter2" />
</list>
</property>
</bean>
<bean id="ItemWriter1" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter" scope="step">
<property name="classifier">
<bean
class="org.springframework.classify.BackToBackPatternClassifier">
<property name="routerDelegate">
<bean class="xxx.xxx.xxx.xxx.Classifier1" scope="step"/>
</property>
<property name="matcherMap">
<map>
<entry key="1" value-ref="WriterA1" />
<entry key="2" value-ref="WriterB2" />
<entry key="3" value-ref="WriterC3" />
<entry key="4" value-ref="WriterD4" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="ItemWriter2" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter" scope="step">
<property name="classifier">
<bean
class="org.springframework.classify.BackToBackPatternClassifier">
<property name="routerDelegate">
<bean class="xxx.xxx.xxx.xxx.Classifier2" scope="step"/>
</property>
<property name="matcherMap">
<map>
<entry key="1" value-ref="WriterA5" />
<entry key="2" value-ref="WriterB6" />
<entry key="3" value-ref="WriterC7" />
<entry key="4" value-ref="WriterD8" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="WriterA1" class="xxx.xxx.xxx.xxx.WriterA1" scope="step">
</bean>
<bean id="WriterB2" class="xxx.xxx.xxx.xxx.WriterB2" scope="step">
</bean>
<bean id="WriterC3" class="xxx.xxx.xxx.xxx.WriterC3" scope="step">
</bean>
<bean id="WriterD4" class="xxx.xxx.xxx.xxx.WriterD4" scope="step">
</bean>
<bean id="WriterA5" class="xxx.xxx.xxx.xxx.WriterA5" scope="step">
</bean>
<bean id="WriterB6" class="xxx.xxx.xxx.xxx.WriterB6" scope="step">
</bean>
<bean id="WriterC7" class="xxx.xxx.xxx.xxx.WriterC7" scope="step">
</bean>
<bean id="WriterD8" class="xxx.xxx.xxx.xxx.WriterD8" scope="step">
</bean>
Please advise.
You have three types of writers. From top to bottom:
abcWriter is a CompositeItemWriter. It implements ItemStream by delegating the ItemStream method calls to the delegates (here ItemWriter1 and ItemWriter2), provided they implement ItemStream. Which is not the case. But even if they implemented ItemStream, you shouldn't separately register ItemWriter1 and ItemWriter2 as streams in the step configuration (there's another independent reason in the next bullet point).
ItemWriter1/ItemWriter2 are ClassifierCompositeItemWriters. This class doesn't implement ItemStream, so you must not register them as streams in the step configuration.
WriterA1 are of type BeanIOFlatFileItemWriter and therefore implement ItemStream. Because the ClassifierCompositeItemWriter that wraps them doesn't call their ItemStream methods (unlike CompositeItemWriter), you must register each one of them as streams in the step configuration.
But this is what you claim you have. Yet your scoped-scoped WriterXX beans are being proxied (with interface proxy mode) through singleton beans that don't implement ItemStream or ItemStreamWriter, instead only ItemWriter. Make sure that the classes you have inside the <bean> elements do implement ItemStream. You can also try creating the scoped proxy bean explicitly (using ScopedProxyFactoryBean and setting the interfaces property). Or you can try putting a breakpoint in ScopedProxyFactoryBean::setBeanFactory breaking when targetBeanName is contains the string WriterXX (it will be something like stepScopedTarget.WriterD8) and try to understand why the ItemStream interface is not being proxied.

I get "Writer must be open before it can be written to" while using ClassifierCompositeItemWriter

The question states my problem. Can you not use FlatFileItemWriters (FFIW) as the writers on anything but simple chunk processing with a single writer? I'm new.
I've attempted to inject an FFIW into ItemProcessors and gotten the same thing. Perhaps I need to write my own custom writers. I was trying to leverage the FFIW to do the work, because all I need is to sift the one input file and populate three outfiles. My routerDelegate works fine, no problems there. Just fails on the write because the file is not open, and I can't see how to manually open it (which I think is the wrong approach, even if I could).
Thanks...
here's my code:
<batch:step id="processCustPermits" next="somethingElse">
<batch:description>Sift permits></batch:description>
<batch:tasklet>
<batch:chunk reader="custPermitReader" writer="custPermitCompositeWriter"
commit-interval="1" />
</batch:tasklet>
</batch:step>
<bean id="custPermitCompositeWriter"
class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
<property name="classifier">
<bean
class="org.springframework.batch.classify.BackToBackPatternClassifier">
<property name="routerDelegate" ref="permitRouterClassifier" />
<property name="matcherMap">
<map>
<entry key="hierarchy" value-ref="custPermitWriter" />
<entry key="omit" value-ref="custPermitOmithWriter" />
<entry key="trash" value-ref="custPermitTrashWriter" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="custPermitWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.heirarchy}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="false" />
</bean>
<bean id="custPermitOmithWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.omits}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="true" />
</bean>
<bean id="custPermitTrashWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="${sap.cust.permit.outfile.trash}" />
<property name="lineAggregator" ref="passThroughLineAggregator" />
<property name="shouldDeleteIfExists" value="true" />
<property name="shouldDeleteIfEmpty" value="true" />
</bean>
Sometimes you just have to read real closely. I added the Streams element to my chunk element and voila!
<batch:step id="processCustPermits" next="somethingElse">
<batch:description>Sort out unwanted permits></batch:description>
<batch:tasklet>
<batch:chunk reader="custPermitReader" writer="custPermitCompositeWriter"
commit-interval="1">
<batch:streams>
<batch:stream ref="custPermitWriter" />
<batch:stream ref="custPermitOmithWriter" />
<batch:stream ref="custPermitTrashWriter" />
</batch:streams>
</batch:chunk>
</batch:tasklet>
</batch:step>
For those who prefer a Java configuration to XML configuration, it is done as follows:
#Bean
public Step processCustPermits(StepBuilderFactory stepBuilderFactory,
#Qualifier("custPermitReader") ItemReader<Wscpos> custPermitReader,
#Qualifier("custPermitCompositeWriter") ItemWriter<Wscpos> custPermitCompositeWriter,
#Qualifier("custPermitWriter") FlatFileItemWriter<Wscpos> custPermitWriter,
#Qualifier("custPermitOmithWriter") FlatFileItemWriter<Wscpos> custPermitOmithWriter,
#Qualifier("custPermitTrashWriter") FlatFileItemWriter<Wscpos> custPermitTrashWriter)
{
return stepBuilderFactory.get("processCustPermits")
.<Wscpos, Wscpos> chunk(1)
.reader(custPermitReader)
.writer(custPermitCompositeWriter)
.stream(writerCustodyMismatch)
.stream(writerNoMatch)
.stream(custPermitTrashWriter)
.build();
}

Spring Batch File Writer Exception handling

I have a Spring Batch process which has following kind of code.
<step id="step1" xmlns="http://www.springframework.org/schema/batch">
<tasklet allow-start-if-complete="true">
<chunk reader="reader1" writer="writer1" commit-interval="10"/>
</tasklet>
</step>
<bean id="writer1" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" ref="resourceFlatFile" />
<property name="shouldDeleteIfExists" value="true" />
<property name="transactional" value="true" />
<property name = "lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator" >
<property name="delimiter" value=""/>
<property name ="fieldExtractor">
<bean class="com.path.MyExtractor" />
</property>
</bean>
</property>
</bean>
Basically my reader gives set of records from database. My writer (writer1) writes it to a flat file. If there is any problem in writing a record to the file, I would like to mark that record status as failed in database. So how to handle these kind of scenarios? Any help is appreciated.
Thanks
My question is if I get any kind of exception
I would recommend you look into using a ItemWriteListener and update the status of the failed records in the onWriteError implementation.