I am working on a Spring batch application where i am using FlatFileItemReader to read the file with delimiter ~ or | and its working fine and its calling the processor once read is completed.
But when i try to use the delimiter as \001 the processor is not called and i am not getting any error also in the console.(Linux environment)
Example file format:
0002~000000000000000470~000006206210008078~PR~7044656907~7044641561~~~~240082202~~~ENG~CH~~19940926~D~~~AL~~~P~USA
This is my reader configuration.
<property name="resource" value="#{stepExecutionContext['fileResource']}" />
<!-- <property name="linesToSkip" value="1"></property> -->
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="${file.delimiter}"/>
<property name="names" value="sor_id,sor_cust_id,acct_id,cust_role_type_cd,cust_full_nm,mailg_adr_line_1,mailg_adr_line_2,mailg_city_nm,mailg_geo_st_cd,mailg_full_pstl_cd,mailg_cntry_cd,mailg_adr_desc,phy_adr_line_1,phy_adr_line_2,phy_city_nm,phy_geo_st_cd,phy_full_pstl_cd,phy_cntry_cd,phy_adr_desc,home_phn_num,work_phn_num,mobile_phn_num,email_adr_txt,ssn,cust_tax_idn_num,gndr_cd,martl_cd,lang_cd,acct_stat_cd,cust_brth_dt,acct_open_dt,sor_acct_stat_cd,sor_acct_stat_desc,vld_phn_num_ind,prod_cd,prft_ctr_cd,bus_legl_strc_cd,acct_use_cd,cntry_of_origin_cd" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.cap1.cdi.batch.SrcMasterFieldSetMapper" />
</property>
</bean>
</property>
</bean>
Is anyone else faced the same kind of issue?
Regards,
Shankar
I am going to answer my own question.
The actual issue was control character was used as delimiter in linux (^A)
In Java when i use string.split("\u0001") it was working. Also passing the same to Spring batch flatfileitemreader as delimiter it works like a charm.
Thanks
Shankar.
Related
Need to read multiple txt files and generate multiple csv files,
I want to read multiple txt files such that i can select some columns from one txt file and some from >other.My txt file is pipe seperated.
i am able to read a single txt file and generate multiple csv files
Thanks in advance.
<!-- ItemReader reads a complete line one by one from input file -->
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="classpath:student.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<!-- Mapper which maps each individual items in a record to properties in POJO -->
<bean class="com.springbatch.ExamResultFieldSetMapper" />
</property>
<property name="lineTokenizer">
<!-- A tokenizer class to be used when items in input record are separated by specific characters -->
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="|" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- CSV ItemWriter which writes the data in CSV format -->
<bean id="CSVWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:output/examResultcsv1.csv"></property>
<property name="shouldDeleteIfExists" value="true"></property>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=","></property>
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="studentName, percentage" />
</bean>
</property>
</bean>
</property>
</bean>
I was trying to share My in-memory jobRepository to the jobExplorer. But it throws an error as,
Nested exception is
org.springframework.beans.ConversionNotSupportedException:
Failed to convert property value of type '$Proxy1 implementing
org.springframework.batch.core.repository.JobRepository,org.
springframework.aop.SpringProxy,org.springframework.aop.framework.Advised'
to required type
Even i tried putting '&' sign before jobRepository when passing to jobExplorer for sharing.But attempt end in vain.
I am using Spring Batch 2.2.1
Is the dependency for jobExplorer is only database not in-memory?
Definition is,
<bean id="jobRepository"
class="com.test.repository.BatchRepositoryFactoryBean">
<property name="cache" ref="cache" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="jobOperator" class="test.batch.LauncherTest.TestBatchOperator">
<property name="jobExplorer" ref="jobExplorer" />
<property name="jobRepository" ref="jobRepository" />
<property name="jobRegistry" ref="jobRegistry" />
<property name="jobLauncher" ref="jobLauncher" />
</bean>
<bean id="jobExplorer" class="test.batch.LauncherTest.TestBatchExplorerFactoryBean">
<property name="repositoryFactory" ref="&jobRepository" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher" class="com.scb.smartbatch.core.BatchLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- To store Batch details -->
<bean id="jobRegistry" class="com.scb.smartbatch.repository.SmartBatchRegistry" />
<bean id="jobRegistryBeanPostProcessor"
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<!--Runtime cache of batch executions -->
<bean id="cache" class="com.scb.cache.TCRuntimeCache" />
thanks for your valuable inputs.
But I used '&' before the job repository reference, which allowed me to use it for my job explorer as a shared resource.
problem solved.
kudos.
Usually you have to wire interface instead of implementation.
Else, probably, you have to add <aop:config proxy-target-class="true"> to create CGLIB-based proxy instead of standard Java-based proxy.
Read Spring official documentation about that
I have written a simple Spring Batch application that reads a CSV file, does some transforming and writes a modified CSV to the disk.
The reading of the file into domain objects works like a charm. I use DelimitedLineTokenizer to tokenize the lines and a BeanWrapperFieldSetMapper to feed the values into a bean:
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{jobParameters['inputResource']}" />
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
<property name="names"
value="ID,NAME,DESCRIPTION,PRICE,DATE" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="targetType" value="myapp.MyDomainObject" />
<property name="customEditors">
<map>
<entry key="java.util.Date" value-ref="dateEditor" />
<entry key="java.math.BigDecimal" value-ref="numberEditor" />
</map>
</property>
</bean>
</property>
</bean>
</property>
</bean>
I especially like the features of BeanWrapperFieldSetMapper to "guess" the field names and the possibility to define CustomEditors which I use to define the special date and number formats used in the input file.
Now I would like to write the modified file in the same format like the input file.
I use the following configuration:
<bean id="writer" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="#{jobParameters['outputResource']}" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="id,name,description,price,date" />
</bean>
</property>
</bean>
</property>
</bean>
There are two things I miss with this configuration:
BeanWrapperFieldSetMapper allowed me to set CustomEditors, but BeanWrapperFieldExtractor has no such possibility. Is there a way to use these?
Is there a way to define the headings in the first line of the file? I have not found any way to write an initial line that is not a bean... It would be great to use the same names here as in BeanWrapperFieldSetMapper such that BeanWrapperFieldExtractor writes the inital line and guesses the bean property namens as BeanWrapperFieldSetMapper does.
The process to load files is so comfortable in Spring Batch. Why is the writing of files so different? Am I missing something?
I have to use Spring Batch 2.1.x because we are using Spring 3.0.x . Therefor an upgrade to 2.2.x would not be an option.
Which is your need? Extract field property as text? You can
use a FormatterLineAggregator if you needs are not too complicated
write your own CustomEditorsFieldExtractor (better)
Generate a complex domain object composed by original domain object and by text-formatted object and use last one as parameter of writer (but breaks your current processor/writer)
Use FlatFileItemWriter.headerCallback: if setted allow custom header write
Writing - in your case - seems a pain respect read process because spring-batch's reading components fits your needs. Standard components fits more used use-case and they cover a lot of scenario. Let us write a custom FieldExtractor sometimes! :)
I would like to execute a Stored Procedure using spring JdbcBatchItemWriter. My current code looks like :
<bean id="xyzWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
......
<property name="sql" value="update abc where x=:paramX" />
......
</bean>
I would like to replace this update sql query with a Stored Proc call. I would like to handle it in the xml file itself. Any help is really appreciated.
Thanks
Did you tried running SP through JdbcBatchItemWriter?
because I also had same requirement and i just tried and it worked for me
<bean id="trackItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="mySQLDatasource"/>
<property name="itemPreparedStatementSetter">
<bean class="com.MyDataPreparedStatmentSetter"/>
</property>
<property name="assertUpdates" value="false" />
<property name="sql" value="Call my_Stored_Proc (?,?,?,?)"/>
</bean>
Hope it helps.
Using JBoss 4.0.5, JBossMQ, and Spring 2.0.8, I am trying to configure Spring to instantiate beans which depend on a remote JMS Queue resource. All of the examples I've come across depend on using JNDI to do lookup for things like the remote ConnectionFactory object.
My problem is when trying to bring up a machine which would put messages into the remote queue, if the remote machine is not up, JNDI lookup simply fails, causing deployment to fail. Is there a way to get Spring to keep trying to lookup this object in the background while not blocking the remainder of deployment?
Iit's difficult to be sure without seeing your spring config, but assuming you're using Spring's JndiObjectFactoryBean to do the JNDI lookup, then you can set the lookupOnStartup property to false, which allows the context to start up even if the JNDI target isn't there. The JNDI resolution will be done the first time the ConnectionFactory is used.
However, this just shifts the problem further up the chain, because if some other component tries to get a JMS Connection on startup, then you're back where you started. You can use the lazy-init="true" attribute on your other beans to prevent this from happening on deployment, but it's easy to accidentally put something in your config which forces everything to initialize.
You're absolutely right. I tried setting lookupOnStartup to false and lazy-init=true . This just defers the problem to the first time that the Queue is attempted to be used. Then an exception as follows is thrown:
[org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.CloseMsg29702787[msgType: m_connectionClosing, msgID: -2147483606, error: null]
java.io.IOException: Client is not connected
Moreover, it looks like the lookup is never attempted again. When the machine with the remote queue is brought back up, no messages are ever processed subsequently. This really does seem like it should be well within the envelope of use cases for J2EE nonsense, and yet I'm not having much luck... It feels like it should even maybe be a solved problem.
For completion's sake, the following is the pertinent portion of my Spring configuration.
<bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
<property name="environment">
<props>
<prop key="java.naming.provider.url">localhost:1099</prop>
<prop key="java.naming.factory.url.pkgs">org.jnp.interfaces:org.jboss.naming</prop>
<prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</prop>
</props>
</property>
</bean>
<bean id="connectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate">
<ref bean="jndiTemplate"/>
</property>
<property name="jndiName">
<value>ConnectionFactory</value>
</property>
</bean>
<bean id="remoteJndiTemplate" class="org.springframework.jndi.JndiTemplate" lazy-init="true">
<property name="environment">
<props>
<prop key="java.naming.provider.url">jnp://10.0.100.232:1099</prop>
<prop key="java.naming.factory.url.pkgs">org.jnp.interfaces:org.jboss.naming</prop>
<prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</prop>
</props>
</property>
</bean>
<bean id="remoteConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean" lazy-init="true">
<property name="jndiTemplate" ref="remoteJndiTemplate"/>
<property name="jndiName" value="ConnectionFactory" />
<property name="lookupOnStartup" value="false" />
<property name="proxyInterface" value="javax.jms.ConnectionFactory" />
</bean>
<bean id="destinationResolver" class="com.foo.jms.FooDestinationResolver" />
<bean id="localVoicemailTranscodingDestination" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="jndiTemplate"/>
<property name="jndiName" value="queue/voicemailTranscoding" />
</bean>
<bean id="globalVoicemailTranscodingDestination" class="org.springframework.jndi.JndiObjectFactoryBean" lazy-init="true" >
<property name="jndiTemplate" ref="remoteJndiTemplate" />
<property name="jndiName" value="queue/globalVoicemailTranscoding" />
</bean>
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate" >
<property name="connectionFactory" ref="connectionFactory"/>
<property name="defaultDestination" ref="localVoicemailTranscodingDestination" />
</bean>
<bean id="remoteJmsTemplate" class="org.springframework.jms.core.JmsTemplate" lazy-init="true">
<property name="connectionFactory" ref="remoteConnectionFactory"/>
<property name="destinationResolver" ref="destinationResolver"/>
</bean>
<bean id="globalQueueStatus" class="com.foo.bar.recording.GlobalQueueStatus" />
<!-- Do not deploy this bean for machines other than transcoding machine -->
<condbean:cond test="${transcoding.server}">
<bean id="voicemailMDPListener"
class="org.springframework.jms.listener.adapter.MessageListenerAdapter" lazy-init="true">
<constructor-arg>
<bean class="com.foo.bar.recording.mdp.VoicemailMDP" lazy-init="true">
<property name="manager" ref="vmMgr" />
</bean>
</constructor-arg>
</bean>
</condbean:cond>
<bean id="voicemailForwardingMDPListener"
class="org.springframework.jms.listener.adapter.MessageListenerAdapter" lazy-init="true">
<constructor-arg>
<bean class="com.foo.bar.recording.mdp.QueueForwardingMDP" lazy-init="true">
<property name="queueStatus" ref="globalQueueStatus" />
<property name="template" ref="remoteJmsTemplate" />
<property name="remoteDestination" ref="globalVoicemailTranscodingDestination" />
</bean>
</constructor-arg>
</bean>
<bean id="prototypeListenerContainer"
class="org.springframework.jms.listener.DefaultMessageListenerContainer"
abstract="true"
lazy-init="true">
<property name="concurrentConsumers" value="5" />
<property name="connectionFactory" ref="connectionFactory" />
<!-- 2 is CLIENT_ACKNOWLEDGE: http://java.sun.com/j2ee/1.4/docs/api/constant-values.html#javax.jms.Session.CLIENT_ACKNOWLEDGE -->
<!-- 1 is autoacknowldge -->
<property name="sessionAcknowledgeMode" value="1" />
<property name="sessionTransacted" value="true" />
</bean>
<!-- Do not deploy this bean for machines other than transcoding machine -->
<condbean:cond test="${transcoding.server}">
<bean id="voicemailMDPContainer" parent="prototypeListenerContainer" lazy-init="true">
<property name="destination" ref="globalVoicemailTranscodingDestination" />
<property name="messageListener" ref="voicemailMDPListener" />
</bean>
</condbean:cond>
<bean id="voicemailForwardMDPContainer" parent="prototypeListenerContainer" lazy-init="true">
<property name="destination" ref="localVoicemailTranscodingDestination" />
<property name="messageListener" ref="voicemailForwardingMDPListener" />
</bean>