Need to read multiple txt files and generate multiple csv files,
I want to read multiple txt files such that i can select some columns from one txt file and some from >other.My txt file is pipe seperated.
i am able to read a single txt file and generate multiple csv files
Thanks in advance.
<!-- ItemReader reads a complete line one by one from input file -->
<bean id="flatFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="classpath:student.txt" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="fieldSetMapper">
<!-- Mapper which maps each individual items in a record to properties in POJO -->
<bean class="com.springbatch.ExamResultFieldSetMapper" />
</property>
<property name="lineTokenizer">
<!-- A tokenizer class to be used when items in input record are separated by specific characters -->
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="|" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- CSV ItemWriter which writes the data in CSV format -->
<bean id="CSVWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource" value="file:output/examResultcsv1.csv"></property>
<property name="shouldDeleteIfExists" value="true"></property>
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=","></property>
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="studentName, percentage" />
</bean>
</property>
</bean>
</property>
</bean>
I am working on a Spring batch application where i am using FlatFileItemReader to read the file with delimiter ~ or | and its working fine and its calling the processor once read is completed.
But when i try to use the delimiter as \001 the processor is not called and i am not getting any error also in the console.(Linux environment)
Example file format:
0002~000000000000000470~000006206210008078~PR~7044656907~7044641561~~~~240082202~~~ENG~CH~~19940926~D~~~AL~~~P~USA
This is my reader configuration.
<property name="resource" value="#{stepExecutionContext['fileResource']}" />
<!-- <property name="linesToSkip" value="1"></property> -->
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="${file.delimiter}"/>
<property name="names" value="sor_id,sor_cust_id,acct_id,cust_role_type_cd,cust_full_nm,mailg_adr_line_1,mailg_adr_line_2,mailg_city_nm,mailg_geo_st_cd,mailg_full_pstl_cd,mailg_cntry_cd,mailg_adr_desc,phy_adr_line_1,phy_adr_line_2,phy_city_nm,phy_geo_st_cd,phy_full_pstl_cd,phy_cntry_cd,phy_adr_desc,home_phn_num,work_phn_num,mobile_phn_num,email_adr_txt,ssn,cust_tax_idn_num,gndr_cd,martl_cd,lang_cd,acct_stat_cd,cust_brth_dt,acct_open_dt,sor_acct_stat_cd,sor_acct_stat_desc,vld_phn_num_ind,prod_cd,prft_ctr_cd,bus_legl_strc_cd,acct_use_cd,cntry_of_origin_cd" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.cap1.cdi.batch.SrcMasterFieldSetMapper" />
</property>
</bean>
</property>
</bean>
Is anyone else faced the same kind of issue?
Regards,
Shankar
I am going to answer my own question.
The actual issue was control character was used as delimiter in linux (^A)
In Java when i use string.split("\u0001") it was working. Also passing the same to Spring batch flatfileitemreader as delimiter it works like a charm.
Thanks
Shankar.
I created a job and archived it into a jar.
When I run it with exec:java it works. When I invoke it with java -cp myjar.jar, MultiResourceItemReader doesn't seem read my resources what fetched from ftp server.
<bean id="merge.reader.resource"
class="org.springframework.batch.item.file.MultiResourceItemReader"
scope="step">
<property name="resources" value="file:job1/merge/fetched/*.xml" />
...
</bean>
What did I do wrong?
$ ls -l
mybatch.jar
job1/merge/fetched/xxxx.xml
$ java -cp mybatch.jar x.y.z.Main
I created a class extending MultiResourceItemReader and checked that file sources are set via setResources method and delegate via setDelegate.
<!--bean id="merge.reader.resource"
class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step"-->
<bean id="merge.reader.resource"
class="xxx.ExtendedMultiResourceItemReader"
scope="step">
<property name="strict" value="true"/>
<property name="resources" value="file:job1/merge/fetched/*.xml"/>
<property name="delegate" ref="merge.reader.item" />
</bean>
<bean id="merge.reader.item"
class="org.springframework.batch.item.xml.StaxEventItemReader">
<property name="fragmentRootElementName" value="XXX"/>
<property name="unmarshaller" ref="merge.reader.unmarshaller"/>
</bean>
<bean id="merge.reader.unmarshaller"
class="org.springframework.oxm.xstream.XStreamMarshaller">
<property name="aliases" ref="merge.reader.binder"/>
<property name="autodetectAnnotations" value="true"/>
</bean>
<util:map id="merge.reader.binder">
<entry key="XXX" value="xxx"/>
</util:map>
I really don't understand why reader don't process files.
I have the following spring batch job configuration. There is a single reader which then passes details to a composite writer which has two specific writers. Both writers share a common parent and need to use the same JobId for the INSERT operations they execute.
<bean id="job" parent="simpleJob">
<property name="steps">
<list>
<bean parent="simpleStep">
<property name="itemReader" ref="policyReader"/>
<property name="itemWriter" ref="stagingCompositeWriter"/>
</bean>
</list>
</property>
</bean>
<bean id="stagingCompositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="stagingLoadWriter"/>
<ref bean="stagingPolicyWriter"/>
</list>
</property>
</bean>
<bean id="abstractStagingWriter" class="a.b.c.AbstractStagingWriter" abstract="true">
<property name="stepExecution" value="#{stepExecutionContext}"/>
<property name="hedgingStagingDataSource" ref="hedgingStagingDataSource"/>
</bean>
<bean id="stagingLoadWriter" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter"/>
<bean id="stagingPolicyWriter" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
When i run my code i get the following error
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:208)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:72)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:88)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:139)
I have tried setting the scope="step" in various place but to no avail. Any suggestions?
You can access the stepExecutionContext only within a bean defined in the scope="step".
Change your bean definition to
<bean id="stagingLoadWriter" scope="step" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter" />
<bean id="stagingPolicyWriter" scope="step" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
I'm reading a comma delimited file which has two fields. The file may not contain the second field at times so Spring DelimitedLineTokenizer should not complain when this happens. By stating the following
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names"
value="planNumber, paymentAmount">
</property>
<property name="delimiter">
<value>,</value>
</property>
</bean>
</property>
Spring does complain
Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 2 actual 1
at org.springframework.batch.item.file.transform.AbstractLineTokenizer.tokenize(AbstractLineTokenizer.java:123)
at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:46)
... 60 more
StringTokenizer would not complain though
set the following property on linetokenizer to false.. this should help avoid the exception getting thrown
<property name="strict" value="false"></property>