I'm reading a comma delimited file which has two fields. The file may not contain the second field at times so Spring DelimitedLineTokenizer should not complain when this happens. By stating the following
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names"
value="planNumber, paymentAmount">
</property>
<property name="delimiter">
<value>,</value>
</property>
</bean>
</property>
Spring does complain
Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 2 actual 1
at org.springframework.batch.item.file.transform.AbstractLineTokenizer.tokenize(AbstractLineTokenizer.java:123)
at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:46)
... 60 more
StringTokenizer would not complain though
set the following property on linetokenizer to false.. this should help avoid the exception getting thrown
<property name="strict" value="false"></property>
Related
I am using spring batch flatfile reader to read a file with header and footer values. Below is sample file and output file should have every record appended with header date and file sequence. Sample input and output as below.
Can someone please suggest
If there is a way to set the header values and footer values as job parameters and used in writer?
Any way to get header and footer values in writer? Thanks in advance!
HH20210218001 --Header starting with "HH", date(20210218), filesequence(001)
name110
name220
name330
name440
name770
FT005 --Footer with "FT" and number of records
Output:(date + file seq(001) + name + age)
20210218001name110
20210218001name330
20210218001name440
20210218001name770
Below is my file reader
<bean id="fileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="resource" value="classpath:input/input.txt"></property>
<!-- <property name="linesToSkip" value="2" /> -->
<property name="lineMapper">
<bean
class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="name,age" />
<property name="columns" value="1-5,6-8"></property>
<property name="strict" value="false" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.test.MyMapper">
</bean>
</property>
</bean>
</property>
</bean>
If there is a way to set the header values and footer values as job parameters and used in writer?
No, it's too late at that point. When the step starts, job parameters are already set and cannot be changed.
Any way to get header and footer values in writer?
From your expected output, you don't need anything from the footer. If you want info from the header in your writer, you can do that in two steps:
step 1: reads (only) the header and puts any needed information in the execution context
step 2: uses a step-scoped writer that gets any required information from the execution context and uses it to write items
I am working on a Spring batch application where i am using FlatFileItemReader to read the file with delimiter ~ or | and its working fine and its calling the processor once read is completed.
But when i try to use the delimiter as \001 the processor is not called and i am not getting any error also in the console.(Linux environment)
Example file format:
0002~000000000000000470~000006206210008078~PR~7044656907~7044641561~~~~240082202~~~ENG~CH~~19940926~D~~~AL~~~P~USA
This is my reader configuration.
<property name="resource" value="#{stepExecutionContext['fileResource']}" />
<!-- <property name="linesToSkip" value="1"></property> -->
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="${file.delimiter}"/>
<property name="names" value="sor_id,sor_cust_id,acct_id,cust_role_type_cd,cust_full_nm,mailg_adr_line_1,mailg_adr_line_2,mailg_city_nm,mailg_geo_st_cd,mailg_full_pstl_cd,mailg_cntry_cd,mailg_adr_desc,phy_adr_line_1,phy_adr_line_2,phy_city_nm,phy_geo_st_cd,phy_full_pstl_cd,phy_cntry_cd,phy_adr_desc,home_phn_num,work_phn_num,mobile_phn_num,email_adr_txt,ssn,cust_tax_idn_num,gndr_cd,martl_cd,lang_cd,acct_stat_cd,cust_brth_dt,acct_open_dt,sor_acct_stat_cd,sor_acct_stat_desc,vld_phn_num_ind,prod_cd,prft_ctr_cd,bus_legl_strc_cd,acct_use_cd,cntry_of_origin_cd" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.cap1.cdi.batch.SrcMasterFieldSetMapper" />
</property>
</bean>
</property>
</bean>
Is anyone else faced the same kind of issue?
Regards,
Shankar
I am going to answer my own question.
The actual issue was control character was used as delimiter in linux (^A)
In Java when i use string.split("\u0001") it was working. Also passing the same to Spring batch flatfileitemreader as delimiter it works like a charm.
Thanks
Shankar.
I have a simple spring batch job - read a file line by line, do something with the input string, and write some output. Output file contains every line of input plus some processing status for that line (success/failure.) The reads a file from: <dir>/<inputFolder>/<inputFileName> and writes processed output to <dir>/<outputFolder>/<inputFileName> All these values are passed as jobParameters
File Reader is like so:
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:#{jobParameters['cwd']}/#{jobParameters['inputFolder']}/#{jobParameters['inputFile']}" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
<property name="fieldSetMapper" >
<bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
</property>
</bean>
</property>
</bean>
Item writer is like so:
<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step" >
<property name="resource" value="#{jobParameters['cwd']}/#{jobParameters['outputFolder']}/#{jobParameters['inputFile']}" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
</property>
</bean>
When I run this batch job, the reader reads the file properly, processor does its job but FileNotFound exception is thrown by the itemWriter
2014/06/27 18-02-31,168:OUT:ERROR[Encountered an error executing the step]
org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt]]
at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:374)
at org.springframework.batch.item.file.FlatFileItemWriter.open(FlatFileItemWriter.java:314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.io.FileNotFoundException: class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt] cannot be resolved to URL because it does not exist
at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:179)
at org.springframework.core.io.AbstractFileResolvingResource.getFile(AbstractFileResolvingResource.java:48)
at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:371)
... 58 more
2014/06/27 18-02-31,168:ERR:ERROR[Encountered an error executing the step]
[org.springframework.batch.item.file.FlatFileItemWriter.getOutputState threw org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt]]]
Batch Execution Failed!
Everytime the batch job runs, the output file does not exist yet. The itemWriter has to create it. Is it not possible using FlatFileItemWriter ?
Adding 'file://' prefix solved my problem. Thanks #LucaBassoRicci.
I have the following spring batch job configuration. There is a single reader which then passes details to a composite writer which has two specific writers. Both writers share a common parent and need to use the same JobId for the INSERT operations they execute.
<bean id="job" parent="simpleJob">
<property name="steps">
<list>
<bean parent="simpleStep">
<property name="itemReader" ref="policyReader"/>
<property name="itemWriter" ref="stagingCompositeWriter"/>
</bean>
</list>
</property>
</bean>
<bean id="stagingCompositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="stagingLoadWriter"/>
<ref bean="stagingPolicyWriter"/>
</list>
</property>
</bean>
<bean id="abstractStagingWriter" class="a.b.c.AbstractStagingWriter" abstract="true">
<property name="stepExecution" value="#{stepExecutionContext}"/>
<property name="hedgingStagingDataSource" ref="hedgingStagingDataSource"/>
</bean>
<bean id="stagingLoadWriter" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter"/>
<bean id="stagingPolicyWriter" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
When i run my code i get the following error
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:208)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:72)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:88)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:139)
I have tried setting the scope="step" in various place but to no avail. Any suggestions?
You can access the stepExecutionContext only within a bean defined in the scope="step".
Change your bean definition to
<bean id="stagingLoadWriter" scope="step" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter" />
<bean id="stagingPolicyWriter" scope="step" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
I have written a simple Spring Batch application that reads a CSV file, does some transforming and writes a modified CSV to the disk.
The reading of the file into domain objects works like a charm. I use DelimitedLineTokenizer to tokenize the lines and a BeanWrapperFieldSetMapper to feed the values into a bean:
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{jobParameters['inputResource']}" />
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
<property name="names"
value="ID,NAME,DESCRIPTION,PRICE,DATE" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="targetType" value="myapp.MyDomainObject" />
<property name="customEditors">
<map>
<entry key="java.util.Date" value-ref="dateEditor" />
<entry key="java.math.BigDecimal" value-ref="numberEditor" />
</map>
</property>
</bean>
</property>
</bean>
</property>
</bean>
I especially like the features of BeanWrapperFieldSetMapper to "guess" the field names and the possibility to define CustomEditors which I use to define the special date and number formats used in the input file.
Now I would like to write the modified file in the same format like the input file.
I use the following configuration:
<bean id="writer" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="#{jobParameters['outputResource']}" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="id,name,description,price,date" />
</bean>
</property>
</bean>
</property>
</bean>
There are two things I miss with this configuration:
BeanWrapperFieldSetMapper allowed me to set CustomEditors, but BeanWrapperFieldExtractor has no such possibility. Is there a way to use these?
Is there a way to define the headings in the first line of the file? I have not found any way to write an initial line that is not a bean... It would be great to use the same names here as in BeanWrapperFieldSetMapper such that BeanWrapperFieldExtractor writes the inital line and guesses the bean property namens as BeanWrapperFieldSetMapper does.
The process to load files is so comfortable in Spring Batch. Why is the writing of files so different? Am I missing something?
I have to use Spring Batch 2.1.x because we are using Spring 3.0.x . Therefor an upgrade to 2.2.x would not be an option.
Which is your need? Extract field property as text? You can
use a FormatterLineAggregator if you needs are not too complicated
write your own CustomEditorsFieldExtractor (better)
Generate a complex domain object composed by original domain object and by text-formatted object and use last one as parameter of writer (but breaks your current processor/writer)
Use FlatFileItemWriter.headerCallback: if setted allow custom header write
Writing - in your case - seems a pain respect read process because spring-batch's reading components fits your needs. Standard components fits more used use-case and they cover a lot of scenario. Let us write a custom FieldExtractor sometimes! :)