I am relatively new to Spring Batch.
I have an input file with a header. This header contains several fields, one of which I am interested in (YYYYMM data).
Here is my config for this :
<bean id="detaillesHeaderReaderCallback" class="fr.generali.ede.daemon.batch.dstaff.detailles.DetaillesHeaderReaderCallback" >
<property name="headerTokenizer" ref="headerTokenizer" />
<property name="fieldSetMapper" ref="fieldSetMapperHeaderLog07" />
<!-- need to write moisComptable to ChunkContext -->
<property name="chunkContext" value="#{chunkExecutionContext}" />
</bean>
<bean id="headerTokenizer"
class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="dummy1,moisComptable,dummy2" />
<property name="columns" value="1-22,23-28,29-146" />
</bean>
After which, in the next step of the job, I want to generate an output file whose name is composed of a static part and that header field :
<bean id="fileItemWriterLog07" class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="resource"
value="file:${batch.coherence.out.path}/DSTAF007_LOG_#{jobExecutionContext['moisComptable']}.txt" />
<property name="shouldDeleteIfExists" value="true" />
<property name="headerCallback" ref="DetaillesHeaderWriterCallbackLog07" />
...
</bean/>
(I have two jobs because I first write to a database, and then read from it.)
As one would guess this doesn't work, the config file is flowed so I get BeanCreationExceptions. But this gives an idea of what I want to achieve.
I have no exception on the ChunkContext (yet ?) but one on the writer resource. Here is the exception :
Field or property 'jobExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
Does anyone have an idea about how to proceed ?
Thanks in advance.
Related
I am using spring batch flatfile reader to read a file with header and footer values. Below is sample file and output file should have every record appended with header date and file sequence. Sample input and output as below.
Can someone please suggest
If there is a way to set the header values and footer values as job parameters and used in writer?
Any way to get header and footer values in writer? Thanks in advance!
HH20210218001 --Header starting with "HH", date(20210218), filesequence(001)
name110
name220
name330
name440
name770
FT005 --Footer with "FT" and number of records
Output:(date + file seq(001) + name + age)
20210218001name110
20210218001name330
20210218001name440
20210218001name770
Below is my file reader
<bean id="fileItemReader"
class="org.springframework.batch.item.file.FlatFileItemReader"
scope="step">
<property name="resource" value="classpath:input/input.txt"></property>
<!-- <property name="linesToSkip" value="2" /> -->
<property name="lineMapper">
<bean
class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean
class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="name,age" />
<property name="columns" value="1-5,6-8"></property>
<property name="strict" value="false" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.test.MyMapper">
</bean>
</property>
</bean>
</property>
</bean>
If there is a way to set the header values and footer values as job parameters and used in writer?
No, it's too late at that point. When the step starts, job parameters are already set and cannot be changed.
Any way to get header and footer values in writer?
From your expected output, you don't need anything from the footer. If you want info from the header in your writer, you can do that in two steps:
step 1: reads (only) the header and puts any needed information in the execution context
step 2: uses a step-scoped writer that gets any required information from the execution context and uses it to write items
I have thoroughly searched the Spring docs and supporting sites, but have not found and answer to this inquiry; if I want to access and store some values in the ExecutionContext, do I have to write custom databaseItemReader and ItemWriter's that implement the ItemStream or can I use the "out-of-the-box" readers and writers and edit the beans in the spring-batch-context.xml file to do this? Any code examples would be greatly appreciated. Thanks!
http://www.springframework.org/schema/batch/spring-batch-3.0.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd">
<import resource="classpath:context-datasource.xml" />
<!-- JobRepository and JobLauncher are configuration/setup classes -->
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- ItemReader which reads from database and returns the row mapped by
rowMapper -->
<bean id="databaseItemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="SELECT PartnerID, ftpUserName, ftpPassword, ftpPath, jobRunTime, jobFrequency FROM tblRosterJobParams" />
<property name="rowMapper">
<bean class="com.explorelearning.batch.ParamResultRowMapper" />
</property>
</bean>
<!-- This was supposed to change to a SavingItemWriter that persists these values to the Step ExecutionContext -->
<bean id="flatFileItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="resource" value="file:csv/ParamResult.txt" />
<property name="lineAggregator">
<!--An Aggregator which converts an object into delimited list of strings -->
<bean
class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value="," />
<property name="fieldExtractor">
<!-- Extractor which returns the value of beans property through reflection -->
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="PartnerID" />
</bean>
</property>
</bean>
</property>
</bean>
<!-- Optional JobExecutionListener to perform business logic before and after the job -->
<bean id="jobListener" class="com.explorelearning.batch.RosterBatchJobListener" />
<!-- Optional StepExecutionListener to perform business logic before and after the job -->
<bean id="stepExecutionListener" class="com.explorelearning.batch.ParamResultStepExecutionListener" />
<!-- Optional ItemProcessor to perform business logic/filtering on the input records -->
<bean id="itemProcessor" class="com.explorelearning.batch.ParamResultItemProcessor" />
<!-- Step will need a transaction manager -->
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<!-- Actual Job -->
<batch:job-repository id="jobRepository" data-source="dataSource" table-prefix="BATCH_"
transaction-manager="transactionManager" isolation-level-for-create="SERIALIZABLE" />
<batch:job id="RosterBatchJob" job-repository="jobRepository">
<batch:step id="readParams" >
<batch:tasklet transaction-manager="transactionManager" allow-start-if-complete="true">
<batch:chunk reader="databaseItemReader" writer="flatFileItemWriter"
processor="itemProcessor" commit-interval="10" />
</batch:tasklet>
</batch:step>
<!--<batch:step id="grabCSVs" next="validateCSVs">
</batch:step>
<batch:step id="validateCSVs" next="filterRecords">
</step>
<batch:step id="filterRecords" next="determineActions">
</batch:step>
<batch:step id="determineActions" next="executeActions">
</batch:step>
<batch:step id="executeActions" next="">
</batch:step> -->
</batch:job>
Updated now that the question has more detail...
Okay, the way I'm reading your context file, you need to:
Get some FTP login info from the Database
Download some CSV files
Validate the files (file-level? or just record-level validation?)
Filter out some garbage records
Determine some "actions"
Execute some "actions"
The best way to accomplish this (in my opinion) is to create the following steps:
Step 1: A simple Tasklet that queries the database for FTP login info and then downloads the CSV files to a local folder
Step 2: Partitioned step that creates one partition per CSV in the folder
Each partition will have a reader (FlatFileItemReader) that reads out records.
Records will then go to an ItemProcessor that returns null if the record is garbage.
Valid items will be written to some DB staging table for further action
Optionally you can use a ClassifierItemWriter to do something w/ the junk records
Step 3: The next step reads the valid data in the staging table and does "Actions"
Step 4: Maybe another step to do something with the junk records
I have a web application using spring-batch and I'm now integrating spring-batch-admin for basic administration.
The problem is that the jobs configuration files (which are shared with the configuration of the existing application) use properties from files in my application's classpath, but spring-batch-admin's context is not able to load them.
The quick solution was to override the placeholderProperties bean in spring-batch-admin just to add my properties files:
<bean id="placeholderProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath:/org/springframework/batch/admin/bootstrap/batch.properties</value>
<value>classpath:batch-default.properties</value>
<value>classpath:batch-${ENVIRONMENT:hsql}.properties</value>
<value>classpath:/path/to/jobs-config.properties</value> <!-- adding my properties here -->
</list>
</property>
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="ignoreResourceNotFound" value="true" />
<property name="ignoreUnresolvablePlaceholders" value="false" />
<property name="order" value="1" />
</bean>
I don't want to move my properties to one of spring-batch-admin's default files. Is there a simpler way to do this?
Answering my own question here...
As described in the documentation, every job configuration file placed under META-INF/spring/batch/jobs/*.xml is loaded by spring-batch-admin as a child context and property placeholders from the parent (i.e. this default bean) are inherited, but the child context can always create its own placeholder bean.
Given that, in my case, the job configuration files are shared with an existing application and use properties from the application classpath, the solution is to create a new job file in META-INF/spring/batch/jobs/*.xml specific for spring-batch-admin:
<!-- placeholder bean with additional properties for the child context -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath:/path/to/job-config.properties</value>
</list>
</property>
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="ignoreResourceNotFound" value="true" />
<property name="ignoreUnresolvablePlaceholders" value="false" />
<property name="order" value="1" />
</bean>
<!-- external job configuration file is imported -->
<import resource="classpath*:/path/to/job.xml" />
I was trying to share My in-memory jobRepository to the jobExplorer. But it throws an error as,
Nested exception is
org.springframework.beans.ConversionNotSupportedException:
Failed to convert property value of type '$Proxy1 implementing
org.springframework.batch.core.repository.JobRepository,org.
springframework.aop.SpringProxy,org.springframework.aop.framework.Advised'
to required type
Even i tried putting '&' sign before jobRepository when passing to jobExplorer for sharing.But attempt end in vain.
I am using Spring Batch 2.2.1
Is the dependency for jobExplorer is only database not in-memory?
Definition is,
<bean id="jobRepository"
class="com.test.repository.BatchRepositoryFactoryBean">
<property name="cache" ref="cache" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="jobOperator" class="test.batch.LauncherTest.TestBatchOperator">
<property name="jobExplorer" ref="jobExplorer" />
<property name="jobRepository" ref="jobRepository" />
<property name="jobRegistry" ref="jobRegistry" />
<property name="jobLauncher" ref="jobLauncher" />
</bean>
<bean id="jobExplorer" class="test.batch.LauncherTest.TestBatchExplorerFactoryBean">
<property name="repositoryFactory" ref="&jobRepository" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher" class="com.scb.smartbatch.core.BatchLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- To store Batch details -->
<bean id="jobRegistry" class="com.scb.smartbatch.repository.SmartBatchRegistry" />
<bean id="jobRegistryBeanPostProcessor"
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<!--Runtime cache of batch executions -->
<bean id="cache" class="com.scb.cache.TCRuntimeCache" />
thanks for your valuable inputs.
But I used '&' before the job repository reference, which allowed me to use it for my job explorer as a shared resource.
problem solved.
kudos.
Usually you have to wire interface instead of implementation.
Else, probably, you have to add <aop:config proxy-target-class="true"> to create CGLIB-based proxy instead of standard Java-based proxy.
Read Spring official documentation about that
I have written a simple Spring Batch application that reads a CSV file, does some transforming and writes a modified CSV to the disk.
The reading of the file into domain objects works like a charm. I use DelimitedLineTokenizer to tokenize the lines and a BeanWrapperFieldSetMapper to feed the values into a bean:
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{jobParameters['inputResource']}" />
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
<property name="names"
value="ID,NAME,DESCRIPTION,PRICE,DATE" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="targetType" value="myapp.MyDomainObject" />
<property name="customEditors">
<map>
<entry key="java.util.Date" value-ref="dateEditor" />
<entry key="java.math.BigDecimal" value-ref="numberEditor" />
</map>
</property>
</bean>
</property>
</bean>
</property>
</bean>
I especially like the features of BeanWrapperFieldSetMapper to "guess" the field names and the possibility to define CustomEditors which I use to define the special date and number formats used in the input file.
Now I would like to write the modified file in the same format like the input file.
I use the following configuration:
<bean id="writer" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step">
<property name="resource" value="#{jobParameters['outputResource']}" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator">
<property name="delimiter" value=";" />
<property name="fieldExtractor">
<bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names" value="id,name,description,price,date" />
</bean>
</property>
</bean>
</property>
</bean>
There are two things I miss with this configuration:
BeanWrapperFieldSetMapper allowed me to set CustomEditors, but BeanWrapperFieldExtractor has no such possibility. Is there a way to use these?
Is there a way to define the headings in the first line of the file? I have not found any way to write an initial line that is not a bean... It would be great to use the same names here as in BeanWrapperFieldSetMapper such that BeanWrapperFieldExtractor writes the inital line and guesses the bean property namens as BeanWrapperFieldSetMapper does.
The process to load files is so comfortable in Spring Batch. Why is the writing of files so different? Am I missing something?
I have to use Spring Batch 2.1.x because we are using Spring 3.0.x . Therefor an upgrade to 2.2.x would not be an option.
Which is your need? Extract field property as text? You can
use a FormatterLineAggregator if you needs are not too complicated
write your own CustomEditorsFieldExtractor (better)
Generate a complex domain object composed by original domain object and by text-formatted object and use last one as parameter of writer (but breaks your current processor/writer)
Use FlatFileItemWriter.headerCallback: if setted allow custom header write
Writing - in your case - seems a pain respect read process because spring-batch's reading components fits your needs. Standard components fits more used use-case and they cover a lot of scenario. Let us write a custom FieldExtractor sometimes! :)