I have setted the commit interval of 1000 . But, it is not working. Still, i am getting items at random intervals to my item writer . please advise.
<batch:step id="step1" next ="step2">
<batch:tasklet task-executor="simpleTaskExecutor" throttle-limit="20">
<batch:chunk reader="itemReader" processor="itemProcessor" writer="itemWriter" commit-interval="1000" />
<batch:listeners>
<batch:listener ref="appJobExecutionListener" />
<batch:listener ref="appJobFailureLoggerListener" />
<batch:listener ref="customStepListener" />
</batch:listeners>
</batch:tasklet>
</batch:step>
You are using an Async executor. (Guessing, you have not shown in the config) The async executor is set to 20, which means you can have up to 20 threads executing in parallel.
If you want to read records up to the commit interval and then process one chunk at a time, you should drop the task-executor from the config
Related
I use the setup below in a project for a job definition.
On the project the batch-jobs are defined in a database. The xml-job definition below serves as a template for creating all these batch jobs at runtime.
This works fine, except in the case of a BeanCreationException in the dataProcessor. When this exception occurs the skip policy is never called and the batch ends immediately instead.
What could be the reason for that? What do I have to do so that every Exception in the dataProcessor is going to use the SkipPolicy?
Thanks a lot in advance
Christian
Version: spring-batch 3.0.7
<batch:job id="MassenGevoJob" restartable="true">
<batch:step id="selectDataStep" parent="selectForMassenGeVoStep" next="executeProcessorStep" />
<batch:step id="executeProcessorStep"
allow-start-if-complete="true" next="decideExitStatus" >
<batch:tasklet>
<batch:chunk reader="dataReader" processor="dataProcessor"
writer="dataItemWriter" commit-interval="10"
skip-policy="batchSkipPolicy">
</batch:chunk>
<batch:listeners>
<batch:listener ref="batchItemListener" />
<batch:listener ref="batchSkipListener" />
<batch:listener ref="batchChunkListener" />
</batch:listeners>
</batch:tasklet>
</batch:step>
<batch:decision decider="failOnPendingObjectsDecider"
id="decideExitStatus">
<batch:fail on="FAILED_PENDING_OBJECTS" exit-code="FAILED_PENDING_OBJECTS" />
<batch:next on="*" to="endFlowStep" />
</batch:decision>
<batch:step id="endFlowStep">
<batch:tasklet ref="noopTasklet"></batch:tasklet>
</batch:step>
<batch:validator ref="batchParameterValidator" />
<batch:listeners>
<batch:listener ref="batchJobListener" />
</batch:listeners>
</batch:job>
A BeanCreationException isn't really skippable because it usually happens before Spring Batch starts. It's also typically a fatal error for your application (Spring couldn't create a component you've defined as being critical to your application). If the creation of that bean is subject to issues and not having it is ok, I'd suggest wrapping it's creation in a factory so that you can control any exceptions that come out of the creation of that bean. For example, if you can't create your custom ItemProcessor, your FactoryBean could return the PassthroughItemProcessor if that's ok.
I'm going to transfer data from csv to mysql database.
I any error occurs during CSV->DB conversion I need to rollback the entire transaction and not only data of current chunk.
in my simple-job.xml is
<batch:job id="jobSimple" job-repository="jobRepository" parent="simpleJob">
<batch:step id="step1">
<batch:tasklet transaction-manager="transactionManager">
<batch:transaction-attributes isolation="DEFAULT" propagation="REQUIRED" timeout="30"/>
<batch:chunk reader="cvsItemReader" processor="Pprocess" writer="databaseItemWriter" commit-interval="100"/>
</batch:tasklet>
</batch:step>
</batch:job>
I created a batch job using spring batch framework, but if the reader return null, the batch job will create an empty file. Anyway to skip to next step and don't create the file.
<batch:job id="MY_BATCH_JOB">
<batch:step id="step1" next="step2">
<batch:tasklet>
<batch:chunk reader="itemReader" writer="itemWriter"
processor="itemProcessor" commit-interval="2">
</batch:chunk>
<batch:listeners>
<batch:listener ref="readListener" />
</batch:listeners>
</batch:tasklet>
</batch:step>
<batch:step id="step2">
<batch:tasklet>
<batch:chunk reader="itemReader2" writer="itemWriter2"
processor="itemProcessor2" commit-interval="2">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
If your ItemWriter is a FlatFileItemWriter, you can set the property shouldDeleteIfEmpty to true so that it doesn't create the file if it is empty.
<bean class="org.springframework.batch.item.file.FlatFileItemWriter">
<property name="shouldDeleteIfEmpty" value="true"></property>
</bean>
If your ItemWriter is a custom item writer, then you can apply your own logic to not create the file.
Either way, it will go to next Step.
I've read through the spring batch docs a few times and searched for a way to skip a job step based on job parameters.
For example say I have this job
<batch:job id="job" restartable="true"
xmlns="http://www.springframework.org/schema/batch">
<batch:step id="step1-partitioned-export-master">
<batch:partition handler="partitionHandler"
partitioner="partitioner" />
<batch:next on="COMPLETED" to="step2-join" />
</batch:step>
<batch:step id="step2-join">
<batch:tasklet>
<batch:chunk reader="xmlMultiResourceReader" writer="joinXmlItemWriter"
commit-interval="1000">
</batch:chunk>
</batch:tasklet>
<batch:next on="COMPLETED" to="step3-zipFile" />
</batch:step>
<batch:step id="step3-zipFile">
<batch:tasklet ref="zipFileTasklet" />
<!-- <batch:next on="COMPLETED" to="step4-fileCleanUp" /> -->
</batch:step>
<!-- <batch:step id="step4-fileCleanUp">
<batch:tasklet ref="fileCleanUpTasklet" />
</batch:step> -->
</batch:job>
I want to be able to skip step4 if desired by specifying in the job paramaters.
The only somewhat related question I could find was how to select which spring batch job to run based on application argument - spring boot java config
Which seems to indicate that 2 distinct job contexts should be created and the decision made outside the batch step definition.
I have already followed this pattern, since I had a csv export as well as xml as in the example. I split the 2 jobs into to separate spring-context.xml files one for each export type, even though the there where not many differences.
At that point I though it was perhaps cleaner since I could find no examples of alternatives.
But having to create 4 separate context files just to make it possible to include step 4 or not for each export case seems a bit crazy.
I must be missing something here.
Can't you do that with a decider? http://docs.spring.io/spring-batch/reference/html/configureStep.html (chapter 5.3.4 Programmatic Flow Decisions)
EDIT: link to the updated url
https://docs.spring.io/spring-batch/trunk/reference/html/configureStep.html#programmaticFlowDecisions
In Spring Batch we can create one step which is dependent on other like:
<batch:job id="firstJob">
<batch:step id="firstStep" next="secondStep">
<batch:tasklet ref="firstTasklet"/>
</batch:step>
<batch:step id="secondStep">
<batch:tasklet ref="secondTasklet"/>
</batch:step>
</batch:job>
In my case, we have dependency as shown below, task C (child) needs to be executed only when A (parent) and B (parent) both are completed:
Is there any way in Spring Batch where we can say something like:
<batch:job id="firstJob">
<batch:step id="A,B" next="C">
<batch:tasklet ref="firstTasklet"/>
</batch:step>
...
</batch:job>
What I thought of is using listener on A and B, and keep track of both listeners in database. When both listeners gets executed, task C can be invoked.
Please help.
Note: I am using Spring Batch version: 2.1.9-RELEASE, if above requirement is available on higher releases, I can update version as well.
You can use "next" tag as many times as you want to define a chain so:
<batch:step id="A" next="B">
<batch:tasklet ref="firstTasklet"/>
</batch:step>
<batch:step id="B" next="C">
<batch:tasklet ref="secondTasklet"/>
</batch:step>
<batch:step id="C">
<batch:tasklet ref="thirdTasklet"/>
</batch:step>
The chain is: A -> B -> C
C step will be executed after B
Probably not useful anymore but:
<job id="job1">
<split id="split1" task-executor="taskExecutor" next="stepC">
<flow>
<step id="stepA" parent="stepA" />
</flow>
<flow>
<step id="stepB" parent="stepB"/>
</flow>
</split>
<step id="stepC" parent="stepC"/>
</job>
So C will execute once A and B have executed.
http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html#scalabilityParallelSteps