I use Spring Batch admin project in which I have got a job which processes files from a particular folder asynchronously. Currently I run it via batch admin ui by passing the relevant job parameters.
Now, I am trying to automate this process by using file inbound channel adapter. I have configured the service activator which would invoke the batch job whenever it receives a file. I have a new requirement now to invoke another batch job once the the first file upload job is complete . To do this, I have created another service activator that uses the output channel of the first service activator. But since the batch job runs asynchronously, then next batch job is getting executed immediately. Is there a way for the second batch job to wait till the first batch job completes.
My current configuation is
<file:inbound-channel-adapter id="filesIn" directory="file:${input.directory}" filename-pattern="*.csv" prevent-duplicates="true">
<integration:poller id="poller" fixed-delay="10000"/>
</file:inbound-channel-adapter>
<integration:channel id="statusChannel"/>
<integration:service-activator input-channel="filesIn" output-channel="statusChannel"
ref="handler" method="process"/>
<bean id="handler" class="AnalysisMessageProcessor">
<property name="job" ref="A-importGlobalSettingsDataJob"/> <!--1st job -->
<property name="requestHandler" ref="jobMessageHandler"/>
</bean>
<bean id="jobMessageHandler" class="org.springframework.batch.integration.launch.JobLaunchingMessageHandler">
<constructor-arg ref="jobLauncher" /> <!--spring batch admins async job launcher -->
</bean>
<integration:service-activator input-channel="statusChannel" ref="jobHandler" method="process"/>
<bean id="jobHandler" class="JobHandler"> <!--This Job handler should get invoked only after the 1st batch job is completed. Currently I am just printing the exit status code of 1st job-->
Any help would be very much appreciated.
You basically have 2 options:
Actively Poll for the Job Execution Status
Trigger the next Batch Job (Part of a second Spring Integration flow?) in an event-driven approach using listeners
For the first approach checkout "Querying the Repository" (Part of the Spring Batch reference documentation):
http://static.springsource.org/spring-batch/reference/html/configureJob.html#queryingRepository
The second option would generally be best. Thus, I believe you may want to look into using a Spring Batch JobExecutionListener
http://static.springsource.org/spring-batch/apidocs/org/springframework/batch/core/JobExecutionListener.html
Please checkout the section "Providing Feedback with Informational Messages" in the following document:
https://github.com/ghillert/spring-batch-admin/blob/BATCHADM-160/spring-batch-integration/README.md#providing-feedback-with-informational-messages
Related
I am running jobs in parallel. My job execution is always null when I use JobRepositoryFactoryBean. I need to use to use this. If I don't use this, then I will not be able to use metadata tables. Because I want to restart my job when it is not completed because of some failure reason. So, I want previous record which I will be fetching from metadata tables. And if I use MapJobRepositoryFactoryBean, the job execution is not null. But then there will not be insertion in metadata tables.
I referred this link:-
My job is always null. Can't inject a batch job with Spring Batch. Why?
But the link is not working for me.
My congifuration is
<bean id="batchScheduler" class="com.abc.BatchScheduler">
<property name="jobLauncher" ref="jobLauncher" />
<property name="jobtwo" ref="JobTwo" />
</bean>
I searched a lot. Please help me out. I am not able to proceed.
I would like to put a hook somewhere in the following code/config to be able to spot a JobInstanceAlreadyCompleteException and then email the production support team that this occurred.
I have tried a JobExecutionListener#beforeJob() method in Spring Batch, but the JobInstanceAlreadyCompleteException is occurring before job execution.
I am using this Spring Batch Integration configuration from the documentation:
<int:channel id="inboundFileChannel"/>
<int:channel id="outboundJobRequestChannel"/>
<int:channel id="jobLaunchReplyChannel"/>
<int-file:inbound-channel-adapter id="filePoller"
channel="inboundFileChannel"
directory="file:/tmp/myfiles/"
filename-pattern="*.csv">
<int:poller fixed-rate="1000"/>
</int-file:inbound-channel-adapter>
<int:transformer input-channel="inboundFileChannel"
output-channel="outboundJobRequestChannel">
<bean class="io.spring.sbi.FileMessageToJobRequest">
<property name="job" ref="personJob"/>
<property name="fileParameterName" value="input.file.name"/>
</bean>
</int:transformer>
I want to handle JobInstanceAlreadyCompleteException in case the same CSV file name appears as the job parameter. Do I extend org.springframework.integration.handler.LoggingHandler?
I notice that class is reporting the error:
ERROR org.springframework.integration.handler.LoggingHandler - org.springframework.messaging.MessageHandlingException: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={input.file.name=C:\Users\csv\file2015.csv}. If you want to run this job again, change the parameters.
The ERROR org.springframework.integration.handler.LoggingHandler is done from the default errorChannel which is reached from the <poller> on your <int-file:inbound-channel-adapter>.
So, to handle it manually your just need to specify your own error-channel there a go ahead with email sending:
<int-file:inbound-channel-adapter>
<int:poller fixed-rate="1000" error-channel="sendErrorToEmailChannel"/>
</int-file:inbound-channel-adapter>
<int-mail:outbound-channel-adapter id="sendErrorToEmailChannel"/>
Of course, you will have to do some ErrorMessage transformation before sending ti over e-mail, but that is already details of the target business logic implementation.
Our Spring Batch application is, upon restart of a failed job, processing the same records again, resulting in duplicate rows, and we want to understand how to avoid this.
The Spring Integration poller which starts the batch job is configured to run every couple of hours. When it runs a second time, the job parameters will be the same, but if the previous run failed (for example, because of a DataTruncation exception), Spring Batch will not complain that the job has already completed.
At the point of failure, several hundred thousand records will already have been processed and copied fromn the source table to the destination table. When the job is run a subsequent time, the same rows will be copied to the destination table, resulting in duplicates. Therefore, it appears that the job is not being resumed, but restarted from the beginning.
The Spring Batch database is Derby (file based), this is setup when the application starts, and it appears state is not maintained between restarts of the actual application (because a job can be run again with the same parameters). However, within one application run, state is maintained. For instance, if the job completes succesfully, the next time the poller runs an exception will be thrown because a job (with those parameters) has already completed.
Our job is definition is as follows:
<batch:job id="publisherJob" >
<batch:step id="step1">
<batch:tasklet >
<batch:chunk reader="itemReader" processor="itemProcessor"
writer="itemWriter" commit-interval="${...}" />
</batch:tasklet>
<batch:listeners>
...
</batch:listeners>
</batch:job>
<bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql" value="select ${...} from ${...} where ${...}" />
<property name="rowMapper" ref="rowMapper" />
</bean>
The WHERE clause includes ORDER BY.
Our understanding was that Spring Batch would retain the state at which processing failed and proceed from that point (if the error in the source table has been fixed), therefore preventing duplicate rows. What has to be configured for this to happen?
Thanks
Spring Batch maintains state in that it remembers how many records were processed, not specifically which ones. Because of that, it's up to you to guarantee the order of the items is reproducible from run to run so that if we process 100 records in run 1 and fail, when we skip the first 100 records in run 2, those are the right 100 records to skip. You didn't provide the configuration for your JdbcCursorItemReader but my assumption is that you are not using an order by in your SQL. If you want restartability, you need some way to guarantee the order of the items. Using an order by in your SQL is the easiest way to accomplish this (there are others like using the process indicator pattern if that's needed).
We have a requirement to pause a job before application maintenance. We are using Quartz 2.2.1 in cluster. Database is oracle.
I have developed a screen with "Pause" functionality. I observed that "pause" works fine until I start the server again. The moment I start server, TRIGGER_STATE of QRTZ_TRIGGERS table resets to "WAITING".
Can anyone please provide a hint.
Thanks a lot in advance.
Rgds - Roy
If you have set overwriteExistingJobs=true (note that default value is false) then each time server starts, it loads the jobs/triggers from the configuration file and replaces existing ones (that have the same job/trigger names), therefore overwriting triggers and their states too as in your case.
You could try to set overwriteExistingJobs=false in the SchedulerFactoryBean. This however may not be convenient for you, since if you ever change job configuration in the server, the existing jobs with old configuration will remain in the database.
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
....
<property name="overwriteExistingJobs" value="false"/>
<property name="triggers">
<list>
....
</list>
</property>
....
</bean>
I use quartz to schedule a custom job to run daily, at a specific time.
However, the machine running mule may be down during that specific scheduled run time, and the custom job cannot get run on that day. So, I wonder if it is possible that I can use jmx to invoke the quartz custom job's execute() method manually.
The easiest is to use a <composite-source> to allow your flow to be triggered both by Quartz and HTTP. That way you can manually trigger it with a simple curl invocation.
Something like:
<flow name="dualTriggerFlow">
<composite-source>
<quartz:inbound-endpoint ...>
...
</quartz:inbound-endpoint>
<http:inbound-endpoint exchange-pattern="request-response"
host="localhost" port="8081" path="/jobs/myjob/trigger" />
</composite-source>
...
Of course it depends on what type of Quartz job you're executing. I'm assuming an event-generator-job.