Running only on job instance at a time this ok : Spring batch restrict single instance of job only
public class jobMailListener implements JobExecutionListener {
// active JobExecution, used as a lock.
private JobExecution _active;
public void beforeJob(JobExecution jobExecution) {
// create a lock
synchronized (jobExecution) {
if (_active != null && _active.isRunning()) {
//***************************//
// Here create/storage in queue it up ?
//****************************//
jobExecution.stop();
} else {
_active = jobExecution;
}
}
}
public void afterJob(JobExecution jobExecution) {
// release the lock
synchronized (jobExecution) {
if (jobExecution == _active) {
_active = null;
}
}
}
}
<batch:job id="envoiMail" restartable="true">
<batch:listeners><batch:listener ref="jobMailListener"/>
<batch:step id="prepareData">...
I would not stop the jobs but create a queue.
Can be used spring integration ?
I thought http://incomplete-code.blogspot.fr/2013/03/spring-batch-running-only-one-job.html#comment-form
But it is not functional.
Related
I have an scheduled job implemented with Spring batch. Right now when it finishes it doesn't start again because it is detected as completed, is it possible to reset its state after completion?
#Component
class JobScheduler {
#Autowired
private Job job1;
#Autowired
private JobLauncher jobLauncher;
#Scheduled(cron = "0 0/15 * * * ?")
public void launchJob1() throws Exception {
this.jobLauncher.run(this.job1, new JobParameters());
}
}
#Configuration
public class Job1Configuration{
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Job job1() {
return this.jobBuilderFactory.get("job1")
.start(this.step1()).on(STEP1_STATUS.NOT_READY.get()).end()
.from(this.step1()).on(STEP1_STATUS.READY.get()).to(this.step2())
.next(this.step3())
.end()
.build();
}
}
I know I can set a job parameter with the time or the id, but this will launch a new execution every 15 minutes. I want to repeat the same execution until is completed without errors, and then, execute a new one.
You can't restart your job because you're setting the job status to COMPLETE by calling end() in .start(this.step1()).on(STEP1_STATUS.NOT_READY.get()).end().
You should instead either fail the job by calling .start(this.step1()).on(STEP1_STATUS.NOT_READY.get()).fail()
or stop the job by calling .start(this.step1()).on(STEP1_STATUS.NOT_READY.get()).stopAndRestart(step1())
Those options will mean the job status is either FAILED or STOPPED instead of COMPLETE which means that if you launch the job with the same JobParameters, it will restart the previous job execution.
See https://docs.spring.io/spring-batch/docs/current/reference/html/step.html#configuringForStop
To launch the job in a way that handles restarting previous instances or starting a new instance, you could look at how the SimpleJobService in spring-batch-admin does it and modify the launch method slightly for your purposes. This requires you to specify an incremental job parameter that is used to launch new instances of your job.
https://github.com/spring-attic/spring-batch-admin/blob/master/spring-batch-admin-manager/src/main/java/org/springframework/batch/admin/service/SimpleJobService.java#L250
#Override
public JobExecution launch(String jobName, JobParameters jobParameters) throws NoSuchJobException,
JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException,
JobParametersInvalidException {
JobExecution jobExecution = null;
if (jobLocator.getJobNames().contains(jobName)) {
Job job = jobLocator.getJob(jobName);
JobExecution lastJobExecution = jobRepository.getLastJobExecution(jobName, jobParameters);
boolean restart = false;
if (lastJobExecution != null) {
BatchStatus status = lastJobExecution.getStatus();
if (status.isUnsuccessful() && status != BatchStatus.ABANDONED) {
restart = true;
}
}
if (job.getJobParametersIncrementer() != null && !restart) {
jobParameters = job.getJobParametersIncrementer().getNext(jobParameters);
}
jobExecution = jobLauncher.run(job, jobParameters);
if (jobExecution.isRunning()) {
activeExecutions.add(jobExecution);
}
} else {
if (jsrJobOperator != null) {
// jobExecution = this.jobExecutionDao
// .getJobExecution(jsrJobOperator.start(jobName, jobParameters.toProperties()));
jobExecution = new JobExecution(jsrJobOperator.start(jobName, jobParameters.toProperties()));
} else {
throw new NoSuchJobException(String.format("Unable to find job %s to launch",
String.valueOf(jobName)));
}
}
return jobExecution;
}
I think the difficulty here comes from mixing scheduling with restartability. I would make each schedule execute a distinct job instance (for example by adding the run time as an identifying job parameter).
Now if a given schedule fails, it could be restarted separately until completion without affecting subsequent schedules. This can be done manually or programmtically in another scheduled method.
This is the solution I came up with after all the comments:
#Component
class JobScheduler extends JobSchedulerLauncher {
#Autowired
private Job job1;
#Scheduled(cron = "0 0/15 * * * ?")
public void launchJob1() throws Exception {
this.launch(this.job1);
}
}
public abstract class JobSchedulerLauncher {
#Autowired
private JobOperator jobOperator;
#Autowired
private JobExplorer jobExplorer;
public void launch(Job job) throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException,
JobParametersInvalidException, NoSuchJobException, NoSuchJobExecutionException, JobExecutionNotRunningException, JobParametersNotFoundException, UnexpectedJobExecutionException {
// Get the last instance
final List<JobInstance> jobInstances = this.jobExplorer.findJobInstancesByJobName(job.getName(), 0, 1);
if (CollectionUtils.isNotEmpty(jobInstances)) {
// Get the last executions
final List<JobExecution> jobExecutions = this.jobExplorer.getJobExecutions(jobInstances.get(0));
if (CollectionUtils.isNotEmpty(jobExecutions)) {
final JobExecution lastJobExecution = jobExecutions.get(0);
if (lastJobExecution.isRunning()) {
this.jobOperator.stop(lastJobExecution.getId().longValue());
this.jobOperator.abandon(lastJobExecution.getId().longValue());
} else if (lastJobExecution.getExitStatus().equals(ExitStatus.FAILED) || lastJobExecution.getExitStatus().equals(ExitStatus.STOPPED)) {
this.jobOperator.restart(lastJobExecution.getId().longValue());
return;
}
}
}
this.jobOperator.startNextInstance(job.getName());
}
}
My job now uses an incrementer, based on this one https://docs.spring.io/spring-batch/docs/current/reference/html/job.html#JobParametersIncrementer:
#Bean
public Job job1() {
return this.jobBuilderFactory.get("job1")
.incrementer(new CustomJobParameterIncrementor())
.start(this.step1()).on(STEP1_STATUS.NOT_READY.get()).end()
.from(this.step1()).on(STEP1_STATUS.READY.get()).to(this.step2())
.next(this.step3())
.end()
.build();
}
In my case my scheduler won't start 2 instances of the same job at the same time, so if I detect a running job in this code it means that the server restarted leaving the job with status STARTED, that's why I stop it and abandon it.
I am using spring batch with Java 7 to do some stuff.
When my batch finish to generate the result file i want to execute some linux command line ( coping it into a CFT server).
I added the following line of code for the listner:
#Bean
public JobExecutionListener envoieCFT() {
JobExecutionListener listener = new JobExecutionListener() {
#Override
public void beforeJob(JobExecution jobExecution) {}
#Override
public void afterJob(JobExecution jobExecution) {
final String command ="scp <file path> <user>#<ip#>:< where to copy the file>";
boolean isWindows = System.getProperty("os.name")
.toLowerCase().startsWith("windows");
Process process;
if (!isWindows) {
try {
process = Runtime.getRuntime()
.exec(String.format(command));
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println("Job finished notification");
}
};
return listener ;
}
it's not working at all, did someonde ever faced the same probleme?
I'm new to Polly and I'm trying to apply the Retry policy, so that I can have it manually handling the retry connection in case of IBMMQ connection issue.
Please, consider the following code:
public class ReconnectException : Exception
{
}
public class QueueMonitor : IObservable<Message>, IDisposable
{
private readonly MQQueue mqQueue;
private readonly MQQueueManager queueManager;
private readonly string queueName;
private IDisposable timer;
private readonly object lockObj = new object();
private bool isChecking;
private readonly TimeSpan checkingFrequency;
private readonly List<IObserver<Message>> observers;
private TimeSpan reconnectInterval;
private readonly IScheduler scheduler;
private readonly int maxReconnectCount;
private static readonly ILog Logger = LogProvider.For<AonQueueManager>();
private readonly Policy pollyPolicy;
public QueueMonitor(IConfiguration configuration, string queueName, IScheduler scheduler = null)
{
this.queueManager = QueueFactory.GetIstance(configuration);
this.queueName = queueName;
this.scheduler = scheduler ?? Scheduler.Default;
checkingFrequency = configuration.GetValue("checkingFrequency", new TimeSpan(0, 0, 5));
reconnectInterval = configuration.GetValue("reconnectInterval", new TimeSpan(0, 0, 5));
maxReconnectCount = configuration.GetValue("maxReconnectCount", 3);
observers = new List<IObserver<Message>>();
pollyPolicy = Policy.Handle<ReconnectException>().WaitAndRetry(maxReconnectCount, _ => TimeSpan.FromSeconds(2));
mqQueue = queueManager.AccessQueue(queueName,
MQC.MQOO_INPUT_AS_Q_DEF // open queue for input
+ MQC.MQOO_FAIL_IF_QUIESCING); // but not if MQM stopping
}
public void Start()
{
var x = pollyPolicy.ExecuteAndCapture(CreateTimer);
}
private void CreateTimer()
{
Logger.DebugFormat("Repeating timer started, checking frequency: {checkingFrequency}", checkingFrequency);
timer = Observable.Interval(checkingFrequency, scheduler).Subscribe(_ =>
{
lock (lockObj)
{
if (isChecking) return;
Logger.Log(LogLevel.Debug, () => "Listening on queues for new messages");
isChecking = true;
var mqMsg = new MQMessage();
var mqGetMsgOpts = new MQGetMessageOptions { WaitInterval = checkingFrequency.Milliseconds };
// 15 second limit for waiting
mqGetMsgOpts.Options |= MQC.MQGMO_WAIT | MQC.MQGMO_FAIL_IF_QUIESCING |
MQC.MQCNO_RECONNECT_Q_MGR | MQC.MQOO_INPUT_AS_Q_DEF;
try
{
mqQueue.Get(mqMsg, mqGetMsgOpts);
if (mqMsg.Format.CompareTo(MQC.MQFMT_STRING) == 0)
{
var text = mqMsg.ReadString(mqMsg.MessageLength);
Logger.Debug($"Message received : [{text}]");
Message message = new Message { Content = text };
foreach (var observer in observers)
observer.OnNext(message);
}
else
{
Logger.Warn("Non-text message");
}
}
catch (MQException ex)
{
if (ex.Message == MQC.MQRC_NO_MSG_AVAILABLE.ToString())
{
Logger.Trace("No messages available");
//nothing to do, emtpy queue
}
else if (ex.Message == MQC.MQRC_CONNECTION_BROKEN.ToString())
{
Logger.ErrorException("MQ Exception, trying to recconect", ex);
throw new ReconnectException();
}
}
finally
{
isChecking = false;
}
}
});
}
public IDisposable Subscribe(IObserver<Message> observer)
{
if (!observers.Contains(observer))
observers.Add(observer);
return new Unsubscriber(observers, observer);
}
public void Dispose()
{
((IDisposable)mqQueue)?.Dispose();
((IDisposable)queueManager)?.Dispose();
timer?.Dispose();
}
}
public class Unsubscriber : IDisposable
{
private readonly List<IObserver<Message>> observers;
private readonly IObserver<Message> observer;
public Unsubscriber(List<IObserver<Message>> observers, IObserver<Message> observer)
{
this.observers = observers;
this.observer = observer;
}
public void Dispose()
{
if (observer != null) observers.Remove(observer);
}
}
The problem I've is that when an exception is thrown inside the lamda ( throw new ReconnectException();), Polly doesn't catch it (and I understand why, since it's on another thread) and the application quits since it's on a different thread.
This code is a part of a library,so I don't know that if in every project the Global exceptions are correctly handed.
How do I get it "catched" by the Polly's code?
Thanks in advance
The code posted in the question applies the policy only to the act of creating the timer (the execution of CreateTimer()), not to the code executed by the timer (the lambda inside the .(Subscribe(_ => { }) call).
This is the same as the behaviour if the call to CreateTimer() was surrounded by a try { } catch { }. The catch would only cover the act of executing the CreateTimer() method, the creation of the timer.
For the Polly policy to govern exceptions thrown within the lambda, it needs to be applied within the lambda, to the relevant block/group of statements which are expected to throw the exception.
For example, you might code:
pollyPolicy.ExecuteAndCapture(() => mqQueue.Get(mqMsg, mqGetMsgOpts));
(with a policy configured to govern the particular MQException/s you want to handle).
Or you can apply the policy to a wider group of statements - just as with a try { } clause.
pollyPolicy.ExecuteAndCapture(() =>
{
// ...
mqQueue.Get(mqMsg, mqGetMsgOpts));
// ...
}
I am trying to fetch execution context of last completed job. Below is the code:
#Component
public class MyJobDataReader implements ItemReader<MyDbEntity>, StepExecutionListener {
#Autowired
#Qualifier("jobService")
JobService jobService;
#Override
public void beforeStep(StepExecution stepExecution) {
Collection<JobExecution> jobExecutions = null;
try {
jobExecutions = jobService.listJobExecutionsForJob("myjob", 0, jobService.countJobExecutionsForJob("myjob"));
} catch (NoSuchJobException e) {
//log
}
JobExecution lastCompletedJobExecution =
jobExecutions
.stream()
.filter(param -> param.getExitStatus().getExitCode().equals(ExitStatus.COMPLETED.getExitCode()))
.findFirst()
.orElse(null);
Long myExecutionContextParam =
lastCompletedJobExecution
.getExecutionContext()
.getLong(contextParam);
}
}
But I am getting the myExecutionContextParam also null , even though I see the value in BATCH_JOB_EXECUTION_CONTEXT table for the fetched job execution.
The issue was that jobService.listJobExecutionsForJob was returning a JobExecution with few fields , then I had to call jobService.getJobExecution(jobExecutionId) to get a fully enriched JobExecution
using Quartz, I'd like few jobs (say about 10) to execute as a chain - i.e. NOT concurrently.
They should be executed after an "accounting day change" event occur but since they all access the same DB, I dont want them to start all together. I want them to be executed sequentially instead (order doesnt matter).
I have an idea to put them into a group - say "account_day_change_jobs" and configure Quartz somehow to do the rest for me :-) Means - run sequentially all jobs from the group. I tried the API doc (both 1.8 and 2.1), tried google but didnt find anything.
Is it possible? Is it even reasonable? Other ideas how to achieve the behavior I want?
Thanks very much for any ideas :-)
Hans
The Trigger Listener class below should re-schedule any jobs that attempt to execute while another job that the listener has been configured for is running.
Ive only lightly tested it but for simple cases it should be suitable.
public class SequentialTriggerListener extends TriggerListenerSupport {
private JobKey activeJob;
private Scheduler activeScheduler;
private Queue<JobDetail> queuedJobs = new ConcurrentLinkedQueue<JobDetail>();
public String getName() {
return "SequentialTriggerListener";
}
public boolean vetoJobExecution(Trigger trigger, JobExecutionContext context) {
synchronized (this) {
if (activeJob != null) {
getLog().debug("Queueing Sequential Job - " + context.getJobDetail().getKey().getName());
JobDetail jd = context.getJobDetail();
activeScheduler = context.getScheduler();
jd = JobBuilder.newJob().usingJobData(jd.getJobDataMap()).withIdentity(getName() + ":" + jd.getKey().getName(), jd.getKey().getGroup())
.ofType(jd.getJobClass()).build();
queuedJobs.add(jd);
return true;
} else {
activeJob = trigger.getJobKey();
getLog().debug("Executing Job - " + activeJob.getName());
return false;
}
}
}
public void triggerMisfired(Trigger trigger) {
triggerFinalized(trigger);
}
public void triggerComplete(Trigger trigger, JobExecutionContext context, CompletedExecutionInstruction triggerInstructionCode) {
triggerFinalized(trigger);
}
protected void triggerFinalized(Trigger trigger) {
synchronized (this) {
try {
if (trigger.getJobKey().equals(activeJob)) {
getLog().debug("Finalized Sequential Job - " + activeJob.getName());
activeJob = null;
JobDetail jd = queuedJobs.poll();
if (jd != null) {
getLog().debug("Triggering Sequential Job - " + jd.getKey().getName());
activeScheduler.scheduleJob(jd,TriggerBuilder.newTrigger().forJob(jd).withIdentity("trigger:" + jd.getKey().getName(), jd.getKey().getGroup())
.startNow().withSchedule(SimpleScheduleBuilder.simpleSchedule().withRepeatCount(0).withIntervalInMilliseconds(1)).build());
}
} else {
// this should not occur as the trigger finalizing should be the one we are tracking.
getLog().warn("Sequential Trigger Listener execution order failer");
}
} catch (SchedulerException ex) {
getLog().warn("Sequential Trigger Listener failure", ex);
}
}
}
}
its has been a long time since I used quartz, however I would try two job listeners registered to listen to two different groups
the basic idea is to have one job fire from a group / list ("todayGroup"), the '("todayGroup") job listener detects the completion for good or bad. then kicks off the next job in the list. However, it saves the 'just finished' job back in the scheduler under the ("tomorrowGroup").
public class MyTodayGroupListener extends JobListenerSupport {
private String name;
private static String GROUP_NAME = "todayGroup";
public MyOtherJobListener(String name) {
this.name = name;
}
public String getName() {
return name;
}
#Override
public void jobWasExecuted(JobExecutionContext context,
JobExecutionException jobException) {
Scheduler sched = context.getScheduler();
// switch the job to the other group so we don't run it again today.
JobDetail current = context.getJobDetail();
JobDetail tomorrows = current.getJobBuilder().withIdentity(current.getKey().getName(), "tomorrow").build();
sched.addJob(tomorrows,true);
//see if there is anything left to run
Set<JobKey> jobKeys = sched.getJobKeys(groupEquals(GROUP_NAME ));
Iterator<JobKey> nextJob = null;
if(jobKeys != null && !jobKeys.isEmpty() ){
nextJob = jobKeys.iterator();
}
if(nextJob != null){
// Define a Trigger that will fire "now" and associate it with the first job from the list
Trigger trigger = newTrigger()
.withIdentity("trigger1", "group1")
.startNow()
.forJob(nextJob =.next())
.build();
// Schedule the trigger
sched.scheduleJob(trigger);
}
}
}
likewise, you'll need two 'group triggers' that will fire the first job from their respective groups at the given time you want.
public class TriggerGroupDisallowConcurrentExecutionTriggerListener : ITriggerListener
{
private IScheduler activeScheduler;
private readonly object locker = new object();
private ConcurrentDictionary<string, JobsQueueInfo> groupsDictionary = new ConcurrentDictionary<string, JobsQueueInfo>();
public string Name => "TriggerGroupDisallowConcurrentExecutionTriggerListener";
public Task TriggerComplete(ITrigger trigger, IJobExecutionContext context, SchedulerInstruction triggerInstructionCode, CancellationToken cancellationToken = default)
{
//JobKey key = context.JobDetail.Key;
//Console.WriteLine($"{DateTime.Now}: TriggerComplete. {key.Name} - {key.Group} - {trigger.Key.Name}");
TriggerFinished(trigger, cancellationToken);
return Task.CompletedTask;
}
public Task TriggerFired(ITrigger trigger, IJobExecutionContext context, CancellationToken cancellationToken = default)
{
//JobKey key = context.JobDetail.Key;
//Console.WriteLine($"{DateTime.Now}: TriggerFired. {key.Name} - {key.Group} - {trigger.Key.Name}");
return Task.CompletedTask;
}
public Task TriggerMisfired(ITrigger trigger, CancellationToken cancellationToken = default)
{
//JobKey key = trigger.JobKey;
//Console.WriteLine($"{DateTime.Now}: TriggerMisfired. {key.Name} - {key.Group} - {trigger.Key.Name}");
TriggerFinished(trigger, cancellationToken);
return Task.CompletedTask;
}
public Task<bool> VetoJobExecution(ITrigger trigger, IJobExecutionContext context, CancellationToken cancellationToken = default)
{
//JobKey key = context.JobDetail.Key;
//Console.WriteLine($"{DateTime.Now}: VetoJobExecution. {key.Name} - {key.Group} - {trigger.Key.Name}");
lock (locker)
{
//if (!groupsDictionary.ContainsKey(context.JobDetail.Key.Group))
//{
groupsDictionary.TryAdd(context.JobDetail.Key.Group, new JobsQueueInfo { QueuedJobs = new ConcurrentQueue<IJobDetail>(), ActiveJobKey = null });
var activeJobKey = groupsDictionary[context.JobDetail.Key.Group].ActiveJobKey;
//}
if (activeJobKey != null && activeJobKey != context.JobDetail.Key)
{
var queuedJobs = groupsDictionary[context.JobDetail.Key.Group].QueuedJobs;
if (queuedJobs.Any(jobDetail => jobDetail.Key.Name == context.JobDetail.Key.Name) == true)
{
//NOTE: Джоба уже есть в очереди, нет необходимости её добавлять повторно
return Task.FromResult(true);
}
else
{
//NOTE: Добавить джобу в очередь на выполнение, и не выполнять её сейчас, т.к. она будет выполнена как только подойдёт её очередь
activeScheduler = context.Scheduler;
var newJob = JobBuilder.Create(context.JobDetail.JobType).WithIdentity(context.JobDetail.Key).Build();
queuedJobs.Enqueue(newJob);
return Task.FromResult(true);
}
}
groupsDictionary[context.JobDetail.Key.Group].ActiveJobKey = trigger.JobKey;
return Task.FromResult(false);
}
}
protected void TriggerFinished(ITrigger trigger, CancellationToken cancellationToken = default)
{
lock (locker)
{
try
{
if (!groupsDictionary.ContainsKey(trigger.JobKey.Group))
{
return;
}
var queuedJobs = groupsDictionary[trigger.JobKey.Group].QueuedJobs;
if (queuedJobs.TryDequeue(out IJobDetail jobDetail))
{
//Console.WriteLine($"dequeue - {jobDetail.Key.Name}");
var task = activeScheduler.TriggerJob(jobDetail.Key, cancellationToken);
task.ConfigureAwait(false);
task.Wait(cancellationToken);
groupsDictionary[trigger.JobKey.Group].ActiveJobKey = jobDetail.Key;
}
else
{
groupsDictionary[trigger.JobKey.Group].ActiveJobKey = null;
}
}
catch (SchedulerException ex)
{
throw;
}
}
}
private class JobsQueueInfo
{
public ConcurrentQueue<IJobDetail> QueuedJobs { get; set; }
public JobKey ActiveJobKey { get; set; }
}
}