Quartz RescheduleJob is creating me new jobs instead of do as spected - quartz-scheduler

I'm facing an issue with Quartz and RescheduleJob function.
I'm having an email processor queue and I'm making a Reschedule for the unprocessed emails. Emails has a retry count (each one), if the email sent process fails, the retry count increases and the email is enqueue. Each time the processor fires, emails on queue are processed.
The issue fact is that on app starts, new schedule jobs appears on minimal time (1 min) with different instances and it not look the same instance.
In the doc it says:
Remove (delete) the Quartz.ITrigger with the given key, and store the
new give one - which must be associated with the same job (the new
trigger must have the job name & group specified) - however, the new
trigger need not have the same name as the old trigger.
But on my logs it comes:
2018-06-21 10:03:21,201 [DefaultQuartzScheduler_Worker-1] INFO Services.BaseService [(null)] - Email Send error. Contact Form | To:esddaada#test.com Subject: Thank You for Contacting. Email moved to Send Queue.
2018-06-21 10:04:05,177 [DefaultQuartzScheduler_Worker-1] WARN Services.EmailJob [(null)] - EmailJob Time elapsed updated to 2 mins.
2018-06-21 10:04:07,874 [DefaultQuartzScheduler_Worker-2] INFO Services.BaseService [(null)] - Email Send error. Contact Form | To:esddaada#test.com Subject: Thank You for Contacting Us. Email moved to Send Queue.
2018-06-21 10:04:59,900 [DefaultQuartzScheduler_Worker-2] WARN Services.EmailJob [(null)] - EmailJob Time elapsed updated to 4 mins.
2018-06-21 10:05:03,595 [DefaultQuartzScheduler_Worker-3] INFO Services.BaseService [(null)] - Email Send error. Contact Form | To:esddaada#test.com Subject: Thank You for Contacting Us. Email moved to Send Queue.
2018-06-21 10:05:08,439 [DefaultQuartzScheduler_Worker-3] WARN Services.EmailJob [(null)] - EmailJob Time elapsed updated to 6 mins.
2018-06-21 10:05:11,155 [DefaultQuartzScheduler_Worker-4] INFO Services.BaseService [(null)] - Email Send error. Contact Form | To:esddaada#test.com Subject: Thank You for Contacting Us. Email moved to Send Queue.
2018-06-21 10:05:18,871 [DefaultQuartzScheduler_Worker-4] WARN Services.EmailJob [(null)] - EmailJob Time elapsed updated to 8 mins.
2018-06-21 10:05:22,539 [DefaultQuartzScheduler_Worker-5] INFO Services.BaseService [(null)] - Email Send error. Contact Form | To:esddaada#test.com Subject: Thank You for Contacting Us. Email moved to Send Queue.
2018-06-21 10:05:22,551 [DefaultQuartzScheduler_Worker-5] ERROR Services.EmailJob [(null)] - Email on Queue Reach max 6 send retry Times! Application: To:esddaada#test.com Subject:Thank You for Contacting Us.
This is my code:
public static void StartEmailJob() // scheduler called on Application_Start()
{
var scheduler = StdSchedulerFactory.GetDefaultScheduler();
var emailJob = JobBuilder.Create<EmailJob>().Build();
var trigger = TriggerBuilder.Create()
.WithIdentity("EmailJob", "group1")
// .StartAt(DateTimeOffset.Now.AddMinutes(1))
.WithSimpleSchedule(sh => sh
.WithIntervalInMinutes(RetryInterval)
.RepeatForever())
.Build();
scheduler.ScheduleJob(emailJob, trigger);
scheduler.Start();
}
[DisallowConcurrentExecution]
public class EmailJob : IJob
{
public void Execute(IJobExecutionContext context)
{
var retryCount = 5;
var emailsQueue = context.Emails;
if (emailsQueue.Any(ml => ml.RetryCount < retryCount))
{
var emails = emailsQueue .Where(ml => ml.RetryCount < retryCount);
var count = emails.FirstOrDefault().RetryCount;
foreach (var email in emails)
{
if (!EmailService.SendEmail(email, out string error))
{
if (email.RetryCount >= retryCount)
{
Logger.Error($"Email on Queue Reach max {retryCount} send retry Times!");
}
}
else
{
//remove email from queue
}
}
try
{
RescheduleJob(context, count, true);
}
catch (Exception exc)
{
Logger.Error("EmailJob RescheduleJob Error!", exc);
}
}
else
{
try
{
RescheduleJob(context, 0, true);
}
catch (Exception exc)
{
Logger.Error("EmailJob RescheduleJob Error!", exc);
}
}
}
}
public void RescheduleJob(IJobExecutionContext jobContext, int retryCount, bool startsNow)
{
var retryInterval = 5;
var scheduler = StdSchedulerFactory.GetDefaultScheduler();
// get the trigger
var intervalReminder = (retryCount == 0 ? 1 : retryCount) * retryInterval;
var triggerBuilder = jobContext.Trigger.GetTriggerBuilder();
//crerates a new trigger with the old one
var newTrigger = triggerBuilder.WithSimpleSchedule(s => s
.WithIntervalInMinutes(intervalReminder)
.RepeatForever())
.Build();
if (newTrigger is ISimpleTrigger simpleTrigger)
{
if (jobContext.JobInstance is ISimpleTrigger trigger)
(newTrigger as ISimpleTrigger).TimesTriggered = trigger.TimesTriggered;
}
// reschedule the job with a new trigger
scheduler.RescheduleJob(jobContext.Trigger.Key, newTrigger);
Logger.Warn($"EmailJob Time elapsed updated to {intervalReminder} mins.");
}
}
The weird thing is that after a time, when the retry time exhausted, it works as expected. Why? I don't know.
Can someone help me?
Thank you.

Finally I change the code to use another column and store the next date retry interval. I remove RescheduleJob and when I check
var retryIntervals = new[] { 15, 25, 45, 60, 93, 110, 150 };
var emails = emailsQueue .Where(ml => ml.RetryCount < retryCount && ml.NextTimeRetry <= DateTimeOffset.UtcNow);
I also change this part to take into consideration that the retry time must be changed
if (!EmailService.SendEmail(email, out string error))
{
email.NextTimeRetry = DateTimeOffset.UtcNow.AddMinutes(retryIntervals[email.RetryCount]);
if (email.RetryCount >= retryCount)
{
Logger.Error($"Email on Queue Reach max {retryCount} send retry Times!");
}
}
For now nobody answer this question but it is good to people know how to deal with this issues.

Related

How To Find Replication Queue is Blocked Programmatically

On AEM CaaS, we are trying to send email notification If replication queue is stuck via custom ReplicationEventHandler. We used the agent manager to get the replication queue and trying to add send email logic when queue is blocked.
We have applied 2 approaches based upon the API Docs which doesn't seems working.
Approach 1 : This sends the emails multiple times, even queue is not blocked
for (Agent agent : agentsMap.values()) {
if (agent.isEnabled() && agent.getId().equals("publish")) {
ReplicationQueue replicationQueue = agent.getQueue();
if(replicationQueue.getStatus().getNextRetryTime() != 0) {
Map<String, String> emailParams = new HashMap<>();
emailParams.put("agentId",agent.getId());
emailParams.put("agentName",agent.getConfiguration().getConfigPath());
sendEmail(emailParams);
log.info("::: Replication Queue Blocked :::");
}
}
}
}
Approach 2 : This doesn't trigger email, even queue is blocked.
if(agent.isValid() && agent.isEnabled()) {
ReplicationQueue replicationQueue = agent.getQueue();
if(!replicationQueue.entries().isEmpty()) {
ReplicationQueue.Entry firstEntry = replicationQueue.entries().get(0);
if(firstEntry.getNumProcessed() > 3) {
// Send Email That Queue Is Blocked
}
} else {
// Queue is Not Empty
}
}
Looking for solution..
Thanks

Google PubSub ACK not received after processing few messages

We are using gcloud pub-sub 1.102.0 release.
Problem :- We have around 8K messages to process. We have 3 subscribers running in 3 different k8s pods and we are doing stream pulling. We are receiving the ACK for few messages (around 100) from there on wards we are not receiving the ACKs (Verified in GCP console). But the message process is going on in background. And we saw couple of duplicate messages as well. In our use case to process a single message it takes around 40 secs to 1 min. Ack deadline has been configured to 10 mins while creating the subscription.
Flow control configured with 20L.
ExecutorProvider configured with 3.
public void runSubsciber() throws Exception {
Subscriber subscriber = null;
MessageReceiver receiver = new MessageReceiver() {
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
messageProcessor.processMessage(message.getData().toStringUtf8());
consumer.ack();
}
};
try {
FlowControlSettings flowControlSettings =
FlowControlSettings.newBuilder()
.setMaxOutstandingElementCount(20L)
.build();
ExecutorProvider executorProvider =
InstantiatingExecutorProvider.newBuilder().setExecutorThreadCount(3).build();
projectId=ServiceOptions.getDefaultProjectId();
ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subId);
subscriber = Subscriber.newBuilder(subscriptionName, receiver)
.setExecutorProvider(executorProvider)
.setFlowControlSettings(flowControlSettings)
.build();
subscriber.addListener(new SubscriberListener(), MoreExecutors.directExecutor());
subscriber.startAsync().awaitRunning();
} catch (Exception e) {
throw new Exception(e);
} finally {
if (subscriber != null) {
//subscriber.stopAsync();
}
}
}
Please help us out, Thanks in advance.

How to process all events emitted by RX Java regardless of error?

I'm using vertx.io web framework to send a list of items to a downstream HTTP server.
records.records() emits 4 records and I have specifically set the web client to connect to the wrong I.P/port.
Processing... prints 4 times.
Exception outer! prints 3 times.
If I put back the proper I.P/port then Susbscribe outer! prints 4 times.
io.reactivex.Flowable
.fromIterable(records.records())
.flatMap(inRecord -> {
System.out.println("Processing...");
// Do stuff here....
Observable<Buffer> bodyBuffer = Observable.just(Buffer.buffer(...));
Single<HttpResponse<Buffer>> request = client
.post(..., ..., ...)
.rxSendStream(bodyBuffer);
return request.toFlowable();
})
.subscribe(record -> {
System.out.println("Subscribe outer!");
}, ex -> {
System.out.println("Exception outer! " + ex.getMessage());
});
UPDATE:
I now understand that on error RX stops right a way. Is there a way to continue and process all records regardless and get an error for each?
Given this article: https://medium.com/#jagsaund/5-not-so-obvious-things-about-rxjava-c388bd19efbc
I have come up with this... Unless you see something wrong with this?
io.reactivex.Flowable
.fromIterable(records.records())
.flatMap
(inRecord -> {
Observable<Buffer> bodyBuffer = Observable.just(Buffer.buffer(inRecord.toString()));
Single<HttpResponse<Buffer>> request = client
.post("xxxxxx", "xxxxxx", "xxxxxx")
.rxSendStream(bodyBuffer);
// So we can capture how long each request took.
final long startTime = System.currentTimeMillis();
return request.toFlowable()
.doOnNext(response -> {
// Capture total time and print it with the logs. Removed below for brevity.
long processTimeMs = System.currentTimeMillis() - startTime;
int status = response.statusCode();
if(status == 200)
logger.info("Success!");
else
logger.error("Failed!");
}).doOnError(ex -> {
long processTimeMs = System.currentTimeMillis() - startTime;
logger.error("Failed! Exception.", ex);
}).doOnTerminate(() -> {
// Do some extra stuff here...
}).onErrorResumeNext(Flowable.empty()); // This will allow us to continue.
}
).subscribe(); // Don't handle here. We subscribe to the inner events.
Is there a way to continue and process all records regardless and get
an error for each?
According to the doc, the observable should be terminated if it encounters an error. So you can't get each error in onError.
You can use onErrorReturn or onErrorResumeNext() to tell the upstream what to do if it encounters an error (e.g. emit null or Flowable.empty()).

CakePHP email timeout & PHP maximum execution time

My application uses an Exchange SMTP server for sending emails. At present we don't have any message queuing in our architecture and emails are sent as part of the HTTP request/response cycle.
Occasionally the Exchange server has issues and times out while the email is sending, and as a result the email doesn't send. Sometimes, Cake recognizes the time out and throws an exception. The application code can catch the exception and report to the user that something went wrong.
However, on other occasions PHP hits its maximum execution time before Cake can throw the exception and so the user just gets an error 500 with no useful information as to what happened.
In an effort to combat this, I overwrote CakeEmail::send() in a custom class CustomEmail (extending CakeEmail) as follows:
public function send($content = null)
{
//get PHP and email timeout values
$phpTimeout = ini_get("max_execution_time");
//if $this->_transportClass is debug, just invoke parent::send() and return
if (!$this->_transportClass instanceof SmtpTransport) {
return parent::send($content);
}
$cfg = $this->_transportClass->config();
$emailTimeout = isset($cfg["timeout"]) && $cfg["timeout"] ? $cfg["timeout"] : 30;
//if PHP max execution time is set (and isn't 0), set it to the email timeout plus 1 second; this should mean the SMTP server should always time out before PHP does
if ($phpTimeout) {
set_time_limit($emailTimeout + 1);
}
//send email
$send = parent::send($content);
//reset PHP timeout to previous value
set_time_limit($phpTimeout);
return $send;
}
However, this isn't alwayus successful and I have had a few instances of this:
Fatal Error: Maximum execution time of 31 seconds exceeded in [C:\path\app\Vendor\pear-pear.cakephp.org\CakePHP\Cake\Network\CakeSocket.php, line 303]
CakeSocket.php line 303 is the $buffer = fread()... line from this CakeSocket::read():
public function read($length = 1024) {
if (!$this->connected) {
if (!$this->connect()) {
return false;
}
}
if (!feof($this->connection)) {
$buffer = fread($this->connection, $length);
$info = stream_get_meta_data($this->connection);
if ($info['timed_out']) {
$this->setLastError(E_WARNING, __d('cake_dev', 'Connection timed out'));
return false;
}
return $buffer;
}
return false;
}
Any ideas?
The problem lies here:
//if PHP max execution time is set (and isn't 0), set it to the email timeout plus 1 second; this should mean the SMTP server should always time out before PHP does
if ($phpTimeout) {
set_time_limit($emailTimeout + 1);
}
In some places in my code I was increasing max time out to more than 30 seconds, but the email timeout was still only 30. This code reverted the PHP timeout to 31 seconds, and I'm guessing other stuff was happening before the email started to send which caused issues.
Fixed code:
//if PHP max execution time is set (and isn't 0), set it to the email timeout plus 1 second; this should mean the SMTP server should always time out before PHP does
if ($phpTimeout && $phpTimeout <= $emailTimeout) {
set_time_limit($emailTimeout + 1);
}

Quartz Scheduler sending email notification multiple time

I am using Quartz for scheduling job. Job is to send reminder email every day at some particular time say 11:00AM. I am able to send reminder mail successfully, but the problem is that it sends more than 1 mails at same time. Sometime it sends 8 mails for 1 reminder request, sometime it sends 5. It seems same job is executed multiple time.
Following is my code,
JobDetail job = JobBuilder.newJob(LmsJob.class)
.withIdentity("lmsJob", org.quartz.Scheduler.DEFAULT_GROUP)
.build();
JobDataMap map = job.getJobDataMap();
map.put("creditMonthlyLeaveBalance", creditMonthlyLeaveBalance);
map.put("dailyUpdationTask", dailyUpdation);
map.put("monthlyPayrollGenerationTask",
monthlyPayrollGenerationTask);
map.put("yearlyMaintenanceOfLeaveBalance",
yearlyMaintenanceOfLeaveBalance);
map.put("emailNotifier", emailNotifier);
try {
CronTrigger trigger = TriggerBuilder
.newTrigger()
.withIdentity("lmsJob", "lmsJobGroup")
.forJob(job)
.startAt(new Date(System.currentTimeMillis()))
.withSchedule(
CronScheduleBuilder
.cronSchedule("00 00 00 ? * *")).build();
scheduler.scheduleJob(job, trigger);
scheduler.start();
// scheduler.shutdown();
} catch (ParseException e) {
e.printStackTrace();
}
Please help me in this, let me know if anything else is needed from my side.
I do not know your entire code,which annotation you gave and stuff.So, I am guessing that you gave annotation like #QuartzEvery("3h"). As far I am guessing, your job is scheduled wrong.To make it run at a particular time of every day,try this...
QuartzManager implements Managed {
.
.
public void start() throws Exception {
.
.
QuartzDailyAt dailyAt = jobType.getAnnotation(QuartzDailyAt.class);
int hours[] = dailyAt.hours();
String hourString =
Arrays.stream(hours).mapToObj(String::valueOf).collect(joining(","));
String cronExpression = String.format("0 0 %s * * ?", hourString);
Trigger trigger = TriggerBuilder.
newTrigger().
startNow().
withSchedule(CronScheduleBuilder.cronSchedule(cronExpression).
inTimeZone(TimeZone.getTimeZone("IST"))).
build();
scheduler.scheduleJob(JobBuilder.newJob(jobType).build(), trigger);
scheduler.start();
.
.
}
.
}
And interface as
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.TYPE)
public #interface QuartzDailyAt {
int[] hours();
}
While running your job,add an annotation at the top of the class like
#QuartzDailyAt(hours = {7,8,9,15,16,17})
public class SomeJob extends QuartzJob {.....}
This gives you to run at every particular intervals of a particular time zone...(above it is IST)