Is there any redis jobStore able to support a quartz cluster?
Have anybody been able to build that?
By other side, what's exactly a quartz cluster? I mean, is it able to have two services running the same quartz.properties file pointing to a redis?
EDIT
I've tried with this redis job store but it seems doesn't supprt quartz clustering:
JobStore class 'net.joelinn.quartz.jobstore.RedisJobStore' props could not be configured. [See nested exception: java.lang.NoSuchMethodException: No setter for property 'isClustered']
quartz.properties:
org.quartz.scheduler.instanceName=office-scheduler-service
org.quartz.scheduler.instanceId=AUTO
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=20000
# thread-pool
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=2
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread=true
org.quartz.jobStore.class = net.joelinn.quartz.jobstore.RedisJobStore
org.quartz.jobStore.host = redisbo
org.quartz.jobStore.misfireThreshold = 60000
you don't need to configure cluster, please check the source code, it is already clustered
Quartz JDBC documentation explains how it handles executing jobs in a cluster of application nodes. RedisJobStore extended that to utilize the Redis storage, and it will work in a cluster mode (Quartz cluster - not Redis cluster) by default without requiring you to enable that.
Basically Quartz uses a shared database to record which scheduler instance is currently working on a job, as opposed to direct node communication among application schedulers. When a scheduler instance picks up a job, it safely registers its instance id with the running job and persists it in the database. This support by the job store is evident in the schema used by RedisJobStore, indicated by the blocked_by fields.
Related
In Quarkus framework how to schedule a job to execute only in one pod rather running in all pods. I tried (concurrentExecution = SKIP) that didn't help.
Run the job only in one pod on multi instant application.
From Quarkus guide: https://quarkus.io/guides/scheduler-reference#concurrent_execution
Note that only executions within the same application instance are
considered. This feature is not intended to work across the cluster
so I suppose you have to move to Quartz to get cluster support out-of-the-box or create your custom synchronization method (eg. using a database or file,etc).
I am building a data pipeline for batch processing. And I find that Spring Cloud Data Flow is a quite attractive framework to use. Without much knowledge in SCDF and Kubernetes, I am not sure whether it is possible to conditionally launch a Spring Cloud Task on a specific machine.
Suppose I have two physical servers that are for running the batch process (Server A and Server B). By default, I would like my Spring cloud task to be launched on Server A. If the Server A is shut down, the task should be deployed on server B. Can Kubernetes / SCDF handle this kind of mechanism? I am wondering whether the nodeselector is the thing that I should look into.
Yes, you can pass deployment.nodeSelector as a deployment property when launching the task.
The deployment.nodeSelector is a Kubernetes deployment property and hence, you need to pass something like this:
task launch mytask --properties "deployer.<taskAppName>.kubernetes.deployment.nodeSelector=foo1:bar1,foo2:bar2"
You can check the list of supported Kubernetes deployer properties here
I'm running Spring XD as single-node for my Sandbox environment with a MySQL DB for the batch tables. If I kill -15 the Spring XD process, then all the current definitions for my jobs and streams are lost (in the case of the jobs, the XD_JOB_REGISTRY is apparently deleted). Consequently, if I start up Spring XD again, I have lost all the previous jobs and streams definitions.
I would like to know whether this is intentional in Spring XD, or maybe due to the fact that I run in single-node mode? Or is it a bug?
EDITED TO ADD THE GIST OF SERVERS.YML:
https://gist.github.com/emedina/486b52f11bc146203534
The job and stream definitions are stored in Zookeeper while the stats for any executed jobs are stored in the database. The single-node server uses an embedded Zookeeper instance by default and that's my guess why your definitions are gone when restarting. Try setting up a separate Zookeeper instance with a permanent data location.
I am using
SchedulerFactory schedulerFactory = new StdSchedulerFactory();
scheduler = schedulerFactory.getScheduler();
scheduler.start();
Trigger asapTrigger = getAsapTrigger();
JobDetail asapJob = getAsapJobDetails();
scheduler.scheduleJob(asapJob, asapTrigger);
This is working but when I go for cluster environment, 2 threads are running for the same job.
I am using annotations not properties file. I want to run only one thread. Can someone help on this. How to configure?
my code almost look like : http://k2java.blogspot.com/2011/04/quartz.html
You have to configure Quartz to run in a clustered environment. Clustering currently only works with the JDBC jobstore, and works by having each node of the cluster to share the same database.
Set the org.quartz.jobStore.isClustered property to true if you have multiple instances of Quartz that use the same set of database tables. This property is used to turn on the clustering features.
Set the org.quartz.jobStore.clusterCheckinInterval property (milliseconds) which is the frequency at which this instance checks in with the other instances of the cluster.
Set the org.quartz.scheduler.instanceId to AUTO so that each node in the cluster will have a unique instanceId.
Please note that each instance in the cluster should use the same copy of the quartz.properties file. Furthermore if you use clustering on separate machines ensure that their clocks are synchronized.
For more information check the official documentation which contains a sample properties file for a clustered scheduler.
I have multiple quartz cron jobs in a load balanced environment. Currently these jobs are running on each node, which is not desirable. I want a node to run only a particular scheduler and if the node crashes, another node should run the scheduler intended for the node that crashed.
How can this be done with spring 2.5.6/tomcat load balancer.
I think there's a few aspects to this question.
Firstly, Quartz has API methods for pausing and resuming the Scheduler, or even individual triggers and jobs
e.g.
http://www.jarvana.com/jarvana/view/opensymphony/quartz/1.6.1/quartz-1.6.1-javadoc.jar!/org/quartz/Scheduler.html#standby()
I would create a spring bean with a reference to the Quartz scheduler or trigger, and a simple isMasterNode boolean member for storing state. I'd then expose 2 [restricted-access] web service calls: makeMaster and makeSlave, which will call Scheduler.resume() or standby/pause, respectively.
Finall, the big question is how & with what you determine that another node has 'crashed'.
If you're using a hardware loadbalancer to manage this, you could configure it to call the 'makeMaster' WS on the new 'primary' node, which in turn calls Scheduler.resume() or similar.
hth