We have clustered Quartz scheduler runner on a couple of application nodes. The application nodes need to be updated, and for high-availability reasons, the update is done as rolling update.
Together with the update, we need to add a new job, and that job needs to start running immediately - i.e. it can't wait until all nodes have been updated. The problem is that I can't control which node will run the new job, and if one of the old nodes runs the job, the job instantiation will faill (with a ClassNotFoundException), the trigger will be set to the state ERROR and the job won't run again.
One solution for this problem would be to do two updates: one to add the class in all nodes, and one to add the trigger. The main reason against this approach is that our ops procedures don't support this.
So is there also a way to schedule the new job and make it run reliably with a single update?
I just tried it and it turned out that Quartz gets a ClassCastException while trying acquire the trigger. The exception is wrapped into a JobPersistenceException and the trigger is left in WAITING state.
So, although this could cause an error log entry in one of the old nodes, Quartz doesn't leave the trigger in a non-working state.
Related
I need to run a DAG every 10 minutes. Sometimes a DAG starts (run2) run and the previous running(run1) is still in "running" state.
My problem is related with xcom deletes. If run2 finishes first (because the batch of data is lighter for example) it will delete all xcom from that dag_id, so the xcom from the run1 ( that is still running) will be deleted in the middle of DAG process.
How can I delete only the xcom from the self DAG? I mean, delete only the xcom created in run2 but keep the xcoms created on run1.
PS:I tried to use max_active_runs=1 but they make a "wait queue" and I don't want it.
Thanks a lot all.
delete_xcom_task = PostgresOperator(
task_id='delete-xcom-task',
postgres_conn_id='postgres',
sql="delete from public.xcom where dag_id='dag_id'",
dag=curation)
I have been trying to get druid to fire a kill task periodically to clean up unused segments.
These are the configuration variables responsible for it
druid.coordinator.kill.on=true
druid.coordinator.kill.period=PT45M
druid.coordinator.kill.durationToRetain=PT45M
druid.coordinator.kill.maxSegments=10
From the above configuration my mental model is, once ingested data is marked unused, kill task will fire and delete the segments that are older that 45 mins while retaining 45 mins worth of data. period and durationToRetain are the config vars that are confusing me, not quite sure how to leverage them. Any help would be appreciated.
The caveat for druid.coordinator.kill.on=true is that segments are deleted from whitelisted datasources. The whitelist is empty by default.
To populate the whitelist with all datasources, set killAllDataSources to true. Once I did that, the kill task fired as expected and deleted the segments from s3 (COS). This was tested for Druid version 0.18.1.
Now, while the above configuration properties can be set when you build your image, the killAllDataSources needs to be set through an API. This can be set via the druid UI too.
When you click the option, a modal appears that has Kill All Data Sources. Click on True and you should see a kill task (Ingestion ---> Tasks below) firing in the interval specified. It would be really nice to have this as a part of runtime.properties or some sort of common configuration file that we can set the value in when build the druid image.
Use crontab it works quite well for us.
If you want to have a control outside the druid over the segments removal, then you must use an scheduled task which runs based on your desire interval and register kill-tasks in druid. It can increase your control over your segments, since when they go away, you cannot recover them. You can use this script to accompany you:
https://github.com/mostafatalebi/druid-kill-task
I'm looking for recommended solution to work around celerybeat being a single point of failure for celery/rabbitmq deployment. I didn't find anything that made sense so far, by searching the web.
In my case, once a day timed scheduler kicks off a series of jobs that could run for half a day or longer. Since there can only be one celerybeat instance, if something happens to it or the server that it's running on, critical jobs will not be run.
I'm hoping there is already a working solution for this, as I can't be the only one who needs reliable (clustered or the like) scheduler. I don't want to resort to some sort of database-backed scheduler, if I don't have to.
There is an open issue in celery github repo about this. Don't know if they are working on it though.
As a workaround you could add a lock for tasks so that only 1 instance of specific PeriodicTask will run at a time.
Something like:
if not cache.add('My-unique-lock-name', True, timeout=lock_timeout):
return
Figuring out lock timeout is well, tricky. We're using 0.9 * task run_every seconds if different celerybeats will try to run them at different times.
0.9 just to leave some margin (e.g. when celery is a little behind schedule once, then it is on schedule which would cause lock to still be active).
Then you can use celerybeat instance on all machines. Each task will be queued for every celerybeat instance but only one task of them will finish the run.
Tasks will still respect run_every this way - worst case scenario: tasks will run at 0.9*run_every speed.
One issue with this case: if tasks were queued but not processed at scheduled time (for example because queue processors was unavailable) - then lock may be placed at wrong time causing possibly 1 next task to simply not run. To go around this you would need some kind of detection mechanism whether task is more or less on time.
Still, this shouldn't be a common situation when using in production.
Another solution is to subclass celerybeat Scheduler and override its tick method. Then for every tick add a lock before processing tasks. This makes sure that only celerybeats with same periodic tasks won't queue same tasks multiple times. Only one celerybeat for each tick (one who wins the race condition) will queue tasks. In one celerybeat goes down, with next tick another one will win the race.
This of course can be used in combination with the first solution.
Of course for this to work cache backend needs to be replicated and/or shared for all of servers.
It's an old question but I hope it helps anyone.
I have some questions about Quartz clustering, specifically about how triggers fire / jobs execute within the cluster.
Does quartz give any preference to nodes when executing jobs? Such as always or never the node that executed the same job the last time, or is it simply whichever node that gets to the job first?
Is it possible to specify the node which should execute the job?
The answer to this will be something of a "it depends".
For quartz 1.x, the answer is that the execution of the job is always (only) on a more-or-less random node. Where "randomness" is really based on whichever node gets to it first. For "busy" schedulers (where there are always a lot of jobs to run) this ends up giving a pretty balanced load across the cluster nodes. For non-busy scheduler (only an occasional job to fire) it may sometimes look like a single node is firing all the jobs (because the scheduler looks for the next job to fire whenever a job execution completes - so the node just finishing an execution tends to find the next job to execute).
With quartz 2.0 (which is in beta) the answer is the same as above, for standard quartz. But the Terracotta folks have built an Enterprise Edition of their TerracottaJobStore which offers more complex clustering control - as you schedule jobs you can specify which nodes of the cluster are valid for the execution of the job, or you can specify node characteristics/requisites, such as "a node with at least 100 MB RAM available". This also works along with ehcache, such that you can specify the job to run "on the node where the data keyed by X is local".
I solved this question for my web application using Spring + AOP + memcached. My jobs do know from the data they traverse if the job has already been executed, so the only thing I need to avoid is two or more nodes running at the same time.
You can read it here:
http://blog.oio.de/2013/07/03/cluster-job-synchronization-with-spring-aop-and-memcached/
I have 4 instances of Quartz Server. All of the instances point to one ADO JobStore. All I want to do is to make each Quartz instance execute the same job at the same time.
I hope it's clear enough.
This isn't supported out of the box. Whenever a trigger fires, it can only be consumed by one instance. You could fire 4 triggers, but it is not guaranteed that the job will not run twice on one instance.
If you want each instance to fire the job once, then you will have to set up 4 separate job stores.
What I do (in Quartz.NET 2.4.1) is that I have multiple identical scheduler instances, which only differ in scheduler instance name (quartz.scheduler.instanceName). They register identical jobs and triggers. Because of different scheduler instance names, the jobs and triggers are duplicated in the job store (scheduler name is part of the primary key in every table of JobStoreTX). This causes logically the same triggers to fire on all scheduler instances at the same time. They are actually separate triggers, though, so each will handle misfires etc separately.