Can we have multiple clusters in Quartz Scheduler?
Assume I have a cluster of server1,server2,server3 and server4 and I'm running 4 jobs (job1,job2,job3, and job4) on this cluster and they are load balanced. [This is working]
Now I want to create groups from the cluster.
Group1 => (server1 and server2)
Group2 => (server3 and server4)
And I want to run
job1 and job2 on group1 (server1 and server2) and
job3 and job4 on group2 (server3 and server4)
Is it possible to do this and how?
Related
I'm running about 30 celery tasks in a group and each of these tasks in a group might create subtasks which they also run in a group.
When I run this in production load, about 6000 tasks and subtasks are generated. Then some of the tasks in the top-level group just disappear and are not processed by the workers.
I'm wondering if there's any limitation on the number of tasks that can run in a group.
I'm learning about azure pipelines. By default you get 1 free parallel job for x number of minutes.
A pipeline contains number of tasks. And atleast 1 job. All the tasks in the pipeline (across multiple jobs) run on the same agent?
Does 1 parallel job means 1 pipeline execution containing 2 or more jobs? or only 2 jobs?
No. Each job will run on a new agent.
1 parallel job means that one job can run at a time. Two parallel jobs means that two parallel jobs can run at a time, each on a separate agent. And so on.
I am using airflow with celery executor. As an example, I have a dag with 3 tasks.
task1 >> task2 >> task3
When my worker nodes = 1, I see that the tasks execute fine and in the right sequence
However, when I increase the worker nodes to 2, I see that task1 is picked by worker node 1 and task2 by worker node 2. In my use case, task2 must execute only after task1. I believed that the celery executor will understand this. My dag fails because of this
Can you please me understand what’s wrong and how do I fix this issue?
The simplest solution is to use the Chain primitive. The whole point of Celery as a distributed task queue system is to run tasks concurrently on different nodes, so do not be alarmed when your tasks are executed on different machines.
I am working on setting up multiple Flink Clusters in Kubernetes to deploy multiple jobs. I would like to deploy the jobs in cluster mode. What would be a good number for the count of task managers in a single Flink Cluster so that the job manager handles task manager registrations well?
I am working in quartz scheduler. We have cluster environment for scheduler, can i schedule a quartz job which will be picked by a specific node/server?
For example : I have 3 Node in cluster: Node1, Node2 and Node3 and have 2 Jobs, job1 and job2.
I want to that every time i schedule job2, it will be picked by only Node2. For Job1 there is no restriction.
Thanks in advance.
Set org.quartz.scheduler.instanceName=node1 in scheduler.properties file.At time of job scheduling, we can set SchedName in job detail and trigger as node1, quartz will pick node1 as server to run job.