On Rundeck, I want my job to do a lot work and would like to slice this data by the number of nodes running the job.
Is there a way to query some of the context variables, where I can know how many nodes will run the current task and in which one we are running now ?
For example:
On my current job execution there is a total of 10 nodes and I'm the shell at node 3.
Is this provided somehow by rundeck context ? or would I need to create some state file from an initial step ?
The goal is to split the work on the amount of nodes running the task.
To see what is running on the N node just go to the activity page (left menu) and see the execution (you can see the execution by nodes by default or just click on the "Nodes" link).
To "split the work on the amount of nodes running the task" the best approach is to create separate jobs pointing to desired nodes and then call it from a parent job using the Job Reference Step, take a look at this answer.
Related
I've got a large computational task, consisting of several steps, that I run on a PC cluster, managed by LSF.
Part of this task includes launching several parallel jobs with identical names. Jobs are somewhat different, therefore it is hard to transform them to a job array.
The next step of this computation, following these jobs, summarizes their results, therefore it must wait until all of them are finished.
I'm trying to use -w ended(job-name) command line switch for bsub, as usual, to specify job dependencies.
However, admins of the cluster have set JOB_DEP_LAST_SUB = 1 in lsb.params.
According to the LSF manual, this makes LSF to wait for only one most recent job with supplied name to complete, instead of all jobs.
Is it possible to override this behavior for my task only without asking admins to reconfigure the whole cluster (this cluster is used by many people, it is very unlikely that they agree)?
I cannot find any clues in the manual.
Looks like it cannot be overridden.
I've changed job names to make them unique by appending random value, then I've changed condition to -w ended(job-name-*)
I'm trying to create a packaging cell for 5 items into 1 package; the 5 items are picked up from a resource (worker) and placed into a packaging machine which generate the package; a conveyor moves the packages from the machine to a buffer and every once in a while (say every 20 packages) the worker stops picking the items and goes to the buffer the put all the packages in a box, ideally ready to be shipped. Once the worker has completed the box he has to go back to his pick&place task.
Now, my issues are:
When the worker stops picking the items from the rackSystem and goes to the buffer, the source blocks have to stop generating agents, otherwise the simulation will stop saying that there are no available cells in the rack;
When the worker gets back to his picking task the source blocks have to start generating agents again.
With the hold blocks in the picture I managed to stop the source blocks when the worker stops picking from the rack, anyway I could not make the process start again when the box is complete. How can i do this?
Everything works fine except from the fact that once the worker returns to the picking location and take the last 5 items from the rack, no more agents are allowed to enter the rack.
Actually from this setup, I think you should do this:
Let your sources create agents continuously. In reality (I suppose) things also do not stop coming in just because the worker is doing something else.
Gather all agents in an infinite queue, as you do
remove the hold blocks
instead, make your RackStore and RackPick objects utilize the worker resource pool (tick the box as below and select your resource pool)
You may need to play with the "customize resource choice" selection as well to ensure that your worker only tries to store items when the RackSystem has a space, though, something like below:
I am using a recent build of Torque/Maui (w/ PBS) to schedule jobs on a cluster with heterogenous hardware. Hardware consists on two set of 10 nodes for which I would like to have two group have elevated priority on one of the sets of nodes. For example:
Node set A of 10 nodes has elevated priority for User Group 1
Node set B of 10 nodes has elevated priority for User Group 2
I am familiar with how this is accomplished for all nodes, which is documented here:
http://docs.adaptivecomputing.com/maui/5.1.3priorityusage.php
However, I am unfamiliar on the best strategy to set this type of priority on a subset of the cluster. From what I can ascertain from the Maui docs it may be done using node sets or partitions, but I am unsure if either of these are correct or there is another strategy all together.
Edit: I would prefer to have a single queue as it simplifies usability and would enable a user to potentially use the entire cluster, albeit with differing priority on node set A and B.
Thanks in advance for the help.
The way I understand the question, you've confused node allocation with job priority. Job priority determines how much more quickly Maui will run a job, as it accrues priority in the priority reservation queue. This will determine how soon a job can run, within the constraints placed on the job, relative to all other jobs in the eligible/idle queue.
That's separate from where Maui decides to place (schedule) jobs. The most natural way to handle this type of use case is with standing reservations. You can create reservations over each set of nodes (via host list, feature, or partition), and then give both groups (or everyone) access to both reservations, but apply negative affinity to everyone outside the group with preferential access.
Example:
SRCFG[rsvA] NODEFEATURES=setA
SRCFG[rsvA] GROUPLIST=group1,ALL-
SRCFG[rsvA] HOSTLIST=ALL
SRCFG[rsvB] NODEFEATURES=setB
SRCFG[rsvB] GROUPLIST=group2,ALL-
SRCFG[rsvB] HOSTLIST=ALL
With this configuration, Maui will create reservation rsvA to include only the nodes with the "setA" property/feature, and jobs from group1 will gravitate (i.e., have positive affinity) to the nodes in that reservation. Likewise, jobs from users in group2 will flow to the nodes in rsvB, with the "setB" property (as defined in the nodes file, or on NODECFG lines in the maui.cfg). This configuration works fine with a single queue, and is essentially user-transparent.
I have some questions about Quartz clustering, specifically about how triggers fire / jobs execute within the cluster.
Does quartz give any preference to nodes when executing jobs? Such as always or never the node that executed the same job the last time, or is it simply whichever node that gets to the job first?
Is it possible to specify the node which should execute the job?
The answer to this will be something of a "it depends".
For quartz 1.x, the answer is that the execution of the job is always (only) on a more-or-less random node. Where "randomness" is really based on whichever node gets to it first. For "busy" schedulers (where there are always a lot of jobs to run) this ends up giving a pretty balanced load across the cluster nodes. For non-busy scheduler (only an occasional job to fire) it may sometimes look like a single node is firing all the jobs (because the scheduler looks for the next job to fire whenever a job execution completes - so the node just finishing an execution tends to find the next job to execute).
With quartz 2.0 (which is in beta) the answer is the same as above, for standard quartz. But the Terracotta folks have built an Enterprise Edition of their TerracottaJobStore which offers more complex clustering control - as you schedule jobs you can specify which nodes of the cluster are valid for the execution of the job, or you can specify node characteristics/requisites, such as "a node with at least 100 MB RAM available". This also works along with ehcache, such that you can specify the job to run "on the node where the data keyed by X is local".
I solved this question for my web application using Spring + AOP + memcached. My jobs do know from the data they traverse if the job has already been executed, so the only thing I need to avoid is two or more nodes running at the same time.
You can read it here:
http://blog.oio.de/2013/07/03/cluster-job-synchronization-with-spring-aop-and-memcached/
I have a computation task which is split in several individual program executions, with dependencies. I'm using Condor 7 as task scheduler (with the Vanilla Universe, due do constraints on the programs beyond my reach, so no checkpointing is involved), so DAG looks like a natural solution. However some of the programs need to run on the same host. I could not find a reference on how to do this in the Condor manuals.
Example DAG file:
JOB A A.condor
JOB B B.condor
JOB C C.condor
JOB D D.condor
PARENT A CHILD B C
PARENT B C CHILD D
I need to express that B and D need to be run on the same computer node, without breaking the parallel execution of B and C.
Thanks for your help.
Condor doesn't have any simple solutions, but there is at least one kludge that should work:
Have B leave some state behind on the execute node, probably in the form of a file, that says something like MyJobRanHere=UniqueIdentifier". Use the STARTD_CRON support to detect this an advertise it in the machine ClassAd. Have D use Requirements=MyJobRanHere=="UniqueIdentifier". A part of D's final cleanup, or perhaps a new node E, it removes the state. If you're running large numbers of jobs through, you'll probably need to clean out left-over state occasionally.
I don't know the answer but you should ask this question on the Condor Users mailing list. The folks who support the DAG functionality in Condor monitor it and will respond. See this page for subscription information. It's fairly low traffic.
It's generally fairly difficult to keep two jobs together on the same host in Condor without locking them to a specific host in advance, DAG or no DAG. I actually can't think of a really viable way to do this that would let B start before C or C start before B. If you were willing to enforce that B must always start before C you could make part of the work that Job B does when it starts running be modify the Requirements portion of Job C's ClassAd so that it has a "Machine == " string where is the name of the machine B landed on. This would also require that Job C be submitted held or not submitted at all until B was running, B would also have to release it as part of its start up work.
That's pretty complicated...
So I just had a thought: you could use Condor's dynamic startd/slots features and collapse your DAG to achieve what you want. In your DAG where you currently have two separate nodes, B and C, you would collapse this down into one node B' that would run both B and C in parallel when it starts on a machine. As part of the job requirements you note that it needs 2 CPUs on a machine. Switch your startd's to use the dynamic slot configuration so machines advertise all of their resources and not just statically allocated slots. Now you have B and C running concurrently on one machine always. There are some starvation issues with dynamic slots when you have a few multi-CPU jobs in a queue with lots of single-CPU jobs, but it's at least a more readily solved problem.
Another option is to tag B' with a special job attribute:
MultiCPUJob = True
And target it just at slot 1 on machines:
Requirements = Slot == 1 && ...your other requirements...
And have a static slot startd policy that says, "If a job with MultiCPUJob=True tries to run on slot 1 on me preempt any job that happens to be in slot 2 on this machine because I know this job will need 2 cores/CPUs".
This is inefficient but can be done with any version of Condor past 6.8.x. I actually use this type of setup in my own statically partitioned farms so if a job needs a machine all to itself for benchmarking it can happen without reconfiguring machines.
If you're interested in knowing more about that preemption option let me know and I can point you to some further configuration reading in the condor-user list archives.
The solution here is to use the fact that you can modify submit descriptions even while DAGMan is running as long as DAGMan has not yet submitted the node. Assume a simple DAG of A -> B -> C. If you want all nodes to run on the same host you can do the following:
Define a POST script on node A.
The post script searches condor_history for the ClusterId of the completed node A. Something like condor_history -l -attribute LastRemoteHost -m1 $JOB_ID ... You'll need to clean up the output and what not, but you'll be left with the host that ran node A.
The post script then searches for and modifies dependent job submit files, inserting into them a job job requirement at the top of the submit file. Just make sure you build your job requirements incrementally so that they pick up this new requirement if it is present.
When the post script completes, DAGMan will then look to submit ready nodes, of which in this example we have one: B. The submission of B will now be done with the new requirement you added in step 3, so that it will run on the same execute host as A.
I do this currently with numerous jobs. It works great.