So, I have an issue with my Slurm GPU queue that leads to a job starvation every now and then.
Basically, I have many nodes with 1 GPU, 2 GPUs, 3 GPUs, and only 2 have 4 GPUs. The situation is as follows:
User A submits a 4 GPU job
Slurm assigns one 4 GPU node to User A's job
Users B, C and D submit 1 GPU jobs and all get allocated to the second 4 GPU node
User E submits one 4 GPU job, it's PENDING since there are no resources to fulfill its need
Users F, G, H, I...etc submit 1 GPU jobs, which get allocated to the 4 GPU node immediately as soon as any of the jobs of users B, C or D finishes
More users keep submitting jobs and the 4 GPU node stays busy with these 1 GPU jobs
User E 4 GPU job stays waiting FOREVER, as the 4 GPUs are never available together
Knowing that I have set the weight of 1 GPU nodes to 1, 2 GPU nodes to 2, 3 GPU nodes to 3 and 4 GPU nodes to 4, so that users priority goes to any available 1 GPU jobs, if not then 2, if not then 3, and lastly the 4.
Any suggestions to eliminate or reduce starvation here (automatically)? I have jobs that wait for weeks!
Related
I need to run a parfor loop in Matlab and assign the number of workers to the loop.
Total Number of Cores: 10 (8 performance and 2 efficiency).
What is the maximum number of cores that I can take up, without crashing the computer and still being able to use the computer for browsing and email? Is leaving only 1 core free enough?
Below is an example in which 3 cores are selected.
n = 3 % Select the number of workers that you need and set n to that integer.
parpool(n)
parfor
i=1:n, c(:,i) = eig(rand(100));
end
I am using a four kernel raspberry pi to implement a small project on networking, in which I need to create several worker processes on the same raspberry pi to accelerate the task. I discovered that although top shows that my CPU usage percentage for each individual worker is not much different, namely 25% for one worker and 23% for four workers each, the single process performance seems to be much lower when I have more workers (a single task takes 3 seconds when I open one worker, but takes 9 seconds when I have 4 workers. There is no communication between the workers, and they perform similar tasks independently. Can anyone explain why this could happen?
I don't understand the costs of having 1 data node vs having 2 or more data nodes.
Will I have the same cost regardless of the number of nodes?
If I have 2 data nodes, that means that I will have double the cost of the instances?
Thanks
Depends on the instance size: i3.2xlarge would be ~2x more expensive than i3.xlarge.
If you use one instance size then yes, 2 nodes would be 2x more expensive than 1 node but you'll get more resilience (if one node goes down your cluster can still get updates and serve data) and rolling restarts.
Though, Opensearch requires an odd number of nodes for master election to work reliably so 3 smaller nodes might be better than 2 larger ones.
Consensus algorithm (e.g. raft) requires the cluster contains an odd number of nodes to avoid the split-brain problem.
Say I have a cluster of 5 nodes, what will happen if only one node fails? The cluster has 4 nodes now, which breaks the odd number rule, will the cluster continue to behave right?
One solution is to drop one more node to make the cluster contain only 3 nodes, but what if the previously failed node comes back? then the cluster has 4 nodes again, and we have to bring the afore-dropped node back in order to keep the cluster odd.
Do implementations of the consensus algorithm handle this problem automatically, or I have to do it in my application code (for example, drop a node)?
Yes, the cluster will continue to work normally. A cluster of N nodes, and if N is odd (N = 2k + 1), can handle k node fails. As long as a majority of nodes is alive, it can work normally. If one node fails, and we still have the majority, everything is fine. Only when you lose majority of nodes, you have a problem.
There is no reason to force the cluster to have an odd number of nodes, and implementations don't consider this as a problem and thus don't handle it (drop nodes).
You can run a consensus algorithm on an even number of nodes, but it usually makes more sense to have it odd.
3 node cluster can handle 1 node fail (the majority is 2 nodes).
4 node cluster can handle 1 node fail (the majority is 3 nodes).
5 node cluster can handle 2 node fail (the majority is 3 nodes).
6 node cluster can handle 2 node fail (the majority is 4 nodes).
I hope this makes it more clear why it makes more sense to have the cluster size to be an odd number, it can handle the same number of node failures with fewer nodes in the cluster.
I was wondering when we are running spmd blocks and create individual lab workers, then how much is the memory allocated to each of them them?
I have an 8 core machine and I used 8 lab workers.
Thanks.
When you launch workers using the matlabpool command in Parallel Computing Toolbox, each worker process starts the same - they're essentially an ordinary MATLAB process but with no desktop visible. They consume memory as and when you create arrays on them. For example, in the following case, each worker uses the same amount of memory to store x:
spmd
x = zeros(1000);
end
But in the following case, each worker consumes a different amount of memory to store their copy of x:
spmd
x = zeros(100 * labindex);
end