Kubernetes node affinity behavior - kubernetes

Question

I'm trying to affect pods to a specific node using affinities but it ends with a strange behavior I can't understand.
Let explain my nodes setup. I have "x" nodes which all have the labels kali=true, two nodes have in addition the labels kali-app=true and one of these nodes have the label kali-app-1=true.
Now I try to deploy a Deployment with a replicas of 2 pods which should result with putting these pods on the kali-app-1 node:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kali
operator: In
values: ['true']
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: kali-app
operator: In
values: ['true']
- weight: 2
preference:
matchExpressions:
- key: kali-app-1
operator: In
values: ['true']
The result is the first pod is put on the kali-app-1 node but the second one is put on the node which only have the kali-app node (namely kali-app-2=true which is an other label I have in my cluster).
Does anyone can explain me this behavior?

Answer 1

Based on the nodeAffinity documentation it works as expected since preferredDuringSchedulingIgnoredDuringExecution field is used for the 2nd and 3rd conditions.
They way it works is:
There are currently two types of node affinity, called
requiredDuringSchedulingIgnoredDuringExecution and
preferredDuringSchedulingIgnoredDuringExecution. You can think of
them as "hard" and "soft" respectively, in the sense that the former
specifies rules that must be met for a pod to be scheduled onto a node
(similar to nodeSelector but using a more expressive syntax), while
the latter specifies preferences that the scheduler will try to
enforce but will not guarantee.
The weight field in preferredDuringSchedulingIgnoredDuringExecution
is in the range 1-100. For each node that meets all of the scheduling
requirements (resource request, RequiredDuringScheduling affinity
expressions, etc.), the scheduler will compute a sum by iterating
through the elements of this field and adding "weight" to the sum if
the node matches the corresponding MatchExpressions. This score is
then combined with the scores of other priority functions for the
node. The node(s) with the highest total score are the most
preferred.
That means that after first pod is scheduled on the required node, another node is more preferred for placing the second pod. (e.g. if the deployment is heavy and has high cpu/memory requests) + generally kubernetes scheduler tries to spread pods across nodes to comply with high availability. Please get familiar with NodeAffinity preferredDuringSchedulingIgnoredDuringExecution doesn't work well GitHub issue.
Therefore if these pods from the deployment have to be scheduled on the same one node with label kali-app-1, you need to use requiredDuringSchedulingIgnoredDuringExecution field to enforce this assignment and do not provide the scheduler with other possible options.