Multiple kube-shcedulers - kubernetes

Question

I'm trying to create a kube-shceduler that will be used by my organisation pod deployment (a data-center). The objective would be to have the pods of any deployment be split equally between the two sites of the data-center. To archive this I created a new kube-shceduler manifest (see below), in which i changed the Ports that are used and added a new configuration file (see below) using the --config parameter. The problem is that even after setting the --port parameter to a new port that isn't already used by the original kube-scheduler, it still tries to use the old one. As such the new kube scheduler can't start:
I1109 20:27:59.225996 1 registry.go:173] Registering SelectorSpread plugin
I1109 20:27:59.226097 1 registry.go:173] Registering SelectorSpread plugin
I1109 20:27:59.926994 1 serving.go:331] Generated self-signed cert in-memory
failed to create listener: failed to listen on 0.0.0.0:10251: listen tcp 0.0.0.0:10251: bind: address already in use
How can I force the use of the specified port OR how can I delete the original kube-scheduler so that there won't be any conflict between the two. The first solution would be preferable.
Configuration kube-scheduler (manifest - /etc/kubernetes/manifest/kube-scheduler.yaml) :
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-scheduler
tier: control-plane
name: kube-scheduler
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=true
#- --port=0
image: k8s.gcr.io/kube-scheduler:v1.19.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10259
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10259
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
status: {}
Configuration kube-custom-scheduler.yaml (/etc/kubernetes/manifest/kube-custom-scheduler.yaml):
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-custom-scheduler
tier: control-plane
name: kube-custom-scheduler
namespace: kube-system
spec:
containers:
- command:
- kube-scheduler
- --config=/etc/kubernetes/scheduler-custom.conf
- --master=true
- --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubeconfig=/etc/kubernetes/scheduler.conf
- --leader-elect=false
- --secure-port=10269
- --scheduler-name=kube-custom-shceduler
- --port=10261
image: k8s.gcr.io/kube-scheduler:v1.19.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10269
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-scheduler
resources:
requests:
cpu: 100m
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /healthz
port: 10269
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/kubernetes/scheduler.conf
name: kubeconfig
readOnly: true
- mountPath: /etc/kubernetes/scheduler-custom.conf
name: customconfig
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/kubernetes/scheduler.conf
type: FileOrCreate
name: kubeconfig
- hostPath:
path: /etc/kubernetes/scheduler-custom.conf
type: FileOrCreate
name: customconfig
status: {}
Custom configuration mentioned in the kube-custom-scheduler.yaml (/etc/kubernetes/scheduler-custom.conf):
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
profiles:
- pluginConfig:
- name: PodTopologySpread
args:
defaultConstraints:
- maxSkew: 1
topologyKey: topology.kube.io/datacenter
whenUnsatisfiable: ScheduleAnyway
Please don't hesistate if you need more information about the cluster.

Answer 1

According to https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/ , the --port would be overwritten by the file specified by --config. Adding HealthzBindAddress and MetricsBindAddress to /etc/kubernetes/scheduler-custom.conf works:
healthzBindAddress: 0.0.0.0:10261
metricsBindAddress: 0.0.0.0:10261
If you want to remove the default scheduler, you can move kube-scheduler.yaml out of /etc/kubernetes/manifests/.