Where to find the Kubernetes Scheduler Configuration file in local system - kubernetes

I am currently working in Minikube cluster and looking to change some flags of kubernetes scheduler configuration, but I can't find it. The file looks something like-
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
algorithmSource:
provider: DefaultProvider
...
disablePreemption: true
What is it's name and where can I find it?

Posting this answer as a community wiki to set a baseline and to provide additional resources/references rather than giving a definitive solution.
Feel free to edit and expand.
I haven't found the file that you are referencing (KubeSchedulerConfiguration) in minikube.
The minikube provisioning process does not create it nor references it in the configuration files (/etc/kubernetes/manifests/kube-scheduler.yaml and the --config=PATH parameter).
I'd reckon you could take a look on other Kubernetes solutions where you can configure how your cluster is created (how kube-scheduler is configured). Some of the options are:
Kubernetes.io: Docs: Setup: Production environment: Tools: Kubeadm: Create cluster and also:
Kubernetes.io: Docs: Setup: Production environment: Tools: Kubeadm: Control plane flags
Github.com: Kubernetes sigs: Kubespray
A side note!
Both: kubespray and minikube are using kubeadm as a bootstrapper!
I would also consider creating additional scheduler that would be responsible for spawning your workload (by referencing in the YAML manifests):
Kubernetes.io: Docs: Tasks: Extend Kubernetes: Configure multiple schedulers
I haven't tested it extensively and in the long term but I've managed to include the YAML manifest that you are referencing for the kube-scheduler.
Disclaimers!
Please consider below example as a workaround!
The method described below is not persistent.
Steps:
Start your minikube instance with the --extra-config
Connect to your minikube instance and edit/add files:
/etc/kubernetes/manifests/kube-scheduler.yaml
newly created KubeSchedulerConfiguration
Delete the failing kube-scheduler Pod and wait for it to be recreated.
Start your minikube instance with the --extra-config
As previously said you can add some additional parameters for your $ minikube start to be passed down to the provisioning process.
In this setup you can either pass it with $ minikube start ... or do it manually later on.
$ minikube start --extra-config=scheduler.config="/etc/kubernetes/sched.yaml"
Above parameter will add the - --config=/etc/kubernetes/sched.yaml to the command of your kube-scheduler. It will look for the file in the mentioned location.
Connect to your minikube instance ($ minikube ssh) and edit/add files:
Your kube-scheduler will fail as you've passed an argument (config) that is incorrect (lack of file). To work around this you will need to:
add: /etc/kubernetes/sched.yaml with your desired configuration
modify: /etc/kubernetes/manifests/kube-scheduler.yaml:
add to: volumeMounts:
- mountPath: /etc/kubernetes/sched.yaml
name: scheduler
readOnly: true
add to volumes:
- hostPath:
path: /etc/kubernetes/sched.conf
type: FileOrCreate
name: scheduler
Delete the failing kube-scheduler Pod and wait for it to be recreated.
You will need to redeploy modified scheduler to get its new config running:
$ kubectl delete pod -n kube-system kube-scheduler-minikube
After some time you should see your kube-scheduler in Ready state.
Additional resources:
Kubernetes.io: Docs: Concepts: Scheduling eviction: Kube-scheduler
Kubernetes.io: Docs: Reference: Command line tools reference: Kube-scheduler

Related

Airflow on Kubernetes - NFS volume won't mount onto worker

I'm fairly new to Kubernetes so apologies for any mixups in terminology.
I'm using the official Airflow helm chart to create a development environment, and have my Dogs (and other) folders in a NFS volume on my local machine. I have configured the values.yaml like so (same for both the scheduler and worker):
# Mount additional volumes into scheduler.
extraVolumes:
- name: dags
nfs:
server: '10.106.0.113'
path: '/home/dev/projects/airflow-jobs/dags'
- name: plugins
nfs:
server: '10.106.0.113'
path: '/home/dev/projects/airflow-jobs/plugins'
- name: scripts
nfs:
server: '10.106.0.113'
path: '/home/dev/projects/airflow-jobs/scripts'
extraVolumeMounts:
- mountPath: '/opt/airflow/dags'
name: 'dags'
- mountPath: '/opt/airflow/plugins'
name: 'plugins'
- mountPath: '/opt/airflow/scripts'
name: 'scripts'
When I then spin this up, only one of the scheduler or worker pod will mount the volume successfully - the other will fail with the following message:
> kubectl describe pod airflow-worker-0
Warning FailedMount 2s kubelet Unable to attach or mount volumes: unmounted volumes=[dags plugins scripts], unattached volumes=[dags plugins scripts logs config kube-api-access-dnsjx]: timed out waiting for the condition
Why am I receiving this error - is it not possible to have two pods using the same NFS store? I had this working before using the same values.yaml file so I don't quite know what has changed!
Figured it out - it was due to the NFS mount being configured as ReadWriteOnce. As per the documentation here, this does allow multiple pods to access the volume, but only if they are located on the same node. So what was happening is that my Scheduler pod would spin up first, mount the volume, and then when the Worker pod followed it would be unable to do so because the Scheduler had reserved the volume. By coincidence, the first time I deployed these two pods must have been assigned the same node.
The simplest solution here would be to mount this as ReadWriteMany, but as I have limited permissions to my cluster and development environment, I simply made some changes to my deployment to ensure that the pods that needed access to this volume were on the same node. Plus, learning experience!
First - get the nodes that each pod is assigned to using kubectl get pods -o wide.
Get all the nodes in the cluster kubectl get nodes --show-labels
Pick a node to assign the two pods that need to share the NFS mount to. This was arbitrary, so lets call it "node123".
Update the labels of the node kubectl label nodes node123 airflow=nfs
Finally, in the values.yaml file, specify the nodeSelector property for the Scheduler and Worker nodes!
# Select certain nodes for airflow worker pods.
nodeSelector:
airflow: nfs
Then re-deploy the chart, and everything works as intended!

Kubernetes Alpha Debug command

I'm trying the debug CLI feature on 1.18 release of Kubernetes but I have an issue while I have executed debug command.
I have created a pod show as below.
kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
After than when running this command: kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Kubernetes is hanging like that :
Defaulting debug container name to debugger-aaaa.
How Can I resolve that issue?
It seems that you need to enable the feature gate on all control plane components as well as on the kubelets. If the feature is enabled partially (for instance, only kube-apiserver and kube-scheduler), the resources will be created in the cluster, but no containers will be created, thus there will be nothing to attach to.
In addition to the answer posted by Konstl
To enable EphemeralContainers featureGate correctly, add on the master nodes to:
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
the following line to container command:
spec:
containers:
- command:
- kube-apiserver # or kube-controller-manager/kube-scheduler
- --feature-gates=EphemeralContainers=true # < -- add this line
Pods will restart immediately.
For enabling the featureGate for kubelet add on all nodes to:
/var/lib/kubelet/config.yaml
the following lines at the bottom:
featureGates:
EphemeralContainers: true
Save the file and run the following command:
$ systemctl restart kubelet
This was enough in my case to be able to use kubectl alpha debug as it's explained in the documentation
Additional usefull pages:
Ephemeral Containers
Share Process Namespace between Containers in a Pod
I'm seeing similar behaviour. Although i'm running 1.17 k8s with 1.18 kubectl. But I was under the impression that the feature was added in 1.16 in k8s and in 1.18 in kubectl.

How to add flag to kubelet

I want to deploy rook for kubernetes. I use 1 master and 3 worker and the host is ubuntu in baremetal. but the container stuck in creating container. after a lot of search i understand i should use this document https://github.com/rook/rook/blob/master/Documentation/flexvolume.md#most-common-readwrite-flexvolume-path that said
Configuring the Rook operator You must provide the above found
FlexVolume path when deploying the rook-operator by setting the
environment variable FLEXVOLUME_DIR_PATH. For example:
env: [...]
- name: FLEXVOLUME_DIR_PATH value: "/var/lib/kubelet/volumeplugins" (In the operator.yaml manifest replace with the
path or if you use helm set the agent.flexVolumeDirPath to the
FlexVolume path)
Configuring the Kubernetes kubelet You need to add the flexvolume flag
with the path to all nodes's kubelet in the Kubernetes cluster:
--volume-plugin-dir=PATH_TO_FLEXVOLUME (Where the PATH_TO_FLEXVOLUME is the above found FlexVolume path)
the question is how can i add flexvolume flag with the path to all nodes's kubelet ?
#yasin lachini,
If you deploy kubernetes cluster on baremetal, you don't need to configure anything. That is because /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ is the kubelet default FlexVolume path and Rook assumes the default FlexVolume path if not set differently.
My env:
rook-ceph/operator.yml (use default FLEXVOLUME_DIR_PATH) :
...
# Set the path where the Rook agent can find the flex volumes
# - name: FLEXVOLUME_DIR_PATH
# value: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
...
After deploy,on node:
# ls /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
ceph.rook.io~rook ceph.rook.io~rook-ceph-system rook.io~rook rook.io~rook-ceph-system
There are two options.
I. set
KUBELET_EXTRA_ARGS=--FLEXVOLUME_DIR_PATH=/var/lib/kubelet/volumeplugins
within the file
/etc/default/kubelet
And restart kubelete service
sudo systemctl restart kubelet
II. You can set kubelet parameters via a config file.
For example:
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
env:
- name: FLEXVOLUME_DIR_PATH
value: "/var/lib/kubelet/volumeplugins"
Then, you just start the Kubelet with the --config flag set to the path of the Kubelet’s config file
sudo kubelet --config=/etc/default/kubelet/custom-conf.config
https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
If you initiated your Cluster with kubeadm you can add flags in this file: /var/lib/kubelet/kubeadm-flags.env and then restart the kubelet with sudo systemctl restart kubelet

Mount propagation in kubernetes

I am using Mount propagation feature of Kubernetes to check the health of mount points of certain type. I create a daemonset and run a script which would do a simple ls on these mount points. I noticed that new mount points are not getting listed from the pods. Is this the expected behaviour.
volumeMounts:
- mountPath: /host
name: host-kubelet
mountPropagation: HostToContainer
volumes:
- name: host-kubelet
hostPath:
path: /var/lib/kubelet
Related Issue : hostPath containing mounts do not update as they change on the host #44713
In brief, Mount propagation allows sharing volumes mounted by a Container to other Containers in the same Pod, or even to other Pods on the same node.
Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts. Its values are:
HostToContainer - one way propagation, from host to container. If you
mount anything inside the volume, the Container will see it there.
Bidirectional - In addition to propagation from host to container,
all volume mounts created by the Container will be propagated back to
the host, so all Containers of all Pods that use the same volume will
see it as well.
Based on documentation the Mount propagation feature is in alpha state for clusters v1.9, and going to be beta on v1.10
I've reproduced your case on kubernetes v1.9.2 and found that it completely ignores MountPropagation configuration parameter. If you try to check current state of the DaemonSet or Deployment, you'll see that this option is missed from the listed yaml configuration
$ kubectl get daemonset --export -o yaml
If you try to run just docker container with mount propagation option you may see it is working as expected:
docker run -d -it -v /tmp/mnt:/tmp/mnt:rshared ubuntu
Comparing docker container configuration with kubernetes pod container in the volume mount section, you may see that the last flag (shared/rshared) is missing in kubernetes container.
And that's why it happens in Google kubernetes clusters and may happen to clusters managed by other providers:
To ensure stability and production quality, normal Kubernetes Engine
clusters only enable features that are beta or higher. Alpha features
are not enabled on normal clusters because they are not
production-ready or upgradeable.
Since Kubernetes Engine automatically upgrades the Kubernetes control
plane, enabling alpha features in production could jeopardize the
reliability of the cluster if there are breaking changes in a new
version.
Alpha level features availability: committed to main kubernetes repo; appears in an official release; feature is disabled by default, but may be enabled by flag (in case you are able to set flags)
Before mount propagation can work properly on some deployments (CoreOS, RedHat/Centos, Ubuntu) mount share must be configured correctly in Docker as shown below.
Edit your Docker’s systemd service file. Set MountFlags as follows:
MountFlags=shared
Or, remove MountFlags=slave if present. Then restart the Docker daemon:
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

How to specify docker credentials for Kubernetes static pods with CRI enabled?

When starting up a Kubernetes cluster, I load etcd plus the core kubernetes processes - kube-proxy, kube-apiserver, kube-controller-manager, kube-scheduler - as static pods from a private registry. This has worked in the past by ensuring that the $HOME environment variable is set to "/root" for kubelet, and then having /root/.docker/config.json defined with the credentials for the private docker registry.
When attempting to run Kubernetes 1.6, with CRI enabled, I get errors in the kubelet log saying it cannot pull the pause:3.0 container from my private docker registry due to no authentication.
Setting --enable-cri=false on the kubelet command line works, but when CRI is enabled, it doesn't seem to use the /root/.docker/config file for authentication.
Is there some new way to provide the docker credentials needed to load static pods when running with CRI enabled?
In 1.6, I managed to make it work with the following recipe in https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
$ kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
You need to specify newly created myregistrykey as the credential under imagePullSecrets field in the pod spec.
apiVersion: v1
kind: Pod
metadata:
name: foo
namespace: awesomeapps
spec:
containers:
- name: foo
image: janedoe/awesomeapp:v1
imagePullSecrets:
- name: myregistrykey
It turns out that there is a deficiency in the CRI capabilities in Kubernetes 1.6. With CRI, the "pause" container - now called the "Pod Sandbox Image" is treated as a special case - because it is an "implementation detail" of the container runtime whether or not you even need it. In the 1.6 release, the credentials applied for other containers, from /root/.docker/config.json, for example, are not used when trying to pull the Pod Sandbox Image.
Thus, if you are trying to pull this image from a private registry, the CRI logic doesn't associate the credentials with the pull request. There is now a Kubernetes issue (#45738) to address this, targeted for 1.7.
In the meantime, an easy workaround is to pre-pull the "Pause" container into the node's local docker image cache before starting up the kubelet process.