I want to deploy rook for kubernetes. I use 1 master and 3 worker and the host is ubuntu in baremetal. but the container stuck in creating container. after a lot of search i understand i should use this document https://github.com/rook/rook/blob/master/Documentation/flexvolume.md#most-common-readwrite-flexvolume-path that said
Configuring the Rook operator You must provide the above found
FlexVolume path when deploying the rook-operator by setting the
environment variable FLEXVOLUME_DIR_PATH. For example:
env: [...]
- name: FLEXVOLUME_DIR_PATH value: "/var/lib/kubelet/volumeplugins" (In the operator.yaml manifest replace with the
path or if you use helm set the agent.flexVolumeDirPath to the
FlexVolume path)
Configuring the Kubernetes kubelet You need to add the flexvolume flag
with the path to all nodes's kubelet in the Kubernetes cluster:
--volume-plugin-dir=PATH_TO_FLEXVOLUME (Where the PATH_TO_FLEXVOLUME is the above found FlexVolume path)
the question is how can i add flexvolume flag with the path to all nodes's kubelet ?
#yasin lachini,
If you deploy kubernetes cluster on baremetal, you don't need to configure anything. That is because /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ is the kubelet default FlexVolume path and Rook assumes the default FlexVolume path if not set differently.
My env:
rook-ceph/operator.yml (use default FLEXVOLUME_DIR_PATH) :
...
# Set the path where the Rook agent can find the flex volumes
# - name: FLEXVOLUME_DIR_PATH
# value: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
...
After deploy,on node:
# ls /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
ceph.rook.io~rook ceph.rook.io~rook-ceph-system rook.io~rook rook.io~rook-ceph-system
There are two options.
I. set
KUBELET_EXTRA_ARGS=--FLEXVOLUME_DIR_PATH=/var/lib/kubelet/volumeplugins
within the file
/etc/default/kubelet
And restart kubelete service
sudo systemctl restart kubelet
II. You can set kubelet parameters via a config file.
For example:
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
env:
- name: FLEXVOLUME_DIR_PATH
value: "/var/lib/kubelet/volumeplugins"
Then, you just start the Kubelet with the --config flag set to the path of the Kubelet’s config file
sudo kubelet --config=/etc/default/kubelet/custom-conf.config
https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
If you initiated your Cluster with kubeadm you can add flags in this file: /var/lib/kubelet/kubeadm-flags.env and then restart the kubelet with sudo systemctl restart kubelet
Related
I am currently working in Minikube cluster and looking to change some flags of kubernetes scheduler configuration, but I can't find it. The file looks something like-
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
algorithmSource:
provider: DefaultProvider
...
disablePreemption: true
What is it's name and where can I find it?
Posting this answer as a community wiki to set a baseline and to provide additional resources/references rather than giving a definitive solution.
Feel free to edit and expand.
I haven't found the file that you are referencing (KubeSchedulerConfiguration) in minikube.
The minikube provisioning process does not create it nor references it in the configuration files (/etc/kubernetes/manifests/kube-scheduler.yaml and the --config=PATH parameter).
I'd reckon you could take a look on other Kubernetes solutions where you can configure how your cluster is created (how kube-scheduler is configured). Some of the options are:
Kubernetes.io: Docs: Setup: Production environment: Tools: Kubeadm: Create cluster and also:
Kubernetes.io: Docs: Setup: Production environment: Tools: Kubeadm: Control plane flags
Github.com: Kubernetes sigs: Kubespray
A side note!
Both: kubespray and minikube are using kubeadm as a bootstrapper!
I would also consider creating additional scheduler that would be responsible for spawning your workload (by referencing in the YAML manifests):
Kubernetes.io: Docs: Tasks: Extend Kubernetes: Configure multiple schedulers
I haven't tested it extensively and in the long term but I've managed to include the YAML manifest that you are referencing for the kube-scheduler.
Disclaimers!
Please consider below example as a workaround!
The method described below is not persistent.
Steps:
Start your minikube instance with the --extra-config
Connect to your minikube instance and edit/add files:
/etc/kubernetes/manifests/kube-scheduler.yaml
newly created KubeSchedulerConfiguration
Delete the failing kube-scheduler Pod and wait for it to be recreated.
Start your minikube instance with the --extra-config
As previously said you can add some additional parameters for your $ minikube start to be passed down to the provisioning process.
In this setup you can either pass it with $ minikube start ... or do it manually later on.
$ minikube start --extra-config=scheduler.config="/etc/kubernetes/sched.yaml"
Above parameter will add the - --config=/etc/kubernetes/sched.yaml to the command of your kube-scheduler. It will look for the file in the mentioned location.
Connect to your minikube instance ($ minikube ssh) and edit/add files:
Your kube-scheduler will fail as you've passed an argument (config) that is incorrect (lack of file). To work around this you will need to:
add: /etc/kubernetes/sched.yaml with your desired configuration
modify: /etc/kubernetes/manifests/kube-scheduler.yaml:
add to: volumeMounts:
- mountPath: /etc/kubernetes/sched.yaml
name: scheduler
readOnly: true
add to volumes:
- hostPath:
path: /etc/kubernetes/sched.conf
type: FileOrCreate
name: scheduler
Delete the failing kube-scheduler Pod and wait for it to be recreated.
You will need to redeploy modified scheduler to get its new config running:
$ kubectl delete pod -n kube-system kube-scheduler-minikube
After some time you should see your kube-scheduler in Ready state.
Additional resources:
Kubernetes.io: Docs: Concepts: Scheduling eviction: Kube-scheduler
Kubernetes.io: Docs: Reference: Command line tools reference: Kube-scheduler
When using the following settings in the kubeadm config file:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- fd10::4:5
I see the following warning when initializing my cluster:
[root#k8s-ansible-2 ansible]# kubeadm init --config /home/ansible/kubeadm-config-new.yaml
W0414 05:52:56.598882 1454 defaults.go:186] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [fd10::4:a]; the provided value is: [fd10::4:5]
and kubeadm actually does configure the recommended value:
[ansible#k8s-ansible-2 ~]$ kubectl get service kube-dns -n kube-system -o yaml | grep clusterIP
clusterIP: fd10::4:a
The Kubelet systemd config file is the following:
[root#k8s-ansible-3 ~]# more /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
ExecStartPre=/bin/sleep 15
In my environment, I need to control which ClusterIP is assigned to which service so my questions are:
Is it a bug or known limitation ?. I was not able to find anything related to that specific behavior.
It seems kubeadm assigns the 10th address from serviceCIDR to coredns service. Could someone confirm if this is the default implementation and it is not a random address from the serviceCIDR pool ?.
Thanks for your support.
I'm trying the debug CLI feature on 1.18 release of Kubernetes but I have an issue while I have executed debug command.
I have created a pod show as below.
kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
After than when running this command: kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Kubernetes is hanging like that :
Defaulting debug container name to debugger-aaaa.
How Can I resolve that issue?
It seems that you need to enable the feature gate on all control plane components as well as on the kubelets. If the feature is enabled partially (for instance, only kube-apiserver and kube-scheduler), the resources will be created in the cluster, but no containers will be created, thus there will be nothing to attach to.
In addition to the answer posted by Konstl
To enable EphemeralContainers featureGate correctly, add on the master nodes to:
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
the following line to container command:
spec:
containers:
- command:
- kube-apiserver # or kube-controller-manager/kube-scheduler
- --feature-gates=EphemeralContainers=true # < -- add this line
Pods will restart immediately.
For enabling the featureGate for kubelet add on all nodes to:
/var/lib/kubelet/config.yaml
the following lines at the bottom:
featureGates:
EphemeralContainers: true
Save the file and run the following command:
$ systemctl restart kubelet
This was enough in my case to be able to use kubectl alpha debug as it's explained in the documentation
Additional usefull pages:
Ephemeral Containers
Share Process Namespace between Containers in a Pod
I'm seeing similar behaviour. Although i'm running 1.17 k8s with 1.18 kubectl. But I was under the impression that the feature was added in 1.16 in k8s and in 1.18 in kubectl.
I have installed my kubernetes cluster(1master, 3worker)in the digital ocean.
The problem is I installed both flannel and calico in this cluster.
I want to remove the calico completely from my cluster or is it ok to have multiple CNI?
Use following two commands to remove calico from your node:
kubectl delete -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl delete -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
PLease change the security group, If its AWS
If its s VM, Expose the port
You should add these
https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
and
https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
to your ansible playbook file state as "absent" then when you run ansible playbook file i.e as below.
- name: Delete Calico (rbac-kdd.yaml) from cluster
kubernetes:
api_endpoint: 123.45.67.89
insecure: true
file_reference: /path/to/rbac-kdd.yaml
state: absent
- name: Delete Calico (calico.yaml) from cluster
kubernetes:
api_endpoint: 123.45.67.89
insecure: true
file_reference: /path/to/calico.yaml
state: absent
Please check your ansible playbook file for cluster installation and change/modify values according your needs.
For more details kubernetes with ansible you can see this:
https://docs.ansible.com/ansible/2.5/modules/kubernetes_module.html
I hope it will be helpful for you
I am using Mount propagation feature of Kubernetes to check the health of mount points of certain type. I create a daemonset and run a script which would do a simple ls on these mount points. I noticed that new mount points are not getting listed from the pods. Is this the expected behaviour.
volumeMounts:
- mountPath: /host
name: host-kubelet
mountPropagation: HostToContainer
volumes:
- name: host-kubelet
hostPath:
path: /var/lib/kubelet
Related Issue : hostPath containing mounts do not update as they change on the host #44713
In brief, Mount propagation allows sharing volumes mounted by a Container to other Containers in the same Pod, or even to other Pods on the same node.
Mount propagation of a volume is controlled by mountPropagation field in Container.volumeMounts. Its values are:
HostToContainer - one way propagation, from host to container. If you
mount anything inside the volume, the Container will see it there.
Bidirectional - In addition to propagation from host to container,
all volume mounts created by the Container will be propagated back to
the host, so all Containers of all Pods that use the same volume will
see it as well.
Based on documentation the Mount propagation feature is in alpha state for clusters v1.9, and going to be beta on v1.10
I've reproduced your case on kubernetes v1.9.2 and found that it completely ignores MountPropagation configuration parameter. If you try to check current state of the DaemonSet or Deployment, you'll see that this option is missed from the listed yaml configuration
$ kubectl get daemonset --export -o yaml
If you try to run just docker container with mount propagation option you may see it is working as expected:
docker run -d -it -v /tmp/mnt:/tmp/mnt:rshared ubuntu
Comparing docker container configuration with kubernetes pod container in the volume mount section, you may see that the last flag (shared/rshared) is missing in kubernetes container.
And that's why it happens in Google kubernetes clusters and may happen to clusters managed by other providers:
To ensure stability and production quality, normal Kubernetes Engine
clusters only enable features that are beta or higher. Alpha features
are not enabled on normal clusters because they are not
production-ready or upgradeable.
Since Kubernetes Engine automatically upgrades the Kubernetes control
plane, enabling alpha features in production could jeopardize the
reliability of the cluster if there are breaking changes in a new
version.
Alpha level features availability: committed to main kubernetes repo; appears in an official release; feature is disabled by default, but may be enabled by flag (in case you are able to set flags)
Before mount propagation can work properly on some deployments (CoreOS, RedHat/Centos, Ubuntu) mount share must be configured correctly in Docker as shown below.
Edit your Docker’s systemd service file. Set MountFlags as follows:
MountFlags=shared
Or, remove MountFlags=slave if present. Then restart the Docker daemon:
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker