I have a multi-master kubernetes v1.1.3 setup. I use the hyperkube binary to run kubelet on the master and have the scheduler and controller manager running in pods with the podmaster acting as the elector. The master's are loadbalanced using HAproxy, and the kubelet on the minions uses that loadbalanced IP as their apiserver endpoint. I am using docker v1.8.2.
I am not having any errors in use, I can deploy pods that are running without issue across the minions and everything seems happy.
However, I am getting constant error spam (every 10 seconds) from the kubelet on the master:
[err] [kubelet] E0201 21:10:54 kubelet.go:1361] Mirror pod not available
[err] [kubelet] E0201 21:10:54 kubelet.go:1361] Mirror pod not available
[err] [kubelet] E0201 21:10:54 kubelet.go:1361] Mirror pod not available
[err] [kubelet] E0201 21:10:54 kubelet.go:1361] Mirror pod not available
[err] [kubelet] E0201 21:11:04 kubelet.go:1361] Mirror pod not available
I am concerned that this is symptomatic of something set up improperly that I am missing. Is this anything to be concerned of? If not, is there anything I could configure to silence the error spam?
edit: here are the options I am running kubelet with:
kubelet on master (mostly standard):
hyperkube kubelet --v=2 --config=/etc/kubernetes/manifests --kubeconfig=<path-to-config> --port=10250 --address=0.0.0.0 --cadvisor-port=4194 --allow-privileged=true
all pod manifests: http://pastebin.com/MJfQz78r
It seems to be something related to deprecated feature static pod - http://kubernetes.io/v1.1/docs/admin/static-pods.html
Source code - https://github.com/kubernetes/kubernetes/blob/v1.1.4/pkg/kubelet/kubelet.go#L1361
IMHO you can ignore it. I didn't find this feature in 1.2alpha7 source code. Maybe you can decrease your log level --v=0 https://github.com/golang/glog
Thank you for your manifests.
Related
I tried to install minikube via below steps on my Monterey:
1.brew install docker
2.download virtualbox 6.1 from https://www.virtualbox.org/ and install
3.curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 sudo install minikube-darwin-amd64 /usr/local/bin/minikube
(it seems homebrew no longer supports virtualbox and minikube?)
then, I tried to start minikube:
minikube start --driver=virtualbox
but got below errors:
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock logs CONTAINERID'
stderr:
W0707 14:15:11.219566 7860 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
💡 Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
🍿 Related issue: https://github.com/kubernetes/minikube/issues/4172
any advice?
Below worked for me today
minikube delete --all
minikube start --force-systemd
Source: https://github.com/kubernetes/minikube/issues/14561
I am currently trying to setup multi master setup on enterprise environment.
OS: RHEL 7.5
Kubernetes Versions: 1.22.1
Documentation followed to setup master node & LB nodes:
https://www.lisenet.com/2021/install-and-configure-a-multi-master-ha-kubernetes-cluster-with-kubeadm-haproxy-and-keepalived-on-centos-7/
While running kubeadm init command we are getting the error and stopped installation.
Can you please help us to fix the issues
enter code hereI0917 11:05:08.005343 26417 waitcontrolplane.go:89] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster
I am trying to setup my very first Kubernetes cluster and it seems to have setup fine until nginx-ingress controller.
Here is my cluster information:
Nodes: three RHEL7 and one RHEL8 nodes
Master is running on RHEL7
Kubernetes server version: 1.19.1
Networking used: flannel
coredns is running fine.
selinux and firewall are disabled on all nodes
Here are my all pods running in kube-system
I then followed instructions on following page to install nginx ingress controller: https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
Instead of deployment, I decided to use daemon-set since I am going to have only few nodes running in my kubernetes cluster.
After following the instructions, pod on my RHEL8 is constantly failing with the following error:
Readiness probe failed: Get "http://10.244.3.2:8081/nginx-ready": dial
tcp 10.244.3.2:8081: connect: connection refused Back-off restarting
failed container
Here is the screenshot shows that RHEL7 pods are working just fine and RHEL8 is failing:
All nodes are setup exactly the same way and there is no difference.
I am very new to Kubernetes and don't know much internals of it. Can someone please point me on how can I debug and fix this issue? I am really willing to learn from issues like this.
This is how I provisioned RHEL7 and RHEL8 nodes
Installed docker version: 19.03.12, build 48a66213fe
Disabled firewalld
Disabled swap
Disabled SELinux
To enable iptables to see bridged traffic, set net.bridge.bridge-nf-call-ip6tables = 1 and net.bridge.bridge-nf-call-iptables = 1
Added hosts entry for all the nodes involved in Kubernetes cluster so that they can find each other without hitting DNS
Added IP address of all nodes in Kubernetes cluster on /etc/environment for no_proxy so that it doesn't hit corporate proxy
Verified docker driver to be "systemd" and NOT "cgroupfs"
Reboot server
Install kubectl, kubeadm, kubelet as per kubernetes guide here at: https://kubernetes.io/docs/tasks/tools/install-kubectl/
Start and enable kubelet service
Initialize master by executing the following:
kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
Apply node-selector patch for mixed OS scheduling
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/l2bridge/manifests/node-selector-patch.yml
kubectl patch ds/kube-proxy --patch "$(cat node-selector-patch.yml)" -n=kube-system
Apply flannel CNI
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Modify net-conf.json section of kube-flannel.yml for a type "host-gw"
kubectl apply -f kube-flannel.yml
Apply node selector patch
kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system
Thanks
According to kubernetes documentation the list of supported host operating systems is as follows:
Ubuntu 16.04+
Debian 9+
CentOS 7
Red Hat Enterprise Linux (RHEL) 7
Fedora 25+
HypriotOS v1.0.1+
Flatcar Container Linux (tested with 2512.3.0)
This article mentioned that there are network issues on RHEL 8:
(2020/02/11 Update: After installation, I keep facing pod network issue which is like deployed pod is unable to reach external network
or pods deployed in different workers are unable to ping each other
even I can see all nodes (master, worker1 and worker2) are ready via
kubectl get nodes. After checking through the Kubernetes.io official website, I observed the nfstables backend is not compatible with the
current kubeadm packages. Please refer the following link in “Ensure
iptables tooling does not use the nfstables backend”.
The simplest solution here is to reinstall the node on supported operating system.
I have setup a Kubernetes cluster with one master (kube-master) and 2 slaves (kube-node-01 and kube-node-02)
All was running fine ... now after debian stretch -> buster upgrade my coredns pods are failing with CrashLoopBackOff for some reason.
I did a kubectl describe and the error is Readiness probe failed: HTTP probe failed with statuscode: 503
The Readiness url looks suspicious to me http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3 ... there is no hostname !? is that correct?
The Liveness property also does not have a hostname.
All vm's are pingable from each other.
Any ideas?
I met similar issue when upgrade my host machine to ubuntu 18.04 which uses systemd-resolved as DNS server. The nameserver field in /etc/resolv.conf is using a local IP address 127.0.0.53 which would cause coreDNS failed to start.
You can take a look at the details from the following link.
https://github.com/coredns/coredns/blob/master/plugin/loop/README.md#troubleshooting-loops-in-kubernetes-clusters
I just hit this problem myself. Apparently the lack of a hostname in the healthcheck url is ok.
What got me ahead was
microk8s.inspect
The output said there's a problem with forwarding on iptables. Since I have firewalld on my system, I temporarily disabled it
systemctl stop firewalld
and then disabled dns in microk8s and enabled it again (for some unknown reason the dns pod didn't get up on it's own)
microk8s.disable dns
microk8s.enable dns
it started without any issues.
I would start troubleshooting with kubelet agent verification on the master and worker nodes in order exclude any intercommunication issue within a cluster nodes whenever the rest core runtime Pods are up and running, as kubelet is the main contributor for
Liveness and Readiness probes.
systemctl status kubelet -l
journalctl -u kubelet
Mentioned in the question health check URLs are fine, as they are predefined in CoreDNS deployment per design.
Ensure that CNI plugin Pods are functioning and cluster overlay network intercepts requests from Pod to Pod communication as CoreDNS is very sensitive on any issue related to entire cluster networking.
In addition to #Hang Du answer about CoreDNS pods loopback issue, I encourage you to get more information regarding CoreDNS problems investigation in official k8s debug documentation.
Have three hosts to run Rancher cluster.
Rancher: 1.6.10
Kubernetes: 1.7.7
Install k8s from catalog on master host.
Set orchestration=true and etcd=true labels to two Rancher agent hosts.
After the k8s stack finished, only the kubelet went wrong. Unhealthy with 0 containers.
Why?
The question has been debugged in the comment section.
Kubernetes Mantra
I have added some additional point to keep it in mind to debug the Kubelet.
The K8s cluster is made of Masters and Workers Node which has several components. Kubelet is one the component which needs to take care properly.
Let's begin by saying that Master node manages or orchestrate the cluster state and Workers node run the pods.However, Without Kubelet It does not work Since It will be part of each node whether it's a Master or Worker.
Performance of the cluster certainly depends on the kubelet.
We can use the following command to check its status and activity or logs.As It is deployed as system-service by systemd.
systemctl status kubelet
journalctl -xeu kubele