Kubernetes service is not reachable throw browser - kubernetes

I've deployed a small K3S cluster with a master and two workers:
The VMs were made with Multipass:
$ multipass ls
Name State IPv4 Image
master-node Running 10.200.68.230 Ubuntu 20.04 LTS
10.42.0.0
10.42.0.1
worker01 Running 10.200.68.67 Ubuntu 20.04 LTS
10.42.1.0
10.42.1.1
worker02 Running 10.200.68.227 Ubuntu 20.04 LTS
10.42.2.0
10.42.2.1
The cluster was made with k3sup:
$ kubectl get node
NAME STATUS ROLES AGE VERSION
master-node Ready control-plane,etcd,master 13m v1.21.3+k3s1
worker01 Ready <none> 10m v1.21.3+k3s1
worker02 Ready <none> 9m46s v1.21.3+k3s1
Workers are all labelled with ols.role=worker.
I'd like to install a NodeRed service on the workers nodes. I've used the following commands:
helm repo add k8s-at-home https://k8s-at-home.com/charts/
helm repo update
helm install node-red k8s-at-home/node-red --set nodeSelector."ols\.role"=worker
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=node-red,app.kubernetes.io/instance=node-red" -o jsonpath="{.items[0].metadata.name}")
while [[ $(kubectl get node $POD_NAME -o 'jsonpath={..status.conditions[?(#.type=="Running")].status}') != "True" ]]; do echo "waiting for pod" && sleep 1; done
kubectl port-forward $POD_NAME 8080:1880&
The service is supposed to be running on port 8080.
Pod's logs look ok:
$ kubectl logs $POD_NAME
> node-red-docker#1.3.5 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
29 Jul 08:20:12 - [info]
Welcome to Node-RED
===================
29 Jul 08:20:12 - [info] Node-RED version: v1.3.5
29 Jul 08:20:12 - [info] Node.js version: v10.24.1
29 Jul 08:20:12 - [info] Linux 5.4.0-80-generic x64 LE
29 Jul 08:20:12 - [info] Loading palette nodes
29 Jul 08:20:12 - [info] Settings file : /data/settings.js
29 Jul 08:20:12 - [info] Context store : 'default' [module=memory]
29 Jul 08:20:12 - [info] User directory : /data
29 Jul 08:20:12 - [warn] Projects disabled : editorTheme.projects.enabled=false
29 Jul 08:20:12 - [info] Flows file : /data/flows.json
29 Jul 08:20:12 - [warn]
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
29 Jul 08:20:12 - [info] Server now running at http://127.0.0.1:1880/
29 Jul 08:20:12 - [info] Starting flows
29 Jul 08:20:12 - [info] Started flows
When I try to reach the webpage (http://192.168.1.14:8080 or even http://127.0.0.1:1880/), the server responds an error: ERR_CONNECTION_REFUSED
The services should be running:
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 32m
node-red ClusterIP 10.43.18.33 <none> 1880/TCP 26m
Is there something else to do to make it work ?

Since your service is Cluster Ip you can not access the service out of Kubernetes cluster.
You have to expose your service as Node port or Loadbalancer.
https://kubernetes.io/docs/concepts/services-networking/service/
however, for testing and debugging locally you can use this command :
kubectl port-forward svc/node-red -n <replace-namespace-name> 1880:1880
once command running open the browser and open URL
HTTP://localhost:1880

Related

Redis sentinel HA on Kubernetes

I am trying to have 1 redis master with 2 redis replicas tied to a 3 Quorum Sentinel on Kubernetes. I am very new to Kubernetes.
My initial plan was to have the master running on a pod tied to 1 Kubernetes SVC and the 2 replicas running on their own pods tied to another Kubernetes SVC. Finally, the 3 Sentinel pods will be tied to their own SVC. The replicas will be tied to the master SVC (because without svc, ip will change). The sentinel will also be configured and tied to master and replica SVCs. But I'm not sure if this is feasible because when master pod crashes, how will one of the replica pods move to the master SVC and become the master? Is that possible?
The second approach I had was to wrap redis pods in a replication controller and the same for sentinel as well. However, I'm not sure how to make one of the pods master and the others replicas with a replication controller.
Would any of the two approaches work? If not, is there a better design that I can adopt? Any leads would be appreciated.
You can deploy Redis Sentinel using the Helm package manager and the Redis Helm Chart.
If you don't have Helm3 installed yet, you can use this documentation to install it.
I will provide a few explanations to illustrate how it works.
First we need to get the values.yaml file from the Redis Helm Chart to customize our installation:
$ wget https://raw.githubusercontent.com/bitnami/charts/master/bitnami/redis/values.yaml
We can configure a lot of parameters in the values.yaml file , but for demonstration purposes I only enabled Sentinel and set the redis password:
NOTE: For a list of parameters that can be configured during installation, see the Redis Helm Chart Parameters documentation.
# values.yaml
global:
redis:
password: redispassword
...
replica:
replicaCount: 3
...
sentinel:
enabled: true
...
Then we can deploy Redis using the configuration from the values.yaml file:
NOTE: It will deploy a three Pod cluster (one master and two slaves) managed by the StatefulSets with a sentinel container running inside each Pod.
$ helm install redis-sentinel bitnami/redis --values values.yaml
Be sure to carefully read the NOTES section of the chart installation output. It contains many useful information (e.g. how to connect to your database from outside the cluster)
After installation, check redis StatefulSet, Pods and Services (headless service can be used for internal access):
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP
redis-sentinel-node-0 2/2 Running 0 2m13s 10.4.2.21
redis-sentinel-node-1 2/2 Running 0 86s 10.4.0.10
redis-sentinel-node-2 2/2 Running 0 47s 10.4.1.10
$ kubectl get sts
NAME READY AGE
redis-sentinel-node 3/3 2m41s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-sentinel ClusterIP 10.8.15.252 <none> 6379/TCP,26379/TCP 2m
redis-sentinel-headless ClusterIP None <none> 6379/TCP,26379/TCP 2m
As you can see, each redis-sentinel-node Pod contains the redis and sentinel containers:
$ kubectl get pods redis-sentinel-node-0 -o jsonpath={.spec.containers[*].name}
redis sentinel
We can check the sentinel container logs to find out which redis-sentinel-node is the master:
$ kubectl logs -f redis-sentinel-node-0 sentinel
...
1:X 09 Jun 2021 09:52:01.017 # Configuration loaded
1:X 09 Jun 2021 09:52:01.019 * monotonic clock: POSIX clock_gettime
1:X 09 Jun 2021 09:52:01.019 * Running mode=sentinel, port=26379.
1:X 09 Jun 2021 09:52:01.026 # Sentinel ID is 1bad9439401e44e749e2bf5868ad9ec7787e914e
1:X 09 Jun 2021 09:52:01.026 # +monitor master mymaster 10.4.2.21 6379 quorum 2
...
1:X 09 Jun 2021 09:53:21.429 * +slave slave 10.4.0.10:6379 10.4.0.10 6379 # mymaster 10.4.2.21 6379
1:X 09 Jun 2021 09:53:21.435 * +slave slave 10.4.1.10:6379 10.4.1.10 6379 # mymaster 10.4.2.21 6379
...
As you can see from the logs above, the redis-sentinel-node-0 Pod is the master and the redis-sentinel-node-1 & redis-sentinel-node-2 Pods are slaves.
For testing, let's delete the master and check if sentinel will switch the master role to one of the slaves:
$ kubectl delete pod redis-sentinel-node-0
pod "redis-sentinel-node-0" deleted
$ kubectl logs -f redis-sentinel-node-1 sentinel
...
1:X 09 Jun 2021 09:55:20.902 # Executing user requested FAILOVER of 'mymaster'
...
1:X 09 Jun 2021 09:55:22.666 # +switch-master mymaster 10.4.2.21 6379 10.4.1.10 6379
...
1:X 09 Jun 2021 09:55:50.626 * +slave slave 10.4.0.10:6379 10.4.0.10 6379 # mymaster 10.4.1.10 6379
1:X 09 Jun 2021 09:55:50.632 * +slave slave 10.4.2.22:6379 10.4.2.22 6379 # mymaster 10.4.1.10 6379
A new master (redis-sentinel-node-2 10.4.1.10) has been selected, so everything works as expected.
Additionally, we can display more information by connecting to one of the Redis nodes:
$ kubectl run --namespace default redis-client --restart='Never' --env REDIS_PASSWORD=redispassword --image docker.io/bitnami/redis:6.2.1-debian-10-r47 --command -- sleep infinity
pod/redis-client created
$ kubectl exec --tty -i redis-client --namespace default -- bash
I have no name!#redis-client:/$ redis-cli -h redis-sentinel-node-1.redis-sentinel-headless -p 6379 -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-sentinel-node-1.redis-sentinel-headless:6379> info replication
# Replication
role:slave
master_host:10.4.1.10
master_port:6379
master_link_status:up
...

Why does starting daskdev/dask into a Pod fail?

Why does kubectl run dask --image daskdev/dask fail?
# starting the container with docker to make sure it basically works
➜ ~ docker run --rm -it --entrypoint bash daskdev/dask:latest
(base) root#5b34ce038eb3:/# python
Python 3.8.0 (default, Nov 6 2019, 21:49:08)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dask
>>>
>>> exit()
(base) root#5b34ce038eb3:/# exit
exit
# now trying to fire up the container on a minikube cluster
➜ ~ kubectl run dask --image daskdev/dask
pod/dask created
# let's see what's going on with the Pod
➜ ~ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
dask 0/1 CrashLoopBackOff 1 13s
dask 0/1 Completed 2 24s
dask 0/1 CrashLoopBackOff 2 38s
# not sure why the logs look like something is missing
➜ ~ kubectl logs dask --tail=100
+ '[' '' ']'
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' '' ']'
+ exec
no environment.yml
So basically, if you will check result of kubectl describe pod dask, you will see that last state was Terminated with Exit Code 0, that literally means you container was launched successfully, did it job and finished also successfully. What else you expect to happen with pod?
IN addition, when you create pod using kubectl run dask --image daskdev/dask- it creates with the restartPolicy: Always by default!!!!
Always means that the container will be restarted even if it exited with a zero exit code (i.e. successfully).
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 02 Apr 2021 15:06:00 +0000
Finished: Fri, 02 Apr 2021 15:06:00 +0000
Ready: False
Restart Count: 3
Environment: <none>
There is no /opt/app/environment.yml in your container. If im not mistake, you should first configure it with prepare.sh. PLease check more here - DASK
section
#docker run --rm -it --entrypoint bash daskdev/dask:latest
(base) root#431d69bb9a80:/# ls -la /opt/app/
total 12
drwxr-xr-x 2 root root 4096 Mar 27 15:43 .
drwxr-xr-x 1 root root 4096 Mar 27 15:43 ..
not sure why the logs look like something is missing ➜ ~ kubectl logs dask --tail=100
...
exec no environment.yml
There is already prepared helm DASK chart. Use it. It works fine:
helm repo add dask https://helm.dask.org/
helm repo update
helm install raffael-dask-release dask/dask
NAME: raffael-dask-release
LAST DEPLOYED: Fri Apr 2 15:43:38 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing DASK, released at name: raffael-dask-release.
This release includes a Dask scheduler, 3 Dask workers, and 1 Jupyter servers.
The Jupyter notebook server and Dask scheduler expose external services to
which you can connect to manage notebooks, or connect directly to the Dask
cluster. You can get these addresses by running the following:
export DASK_SCHEDULER="127.0.0.1"
export DASK_SCHEDULER_UI_IP="127.0.0.1"
export DASK_SCHEDULER_PORT=8080
export DASK_SCHEDULER_UI_PORT=8081
kubectl port-forward --namespace default svc/raffael-dask-release-scheduler $DASK_SCHEDULER_PORT:8786 &
kubectl port-forward --namespace default svc/raffael-dask-release-scheduler $DASK_SCHEDULER_UI_PORT:80 &
export JUPYTER_NOTEBOOK_IP="127.0.0.1"
export JUPYTER_NOTEBOOK_PORT=8082
kubectl port-forward --namespace default svc/raffael-dask-release-jupyter $JUPYTER_NOTEBOOK_PORT:80 &
echo tcp://$DASK_SCHEDULER:$DASK_SCHEDULER_PORT -- Dask Client connection
echo http://$DASK_SCHEDULER_UI_IP:$DASK_SCHEDULER_UI_PORT -- Dask dashboard
echo http://$JUPYTER_NOTEBOOK_IP:$JUPYTER_NOTEBOOK_PORT -- Jupyter notebook
NOTE: It may take a few minutes for the LoadBalancer IP to be available. Until then, the commands above will not work for the LoadBalancer service type.
You can watch the status by running 'kubectl get svc --namespace default -w raffael-dask-release-scheduler'
NOTE: It may take a few minutes for the URLs above to be available if any EXTRA_PIP_PACKAGES or EXTRA_CONDA_PACKAGES were specified,
because they are installed before their respective services start.
NOTE: The default password to login to the notebook server is `dask`. To change this password, refer to the Jupyter password section in values.yaml, or in the README.md.
If you still want create manually pod, use below... Main idea is set restartPolicy: Never.
apiVersion: v1
kind: Pod
metadata:
name: dask-tesssssst
labels:
foo: bar
spec:
restartPolicy: Never
containers:
- image: daskdev/dask:latest
imagePullPolicy: Always
name: dask-tesssssst
Please check DASK KubeCluster official documentation for more examples. Last one I took exactly from there.

Kubernetes worker node is NotReady due to CNI plugin not initialized

I'm using kind to run a test kubernetes cluster on my local Macbook.
I found one of the nodes with status NotReady:
$ kind get clusters
mc
$ kubernetes get nodes
NAME STATUS ROLES AGE VERSION
mc-control-plane Ready master 4h42m v1.18.2
mc-control-plane2 Ready master 4h41m v1.18.2
mc-control-plane3 Ready master 4h40m v1.18.2
mc-worker NotReady <none> 4h40m v1.18.2
mc-worker2 Ready <none> 4h40m v1.18.2
mc-worker3 Ready <none> 4h40m v1.18.2
The only interesting thing in kubectl describe node mc-worker is that the CNI plugin not initialized:
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 11 Aug 2020 16:55:44 -0700 Tue, 11 Aug 2020 12:10:16 -0700 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 11 Aug 2020 16:55:44 -0700 Tue, 11 Aug 2020 12:10:16 -0700 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 11 Aug 2020 16:55:44 -0700 Tue, 11 Aug 2020 12:10:16 -0700 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Tue, 11 Aug 2020 16:55:44 -0700 Tue, 11 Aug 2020 12:10:16 -0700 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady
message:Network plugin returns error: cni plugin not initialized
I have 2 similar clusters and this only occurs on this cluster.
Since kind uses the local Docker daemon to run these nodes as containers, I have already tried to restart the container (should be the equivalent of rebooting the node).
I have considered deleting and recreating the cluster, but there ought to be a way to solve this without recreating the cluster.
Here are the versions that I'm running:
$ kind version
kind v0.8.1 go1.14.4 darwin/amd64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
How do you resolve this issue?
Most likely cause:
The docker VM is running out of some resource and cannot start CNI on that particular node.
You can poke around in the HyperKit VM by connecting to it:
From a shell:
screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
If that doesn't work for some reason:
docker run -it --rm --privileged --pid=host alpine nsenter -t 1 -m -u -n -i sh
Once in the VM:
# ps -Af
# free
# df -h
...
Then you can always update the setting on the docker UI:
Finally, your node after all is running in a container. So you can connect to that container and see what kubelet errors you see:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6d881be79f4a kindest/node:v1.18.2 "/usr/local/bin/entr…" 32 seconds ago Up 29 seconds 127.0.0.1:57316->6443/tcp kind-control-plane
docker exec -it 6d881be79f4a bash
root#kind-control-plane:/# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/kind/systemd/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2020-08-12 02:32:16 UTC; 35s ago
Docs: http://kubernetes.io/docs/
Main PID: 768 (kubelet)
Tasks: 23 (limit: 2348)
Memory: 32.8M
CGroup: /docker/6d881be79f4a8ded3162ec6b5caa8805542ff9703fabf5d3d2eee204a0814e01/system.slice/kubelet.service
└─768 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet
/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --fail-swap-on=false --node-ip= --fail-swap-on=false
...
✌️
I encountered this scenario. Master is Ready but the worker node's status are not. After some investigation, i found out that the /opt/cni/bin is empty - there is no network plugin for my worker node hosts. Thus, i installed this "kubernetes-cni.x86_64" and restarted kubelet service. This solved the "NotReady" status of my worker nodes.
Stop and disable apparmor & restart the containerd service on that node will solve your issue
root#node:~# systemctl stop apparmor
root#node:~# systemctl disable apparmor
root#node:~# systemctl restart containerd.service

How to debug when Kubernetes nodes are in 'Not Ready' state

I initialized the master node and add 2 worker nodes, but only master and one of the worker node show up when I run the following command:
kubectl get nodes
also, both these nodes are in 'Not Ready' state.
What are the steps should I take to understand what the problem could be?
I can ping all the nodes from each of the other nodes.
The version of Kubernetes is 1.8.
OS is Cent OS 7
I used the following repo to install Kubernetes:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes] name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
yum install kubelet kubeadm kubectl kubernetes-cni
First, describe nodes and see if it reports anything:
$ kubectl describe nodes
Look for conditions, capacity and allocatable:
Conditions:
Type Status
---- ------
OutOfDisk False
MemoryPressure False
DiskPressure False
Ready True
Capacity:
cpu: 2
memory: 2052588Ki
pods: 110
Allocatable:
cpu: 2
memory: 1950188Ki
pods: 110
If everything is alright here, SSH into the node and observe kubelet logs to see if it reports anything. Like certificate erros, authentication errors etc.
If kubelet is running as a systemd service, you can use
$ journalctl -u kubelet
Steps to debug:-
In case you face any issue in kubernetes, first step is to check if kubernetes self applications are running fine or not.
Command to check:- kubectl get pods -n kube-system
If you see any pod is crashing, check it's logs
if getting NotReady state error, verify network pod logs.
if not able to resolve with above, follow below steps:-
kubectl get nodes # Check which node is not in ready state
kubectl describe node nodename #nodename which is not in readystate
ssh to that node
execute systemctl status kubelet # Make sure kubelet is running
systemctl status docker # Make sure docker service is running
journalctl -u kubelet # To Check logs in depth
Most probably you will get to know about error here, After fixing it reset kubelet with below commands:-
systemctl daemon-reload
systemctl restart kubelet
In case you still didn't get the root cause, check below things:-
Make sure your node has enough space and memory. Check for /var directory space especially.
command to check: -df -kh, free -m
Verify cpu utilization with top command. and make sure any process is not taking an unexpected memory.
I was having similar issue because of a different reason:
Error:
cord#node1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 17h v1.13.5
node2 Ready <none> 17h v1.13.5
node3 NotReady <none> 9m48s v1.13.5
cord#node1:~$ kubectl describe node node3
Name: node3
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
Ready False Thu, 18 Apr 2019 01:15:46 -0400 Thu, 18 Apr 2019 01:03:48 -0400 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 192.168.2.6
Hostname: node3
cord#node3:~$ journalctl -u kubelet
Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649047 54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649086 54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:50 node3 kubelet[54132]: E0418 01:24:50.649402 54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650816 54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650845 54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:55 node3 kubelet[54132]: E0418 01:24:55.651056 54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:57 node3 kubelet[54132]: I0418 01:24:57.248519 54132 setters.go:72] Using node IP: "192.168.2.6"
Issue:
My file: 10-calico.conflist was incorrect. Verified it from a different node and from sample file in the same directory "calico.conflist.template".
Resolution:
Changing the file, "10-calico.conflist" and restarting the service using "systemctl restart kubelet", resolved my issue:
NAME STATUS ROLES AGE VERSION
node1 Ready master 18h v1.13.5
node2 Ready <none> 18h v1.13.5
node3 Ready <none> 48m v1.13.5
I recently started using VMWare Octant https://github.com/vmware-tanzu/octant. This is a better UI than the Kubernetes Dashboard. You can view the Kubernetes cluster and look at the details of the cluster and the PODS. This will allow you to check the logs and open a terminal into the POD(s).
I found applying the network and rebooting both the nodes did the trick for me.
kubectl apply -f [podnetwork].yaml
I recently had this issue and checking out the known-issues from kind website here https://kind.sigs.k8s.io/docs/user/known-issues/ it would tell you specifically the main problem mostly comes from the lack of memory allocated to docker. They actually advice to allocate 8GB to docker, I allocated 6GB up from 3GB and it worked fine for me this is kind version I am running atm
$ kind version
kind v0.10.0 go1.15.7 darwin/amd64
and this is docker version
$ docker version
Client:
Cloud integration: 1.0.17
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:55:20 2021
OS/Arch: darwin/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:10 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
I hope this helps you or anyone facing the same issue.
and here is the output from kind
$ k get node
NAME STATUS ROLES AGE VERSION
test2-control-plane Ready control-plane,master 4m42s v1.20.2

kube-discovery fails to start when using kubeadm

I'm trying to install a cluster using kubeadm, using this guide.
I'm installing it on bare metal Ubuntu 16.04 server.
Docker is already preinstalled:
root#host# docker -v
Docker version 1.12.3, build 6b644ec
After executing the following:
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl kubernetes-cni
I run 'kubeadm init', and it hangs on the kube-discovery addon:
root#host# kubeadm init
Running pre-flight checks
<master/tokens> generated token: "<token>"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2016-11-22 15:27:25 +0000 UTC Not After: 2026-11-20 15:27:25 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2016-11-22 15:27:25 +0000 UTC Not After: 2017-11-22 15:27:25 +0000 UTC
Alternate Names: [<ipaddress> kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready
<master/apiclient> all control plane components are healthy after 44.584082 seconds
<master/apiclient> waiting for at least one node to register and become ready
<master/apiclient> first node is ready after 1.003104 seconds
<master/apiclient> attempting a test deployment
<master/apiclient> test deployment succeeded
<master/discovery> created essential addon: kube-discovery, waiting for it to become ready
I can see that this pod is restarting:
root#host# kubectl get pods --all-namespaces=true
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system dummy-2088944543-dsjtb 1/1 Running 0 29m
kube-system etcd-host.test.com 1/1 Running 0 29m
kube-system kube-apiserver-host.test.com 1/1 Running 0 30m
kube-system kube-controller-manager-host.test.com 1/1 Running 0 29m
kube-system kube-discovery-1150918428-ulap3 0/1 CrashLoopBackOff 10 29m
kube-system kube-scheduler-host.test.com 1/1 Running 0 29m
root#host# kubectl logs kube-discovery-1150918428-ulap3 --namespace=kube-system
2016/11/22 13:31:32 root CA certificate does not exist: /tmp/secret/ca.pem
Do I need to provide it a certificate?
What specific version of kubernetes are you trying to install? You can check it with:
apt-get policy kubelet