I'm using Arch linux
I had virtualbox 5.2.12 installed
I had the minikube 0.27.0-1 installed
I had the Kubernetes v1.10.0 installed
When i try start the minkube with sudo minikube start i get this error
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0527 12:58:18.929483 22672 start.go:281] Error restarting cluster: running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: running command:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: exit status 1
I already try start minekube with others option like:
sudo minikube start --kubernetes-version v1.10.0 --bootstrapper kubeadm
sudo minikube start --bootstrapper kubeadm
sudo minikube start --vm-driver none
sudo minikube start --vm-driver virtualbox
sudo minikube start --vm-driver kvm
sudo minikube start --vm-driver kvm2
Always I get the same error. Can someone help me?
Minikube VM is usually started for simple experiments without any important payload.
That's why it's much easier to recreate minikube cluster than trying to fix it.
To delete existing minikube VM execute the following command:
minikube delete
This command shuts down and deletes the minikube virtual machine. No data or state is preserved.
Check if you have all dependencies at place and run command:
minikube start
This command creates a “kubectl context” called “minikube”. This context contains the configuration to communicate with your minikube cluster. minikube sets this context to default automatically, but if you need to switch back to it in the future, run:
kubectl config use-context minikube
Or pass the context on each command like this:
kubectl get pods --context=minikube
More information about command line arguments can be found here.
Update:
The below answer did not work due to what I suspect are differences in versions between my environment and the information I found and I'm not willing to sink more time into this problem. The VM itself does startup so if you have important information in it, ie: other docker containers you can login to the VM and extract such data from it before minikube delete
Same problem, I ssh'ed into the VM and ran
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml
The result was
failure loading apiserver-kubelet-client certificate: the certificate has expired
So like any good engineer, I googled that and found this:
Source: https://github.com/kubernetes/kubeadm/issues/581
If you are using a version of kubeadm prior to 1.8, where I understand certificate rotation #206 was put into place (as a beta
feature) or your certs already expired, then you will need to manually
update your certs (or recreate your cluster which it appears some (not
just #kachkaev) end up resorting to).
You will need to SSH into your master node. If you are using kubeadm
= 1.8 skip to 2.
Update Kubeadm, if needed. I was on 1.7 previously. $ sudo curl -sSL
https://dl.k8s.io/release/v1.8.15/bin/linux/amd64/kubeadm >
./kubeadm.1.8.15 $ chmod a+rx kubeadm.1.8.15 $ sudo mv
/usr/bin/kubeadm /usr/bin/kubeadm.1.7 $ sudo mv kubeadm.1.8.15
/usr/bin/kubeadm Backup old apiserver, apiserver-kubelet-client, and
front-proxy-client certs and keys. $ sudo mv
/etc/kubernetes/pki/apiserver.key
/etc/kubernetes/pki/apiserver.key.old $ sudo mv
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.crt.old $ sudo mv
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.crt.old $ sudo mv
/etc/kubernetes/pki/apiserver-kubelet-client.key
/etc/kubernetes/pki/apiserver-kubelet-client.key.old $ sudo mv
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/front-proxy-client.crt.old $ sudo mv
/etc/kubernetes/pki/front-proxy-client.key
/etc/kubernetes/pki/front-proxy-client.key.old Generate new apiserver,
apiserver-kubelet-client, and front-proxy-client certs and keys. $
sudo kubeadm alpha phase certs apiserver --apiserver-advertise-address
$ sudo kubeadm alpha phase certs
apiserver-kubelet-client $ sudo kubeadm alpha phase certs
front-proxy-client Backup old configuration files $ sudo mv
/etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old $ sudo mv
/etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old $ sudo
mv /etc/kubernetes/controller-manager.conf
/etc/kubernetes/controller-manager.conf.old $ sudo mv
/etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old
Generate new configuration files. There is an important note here. If
you are on AWS, you will need to explicitly pass the --node-name
parameter in this request. Otherwise you will get an error like:
Unable to register node "ip-10-0-8-141.ec2.internal" with API server:
nodes "ip-10-0-8-141.ec2.internal" is forbidden: node ip-10-0-8-141
cannot modify node ip-10-0-8-141.ec2.internal in your logs sudo
journalctl -u kubelet --all | tail and the Master Node will report
that it is Not Ready when you run kubectl get nodes.
Please be certain to replace the values passed in
--apiserver-advertise-address and --node-name with the correct values for your environment.
$ sudo kubeadm alpha phase kubeconfig all
--apiserver-advertise-address 10.0.8.141 --node-name ip-10-0-8-141.ec2.internal [kubeconfig] Wrote KubeConfig file to disk:
"admin.conf" [kubeconfig] Wrote KubeConfig file to disk:
"kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk:
"controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk:
"scheduler.conf"
Ensure that your kubectl is looking in the right place for your config
files. $ mv .kube/config .kube/config.old $ sudo cp -i
/etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id
-u):$(id -g) $HOME/.kube/config $ sudo chmod 777 $HOME/.kube/config $ export KUBECONFIG=.kube/config Reboot your master node $ sudo
/sbin/shutdown -r now Reconnect to your master node and grab your
token, and verify that your Master Node is "Ready". Copy the token to
your clipboard. You will need it in the next step. $ kubectl get nodes
$ kubeadm token list If you do not have a valid token. You can create
one with:
$ kubeadm token create The token should look something like
6dihyb.d09sbgae8ph2atjw
SSH into each of the slave nodes and reconnect them to the master $
sudo curl -sSL
https://dl.k8s.io/release/v1.8.15/bin/linux/amd64/kubeadm >
./kubeadm.1.8.15 $ chmod a+rx kubeadm.1.8.15 $ sudo mv
/usr/bin/kubeadm /usr/bin/kubeadm.1.7 $ sudo mv kubeadm.1.8.15
/usr/bin/kubeadm $ sudo kubeadm join --token= : --node-name
Repeat Step 9 for each connecting node. From the master node, you can
verify that all slave nodes have connected and are ready with: $
kubectl get nodes Hopefully this gets you where you need to be
#davidcomeyne.
As root try first a full clean up with:
minikube delete --all --purge
Then start up minikube and you need to copy the root certs for another users.
Related
I want to setup an etcd cluster runnin on multiple nodes. I have running 2 unbuntu 18.04 machines running on a Hyper-V terminal.
I followed this guide on the official kubernetes site:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/
Therefore, I changed the several scripts and executed this scripts on HOST0 and HOST1
export HOST0=192.168.101.90
export HOST1=192.168.101.91
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/
ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1")
for i in "${!ETCDHOSTS[#]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
local:
serverCertSANs:
- "${HOST}"
peerCertSANs:
- "${HOST}"
extraArgs:
initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380
initial-cluster-state: new
name: ${NAME}
listen-peer-urls: https://${HOST}:2380
listen-client-urls: https://${HOST}:2379
advertise-client-urls: https://${HOST}:2379
initial-advertise-peer-urls: https://${HOST}:2380
EOF
done
After that, I executed this command on HOST0
kubeadm init phase certs etcd-ca
I created all the nessecary on HOST0
# cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.k
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0
# clean up certs that should not be copied off this host
find /tmp/${HOST1} -name ca.key -type f -delete
After that, I copied the files to the second ETCTD node (HOST1). Before that I created a root user mbesystem
USER=mbesystem
HOST=${HOST1}
scp -r /tmp/${HOST}/* ${USER}#${HOST}:
ssh ${USER}#${HOST}
USER#HOST $ sudo -Es
root#HOST $ chown -R root:root pki
root#HOST $ mv pki /etc/kubernetes/
I'll check all the files were there on HOST0 and HOST1.
On HOST0 I started the etcd cluster using:
kubeadm init phase etcd local --config=/tmp/192.168.101.90/kubeadmcfg.yaml
On Host1 I started using:
kubeadm init phase etcd local --config=/home/mbesystem/kubeadmcfg.yaml
After I executed:
docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://192.168.101.90:2379 endpoint health --cluster
I discovered my cluster is not healty, I'll received a connection refused.
I can't figure out what went wrong. Any help will be appreciated.
I've looked into it, reproduced what was in the link that you provided: Kubernetes.io: Setup ha etcd with kubeadm and managed to make it work.
Here is some explanation/troubleshooting steps/tips etc.
First of all etcd should be configured with odd number of nodes. What I mean by that is it should be created as 3 or 5 nodes cluster.
Why an odd number of cluster members?
An etcd cluster needs a majority of nodes, a quorum, to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1. For any odd-sized cluster, adding one node will always increase the number of nodes necessary for quorum. Although adding a node to an odd-sized cluster appears better since there are more machines, the fault tolerance is worse since exactly the same number of nodes may fail without losing quorum but there are more nodes that can fail. If the cluster is in a state where it can't tolerate any more failures, adding a node before removing nodes is dangerous because if the new node fails to register with the cluster (e.g., the address is misconfigured), quorum will be permanently lost.
-- Github.com: etcd documentation
Additionally here are some troubleshooting steps:
Check if docker is running You can check it by running command (on systemd installed os):
$ systemctl show --property ActiveState docker
Check if etcd container is running properly with:
$ sudo docker ps
Check logs of etcd container if it's running with:
$ sudo docker logs ID_OF_CONTAINER
How I've managed to make it work:
Assuming 2 Ubuntu 18.04 servers with IP addresses of:
10.156.0.15 and name: etcd-1
10.156.0.16 and name: etcd-2
Additionally:
SSH keys configured for root access
DNS resolution working for both of the machines ($ ping etcd-1)
Steps:
Pre-configuration before the official guide.
I did all of the below configuration with the usage of root account
Configure the kubelet to be a service manager for etcd.
Create configuration files for kubeadm.
Generate the certificate authority.
Create certificates for each member
Copy certificates and kubeadm configs.
Create the static pod manifests.
Check the cluster health.
Pre-configuration before the official guide
Pre-configuration of this machines was done with this StackOverflow post with Ansible playbooks:
Stackoverflow.com: 3 kubernetes clusters 1 base on local machine
You can also follow official documentation: Kubernetes.io: Install kubeadm
Configure the kubelet to be a service manager for etcd.
Run below commands on etcd-1 and etcd-2 with root account.
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
# Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
EOF
$ systemctl daemon-reload
$ systemctl restart kubelet
Create configuration files for kubeadm.
Create your configuration file on your etcd-1 node.
Here is modified script that will create kubeadmcfg.yaml for only 2 nodes:
export HOST0=10.156.0.15
export HOST1=10.156.0.16
# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/
ETCDHOSTS=(${HOST0} ${HOST1})
NAMES=("etcd-1" "etcd-2")
for i in "${!ETCDHOSTS[#]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1beta2"
kind: ClusterConfiguration
etcd:
local:
serverCertSANs:
- "${HOST}"
peerCertSANs:
- "${HOST}"
extraArgs:
initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380
initial-cluster-state: new
name: ${NAME}
listen-peer-urls: https://${HOST}:2380
listen-client-urls: https://${HOST}:2379
advertise-client-urls: https://${HOST}:2379
initial-advertise-peer-urls: https://${HOST}:2380
EOF
done
Take a special look on:
export HOSTX on the top of the script. Paste the IP addresses of your machines there.
NAMES=("etcd-1" "etcd-2"). Paste the names of your machines (hostname) there.
Run this script from root account and check if it created files in /tmp/IP_ADDRESS directory.
Generate the certificate authority
Run below command from root account on your etcd-1 node:
$ kubeadm init phase certs etcd-ca
Create certificates for each member
Below is a part of the script which is responsible for creating certificates for each member of etcd cluster. Please modify the HOST0 and HOST1 variables.
#!/bin/bash
HOST0=10.156.0.15
HOST1=10.156.0.16
kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0
Run above script from root account and check if there is pki directory inside /tmp/10.156.0.16/.
There shouldn't be any pki directory inside /tmp/10.156.0.15/ as it's already in place.
Copy certificates and kubeadm configs.
Copy your kubeadmcfg.yaml of etcd-1 from /tmp/10.156.0.15 to root directory with:
$ mv /tmp/10.156.0.15/kubeadmcfg.yaml /root/
Copy the content of /tmp/10.156.0.16 from your etcd-1 to your etcd-2 node to /root/ directory:
$ scp -r /tmp/10.156.0.16/* root#10.156.0.16:
After that check if files copied correctly, have correct permissions and copy pki folder to /etc/kubernetes/ with command on etcd-2:
$ mv /root/pki /etc/kubernetes/
Create the static pod manifests.
Run below command on etcd-1 and etcd-2:
$ kubeadm init phase etcd local --config=/root/kubeadmcfg.yaml
All should be running now.
Check the cluster health.
Run below command to check cluster health on etcd-1.
docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert /etc/kubernetes/pki/etcd/ca.crt --endpoints https://10.156.0.15:2379 endpoint health --cluster
Modify:
--endpoints https://10.156.0.15:2379 with correct IP address of etcd-1
It should give you a message like this:
https://10.156.0.15:2379 is healthy: successfully committed proposal: took = 26.308693ms
https://10.156.0.16:2379 is healthy: successfully committed proposal: took = 26.614373ms
Above message concludes that the etcd is working correctly but please be aware of an even number of nodes.
Please let me know if you have any questions to that.
I am new to kubernetes . I got the below error while interacting with the cluster kubectl get nodes .
ERROR:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
After search in the internet i fixed my issues .
#sudo cp /etc/kubernetes/admin.conf $HOME/
#sudo chown $(id -u):$(id -g) $HOME/admin.conf
#export KUBECONFIG=$HOME/admin.conf
Your kubectl is probably not referring to right kubeconfig file or the kubeconfig file does not right details.
When there is clear instructions by kubeadm init to execute following commands as an regular user, if you miss runing them you end up with issue reported.
To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should check back the logs at kubeadm init time and you will find similar as below asking to execute the command.
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
Maybe you not set environment variables, try this:
export KUBERNETES_MASTER=http://MasterIP:8080
MasterIP is your Kubernetes master IP
Issue is with the use-context in kubectl command.Please check the same in kubeconfig file.
As I am new to Kubernetes, I still haven't figured out how to deploy an application on my local network using one or various physical machines.
Usually the tutorials on the internet describe situations using minikube, but it is only for local machine tests, isn't it? Or situations where the deploy is performed on cloud platforms, like google.
I would really appreciate some support in where to begin? In my case will I need to install only Kubernetes on the machines? Is it a trivial task?
One way is to install kubernetes is via kubeadm.
I've done this with my own VM. But should also work for your physical machines.
You can use kubeadm, here is a doc which will help you to install it.
Each server will need to have kubeadm, kubelet and kubectl installed.
On master node you will do kubeadm init and once the whole process is finished it will provide you with an output which you can run to add worker nodes.
Of course those servers needs to see each other on your network.
There are also other options to install Kubernetes. For example using kops or kubespray.
From https://kubernetes.io/docs/setup/learning-environment/minikube/
Minikube is a tool that makes it easy to run Kubernetes locally.
Minikube runs a single-node Kubernetes cluster inside a Virtual
Machine (VM) on your laptop for users looking to try out Kubernetes or
develop with it day-to-day
If you want to set up your own cluster including, there's quite a few options including kubeadm or kubespray.
Personally I've used Vagrant and Ansible to set up a local Kubernetes cluster using kubeadm - there's a good tutorial here... and here's my implementation :)
Kubeadm allows you to create cluster using any machine that runs Docker.
This guide will help you to create your first cluster with 1 master and 1 worker nodes. (This is based on Debian)
Execute the following command on all 2 machines:
Installing Docker from official repository
$ sudo apt-get update
$ sudo apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
$ curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo groupadd docker
$ sudo usermod -aG docker $USER
Install kubeadm, kubectl and kubelet from official repository
$ sudo apt-get update && sudo apt-get install -y apt-transport-https curl
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo su -c "cat > /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF"
$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl
Execute the following command only on your master node
Initialize your cluster
$ sudo kubeadm init
When the init finishes, kubeadm will give you the command to add your workers to your cluster. Save this command for later use.
EXAMPLE:
kubeadm join 10.128.0.17:6443 --token 27evky.hoel95h16poqici6 \
--discovery-token-ca-cert-hash sha256:521f69cb935951bbfee142432108caeaeaf1682d8647208ba2f558624890ab63
After the kubeadm init command completes, run the following commands to start using your new cluster
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
Your master is not ready yet, we need to create a cni network (You can choose different CNIs)
$ kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
Check if your master node is ready. It needs a while to start all dependencies
$ kubectl get nodes
EXAMPLE:
user#kubemaster:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubemaster NotReady master 32s v1.16.2
To check more details about your node:
$ kubectl describe node <NODENAME>
When your master node is ready you can proceed and run the kubectl join command (saved previously) on your worker nodes:
$ sudo kubeadm join 10.128.0.17:6443 --token 27evky.hoel95h16poqici6 --discovery-token-ca-cert-hash sha256:521f69cb935951bbfee142432108caeaeaf1682d8647208ba2f558624890ab63
Check if your worker node is ready (command must be executed on master node)
$ kubectl get nodes
If you want to have more workers, just repeat the kubectl join command on the new workers.
My kubernetes PKI expired (API server to be exact) and I can't find a way to renew it. The error I get is
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922595 8751 server.go:417] Version: v1.14.2
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922784 8751 plugins.go:103] No cloud provider specified.
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922800 8751 server.go:754] Client rotation is on, will bootstrap in background
May 27 08:43:51 node1 kubelet[8751]: E0527 08:43:51.925859 8751 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2019-05-24 13:24:42 +0000 UTC
May 27 08:43:51 node1 kubelet[8751]: F0527 08:43:51.925894 8751 server.go:265] failed to run Kubelet: unable to load bootstrap
kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
The documentation on https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ describes how to renew but it only works if the API server is not expired. I have tried to do a
kubeadm alpha cert renew all
and do a reboot but that just made the entire cluster fail so I did a rollback to a snapshot (my cluster is running on VMware).
The cluster is running and all containers seem to work but I can't access it via kubectl so I can't really deploy or query.
My kubernetes version is 1.14.2.
So the solution was to (first a backup)
$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot
then
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
that did the job for me and thanks for your hints :)
This topic is also discussed in:
https://github.com/kubernetes/kubeadm/issues/581
after 1.15 kubeadm upgrade automatically will renewal the certificates for you!
also 1.15 added a command to check cert expiration in kubeadm
Kubernetes: expired certificate
Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
Check certificate expiration:
kubeadm alpha certs check-expiration
Automatic certificate renewal:
kubeadm renews all the certificates during control plane upgrade.
Manual certificate renewal:
You can renew your certificates manually at any time with the kubeadm alpha certs renew command.
This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.
Overall for Kubernetes v1.14 I find this procedure the most helpful:
https://stackoverflow.com/a/56334732/1147487
I was using Kubernetes v15.1 and updated my certificates as explained above, but I still got the same error.
The /etc/kubernetes/kubelet.conf was still referring to the expired/old "client-certificate-data".
After some research I found out that kubeadm is not updating the /etc/kubernetes/kubelet.conf file if the certificate-renew was not set to true. So please be aware of a bug of kubeadm below version 1.17 (https://github.com/kubernetes/kubeadm/issues/1753).
kubeadm only upgrades if the cluster upgrade was done with certificate-renewal=true. So I manually had to delete the /etc/kubernetes/kubelet.conf and regenerated it with kubeadm init phase kubeconfig kubelet which finally fixed my problem.
Try to do cert renewal via kubeadm init phase certs command.
You can check certs expiration via the following command:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text
openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text
First, ensure that you have most recent backup of k8s certificates inventory /etc/kubernetes/pki/*.
Delete apiserver.* and apiserver-kubelet-client.* cert files in /etc/kubernetes/pki/ directory.
Spawn a new certificates via kubeadm init phase certs command:
sudo kubeadm init phase certs apiserver
sudo kubeadm init phase certs apiserver-kubelet-client
Restart kubelet and docker daemons:
sudo systemctl restart docker; sudo systemctl restart kubelet
You can find more related information in the official K8s documentation.
This will update all certs under /etc/kubernetes/ssl
kubeadm alpha certs renew all --config=/etc/kubernetes/kubeadm-config.yaml
and do this to restart server commpenont:
kill -s SIGHUP $(pidof kube-apiserver)
kill -s SIGHUP $(pidof kube-controller-manager)
kill -s SIGHUP $(pidof kube-scheduler)
[root#nrchbs-slp4115 ~]# kubectl get apiservices |egrep metrics
v1beta1.metrics.k8s.io kube-system/metrics-server True 125m
[root#nrchbs-slp4115 ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 20d
metrics-server ClusterIP 10.99.2.11 <none> 443/TCP 125m
[root#nrchbs-slp4115 ~]# kubectl get ep -n kube-system
NAME ENDPOINTS AGE
kube-controller-manager <none> 20d
kube-dns 10.244.0.5:53,10.244.0.6:53,10.244.0.5:53 + 3 more... 20d
kube-scheduler <none> 20d
metrics-server 10.244.2.97:443 125m
[root#nrchbs-slp4115 ~]#
To help anyone else with Multi-Master setup as I was searching for the answer after the first master has been updated on the second master I did this I found this from another question:
kubeadm only upgrades if the cluster upgrade was done with certificate-renewal=true. So I manually had to delete the /etc/kubernetes/kubelet.conf and regenerated it with kubeadm init phase kubeconfig kubelet which finally fixed my problem.
I use a config.yaml to configure the Masters so for me, the answer was:
sudo -i
mkdir -p ~/k8s_backup/etcd
cd /etc/kubernetes/pki/
mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/k8s_backup
cd /etc/kubernetes/pki/etcd
mv {healthcheck-client.crt,healthcheck-client.key,peer.crt,peer.key,server.crt,server.key} ~/k8s_backup/etcd/
kubeadm init phase certs all --ignore-preflight-errors=all --config /etc/kubernetes/config.yaml
cd /etc/kubernetes
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/k8s_backup
kubeadm init phase kubeconfig all --config /etc/kubernetes/config.yaml --ignore-preflight-errors=all
For good measure I reboot
shutdown now -r
The best solution is as Kim Nielsen wrote.
$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot
With the next command, you can check when the new certifications will be expired:
$ kubeadm alpha certs check-expiration --config=/etc/kubernetes/kubeadm-config.yaml
or
$ kubeadm certs check-expiration --config=/etc/kubernetes/kubeadm-config.yaml
However, if you have more than one master, you need to copy these files on them.
Log in to the second master and do (backup):
$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
Then log in to the first master (where you created new certificates) and do the next commands (node2 should be changed with IP address of second master machine):
$ rsync /etc/kubernetes/pki/*.crt -e ssh root#node2:/etc/kubernetes/pki/
$ rsync /etc/kubernetes/pki/*.key -e ssh root#node2:/etc/kubernetes/pki/
$ rsync /etc/kubernetes/*.conf -e ssh root#node2:/etc/kubernetes/
In my case, I've got same issue after certificate renewal. My cluster was built using Kubespray. My kubelet stopped working and it was saying that I do not have /etc/kubernetes/bootstrap-kubelet.conf file so I looked what this config does.
--bootstrap-kubeconfig string
| Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
I understood that this file might not be needed.
A note that I renewed k8s 1.19 with:
kubeadm alpha certs renew apiserver-kubelet-client
kubeadm alpha certs renew apiserver
kubeadm alpha certs renew front-proxy-client
... and that was not sufficient.
The solution was
cp -r /etc/kubernetes /etc/kubernetes.backup
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf
kubeadm alpha kubeconfig user --client-name system:node:YOUR_MASTER_HOSTNAME_IS_HERE --org system:nodes > /etc/kubernetes/kubelet.conf
kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
I am able to solve this issue with below steps:
openssl req -new -key <existing.key> -sub "/CN=system:node:<HOST_NAME>/O=system:nodes" -out new.csr
<existing.key> - use it from kubelet.conf
<HOST_NAME> - hostname in lower case ( refer expired certificate with command openssl ex: openssl x509 -in old.pem -text -noout)
openssl x509 -req -in new.csr -CA <ca.crt> -CAkey <ca.key> -CAcreateserial -out new.crt -days 365
<ca.crt> - from master node
<ca.key> - from
new.crt - is the renewed certificate replace with expired certificate.
Once replaced , restart kubelet and docker or any other container service.
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) >/etc/kubernetes/kubelet.conf
Renew the kubelet.conf helps me solve this issue.
I installed kubernetes cluster using kubeadm following this guide. After some period of time, I decided to reinstall K8s but run into troubles with removing all related files and not finding any docs on official site how to remove cluster installed via kubeadm.
Did somebody meet the same problems and know the proper way of removing all files and dependencies? Thank you in advance.
For more information, I removed kubeadm, kubectl and kubelet using apt-get purge/remove but when I started installing the cluster again I got next errors:
[preflight] Some fatal errors occurred:
Port 6443 is in use
Port 10251 is in use
Port 10252 is in use
/etc/kubernetes/manifests is not empty
/var/lib/kubelet is not empty
Port 2379 is in use
/var/lib/etcd is not empty
In my "Ubuntu 16.04", I use next steps to completely remove and clean Kubernetes (installed with "apt-get"):
kubeadm reset
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
sudo apt-get autoremove
sudo rm -rf ~/.kube
And restart the computer.
use kubeadm reset command. this will un-configure the kubernetes cluster.
If you are clearing the cluster so that you can start again, then, in addition to what #rib47 said, I also do the following to ensure my systems are in a state ready for kubeadm init again:
kubeadm reset -f
rm -rf /etc/cni /etc/kubernetes /var/lib/dockershim /var/lib/etcd /var/lib/kubelet /var/run/kubernetes ~/.kube/*
iptables -F && iptables -X
iptables -t nat -F && iptables -t nat -X
iptables -t raw -F && iptables -t raw -X
iptables -t mangle -F && iptables -t mangle -X
systemctl restart docker
You then need to re-install docker.io, kubeadm, kubectl, and kubelet to make sure they are at the latest versions for your distribution before you re-initialize the cluster.
EDIT: Discovered that calico adds firewall rules to the raw table so that needs clearing out as well.
kubeadm reset
/*On Debian base Operating systems you can use the following command.*/
# on debian base
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
/*On CentOs distribution systems you can use the following command.*/
#on centos base
sudo yum remove kubeadm kubectl kubelet kubernetes-cni kube*
# on debian base
sudo apt-get autoremove
#on centos base
sudo yum autoremove
/For all/
sudo rm -rf ~/.kube
The guide you linked now has a Tear Down section:
Talking to the master with the appropriate credentials, run:
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
Then, on the node being removed, reset all kubeadm installed state:
kubeadm reset
I use the following scripts to completely uninstall an existing Kubernetes cluster and its running docker containers
sudo kubeadm reset
sudo apt purge kubectl kubeadm kubelet kubernetes-cni -y
sudo apt autoremove
sudo rm -fr /etc/kubernetes/; sudo rm -fr ~/.kube/; sudo rm -fr /var/lib/etcd; sudo rm -rf /var/lib/cni/
sudo systemctl daemon-reload
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
# remove all running docker containers
docker rm -f `docker ps -a | grep "k8s_" | awk '{print $1}'`
If wanting to make it easily repeatable, it would make sense to make this into a script. This is assuming you are using a Debian based OS:
#!/bin/sh
# Kube Admin Reset
kubeadm reset
# Remove all packages related to Kubernetes
apt remove -y kubeadm kubectl kubelet kubernetes-cni
apt purge -y kube*
# Remove docker containers/ images ( optional if using docker)
docker image prune -a
systemctl restart docker
apt purge -y docker-engine docker docker.io docker-ce docker-ce-cli containerd containerd.io runc --allow-change-held-packages
# Remove parts
apt autoremove -y
# Remove all folder associated to kubernetes, etcd, and docker
rm -rf ~/.kube
rm -rf /etc/cni /etc/kubernetes /var/lib/dockershim /var/lib/etcd /var/lib/kubelet /var/lib/etcd2/ /var/run/kubernetes ~/.kube/*
rm -rf /var/lib/docker /etc/docker /var/run/docker.sock
rm -f /etc/apparmor.d/docker /etc/systemd/system/etcd*
# Delete docker group (optional)
groupdel docker
# Clear the iptables
iptables -F && iptables -X
iptables -t nat -F && iptables -t nat -X
iptables -t raw -F && iptables -t raw -X
iptables -t mangle -F && iptables -t mangle -X
NOTE:
This will destroy everything related to Kubernetes, etcd, and docker on the Node/server this command is run against!