How to backup and restore a kubernetes master node? - kubernetes

Question

There is a k8s single master node, I need to back it up and restore on a different server with different ip addresses. I googled this topic and found a solution -
https://elastisys.com/2018/12/10/backup-kubernetes-how-and-why/
Everything looked easy; so, I followed the instruction and got a copy of the certificates and a snapshot of the etcd database. Then I used the second script to restore the node on a different server. It did not go well this time. It gave me a bunch of errors related to mismatching the certificates and server's local ip addresses.
As far as I understood, when a kubernetes cluster is initializing, it creates a set of certificates assigned to the original server's ip addresses and I cannot just back it up and restore somewhere else.
So, how to backup a k8s master node and restore it?

Answer 1

Make sure that you added an extra flag to the kubeadm init command (--ignore-preflight-errors=DirAvailable--var-lib-etcd) to acknowledge that we want to use the pre-existing data.
Do the following steps:
replace the IP address in all config files in /etc/kubernetes
back up /etc/kubernetes/pki
identify certs in /etc/kubernetes/pki that have the old IP address
as an alt name - 1st step
delete both the cert and key for each of them (for me it was just
apiserver and etcd/peer)
regenerate the certs using kubeadm alpha phase certs - 2nd step
identify configmap in the kube-system namespace that referenced
the old IP - 3rd step
manually edit those configmaps
restart kubelet and docker (to force all containers to be
recreated)
1.
/etc/kubernetes/pki# for f in $(find -name "*.crt"); do openssl x509 -in $f -text -noout > $f.txt; done
/etc/kubernetes/pki# grep -Rl 12\\.34\\.56\\.78 .
./apiserver.crt.txt
./etcd/peer.crt.txt
/etc/kubernetes/pki# for f in $(find -name "*.crt"); do rm $f.txt; done
2.
/etc/kubernetes/pki# rm apiserver.crt apiserver.key
/etc/kubernetes/pki# kubeadm alpha phase certs apiserver
...
/etc/kubernetes/pki# rm etcd/peer.crt etcd/peer.key
/etc/kubernetes/pki# kubeadm alpha phase certs etcd-peer
...
3.
$ kubectl -n kube-system get cm -o yaml | less
...
$ kubectl -n kube-system edit cm ...
Take a look here: master-backup.
UPDATE:
During replacing master nodes and changing IP you cannot contact the api-server to change the configmaps in step 4. Moreover if you have single master k8s cluster connection between worker nodes will be interrupted till new master will be up.
To ensure connection between master and worker nodes during master replacement you have to create HA cluster.
The certificate is signed for {your-old-IP-here} and secure communication can't then happen to {your-new-ip-here}
You can add more IPs in the certificate in beforehand though...
The api-server certificate is signed for hostname kubernetes, so you can add that as an alias to the new IP address in /etc/hosts then do kubectl --server=https://kubernetes:6443 .... .