How to access kubernetes keys in etcd - kubernetes

Question
How to get the Kubernetes related keys from etcd? Tried to list keys in etcd but could not see related keys. Also where is etcdctl installed?
$ etcdctl
bash: etcdctl: command not found..
$ sudo netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 386/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 386/etcd
$ curl -s http://localhost:2379/v2/keys | python -m json.tool
{
"action": "get",
"node": {
"dir": true
}
}
Background
Installed Kubernetes 1.8.5 by following Using kubeadm to Create a Cluster on CentOS 7. When I looked at Getting started with etcd, v2/keys looks to be the end point.

Usually you need to get etcdctl by yourself. Just download the latest etcdctl archive from etcd releases page.
Also, starting from Kubernetes version 1.6 it uses etcd version 3, so to get a list of all keys is:
ETCDCTL_API=3 etcdctl --endpoints=<etcd_ip>:2379 get / --prefix --keys-only
You can find all etcdctl v3 actions using:
ETCDCTL_API=3 etcdctl --endpoints=<etcd_ip>:2379 --help
EDIT (thanks to #leodotcloud):
In case ETCD is configured with TLS certificates support:
ETCDCTL_API=3 etcdctl --endpoints <etcd_ip>:2379 --cacert <ca_cert_path> --cert <cert_path> --key <cert_key_path> get / --prefix --keys-only

Access the docker container, and run the following commmand:
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only

For Minikube
(v1.17.0)
You can see the arguments exploring the pod: kubectl describe pod -n kube-system etcd-PODNAME |less
Here you can see the certificates path and much more.
To fastly query your etcd dictionary you can use this alias:
alias etcdctl_mini="MY_IP=$(hostname -I |awk '{print $1}'|tr -d ' '); \
ETCDCTL_API=3; \
sudo -E etcdctl --endpoints ${MY_IP}:2379 \
--cacert='/var/lib/minikube/certs/etcd/ca.crt' \
--cert='/var/lib/minikube/certs/etcd/peer.crt' \
--key='/var/lib/minikube/certs/etcd/peer.key'"
$ etcdctl_mini put foo bar

I needed to use etcdctl with etcd installed on CoreOS (Container Linux).
In my case the following worked (executed from CoreOS shell prompt):
$ sudo ETCDCTL_API=3 etcdctl --cacert /etc/ssl/etcd/etcd/peer-ca.crt --cert /etc/ssl/etcd/etcd/peer.crt --key /etc/ssl/etcd/etcd/peer.key get --prefix / --keys-only
I used sudo as a quick solution to the permission problem "Error: open /etc/ssl/etcd/etcd/peer.crt: permission denied".

You can also try following (assuming etcd pod name is etcd-minikube).
Minikube access using etcdctl was already explained above.
$kubectl -it exec etcd-minikube -n kube-system -- etcdctl --cacert='/var/lib/minikube/certs/etcd/ca.crt' --cert='/var/lib/minikube/certs/etcd/peer.crt' --key='/var/lib/minikube/certs/etcd/peer.key' put foo bar
OK
$kubectl -it exec etcd-minikube -n kube-system -- etcdctl --cacert='/var/lib/minikube/certs/etcd/ca.crt' --cert='/var/lib/minikube/certs/etcd/peer.crt' --key='/var/lib/minikube/certs/etcd/peer.key' get foo
foo
bar

Related

Unable to connect internet/google.com from pod. Docker and k8 are able to pull images

I am trying to learn Kubernetes.
Create a single-node Kubernetes Cluster on Oracle Cloud using these steps here
cat /etc/resolv.conf
>> nameserver 169.254.169.254
kubectl run busybox --rm -it --image=busybox --restart=Never -- sh
cat /etc/resolv.conf
>> nameserver 10.33.0.10
nslookup google.com
>>Server: 10.33.0.10
Address: 10.33.0.10:53
;; connection timed out; no servers could be reached
ping 10.33.0.10
>>PING 10.33.0.10 (10.33.0.10): 56 data bytes
kubectl get svc -n kube-system -o wide
>> CLUSTER-IP - 10.33.0.10
kubectl logs --namespace=kube-system -l k8s-app=kube-dns
>>[ERROR] plugin/errors: 2 google.com. A: read udp 10.32.0.9:57385->169.254.169.254:53: i/o timeout
Not able to identify if this is an error of coredns or pod networking. Any direction would really help
Kubernetes has deprecated Docker as a container runtime after v1.20.
Kubernetes Development decision to deprecate Docker as an underlying runtime in favor of runtimes that use the Container Runtime Interface (CRI) created for Kubernetes.
To support this Mirantis and Docker came to the rescue by agreeing to partner in the maintenance of the shim code standalone.
More details here here
sudo systemctl enable docker
# -- Installin cri-dockerd
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4)
echo $VER
wget https://github.com/Mirantis/cri-dockerd/releases/download/${VER}/cri-dockerd-${VER}-linux-arm64.tar.gz
tar xvf cri-dockerd-${VER}-linux-arm64.tar.gz
install -o root -g root -m 0755 cri-dockerd /usr/bin/cri-dockerd
cp cri-dockerd /usr/bin/
# -- Verification
cri-dockerd --version
# -- Configure systemd units for cri-dockerd
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo cp cri-docker.socket cri-docker.service /etc/systemd/system/
sudo cp cri-docker.socket cri-docker.service /usr/lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket
# -- Using cri-dockerd on new Kubernetes cluster
systemctl status docker | grep Active
I ran into similar issue with almost same scenario described above. The accepted solution https://stackoverflow.com/a/72104194/1119570 is wrong. This issue is a pure networking issue that is not related to any of EKS upgrade in any way.
The root cause for our issue was the fact that the Worker Node AWS EKS Linux 1.21 AMI being hardened by our security department which turns off the following setting in this file /etc/sysctl.conf:
net.ipv4.ip_forward = 0
After switching this setting to:
net.ipv4.ip_forward = 1 and rebooting the EC2 Node, everything started working properly. Hope this helps!

rancher rke up errors on etcd host health checks remote error: tls: bad certificate

rke --debug up --config cluster.yml
fails with health checks on etcd hosts with error:
DEBU[0281] [etcd] failed to check health for etcd host [x.x.x.x]: failed to get /health for host [x.x.x.x]: Get "https://x.x.x.x:2379/health": remote error: tls: bad certificate
Checking etcd healthchecks
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5"); do
echo "Validating connection to ${endpoint}/health";
curl -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health";
done
Running on that master node
Validating connection to https://x.x.x.x:2379/health
{"health":"true"}
Validating connection to https://x.x.x.x:2379/health
{"health":"true"}
Validating connection to https://x.x.x.x:2379/health
{"health":"true"}
Validating connection to https://x.x.x.x:2379/health
{"health":"true"}
you can run it manually and see if it responds correctly
curl -w "\n" --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-etcd-x-x-x-x.pem --key /etc/kubernetes/ssl/kube-etcd-x-x-x-x-key.pem https://x.x.x.x:2379/health
Checking my self signed certificates hashes
# md5sum /etc/kubernetes/ssl/kube-ca.pem
f5b358e771f8ae8495c703d09578eb3b /etc/kubernetes/ssl/kube-ca.pem
# for key in $(cat /home/kube/cluster.rkestate | jq -r '.desiredState.certificatesBundle | keys[]'); do echo $(cat /home/kube/cluster.rkestate | jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5sum) $key; done | grep kube-ca
f5b358e771f8ae8495c703d09578eb3b - kube-ca
versions on my master node
Debian GNU/Linux 10
rke version v1.3.1
docker version Version: 20.10.8
kubectl v1.21.5
v1.21.5-rancher1-1
I think my cluster.rkestate gone bad, are there any other locations where rke tool checks for certificates?
Currently I cannot do anything with this production cluster, and want to avoid downtime. I experimented on testing cluster different scenarios, I could do as last resort to recreate the cluster from scratch, but maybe I can still fix it...
rke remove && rke up
rke util get-state-file helped me to reconstruct bad cluster.rkestate file
and I was able to successfully rke up and add new master node to fix whole situation.
The problem can be solved by doing the following steps:
Remove kube_config_cluster.yml file where you run rke up command. (Since some data are missing in your K8s nodes)
Remove cluster.rkestate file.
Re-run rke up command.

Error installing TimescaleDB with K8S / Helm : MountVolume.SetUp failed for volume "certificate" : secret "timescaledb-certificate" not found

I just tried to install timescaleDB Single with Helm in minikube on Ubuntu 20.04.
After installing via:
helm install timescaledb timescaledb/timescaledb-single --namespace espace-client-v2
I got the message:
➜ ~ helm install timescaledb timescaledb/timescaledb-single --namespace espace-client-v2
NAME: timescaledb
LAST DEPLOYED: Fri Aug 7 17:17:59 2020
NAMESPACE: espace-client-v2
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
TimescaleDB can be accessed via port 5432 on the following DNS name from within your cluster:
timescaledb.espace-client-v2.svc.cluster.local
To get your password for superuser run:
# superuser password
PGPASSWORD_POSTGRES=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" | base64 --decode)
# admin password
PGPASSWORD_ADMIN=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_admin_PASSWORD}" | base64 --decode)
To connect to your database, chose one of these options:
1. Run a postgres pod and connect using the psql cli:
# login as superuser
kubectl run -i --tty --rm psql --image=postgres \
--env "PGPASSWORD=$PGPASSWORD_POSTGRES" \
--command -- psql -U postgres \
-h timescaledb.espace-client-v2.svc.cluster.local postgres
# login as admin
kubectl run -i --tty --rm psql --image=postgres \
--env "PGPASSWORD=$PGPASSWORD_ADMIN" \
--command -- psql -U admin \
-h timescaledb.espace-client-v2.svc.cluster.local postgres
2. Directly execute a psql session on the master node
MASTERPOD="$(kubectl get pod -o name --namespace espace-client-v2 -l release=timescaledb,role=master)"
kubectl exec -i --tty --namespace espace-client-v2 ${MASTERPOD} -- psql -U postgres
It seemed to have installed well.
But then, when executing:
PGPASSWORD_POSTGRES=$(kubectl get secret --namespace espace-client-v2 timescaledb-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" | base64 --decode)
Error from server (NotFound): secrets "timescaledb-credentials" not found
After that, I realized pod has not even been created, and it gives me the following errors
MountVolume.SetUp failed for volume "certificate" : secret "timescaledb-certificate" not found
Unable to attach or mount volumes: unmounted volumes=[certificate], unattached volumes=[storage-volume wal-volume patroni-config timescaledb-scripts certificate socket-directory timescaledb-token-svqqf]: timed out waiting for the condition
What should I do ?
I could do it. If the page https://github.com/timescale/timescaledb-kubernetes doesn't give much details about installation process, you can go here:
https://github.com/timescale/timescaledb-kubernetes/tree/master/charts/timescaledb-single
I had to use kustomize to generate content:
./generate_kustomization.sh my-release
and then it generate several files:
credentials.conf kustomization.yaml pgbackrest.conf timescaledbMap.yaml tls.crt tls.key
then I did:
kubectl kustomize ./
which generated a k8s yml file, which I saved with the name timescaledbMap.yaml
Finally, I did:
kubectl apply -f timescaledbMap.yaml
Then it created all necesarry secrets, and I could install chart
. Hope it helps others.

How to backup etcd on a Kubernetes cluster created with kubeadm - rpc error: code = 13

I have a K8s cluster created with kubeadm that consists of a master node and two workers.
I am following this documentation article regarding the etcd backup: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster
I have to use etcdctl to backup the etcd db so I sh into the etcd pod running on the master node to do it from there: kubectl exec -it -n kube-system etcd-ip-x-x-x-x sh
NOTE: The master node hosts the etcd database in this path /var/lib/etcd which is mounted on the pod as a VolumeMount in /var/lib/etcd.
Following the doc I run: ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 snapshot save snapshotdb and it returns the following error:
Error: rpc error: code = 13 desc = transport: write tcp 127.0.0.1:44464->127.0.0.1:2379: write: connection reset by peer
What is the problem here?
I managed to make it work adding the certificates info to the command:
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save ./snapshot.db
You can also check the Etcd configuration file in the following line on a node that Etcd is running.
/etc/kubernetes/manifests/etcd.yaml
After getting the command in the command section of the Etcd configuration,
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key
you can run Etcd commands like below commands.
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key member list
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save ./snapshot.db
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key snapshot status ./snapshot.db

Configure Client commands from command line

In IBM Cloud Private EE, I need to go to the Web UI User > Configure client, copy the kubectl config commands and then run these 5 commands on my client machine.
I deployed the IBM Cloud private EE on 5 VMs and have access to the master node. I am wondering if there is a way to capture these kubectl config commands directly from the docker containers without having a need to go to the Web UI.
For example: I did not want to download the kubectl client from google (as I just want to use same kubectl version which is in the ICP containers) and I used the following command to get it from the container itself.
docker run --rm -v $(pwd):/data -e LICENSE=accept \
ibmcom/icp-inception:2.1.0.1-ee \
cp -r /usr/local/bin/kubectl /data
Then, I copied this to all VM guests so that I could access kubectl from any guest.
chmod +x kubectl
for host in $(awk '/192.168.142/ {print $3}' /etc/hosts)
do
scp kubectl $host:/bin
done
Where - 192.168.142 is the subnet of my VM guests.
But, I could not figure out how to get Configure Client commands without having to go to the Web UI. I need this to automate client kubectl command so that my environment is ready for kubectl commands through simple scripts.
You should use Vagrant to automate those steps.
For instance, IBM/deploy-ibm-cloud-private/Vagrantfile has this section:
install_kubectl = <<SCRIPT
echo "Pulling #{image_repo}/kubernetes:v#{k8s_version}..."
sudo docker run -e LICENSE=#{license} --net=host -v /usr/local/bin:/data #{image_repo}/kubernetes:v#{k8s_version} cp /kubectl /data &> /dev/null
kubectl config set-credentials icpadmin --username=admin --password=admin &> /dev/null
kubectl config set-cluster icp --server=http://127.0.0.1:8888 --insecure-skip-tls-verify=true &> /dev/null
kubectl config set-context icp --cluster=icp --user=admin --namespace=default &> /dev/null
kubectl config use-context icp &> /dev/null
SCRIPT
See more at "Kubernetes, IBM Cloud Private, and Vagrant, oh my!", from Tim Pouyer.
#VonC provided useful tips. This is how the service account token can be obtained.
Get the token from a running container - Tip from this link.
RUNNIGCONTAINER=$(docker ps | grep k8s_cloudiam-apikeys_auth | awk '{print $1}')
TOKEN=$(docker exec -t $RUNNIGCONTAINER cat /var/run/secrets/kubernetes.io/serviceaccount/token)
I already know the name of the IBM Cloud Private cluster name, master node and the default user name. The only missing link was the token. Please note that the script used by Tim is using password and the only difference was - I wanted to use token instead of the password.
So use the scripts.
kubectl config set-cluster ${CLUSTERNAME}.icp --server=https://$MASTERNODE:8001 --insecure-skip-tls-verify=true
kubectl config set-context ${CLUSTERNAME}.icp-context --cluster=${CLUSTERNAME}.icp
kubectl config set-credentials admin --token=$TOKEN
kubectl config set-context ${CLUSTERNAME}.icp-context --user=$DEFAULTUSERNAME --namespace=default
kubectl config use-context ${CLUSTERNAME}.icp-context
# get token
icp_auth_token=`curl -s -k -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" \
-d "grant_type=password&username=${myuser}&password=${mypass}&scope=openid" \
https://${icp_server}:8443/idprovider/v1/auth/identitytoken --insecure | \
sed 's/{//g;s/}//g;s/\"//g' | \
awk -F ':' '{print $7}'`
# setup context
kubectl config set-cluster ${icp_server} --server=https://${icp_server}:8001 --insecure-skip-tls-verify=true
kubectl config set-credentials ${icp_server}-user --token=${icp_auth_token}
kubectl config set-context ${icp_server}-context --cluster=${icp_server} --user=${icp_server}-user
kubectl config use-context ${icp_server}-context