trying to shell into the container by kubectl exec -it xxxxxx
but it returns
rpc error: code = 5 desc = open /var/run/docker/libcontainerd/containerd/faf3fd49262cc738e16368001eba5e1113abcb8a87e7b818cb84af3799906149/30fe901c16e0465aa15b596bf3e4f244fb12a7e4133b6e4da5aa35167a8dfb30/shim-log.json: no such file or directory
trying to reboot the node but not help
Thanks #Prafull Ladha
Eventually I restarted the docker (systemctl restart docker) of that Node which my pods could not be shelled, and it resumes to normal
The problem is with containerd, Once the containerd restart in the background, the docker daemon still try to process event streams against the old socket handles. After that, the error handeling when client can't connect to the containerd leads to the CPU spike on machine.
This is the open issue with docker and currently the workaround is to restart the docker.
sudo systemctl restart docker
It appears like some issue with the docker daemon. it would help if you add the logs from the container to research the root cause.
deploy alpine pod and see if you can get into the container. This is to isolate if the problem is with the platform or the pod that you are running.
kubectl run pingpong --image alpine ping 8.8.8.8
kubectl exec -it <pingpong-pod-name> sh
most likely something wrong with the pod that you are running. share the container logs for further help
Related
I have installed OracleVM, and created an Ubuntu machine. I have not tried any major cluster or tried deploying anything, I have started reading about Kubernetes and as an example just tried creating a simple pod. But I am getting host error, can anyone tell me where am I going wrong?
I have tried the simple kubectl run command
The snip of my issue
You need to disable swap on the system for kubelet to work. You can disable swap with sudo swapoff -a and restart kubelet service sudo systemctl restart kubelet
I am using the mssql docker image (Linux) for sql server 2019. The default user is not root but mssql.
I need to perform some operations as root inside the container:
docker exec -it sql bash
mssql#7f5a78a63728:/$ sudo <command>
bash: sudo: command not found
Then I start the shell as root:
docker exec -it --user=root sql bash
root#7f5a78a63728:/# <command>
...
This works.
Now I need to do this in a container deployed in an AKS cluster
kubectl exec -it rms-sql-1-sql-server-deployment-86cc45dc5c-tgtm2 -- bash
mssql#rms-sql-1-sql-server-host:/$ sudo <command>
bash: sudo: command not found
as expected. But then:
kubectl exec -it --user=root rms-sql-1-sql-server-deployment-86cc45dc5c-tgtm2 -- bash
error: auth info "root" does not exist
So when the container is in an AKS cluster, starting a shell as root doesn't work.
I then try to ssh into the node and use docker from inside:
kubectl debug node/aks-agentpool-30797540-vmss000000 -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11
Creating debugging pod node-debugger-aks-agentpool-30797540-vmss000000-xfrsq with container debugger on node aks-agentpool-30797540-vmss000000.
If you don't see a command prompt, try pressing enter.
root#aks-agentpool-30797540-vmss000000:/# docker ...
bash: docker: command not found
Looks like a Kubernetes cluster node doesn't have docker installed!
Any clues?
EDIT
The image I used locally and in Kubernetes is exactly the same,
mcr.microsoft.com/mssql/server:2019-latest untouched
David Maze has well mentioned in the comment:
Any change you make in this environment will be lost as soon as the Kubernetes pod is deleted, including if you need to update the underlying image or if its node goes away outside of your control. Would building a custom image with your changes be a more maintainable solution?
Generally, if you want to change something permanently you have to create a new image. Everything you described behaved exactly as it was supposed to. First you have exec the container in docker, then logged in as root. However, in k8s it is a completely different container. Perhaps a different image is used. Second, even if you made a change, it would exist until the container dies. If you want to modify something permanently, you have to create your new image with all the components and the configuration you need. For more information look at pod lifecycle.
I use minikube v1.6.2, kubectl 1.17.
I start minikube without Virtualbox, with:
sudo minikube start --vm-driver none
Now, to stop it, I do:
sudo minikube stop
minikube stop # I don't know which one is the good one, but I do both
but, after that, when I do:
kubectl get po
I still get the pods listing. The only way to stop it is to actually reboot my machine.
Why is it happening, and how should I fix it ?
minikube stop when used with --vm-driver=none does not do any cleanup of the pods. As mentioned here:
When minikube starts without a hypervisor, it installs a local kubelet
service on your host machine, which is important to know for later.
Right now it seems that minikube start is the only command aware of
--vm-driver=none. Running minikube stop keeps resulting in errors related to docker-machine, and as luck would have it also results in
none of the Kubernetes containers terminating, nor the kubelet service
stopping.
Of course, if you wish to actually terminate minikube, you will need
to execute service kubelet stop and then ensure the k8s containers are
removed from the output in docker ps.
If you wish to know the overview of none (bare-metal) driver you can find it here.
Also as a workaround you can stop and remove all Docker containers that have 'k8s' in their name by executing the following command: docker stop (docker ps -q --filter name=k8s) and docker rm (docker ps -aq --filter name=k8s).
Please let me know if that helped.
When I define multiple containers in a pod/pod template like one container running agent and another php-fpm, how can they access each other? I need the agent container to connect to php-fpm by shell and need to execute few steps interactively through agent container.
Based on my understanding, we can package kubectl into the agent container and use kubectl exec -it <container id> sh to connect to the container. But I don't want Agent container to have more privilege than to connect to the target container with is php-fpm.
Is there a better way for agent container to connect to php-fpm by a shell and execute commands interactively?
Also, I wasn't successful in running kubectl from a container when using minikube due to following errors
docker run -it -v ~/.kube:/root/.kube lachlanevenson/k8s-kubectl get nodes
Error in configuration:
* unable to read client-cert /Users/user/.minikube/apiserver.crt for minikube due to open /Users/user/.minikube/apiserver.crt: no such file or directory
* unable to read client-key /Users/user/.minikube/apiserver.key for minikube due to open /Users/user/.minikube/apiserver.key: no such file or directory
* unable to read certificate-authority /Users/user/.minikube/ca.crt for minikube due to open /Users/user/.minikube/ca.crt: no such file or directory
docker run -it -v ~/.kube:/root/.kube lachlanevenson/k8s-kubectl get nodes
First off, every Pod within a k8s cluster has its own k8s credentials provided by /var/run/secrets/kubernetes.io/serviceaccount/token, and thus there is absolutely no need to attempt to volume mount your home directory into a docker container
The reason you are getting the error about client-cert is because the contents of ~/.kube are merely strings that point to the externally defined ssl key, ssl certificate, and ssl CA certificate defined inside ~/.kube/config -- but I won't speak to fixing that problem further since there is no good reason to be using that approach
I have setup docker toolbox on windows 10. While building the project I encountered the following error :- Bind for 0.0.0.0:8081 failed: port is already allocated ? The sudo service docker restart command isn't working. Please provide me a solution for the same.
Generally speaking, you need to stop running the current container. For that you are going to know current CONTAINER ID:
$ docker container ls
You get something like:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
97a32e8928ef friendlyhello "python app.py" 51 seconds ago Up 50 seconds 0.0.0.0:4000->80/tcp romantic_tesla
Then you stop the container by:
$ docker stop 97a32e8928ef
Finally, you try to do what you wanted to do, for example:
$ docker run -p 4000:80 friendlyhello
Commonly if this error happens, I restarting my winnat with commands:
$ net stop winnat
// build your project
$ net start winnat
If that doesnt help. I restart whole docker with commands:
wsl --unregister docker-desktop
wsl --unregister docker-desktop-data
Then docker offers to restart docker-service.