I am testing a log previous command and for that I need a pod to restart.
I can get my pods using a command like
kubectl get pods -n $ns -l $label
Which shows that my pods did not restart so far. I want to test the command:
kubectl logs $podname -n $ns --previous=true
That command fails because my pod did not restart making the --previous=true switch meaningless.
I am aware of this command to restart pods when configuration changed:
kubectl rollout restart deployment myapp -n $ns
This does not restart the containers in a way that is meaningful for my log command test but rather terminates the old pods and creates new pods (which have a restart count of 0).
I tried various versions of exec to see if I can shut them down from within but most commands I would use are not found in that container:
kubectl exec $podname -n $ns -- shutdown
kubectl exec $podname -n $ns -- shutdown now
kubectl exec $podname -n $ns -- halt
kubectl exec $podname -n $ns -- poweroff
How can I use a kubectl command to forcefully restart the pod with it retaining its identity and the restart counter increasing by one so that my test log command has a previous instance to return the logs from.
EDIT:
Connecting to the pod is well described.
kubectl -n $ns exec --stdin --tty $podname -- /bin/bash
The process list shows only a handful running processes:
ls -1 /proc | grep -Eo "^[0-9]{1,5}$"
proc 1 seems to be the one running the pod.
kill 1 does nothing, not even kill the proc with pid 1
I am still looking into this at the moment.
There are different ways to achieve your goal. I'll describe below most useful options.
Crictl
Most correct and efficient way - restart the pod on container runtime level.
I tested this on Google Cloud Platform - GKE and minikube with docker driver.
You need to ssh into the worker node where the pod is running. Then find it's POD ID:
$ crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
9863a993e0396 87a94228f133e 3 minutes ago Running nginx-3 2 6d17dad8111bc
OR
$ crictl pods -s ready
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
6d17dad8111bc About an hour ago Ready nginx-3 default 2 (default)
Then stop it:
$ crictl stopp 6d17dad8111bc
Stopped sandbox 6d17dad8111bc
After some time, kubelet will start this pod again (with different POD ID in CRI, however kubernetes cluster treats this pod as the same):
$ crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
f5f0442841899 87a94228f133e 41 minutes ago Running nginx-3 3 b628e1499da41
This is how it looks in cluster:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-3 1/1 Running 3 48m
Getting logs with --previous=true flag also confirmed it's the same POD for kubernetes.
Kill process 1
It works with most images, however not always.
E.g. I tested on simple pod with nginx image:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 27h
$ kubectl exec -it nginx -- /bin/bash
root#nginx:/# kill 1
root#nginx:/# command terminated with exit code 137
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 1 27h
Useful link:
Debugging Kubernetes nodes with crictl
For some troubleshooting, I want to connect to my coredns pod. Is this possible?
$ microk8s kubectl get pod --namespace kube-system
NAME READY STATUS RESTARTS AGE
hostpath-provisioner-5c65fbdb4f-w6fmn 1/1 Running 1 7d22h
coredns-7f9c69c78c-mcdl5 1/1 Running 1 7d23h
calico-kube-controllers-f7868dd95-hbmjt 1/1 Running 1 7d23h
calico-node-rtprh 1/1 Running 1 7d23h
When I try, I get the following error msg:
$ microk8s kubectl --namespace kube-system exec --stdin --tty coredns-7f9c69c78c-mcdl5 -- /bin/bash
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "f1d08ed8494894d1281cd5c43dee36119225ab1ba414def333659538e5edc561": OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
User AndD has good mentioned in the comment:
Coredns Pod have no shell, I think. Check this to kind-of exec with a sidecar: How to get into CoreDNS pod kuberrnetes?
Yes. This image has no shell. You can read more about this situation in this thread:
The image does not contain a shell. Logs can be viewed with kubectl.
You have asked:
I want to connect to my coredns pod, is this possible?
Theoretically yes, but you need to make a workaroud with docker. It is described in this answer:
In short, do this to find a node where a coredns pod is running:
kubectl -n kube-system get po -o wide | grep coredns
ssh to one of those nodes, then:
docker ps -a | grep coredns
Copy the Container ID to clipboard and run:
ID=<paste ID here>
docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh
You will now be inside the "sidecar" container and can poke around. I.e.
cat /etc/coredns/Corefile
Additionally, you can check the logs, with kubectl. See also official documentation about DNS debugging.
I successfully deployed my Kubernetes app using kubectl apply -f deployment.yaml.
When I try to hit the URL endpoint, I'm getting an nginx 404 Not Found error page.
My next step is to open a bash shell on the docker instance that is running my app. How do I do this in Kubernetes?
How do I ssh into the docker container running my app, or docker exec bash to an app I've deployed to Kubernetes?
If I were running in docker I would run docker ps to find the container ID and then run docker exec -t ### bash to open a shell on my container to look around to troubleshoot why something isn't working.
What is the equivalent way to do this in Kubernetes?
Searching for a solution
I searched and found this URL, which says how to get a shell on your app.
The summary of that URL is:
kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml
kubectl get pod shell-demo
kubectl exec --stdin --tty shell-demo -- /bin/bash
But when I tried the equivalent commands, I got an error see below:
kubectl get pods --namespace my-app-namespace
NAME READY STATUS RESTARTS AGE
dpl-my-app-787bc5b7d-4ftkb 1/1 Running 0 2h
Then I tried:
kubectl exec --stdin --tty my-app-namespace -- /bin/bash
Error from server (NotFound): pods "my-app-namespace" not found
exit status 1
I figured this happened because I was trying to exec into the namespace not the pod, so I also tried with the dpl-my-app-... (see below) but got the same error.
kubectl exec --stdin --tty dpl-my-app-787bc5b7d-4ftkb -- /bin/bash
Error from server (NotFound): pods "dpl-my-app-787bc5b7d-4ftkb" not found
exit status 1
What is the command I need to get the pod instance so that kubectl exec will work?
As correctly stated by #David Maze:
Your kubectl get pods command has a --namespace option; you need to repeat this in the kubectl exec command. – David Maze 12 hours ago
If you've created your Deployment: dpl-my-app in a namespace: my-app-namespace you should also specify the --namespace/-n parameter in all of your commands.
A side note!
There is a tool to change namespaces, called: kubens
With a following command:
kubectl exec --stdin --tty my-app-namespace -- /bin/bash
You've correctly identified the issue that you are trying to exec into a namespace but not into a Pod
With a following command:
kubectl exec --stdin --tty dpl-my-app-787bc5b7d-4ftkb -- /bin/bash
You've tried to exec into a Pod named dpl-my-app-787bc5b7d-4ftkb but in a default namespace. Not in a namespace your Pod is residing.
To exec into your Pod in a specific namespace you should use following command:
kubectl exec --stdin --tty --namespace my-app-namespace dpl-my-app-787bc5b7d-4ftkb -- /bin/bash
Please notice the --namespace is before -- where the commands to the Pod should be placed (like -- /bin/bash).
Additional resources:
Kubernetes.io: Docs: Concepts: Overview: Working with objects: Namespaces
Kubernetes.io: Docs: Tasks: Debug application cluster: Get shell running container
i try to execute this command :
alias kubectl_winpty='winpty kubectl' ( using winpty due to [bug][1]
kubectl_winpty exec -it vault-0 -n vault-xxx -- sh export VAULT_CACERT=/vault/userconfig/vault-tls/vault.ca
but I'm getting this error :
sh: can't open 'export': No such file or directory
command terminated with exit code 2
the pod do exist and running
$ kubectl get pods vault-0 -n vault-xxx
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 17m
I guess sh export makes shell to assume export as a filename (some script that shell will run). Hence the error can't open 'export': No such file or directory.
Try to make use of -c option.
kubectl_winpty exec -it vault-0 -n vault-xxx -- sh -c "export VAULT_CACERT=/vault/userconfig/vault-tls/vault.ca"
Also it's better to set env variable when you create this pod.
Also make sure your pod is healthy and in ready state, which is not the case as per kubectl get output.
My CI tool uses lifecycles so if Dev deployments works, it goes to QA.
I have an end to end test container that i want to run in kubernetes, but how do i get the exit code from the container?
Can i somehow run the container and get back the exit code in one command?
kubectl run -it
doesn't seem to get the exit code and has some extra things to say after the container is done.
To get the exit code from a Pod (container) you can get the pod details with the command:
kubectl get pod termination-demo --output=yaml
Output:
apiVersion: v1
kind: Pod
...
lastState:
terminated:
containerID: ...
exitCode: 0
finishedAt: ...
message: |
Sleep expired
...
To know more, you can check the documentation.
To make it easier as you wish you can run:
kubectl get pod busybox-term -ojson | jq .status.containerStatuses[].lastState.terminated.exitCode
Or if you don't want to install jq, you can run:
kubectl get pod busybox-term --output="jsonpath={.status.containerStatuses[].lastState.terminated.exitCode}"
This way was mentioned by mWatney previously, so I've just put some additional details here:
This way you can return exit code 0-255 (after 255 it starts over, 256==0) from a Pod.
-it and --restart=Never are required, --rm is optional, but useful to remove failed pods.
--restart=Never tells the generator to create a Pod object instead of Deployment.
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 0"
pod "exitcode" deleted
$ echo $?
0
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 1"
pod "exitcode" deleted
pod default/exitcode terminated (Error)
$ echo $?
1
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 8"
pod "exitcode" deleted
pod default/exitcode terminated (Error)
$ echo $?
8
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 250"
pod "exitcode" deleted
pod default/exitcode terminated (Error)
$ echo $?
250
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 255"
pod "exitcode" deleted
pod default/exitcode terminated (Error)
$ echo $?
255
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 256"
pod "exitcode" deleted
$ echo $?
0
# exit code can also be assigned to a variable
$ kubectl run -it --rm exitcode --image=nginx --restart=Never -- bash -c "exit 255" ; a=$? && echo $a
pod "exitcode" deleted
pod default/exitcode terminated (Error)
255
Update for LoganMzz:
$ kubectl run -it --rm --image=busybox --restart=Never foobar -- ash -c 'exit 10'; echo rc=$?
pod "foobar" deleted
pod default/foobar terminated (Error)
rc=10
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
10.10.0.21 Ready master 2d1h v1.18.5 10.10.0.21 <none> Ubuntu 16.04.6 LTS 4.4.0-184-generic docker://18.9.7
10.10.0.22 Ready <none> 2d1h v1.18.5 10.10.0.22 <none> Ubuntu 16.04.6 LTS 4.4.0-184-generic docker://18.9.7
I recommend the jsonpath approach in #mark-watney 's answer because you do not need any additional tools. But I will expand that if you have multiple containers, you will need to add a filter. For example:
kubectl get pods pod-name -o jsonpath='{.status.containerStatuses[?(#.name=="container-name")].state.terminated.exitCode}'
kubectl get po pod_name -ojson | jq .status.containerStatuses[].state.terminated.exitCode
You can pipe the output of kubectl get to jq which can parse the json and print the exit code, you may skip the -c container_name if single container is present.
kubectl get pod pod_name -c container_name-n namespace -ojson | jq .status.containerStatuses[].state.terminated.exitCode
I would like to run a command in a temporary pod just to verify that my network policies are correct. The exit code of the command run inside the container should then indicate whether access was allowed or denied. For instance:
kubectl run --rm -it --image centos:7 --restart=Never xxx -- exit 1
kubectl get pod xxx -ojson | jq .status.containerStatuses[].state.terminated.exitCode
where 'exit 1' is replaced by a command that implements my check.
But of course, since --rm was used for running the container, the POD no longer exists after the command so retrieving the exit status fails. The only way around it is then to not use the --rm option and to cleanup the temporary pod afterwards.
Another, perhaps better alternative is to echo an output string from the command and then check the output. The command could use a 'trap' to catch any exit and echo the exit status
trap 'echo EXITCODE $?; exit' EXIT
Above, EXIT traps any exit, and 'exit' does an exit with the original exit code.
Then get the exitcode using
code="$( kubectl ... | awk '/EXITCODE/ { print $2 }')"