View log files of crashed pods in kubernetes - kubernetes

Question

Any idea to view the log files of a crashed pod in kubernetes?
My pod is listing it's state as "CrashLoopBackOff" after started the replicationController. I search the available docs and couldn't find any.

Answer 1

Assuming that your pod still exists:
kubectl logs <podname> --previous
$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.

Answer 2

In many cases, kubectl logs <podname> --previous is returning:
Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found
So you can try to check in the namespace's events (kubectl get events ..) like #alltej showed.
If you don't find the reason for the error with kubectl logs / get events and you can't view it with external logging tool I would suggest:
1 ) Check on which node that pod was running on with:
$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME STATUS NODE
failed-pod-name Pending dns-of-node
(If you remove the <pod-name> you can see other pods in the namespace).
2 ) SSH into that node and:
A ) Search for the failed pod container name in /var/log/containers/ and dump its .log file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.
B ) If previous step doesn't help try searching for latest System level errors by running: sudo journalctl -u kubelet -n 100 --no-pager.

Answer 3

kubectl logs command only works if the pod is up and running. If they are not, you can use the kubectl events command.
kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'
By default it does not sort the events, hence the --sort-by flag.

Answer 4

There was a bug in kubernetes that prevents logs obtaining for pods in CrashLoopBackOff state. Looks like it was fixed. Here issue on github with additional information

Answer 5

As discussed on another StackOverflow question, I wrote an open source tool to do this
The main difference with the other answers is that this is triggered automatically when a pod crashes, so it can help avoid scenarios where you start debugging this much later on and the pod itself no longer exists and logs can't be fetched.

Answer 6

If the pod does not exist anymore:
kubectl describe pod {RUNTIME_NAME_OF_POD}
In the output you should have the section "Events" which contains the error messages that prevented the pod to start.

Answer 7

Container Failures could be due to resource limits reached
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Wed, 18 Jan 2023 11:28:14 +0530
Finished: Wed, 18 Jan 2023 11:28:18 +0530
Ready: False
Restart Count: 13
OR
The application ended due to an error
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Wed, 18 Jan 2023 11:50:59 +0530
Finished: Wed, 18 Jan 2023 11:51:03 +0530
Ready: False
Debugging container failure:
Looking at pod status which will contain the above status information
sugumar$ kubectl get pod POD_NAME -o yaml
sugumar$ kubectl get events -w | grep POD_NAME_STRING
For default container logs
sugumar$ kubectl logs -f POD_NAME
For specific container: reason for application failure
sugumar$ kubectl logs -f POD_NAME --container CONTAINER_NAME
Looking at events
sugumar$ kubectl describe deployment DEPLOYMENT_NAME
sugumar$ kubectl describe pod POD_NAME