kubectl logs -f pod shows all logs from the beginning and it becomes a problem when the log is huge and we have to wait for a few minutes to get the last log. Its become more worst when connecting remotely. Is there a way that we can tail the logs for the last 100 lines of logs and follow them?
In a cluster best practices are to gather all logs in a single point through an aggregator and analyze them with a dedicated tool. For that reason in K8S, log command is quite basic.
Anyway kubectl logs -h shows some options useful for you:
# Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
# Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
Some tools with your requirements (and more) are available on github, some of which are:
https://github.com/boz/kail
https://github.com/wercker/stern
Try kubectl logs -f pod --tail=10
To fetch tail lines from logs of a pod with multi containers.
kubectl logs <pod name> --all-containers=true --tail=10
To fetch tail lines from logs of pods within an application:
kubectl logs --selector app=<your application> --tail=10
(ex:if your application has 3 pods then output of above command can be 30 logs 10 of each pod logs)
You can ues this way to get the first 10 lines
kubectl logs my-pod-name -n my-ns | head -n 10
You can also follow logs from the end if you are testing something:
kubectl logs my-pod-name --follow
This will work just like running tail -f in bash or other shells.
Related
I have a deployment that fails within one second. And the logs are destroyed as the deployment does a rollback.
Is there anything similar to logs -f that works before a deployment have started and wait until it starts?
Check previous logs with kubectl logs -p <pod-name> to spot application issues.
Also, check the exit code of your container with:
kubectl describe pod <pod-name> | grep "Exit Code"
Finally, if it is a scheduling problem, check out the event log of the corresponding ReplicaSet:
kubectl describe replicaset <name-of-deployment>
I was looking for a way to stream the logs of all pods of a specific deployment of mine.
So, some days ago I've found this SO answer giving me a magical command:
kubectl logs -f deployment/<my-deployment> --all-containers=true
However, I've just discovered, after a lot of time debugging, that this command actually shows the logs of just one pod, and not all of the deployment.
So I went to Kubectl's official documentation and found nothing relevant on the topic, just the following phrase above the example that uses the deployment, as a kind of selector, for log streaming:
...
# Show logs from a kubelet with an expired serving certificate
kubectl logs --insecure-skip-tls-verify-backend nginx
# Return snapshot logs from first container of a job named hello
kubectl logs job/hello
# Return snapshot logs from container nginx-1 of a deployment named nginx
kubectl logs deployment/nginx -c nginx-1
So why is that the first example shown says "Show logs" and the other two say "Return snapshot logs"?
Is it because of this "snapshot" that I can't retrieve logs from all the pods of the deployment?
I've searched a lot for more deep documentation on streaming logs with kubectl but couldn't find any.
To return all pod(s) log of a deployment you can use the same selector as the deployment. You can retrieve the deployment selector like this kubectl get deployment <name> -o jsonpath='{.spec.selector}' --namespace <name>, then you retrieve logs using the same selector kubectl logs --selector <key1=value1,key2=value2> --namespace <name>
What is the purpose of kubectl logs deploy/my-deployment shown at https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-deployments-and-services?
I would think it will show me logs from all the pods deployed as part of the my-deployment object. However, even though I have 2 pods in my deployment, that command shows logs from only one of them.
If your deployment has multiple pod replicas, then kubectl logs deployment/... will just pick one on its own.
Here is an example:
kubectl get pods -n kube-system | grep coredns
coredns-78fcd69978-dqf95 1/1 Running 0 42h
coredns-78fcd69978-vgvf2 1/1 Running 0 42h
kubectl logs deployment/coredns -n kube-system
Found 2 pods, using pod/coredns-78fcd69978-vgvf2
As you can see from the documentation you linked:
kubectl logs deploy/my-deployment # dump Pod logs for a Deployment (single-container case)
kubectl logs deploy/my-deployment -c my-container # dump Pod logs for a Deployment (multi-container case)
kubectl logs deploy/my-deployment is used when you have just one container. So in your case is probably taking the first one. If you have multiple containers you have to specify one with -c option.
If you want to have logs from multiple pods, you can use Stern
By following documentation provided, when there are multiple Pods using the below command, it displays logs from only one Pod at a time it will pick randomly one at a point of time.
kubectl get pods -n kube-system | grep coredns
If there are multiple containers then one can specify by using “-c” and mention the container name.
By following the Stren documentation, one can get the logs from multiple containers within a pod. Using the below command will display the multiple container data.
kubectl logs deploy/my-deployment -c my-container
This should work:
kubectl -n <namespace> logs -l <label_selector> --all-containers=true -f --tail=25
I have a deployment of pods which failed 22h ago, how often does Kubernetes log-rotate its logs?
Is there any possibility to view the logs of the deployment but 22 hours ago?
Thanks
I think we can not retrieve logs from a pod that is not in ready state.
We can get the logs of the container inside the pod , By logging into the worker node where pod was running .
docker ps -a | grep <pod name>
docker logs <container name/id from above output
You can use kubectl logs --previous to retrieve logs from a previous instantiation of a container.
Kubernetes does NOT provide built-in log rotation.
Check official Debug Running Pods documentation:
If your container has previously crashed, you can access the previous
container's crash log with:
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
In my opinion you are asking not about logs on pod, you are more interested in full debug. Your starting point is again official documentation Troubleshoot Applications-Debugging Pods. ANd start check with kubectl describe pods ${POD_NAME}
4.All I wrote above is great, however sometimes the only way to get the logs is #confused genius answer.
I am running selenium hubs and my pods are getting terminated frequently. I would like to look at the logs of the pods which are terminated. How to do it?
NAME READY STATUS RESTARTS AGE
chrome-75-0-0e5d3b3d-3580-49d1-bc25-3296fdb52666 0/2 Terminating 0 49s
chrome-75-0-29bea6df-1b1a-458c-ad10-701fe44bb478 0/2 Terminating 0 23s
chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5 0/2 ContainerCreating 0 7s
kubectl logs chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5
Error from server (NotFound): pods "chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5" not found
$ kubectl logs chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5 --previous
Error from server (NotFound): pods "chrome-75-0-8929d8c8-1f7b-4eba-96f2-918f7a0d77f5" not found
Running kubectl logs -p will fetch logs from existing resources at API level. This means that terminated pods' logs will be unavailable using this command.
As mentioned in other answers, the best way is to have your logs centralized via logging agents or directly pushing these logs into an external service.
Alternatively and given the logging architecture in Kubernetes, you might be able to fetch the logs directly from the log-rotate files in the node hosting the pods. However, this option might depend on the Kubernetes implementation as log files might be deleted when the pod eviction is triggered.
From kubernetes docs:
Examples
# Return snapshot logs from pod nginx with only one container
kubectl logs nginx
# Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1
# Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1
# Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
# Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
Options
-c, --container="": Print the logs of this container
-f, --follow[=false]: Specify if the logs should be streamed.
--limit-bytes=0: Maximum bytes of logs to return. Defaults to no limit.
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.
--since=0: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.
--since-time="": Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.
--tail=-1: Lines of recent log file to display. Defaults to -1, showing all log lines.
--timestamps[=false]: Include timestamps on each line in the log output
That is just a simple way of doing it. But in production , I would send all the logs of all the pods to a centeral Log management system such as ELK by deploying a log sending client on the kubernetes cluster as daemon-set such as fluentbit , that will keep sending logs to ELk where I am able to filter things base don the namespace , pod , container , or any other label.
kubectl get event -o custom-columns=NAME:.metadata.name -n <namespace> --no-headers
use the above command to get the list of terminated pods in your namespace and use
kubectl logs -f pod-name -n <namespace> -p
to see the terminated pod's logs
P.S.: The above command to fetch terminated pod details will give you the pods which were terminated 1 hour ag
You can try --previous flag on the logs
i.e.
kubectl --namespace namespace logs pod_name --previous
This will show the logs from dump pod logs (stdout) for a previous container accroding to kubernetes docs
A combination of a flag --previous and a container name for a container that was terminated with reason: CrashLoopBackOff:
First find a pod in a namespace. Its status is CrashLoopBackOff:
kubectl get pods -n namespace_name
NAME READY STATUS RESTARTS AGE
crashing_pod_name 0/9 Init:CrashLoopBackOff 17 (105s ago) 63m
Then use describe to find out a name of a container that failed:
kubectl describe pod -n namespace_name crashing_pod_name
Find a name of a container that is terminated with a reason CrashLoopBackOff
Then list logs:
kubectl logs -n namespace_name crashing_pod_name -c failing_container_name --previous