If I run, for example, kubectl logs --namespace kube-system kube-apiserver-XXXX | head -n 25 I can see the output with only a timestamp and no date. I can't tell if these are from the inception of the pod or not.
Generally speaking, how long do a pod's logs last in Kubernetes?
From k8s unofficial docs
Kubernetes performs log rotation daily, or if the log file grows beyond 10MB in size. Each rotation belongs to a single container; if the container repeatedly fails or the pod is evicted, all previous rotations for the container are lost. By default, Kubernetes keeps up to five logging rotations per container.
The container runtime attaches the current timestamp to every line produced by the application. You can display these timestamps by using the --timestamps=true option while running kubectl logs.
Related
I was wondering if there is a kubectl command to quickly get the history of all STATUS for a given pod?
for example: Lets say a pod - my-test-pod went from ContainerCreating to Running to OomKill to Terminating:
I was wondering if there is a command that experts use to get this lineage. Appreciate a nudge..
Using kubectl get events you can only see events of last 1 hour. If you want to persist events for a longer duration you can sue eventrouter.The event router serves as an active watcher of event resource in the kubernetes system, which takes those events and pushes them to a user specified sink. This is useful for a number of different purposes, but most notably long term behavioral analysis of your workloads running on your kubernetes cluster.
kubectl get events or kubectl describe pod which shows the events for the pod at the bottom. However events are only kept for a little while, so it's not a permanent history. For that you would need some webhooks or a tool like Prometheus.
Pods on our k8s cluster are scheduled with Airflow's KubernetesExecutor, which runs all Tasks in a new pod.
I have a such a Task for which the pod instantly (after 1 or 2 seconds) crashes, and for which of course I want to see the logs.
This seems hard. As soon the pod crashes, it gets deleted, along with the ability to retrieve crash logs. I already tried all of:
kubectl logs -f <pod> -p: cannot be used since these pods are named uniquely
(courtesy of KubernetesExecutor).
kubectl logs -l label_name=label_value: I
struggle to apply the labels to the pod (if this is a known/used way of working, I'm happy to try further)
An shared nfs is mounted on all pods on a fixed log directory. The failing pod however, does not log to this folder.
When I am really quick I run kubectl logs -f -l dag_id=sample_dag --all-containers (dag_idlabel is added byAirflow)
between running and crashing and see Error from server (BadRequest): container "base" in pod "my_pod" is waiting to start: ContainerCreating. This might give me some clue but:
these are only but the last log lines
this is really backwards
I'm basically looking for the canonical way of retrieving logs from transient pods
You need to enable remote logging. Code sample below is for using S3. In airflow.cfg set the following:
remote_logging = True
remote_log_conn_id = my_s3_conn
remote_base_log_folder = s3://airflow/logs
The my_s3_conn can be set in airflow>Admin>Connections. In the Conn Type dropdown, select S3.
(Before I start, I'm using minikube v27 on Windows 10.)
I have created a deployment with the nginx 'hello world' container with a desired count of 2:
I actually went into the '2 hours' old pod and edited the index.html file from the welcome message to "broken" - I want to play with k8s to seem what it would look like if one pod was 'faulty'.
If I scale this deployment up to more instances and then scale down again, I almost expected k8s to remove the oldest pods, but it consistently removes the newest:
How do I make it remove the oldest pods first?
(Ideally, I'd like to be able to just say "redeploy everything as the exact same version/image/desired count in a rolling deployment" if that is possible)
Pod deletion preference is based on a ordered series of checks, defined in code here:
https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/controller/controller_utils.go#L737
Summarizing- precedence is given to delete pods:
that are unassigned to a node, vs assigned to a node
that are in pending or not running state, vs running
that are in not-ready, vs ready
that have been in ready state for fewer seconds
that have higher restart counts
that have newer vs older creation times
These checks are not directly configurable.
Given the rules, if you can make an old pod to be not ready, or cause an old pod to restart, it will be removed at scale down time before a newer pod that is ready and has not restarted.
There is discussion around use cases for the ability to control deletion priority, which mostly involve workloads that are a mix of job and service, here:
https://github.com/kubernetes/kubernetes/issues/45509
what about this :
kubectl scale deployment ingress-nginx-controller --replicas=2
Wait until 2 replicas are up.
kubectl delete pod ingress-nginx-controller-oldest-replica
kubectl scale deployment ingress-nginx-controller --replicas=1
I experienced zero downtime doing so while removing oldest pod.
I have been running a pod for more than a week and there has been no restart since started. But, still I am unable to view the logs since it started and it only gives logs for the last two days. Is there any log rotation policy for the container and how to control the rotation like based on size or date?
I tried the below command but shows only last two days logs.
kubectl logs POD_NAME --since=0
Is there any other way?
Is there any log rotation policy for the container and how to control the rotation like based on size or date
The log rotation is controlled by the docker --log-driver and --log-opts (or their daemon.json equivalent), which for any sane system has file size and file count limits to prevent a run-away service from blowing out the disk on the docker host. that answer also assumes you are using docker, but that's a fairly safe assumption
Is there any other way?
I strongly advise something like fluentd-elasticsearch, or graylog2, or Sumologic, or Splunk, or whatever in order to egress those logs from the hosts. No serious cluster would rely on infinite log disks nor on using kubectl logs in a for loop to search the output of Pods. To say nothing of egressing the logs from the kubernetes containers themselves, which is almost essential for keeping tabs on the health of the cluster.
I am using the following command to check logs in Kubernetes.
kubectl logs pod_name -n namespace
It is printing all the logs from the beginning.
Is there anyway to tail the logs or check logs between the given window?
Is it possible to rotate docker logs based on size or date?
Yes, we can extract the log by using the since like below -
kubectl logs --since=48h podname > 24Logs.txt
Then you can easily check the logs for specific time within last 48 hours.
1: yes, you can tail or filter by date.
As easy as running kubectl logs --help
Options:
-c, --container='': Print the logs of this container
-f, --follow=false: Specify if the logs should be streamed.
--include-extended-apis=true: If true, include definitions of new APIs via calls to the API server. [default true]
--interactive=false: If true, prompt the user for input when required.
--limit-bytes=0: Maximum bytes of logs to return. Defaults to no limit.
--pod-running-timeout=20s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
pod is running
-p, --previous=false: If true, print the logs for the previous instance of the container in a pod if it exists.
-l, --selector='': Selector (label query) to filter on.
--since=0s: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of
since-time / since may be used.
--since-time='': Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time /
since may be used.
--tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise
10, if a selector is provided.
--timestamps=false: Include timestamps on each line in the log output
2: Docker stores the container logs in host in the path /var/lib/docker/containers/{ContainerId} so you could copy/truncate the logs directly.
That won't have any impact in the container or the pod.
Is there anyway to tail the logs or check logs between the given window?
To tail the logs, use the -foption
kubectl logs pod_name -n namespace -f
Is it possible to role docker logs based on size or date?
You can query logs x-lines ago or since a time range. Take a look at the —tail and —since options
kubectl logs [-f] [-p] POD [-c CONTAINER]
Examples
Return snapshot logs from pod nginx with only one container
kubectl logs nginx
Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1
Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1
Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
https://kubernetes-v1-4.github.io/docs/user-guide/kubectl/kubectl_logs/
The "tail" functionality of "kubectl logs" can be used with this convenient GUI frontend: https://retrospective.centeractive.com/blog_retrospective_5_0_0.html
The frontend leverages several functions of "kubectl", for example:
allows to filter the "tail" output in several ways ("check logs between the given window" from question #1)
allows the visual configuration of a group of Kuberenetes pods via labels. The log data the pods in a group can then be "multi-tailed" in a single view.
Disclosure: I helped in making this frontend.
kubectl logs pod_name --since=2m --timestamps