I am using a WordPress site build with Kubernetes deployment. In the deployment I have only one container pod. Could you please clarify the possible reasons on the logs are not getting via kubectl logs command. I have tried docker logs on the container also, which is also showing the same output.
#~$ kubectl logs -f <pod_name> -n <namespace>
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using xx.xx.xx.xx. Set the 'ServerName' directive globally to suppress this message
This is the only content in logs even after accessing the domain.
# docker logs -f <k8s_container_name>
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using xx.xx.xx.xx. Set the 'ServerName' directive globally to suppress this message
kubectl logs show the default STDOUT and STDERR
You need to write the file_put_contents('php://stderr', 'Your text goes to STDERR',FILE_APPEND);
at least append in file and you read from there or else if you developing php WordPress you can push logs to external logging also.
Read more at : https://www.php.net/manual/en/features.commandline.io-streams.php & https://docs.docker.com/config/containers/logging/
Related
kubectl logs -f <pod-name>
This command shows the logs from the container log file.
Basically, I want to check the difference between "what is generated by the container" and "what is written to the log file".
I see some unusual binary logs, so I just want to find out if the container is creating those binary logs or the logs are not properly getting written to the log file.
"Unusual logs":
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\
Usually, containerized applications, do not write to the log files but send messages to stdout/stderr, there is no point in storing log files inside containers, as they will be deleted when the pod is deleted.
What you see when running
kubectl logs -f <pod-name>
are messages sent to stdout/stderr. There are no container specific logs here, only application logs.
If, for some reason, your application does write to the log file, you can check it by execing into pod with e.g.
kubectl exec -it <pod-name> -- /bin/bash
and read logs as you would in shell.
Edit
Application logs
A container engine handles and redirects any output generated to a containerized application's stdout and stderr streams. For example, the Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in JSON format.
Those logs are also saved to
/var/log/containers/
/var/log/pods/
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs.
Everything you see by issuing the command
kubectl logs <pod-name>
is what application sent to stdout/stderr, or what was redirected to stdout/stderr. For example nginx:
The official nginx image creates a symbolic link from /var/log/nginx/access.log to /dev/stdout, and creates another symbolic link from /var/log/nginx/error.log to /dev/stderr, overwriting the log files and causing logs to be sent to the relevant special device instead.
Node logs
Components that do not run inside containers (e.g kubelet, container runtime) write to journald. Otherwise, they write to .log fies inside /var/log/ directory.
Excerpt from official documentation:
For now, digging deeper into the cluster requires logging into the relevant machines. Here are the locations of the relevant log files. (note that on systemd-based systems, you may need to use journalctl instead)
Master
/var/log/kube-apiserver.log - API Server, responsible for serving the API
/var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions
/var/log/kube-controller-manager.log - Controller that manages replication controllers
Worker Nodes
/var/log/kubelet.log - Kubelet, responsible for running containers on the node
/var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing
The only way I could imagine this to work is to make use of some external logging facility like Syslog or Elastisearch or anything else. Configure your application to send logs directly to logging facility (avoiding agents like fluentd or logstash which parse logs from files).
All modern languages have support for external logging. You can also configure Docker to send logs to syslog server.
Simple way to check log is kubernets:
=> If pod have single container
kubectl logs POD_NAME
=> If pod have multiple containers
kubectl logs POD_NAME -c CONTAINER_NAME -n NAMESPACE
I am trying to find out in which user's context do the containers run and looking for a way to get this data without exec into each container. Since I am working on a cluster with many pods, it is nearly impossible to go inside each container and get the user.
Is there a command that I can use for this?
You can use Falco for this.Falco communicates with the provided K8s API server to decorate events with the K8s pod/namespace/deployment/etc. associated with the event. Below is an example of Falco alerts from here. You can see user=root
output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline parent=%proc.pname %container.info)"
$ falco
15:42:35.347416068: Warning Namespace change (setns) by unexpected program (user=root command=test_program parent=hyperkube k8s-kubelet (id=4a4021c50439))
$ falco -pk -k <k8s api server url>
15:42:35.347416068: Warning Namespace change (setns) by unexpected program (user=root command=test_program parent=hyperkube k8s.pod=jclient-3160134038-qqaaz container=4a4021c50439)
$ falco -p "This is Some Extra" -k <k8s api server url>
15:42:35.347416068: Warning Namespace change (setns) by unexpected program (user=root command=test_program parent=hyperkube k8s-kubelet (id=4a4021c50439)) This is Some Extra
Hi i have a server configured with kubernetes (without using minikube), i can execute kubectl commands without problems, like kubectl get all, kubectl delete pod, kubectl delete apply ...
I would want to know how to allow another user from my server to execute kubectl commands, because if i change to another user and i try to execute kubectl get all -s localhost:8443 i get:
Error from server (BadRequest): the server rejected our request for an unknown reason
I have read the Kubernetes Authorization Documentation, but im not sure if it is what im looking for.
This is happening because there is no kubeconfig file for the user.You need to have the same kubeconfig file for the other user either in the default location $HOME/.kube/config or in any location pointed by KUBECONFIG environment variable.
You can copy the existing kubeconfig file for the working user to the above location for the non working user.
I am using the following command to check logs in Kubernetes.
kubectl logs pod_name -n namespace
It is printing all the logs from the beginning.
Is there anyway to tail the logs or check logs between the given window?
Is it possible to rotate docker logs based on size or date?
Yes, we can extract the log by using the since like below -
kubectl logs --since=48h podname > 24Logs.txt
Then you can easily check the logs for specific time within last 48 hours.
1: yes, you can tail or filter by date.
As easy as running kubectl logs --help
Options:
-c, --container='': Print the logs of this container
-f, --follow=false: Specify if the logs should be streamed.
--include-extended-apis=true: If true, include definitions of new APIs via calls to the API server. [default true]
--interactive=false: If true, prompt the user for input when required.
--limit-bytes=0: Maximum bytes of logs to return. Defaults to no limit.
--pod-running-timeout=20s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
pod is running
-p, --previous=false: If true, print the logs for the previous instance of the container in a pod if it exists.
-l, --selector='': Selector (label query) to filter on.
--since=0s: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of
since-time / since may be used.
--since-time='': Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time /
since may be used.
--tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise
10, if a selector is provided.
--timestamps=false: Include timestamps on each line in the log output
2: Docker stores the container logs in host in the path /var/lib/docker/containers/{ContainerId} so you could copy/truncate the logs directly.
That won't have any impact in the container or the pod.
Is there anyway to tail the logs or check logs between the given window?
To tail the logs, use the -foption
kubectl logs pod_name -n namespace -f
Is it possible to role docker logs based on size or date?
You can query logs x-lines ago or since a time range. Take a look at the —tail and —since options
kubectl logs [-f] [-p] POD [-c CONTAINER]
Examples
Return snapshot logs from pod nginx with only one container
kubectl logs nginx
Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1
Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1
Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
https://kubernetes-v1-4.github.io/docs/user-guide/kubectl/kubectl_logs/
The "tail" functionality of "kubectl logs" can be used with this convenient GUI frontend: https://retrospective.centeractive.com/blog_retrospective_5_0_0.html
The frontend leverages several functions of "kubectl", for example:
allows to filter the "tail" output in several ways ("check logs between the given window" from question #1)
allows the visual configuration of a group of Kuberenetes pods via labels. The log data the pods in a group can then be "multi-tailed" in a single view.
Disclosure: I helped in making this frontend.
kubectl logs pod_name --since=2m --timestamps
Is there any way to get hold of the log file of the pod in Kubernetes cluster?
I know I can fetch logs using "kubectl exec log -f $POD_NAME" command but I want to get access to log file directly.
It depends on the logging driver you're using
I'm assuming you're using the default json logging driver here, but you can see the node the pod is scheduled on by using kubectl get po -o wide
Then, logon to that node and you'll see the docker logs of the container under /var/lib/docker/containers/<long_container_id>/<long_container_id>-json.log
You will need to use docker ps and docker inspect to determine the long container id.
Run kubectl get pod <pod_name> -n <namespace> -o jsonpath='{.spec.nodeName}' to get the node this Pod is running on.
ssh into the node and you'll find the logs for the Pod at /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/.
The files within the /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory are symlinks to where your container runtime writes its container log files. So unlike jaxxstorm's answer, it doesn't matter which container runtime you're running.
I normally retrieve it from /var/log/containers where you will find all the containers' logs deployed on that particular machine