When I call kubectl logs pod_name, I get both the stdout/err combined. Is it possible to specify that I only want stdout or stderr? Likewise I am wondering if it is possible to do so through the k8s rest interface. I've searched for several hours and read through the repository but could not find anything.
Thanks!
No, this is not possible. To my knowlegde, the moment of writing this, kubernetes supports only one logs api endpoint that returns all logs (stdout and stderr combined).
If you want to access them separately you should consider using different logging driver or query logs directly from docker.
Related
My service (tomcat/java) is running on a kubernetes cluster (AKS).
I would like to write the log files (tomcat access logs, application logs with logback) to an AzureFile volume.
I do not want to write the access logs to the stdout, because I do not want to mix the access logs with the application logs.
Question
I expect that all logging is done asynchronously, so that writing to the slow AzureFile volume should not affect the performance.
Is this correct?
Update
In the end I want to collect the logfiles so that I can send all logs to ElasticSearch.
Especially I need a way to collect the access logs.
If you want to send your access logs to Elastic Search, you just need to extend the AbstractAccessLogValve and implement the log method.
The AbstractAccessLogValve already contains the logic to format the messages, so you need just to add the logic to send the formatted message.
Yes, you are right but still here depends on how you are writing the logs. If asynchronously you are writing long process will take and your files system is slow. If it's NFS there is also the chance of network latency etc.
i have seen performance issues if attaching NFS & Bucket volume direct to multiple PODs.
If your writing is slow asyn thread might take time to complete job and take higher resources also however it still depends on code and way of written code.
Ideally, people use to store in Elasticsearch for fast retrieval easy management.
People use different stacks based on requirement but mostly all of them backed by elasticsearch for example Graylog, ELK.
For sending or writing logs to these stack people use the UDP I personally prefer GELF UDP and throws a logs at Graylog and forget.
Our setup:
We are using kubernetes in GCP.
We have pods that write logs to a shared volume, with a sidecar container that sucks up our logs for our logging system.
We cannot just use stdout instead for this process.
Some of these pods are long lived and are filling up disk space because of no log rotation.
Question:
What is the easiest way to prevent the disk space from filling up here (without scheduling pod restarts)?
I have been attempting to install logrotate using: RUN apt-get install -y logrotate in our Dockerfile and placing a logrotate config file in /etc/logrotate.d/dynamicproxy but it doesnt seem to get run. /var/lib/logrotate/status never gets generated.
I feel like I am barking up the wrong tree or missing something integral to getting this working. Any help would be appreciated.
We ended up writing our own daemonset to properly collect the logs from the nodes instead of the container level. We then stopped writing to shared volumes from the containers and logged to stdout only.
We used fluentd to the logs around.
https://github.com/splunk/splunk-connect-for-kubernetes/tree/master/helm-chart/splunk-kubernetes-logging
In general, you should write logs to stdout and configure log collection tool like ELK stack. This is the best practice.
However, if you want to run logrotate as a separate process in your container - you may use Supervisor, which serves as a very simple init system and allows you to run as many parallel process in container as you want.
Simple example for using Supervisor for rotating Nginx logs can be found here: https://github.com/misho-kr/docker-appliances/tree/master/nginx-nodejs
If you write to the filesystem the application creating the logs should be responsible for rotation. If you are running a java application with logback or log4j it is simple configuration change. For other languages/frameworks it is usually similar.
If that is not an option you could use a specialized tool to handle the rotation and piping the output to it. One example would be http://cr.yp.to/daemontools/multilog.html
As method of last resort you could investigate to log into a named pipe (FIFO) instead of a real file and have some other process handling the retrieval and writing of the data - including the rotation.
I have a python application that is running in Kubernetes. The app has a ping health-check which is called frequently via a REST call and checks that the call returns an HTTP 200. This clutters the Kubernetes logs when I view it through the logs console.
The function definition looks like this:
def ping():
return jsonify({'status': 'pong'})
How can I silence a specific call from showing up in the log? Is there a way I can put this in code such as a python decorator on top of the health check function? Or is there an option in the Kubernetes console where I can configure to ignore this call?
In kubernetes, everything in container which you have on stdout or stderr will come into the kubernetes logs. The only way to exclude the logs of health-check ping call remove from kubernetes logs is that, In your application you should redirect output of those ping calls to somefile in say /var/log/. This will effectively remove the output of that health-check ping from the stdout.
Once the output is not in stdout or stderr of the pods, pod logs will not have logs from that special health-check
You can also use the sidecar containers to streamline your application logs, like if you don't want all of the logs of application in kubectl logs output. You can write those file.
As stated in Official docs of kubernetes:
By having your sidecar containers stream to their own stdout and stderr streams, you can take advantage of the kubelet and the logging agent that already run on each node. The sidecar containers read logs from a file, a socket, or the journald. Each individual sidecar container prints log to its own stdout or stderr stream.
This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead.
For more information on Kubernetes logging, please refer official docs:
https://kubernetes.io/docs/concepts/cluster-administration/logging/
clutters the Kubernetes logs when I view it through the logs console.
Since you said "view", I know it might not be the most accurate way to do this, but works ok:
kubectl logs [pod] [options] | awk '!/some health-check pattern|or another annoying pattern to exclude|another pattern you dont want to see|and so on/ {print $0}'
Used awk for filtering the logs, if you're unaware, you can read about it here
!: means exclude the following pattern
/pattern/: enclose your pattern between // (standard awk syntax)
{print $0}: means print the current line (you can also use just print)
Note: There is always a risk of excluding something necessary. Use wisely.
Just invert the logic log on fail in the app, modify the code or wrap with a custom decorator
Is there any documentation out there on sending logs from containers in K8s to an external ELK cluster running on EC2 instances?
We're in the process of trying to Kubernetes set up and I'm trying to figure out how to get the logging to work correctly. We already have an ELK stack setup on EC2 for current versions of the application but most of the documentation out there seems to be referring to ELK as it's deployed to the K8s cluster.
I am also working on the same cause.
First you should know what driver is being used by your docker containers to manage the logs (json driver/ journald etc - read here).
After that you should use some log collector in your architecture to send the logs to the Logstash endpoint. You can use filebeat/fluent bit. They are light weight alternatives to logstash/fluentd respectively. You must use one of them and not directly send your logs to logstash via syslog since these log shippers have a special functionality of enriching your logs with kubernetes metadata of the respective containers.
There might be lot of challenges after that. Parsing log data (multiline logs for example) etc. For an efficient pipeline, it’s better to do most of the work (i.e. extracting the date object from the logs etc) at the log sender side, than using the common logstash for this purpose that might be a bottle-neck.
Note that in case the container logs are not sent to stdout/stderr but written else-where, you might need to run filebeat/fluent-bit as side-car with your containers.
As for the links for documentation are concerned, I myself didn’t find anything documented in a single place on this, but the keywords that I mentioned over, reading about them I got to know many things.
Hope this helps.
Before looking at Kubernetes, we are writing all our logs to stdout(according to 12-factor-app) and using logspout to collect the logs to Logstash. And in Logstash we then route logs to different targets:
InfluxDB+Grafana: to monitor application metrics(e.g., how long does a certain calculation takes)
Riemann: to alert if some performance thresholds are crossed
How these things can be done in Kubernetes?
I know that with Heapster you can see JVM level graphs(memory usages, etc) or even maybe Heapster can send events to Riemann in order to alert some system level statistics(e.g., disk is full). But for stuff on the application level, what would be the right approach then?
Heapster should be grabbing the stdout from the containers as well and can send the data to different backends (sinks). It would essentially be an API call with the data. Check out: https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md
I'm not 100% sure on stdout being the only method for a 12fa, but we use a in-house logging lib that also streams the stdout to our logging engine (graylog). That happens inside the app so that the log messages are preserved as a full 'event' vs heapster or other stdout scrapings treating each line as an event.