I have a scenario where I need to push application logs running on EKS Cluster to separate cloudwatch log streams. I have followed the below link, which pushes all logs to cloudwatch using fluentd. But the issue is, it pushes logs to a single log stream only.
https://github.com/aws-samples/aws-workshop-for-kubernetes
It also pushes all the logs under /var/lib/docker/container/*.log. How Can I filter this to can only application specific logs?
Collectord now supports AWS CloudWatch Logs (and S3/Athena/Glue). It gives you flexibility to choose to what LogGroup and LogStream you want to forward the data (if the default does work for you).
Installation instructions for CloudWatch
How you can specify LogGroup and LogStream with annotations
Highly recommend to read Setting up comprehensive centralized logging with AWS Services for Kubernetes
Related
I had a few pods restarting in my EKS cluster. I could see that they were SIGKILL'ed by K8s. Now I would like to know the reason but I can't because the Kubernetes events TTL is only one hour.
I am checking the control plane logs for the EKS cluster in CloudWatch now but don't know which of them contains these messages as well.
Which of the logs does contain these events form K8s?
Yes you are right, the default value of --event-ttl is 60m00s, and unfortunately, there is currently no any native option to change that value in EKS. The github issue is still opened without any promising timeframes.
As per guide you sent and as per Streaming EKS Metrics and Logs to CloudWatch, if you configured everything correctly, you can find logs under “Container Insights” from the drop-down menu.
Logs you might want to check are
Control plane logs consist of scheduler logs, API server logs, and
audit logs.
Data plane logs consist of kubelet and container runtime
engine logs.
Can you please specify what exact logs you have in your cloudwatch control plane logs and what you already checked? Maybe that will help
I currently have a Cronjob that has a job that schedule at some period of time and run in a pattern. I want to export the logs of each pod runs to a file in the path as temp/logs/FILENAME
with the FILENAME to be the timestamp of the run being created. How am I going to do that? Hopefully to provide a solution. If you would need to add a script, then please use python or shell command. Thank you.
According to Kubernetes Logging Architecture:
In a cluster, logs should have a separate storage and lifecycle
independent of nodes, pods, or containers. This concept is called
cluster-level logging.
Cluster-level logging architectures require a separate backend to
store, analyze, and query logs. Kubernetes does not provide a native
storage solution for log data. Instead, there are many logging
solutions that integrate with Kubernetes.
Which brings us to Cluster-level logging architectures:
While Kubernetes does not provide a native solution for cluster-level
logging, there are several common approaches you can consider. Here
are some options:
Use a node-level logging agent that runs on every node.
Include a dedicated sidecar container for logging in an application pod.
Push logs directly to a backend from within an application.
Kubernetes does not provide log aggregation of its own. Therefore, you need a local agent to gather the data and send it to the central log management. See some options below:
Fluentd
ELK Stack
You can find all logs that PODs are generating at /var/log/containers/*.log
on each Kubernetes node. You could work with them manually if you prefer, using simple scripts, but you will have to keep in mind that PODs can run on any node (if not restricted), and nodes may come and go.
Consider sending your logs to an external system like ElasticSearch or Grafana Loki and manage them there.
So there's this page about auditing-logs and I'm very confused about:
The k8s.io service is used for Kubernetes audit logs. These logs are generated by the Kubernetes API Server component and they contain information about actions performed using the Kubernetes API. For example, any changes you make on a Kubernetes resource by using the kubectl command are recorded by the k8s.io service. For more information, see Auditing in the Kubernetes documentation.
The container.googleapis.com service is used for GKE control plane audit logs. These logs are generated by the GKE internal components and they contain information about actions performed using the GKE API. For example, any changes you perform on a GKE cluster configuration using a gcloud command are recorded by the container.googleapis.com service.
which one shall I pick to get:
/var/log/kube-apiserver.log - API Server, responsible for serving the API
/var/log/kube-controller-manager.log - Controller that manages replication controllers
or these are all similar to EKS where audit logs means a separate thing?
Audit (audit) – Kubernetes audit logs provide a record of the individual users, administrators, or system components that have affected your cluster. For more information, see Auditing in the Kubernetes documentation.
If the cluster still exists, you should be able to do the following on GKE
kubectl proxy
curl http://localhost:8001/logs/kube-apiserver.log
AFAIK, there's no way to get server logs for clusters that have been deleted.
Logs for GKE control-plane components are available since November 29, 2022 for clusters with versions 1.22.0 and later.
You simply need to activate it on the clusters. Either
via CLI:
gcloud container clusters update [CLUSTER_NAME] \
--region=[REGION] \
--monitoring=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
via GCP web-console: Open the cluster-details, in the section "Features" edit the entry "Cloud Logging" and add the "Control Plane" components.
See documentation for details.
Note the notes in the solutions documentation, especially about reaching the logging.googleapis.com/write_requests quota (quick link).
You cannot. GKE does not make them available. Audit logs are different, those are a record of API actions.
in unmanaged cluster in order to export the k8s audit log we can use the AuditSink object and redirect the logs to any webhook we would like to . in order to do so we should changed the API server.
in managed cluster the API server is not accessible - is there any way to send the data to webhook as well?
if you can add an example it will be great since i saw the sub/pub option of GCP for example and it seems that i cant use my webhook
Within a managed GKE cluster, the audit logs are sent to Stackdriver Logging. At this time, there is no way to send the logs directly from GKE to a webhook; however, there is a workaround.
You can export the GKE Audit logs from Stackdriver Logging to Pub/Sub using a log sink. You will need to define which GKE Audit logs you will like to export to Pub/Sub.
Once the logs are exported to Pub/Sub, you will then be able to push them from Pub/Sub using your webhook. Cloud Pub/Sub is highly programmable and you can control the data you exchange. Please take a look at this link for an example about webhooks in Cloud Pub/Sub.
Is there some AWS command get to get logs from all services/tasks from an ECS cluster? something like:
aws ecs logs --cluster dev
or
aws ecs describe-clusters --cluster dev logs
for example, there must be some API to retrieve the logs that are shown in the UI here:
No there is no such out of the box option that takes logs from all service based on the cluster, as evey container running in their own space (EC2 instance).
So there can similar option that you can try, but before that, you need to understand the logging mechanism of AWS ECS.
logDriver
The log driver to use for the container. The valid values listed for
this parameter are log drivers that the Amazon ECS container agent can
communicate with by default.
For tasks using the Fargate launch type, the supported log drivers are awslogs, splunk, and awsfirelens.
For tasks using the EC2 launch type, the supported log drivers are awslogs, fluentd, gelf, json-file, journald, logentries, syslog, splunk, and awsfirelens.
So if you are running multiple container on same ec2 instance then syslog make sense for you.
Syslog logging driver
The syslog logging driver routes logs to a syslog server. The syslog
protocol uses a raw string as the log message and supports a limited
set of metadata. The syslog message must be formatted in a specific
way to be valid. From a valid message, the receiver can extract the
following information:
But the best approach is to have a single log group against each container. as syslog is not working in case of fargate so better to go with log group per container.