There are three levels of metrics collection to consider in Kubernetes - Node, Pod and the Application that runs in the pod.
For Node and Application metrics I have solutions that work wonderfully, but I am stuck on pod metrics.
I have tried cAdvisor and Kube state metrics but none of them give me what I want. Kube state metrics only gives information that is already known like pod CPU limits and requests. cAdvisor doesn't insert pod labels to container names so I have no means of knowing which pod is misbehaving.
Given a pod, I'd like to know it's CPU, memory and storage usage both with respect to the pod itself and also with respect to the node it is scheduled on.
I am using prometheus to collect metrics via the prometheus operator CRD.
Can anyone help suggest an open source metrics exporter that would do the job I mentioned above?
The standard metric collector is Heapster. It comes preinstalled in many vendors like GKE also. With Heapster installed, you can just do kubectl top pods to see cpu/mem metrics on the client side. You can plug it with some sink to store the results for archival.
https://github.com/kubernetes/heapster
Related
I'm trying to find the sutiable expression to set alert that can calculate the Pods utlization that are reaching 80% for both memory and CPU on all the pods that are exisiting on namespace so i appreciate if can someone help me to achieve that
The Kubernetes ecosystem includes two complementary add-ons for aggregating and reporting valuable monitoring data from your cluster: Metrics Server and kube-state-metrics.
Metrics Server collects resource usage statistics from the kubelet on each node and provides aggregated metrics through the Metrics API. Metrics Server stores only near-real-time metrics in memory, so it is primarily valuable for spot checks of CPU or memory usage, or for periodic querying by a full-featured monitoring service that retains data over longer timespans.
kube-state-metrics is a service that makes cluster state information easily consumable. Whereas Metrics Server exposes metrics on resource usage by pods and nodes, kube-state-metrics listens to the Control Plane API server for data on the overall status of Kubernetes objects (nodes, pods, Deployments, etc.), as well as the resource limits and allocations for those objects. It then generates metrics from that data that are available through the Metrics API.
Once you have installed the same, you can use the following command to get the metrics
kubectl top pod <pod-name> -n <namespace> --containers
I like to see how my services will work on kubernethes so I can optimize my code and set good values for request/limit on both CPU and memory.
To do that I have tried kubectl top bit ot only gives me the current usage.
kubectl top pod podname
How do I get the init, min and max usage?
If it is not possible to get all those values, is there any way to get max usage?
In order to see stats you may want to use one of these monitoring tools:
cAdvisor
Container Advisor is a great monitoring tool that provides
container-level metrics and exposes resource usage and performance
data from running containers. It provides quick insight into CPU
usage, memory usage, and network receive/transmit of running
containers. cAdvisor is embedded into the kubelet, hence you can
scrape the kubelet to get container metrics, store the data in a
persistent time-series store like Prometheus/InfluxDB, and then
visualize it via Grafana.
Metrics Server
Metrics Server is a cluster-wide aggregator of resource usage data and
collects basic metrics like CPU and memory usage for Kubernetes nodes,
pods, and containers. It’s used by Horizontal Pod Autoscaler and the
Kubernetes dashboard itself, and users can access these metrics
directly by using the kubectl top command. Metrics Server replaces
Heapster as the primary metrics aggregator in the cluster, which has
been marked as deprecated in the newer version of Kubernetes.
Node Exporter
Node Exporter is the Prometheus exporter for hardware and operating
system metrics. It allows you to monitor node-level metrics such as
CPU, memory, filesystem space, network traffic, and other monitoring
metrics, which Prometheus scraps from a running node exporter
instance. You can then visualize these metrics in Grafana.
Kube-State-Metrics
Kube-state-metrics is an add-on agent that listens to the Kubernetes
API server. It generates metrics about the state of the Kubernetes
objects inside the cluster like deployments, replica sets, nodes, and
pods.
Metrics generated by kube-state-metrics are different from resource
utilization metrics, which are primarily geared more towards CPU,
memory, and network usage. Kube-state-metrics expose critical metrics
about the condition of your Kubernetes cluster:
Resource requests and limits
Number of objects–nodes, pods, namespaces, services, deployments
Number of pods in a running/terminated/failed state
Prometheus
Prometheus is a free software application used for event monitoring
and alerting. It records real-time metrics in a time series database
built using a HTTP pull model, with flexible queries and real-time
alerting
You can visualize Prometheus monitoring data with Grafana
and its dashboard collection.
You can find detailed Monitor Your Kubernetes Cluster With Prometheus and Grafana instruction how to use them together
I'm working on memory monitoring using Prometheus (prometheus-operator Helm chart).
While investigating values I've noticed that memory usage (container_memory_working_set_bytes ) is being scraped from two endpoints:
/metrics/cadvisor
/metrics/resource/v1alpha1 (/metrics/resource from kubernetes 1.18)
I've figured out how to disable one of the endpoints in the chart but I'd like to understand the purpose of both.
I understand that /metrics/cadvisor returns three values - pod's container (or more if a pod has multiple containers), some special container POD (is it some internal memory usage to run a POD service?) and a sum of all containers (then the result has empty label container="").
On the other hand /metrics/resource/v1alpha1 returns only memory usage of a pod's containers (without container="POD" and without sum of these container="")
Is /metrics/resource/v1alpha1 then planned to replace /metrics/cadvisor as a single source of metrics?
Seeing that both endpoints (both are enabled by default in prometheus-operator) return the same metrics any sum() queries can return values 2 as big as a real memory usage.
Appreciate any clarification in this subject!
Answer is partial
I understand that /metrics/cadvisor returns three values - pod's
container (or more if a pod has multiple containers), some special
container POD (is it some internal memory usage to run a POD service?)
and a sum of all containers (then the result has empty label
container="").
container_name=="POD" is the "pause" container for the pods. The pause container is a container which holds the network namespace for the pod. Kubernetes creates pause containers to acquire the respective pod’s IP address and set up the network namespace for all other containers that join that pod. This container is a part of whole ecosystem and it starts first in pods to configure PODs network in the first place prior to scheduling another pods. After pod has been started - there is nothing to do for pause container.
Pause container code for your reference: https://github.com/kubernetes/kubernetes/tree/master/build/pause
Example of pause containers:
docker ps |grep pause
k8s_POD_etcd-master-1_kube-system_ea5105896423fc919bf9bfc0ab339888_0
k8s_POD_kube-scheduler-master-1_kube-system_155707e0c19147c8dc5e997f089c0ad1_0
k8s_POD_kube-apiserver-master-1_kube-system_fe660a7e8840003352195a8c40a01ef8_0
k8s_POD_kube-controller-manager-master-1_kube-system_807045fe48b23a157f7fe1ef20001ba0_0
k8s_POD_kube-proxy-76g9l_kube-system_e2348a94-eb96-4630-86b2-1912a9ce3a0f_0
k8s_POD_kube-flannel-ds-amd64-76749_kube-system_bf441436-bca3-4b49-b6fb-9e031ef7513d_0
container_name!=="POD" It filters out metric streams for the pause container, not metadata generally. Most people, if they want to graph the containers in their pod, don't want to see resource usage for the pause container, as it doesn't do much. The name of the pause container is an implementation detail of some container runtimes, but doesn't apply to all, and isn't guaranteed to stick around.
Official (obsolete v1.14) page shows differences between cadvisor and metrics resource monitoring:
Kubelet
The Kubelet acts as a bridge between the Kubernetes master and
the nodes. It manages the pods and containers running on a machine.
Kubelet translates each pod into its constituent containers and
fetches individual container usage statistics from the container
runtime, through the container runtime interface. For the legacy
docker integration, it fetches this information from cAdvisor. It then
exposes the aggregated pod resource usage statistics through the
kubelet resource metrics api. This api is served at
/metrics/resource/v1alpha1 on the kubelet’s authenticated and
read-only ports.
cAdvisor
cAdvisor is an open source container resource usage and
performance analysis agent. It is purpose-built for containers and
supports Docker containers natively. In Kubernetes, cAdvisor is
integrated into the Kubelet binary. cAdvisor auto-discovers all
containers in the machine and collects CPU, memory, filesystem, and
network usage statistics. cAdvisor also provides the overall machine
usage by analyzing the ‘root’ container on the machine.
Also you should know that kubelet exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Those 3 metrics do not have same lifecycle.
As per helm prometheus values yaml - there are 3 options and you can disable what you dont need
## Enable scraping /metrics/cadvisor from kubelet's service
##
cAdvisor: true
## Enable scraping /metrics/probes from kubelet's service
##
probes: true
## Enable scraping /metrics/resource from kubelet's service
##
resource: true
# From kubernetes 1.18, /metrics/resource/v1alpha1 renamed to /metrics/resource
resourcePath: "/metrics/resource/v1alpha1"
My opinion /metrics/resource/ wont replace google's cadvisor. Just disable in your case what you dont need. It just depends on your needs. For example, I found an article Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles where 4 diff tools being used to monitor everything.
metrics-server – CPU, memory, file-descriptors, disks, etc of the cluster
cAdvisor – a Docker daemon metrics – containers monitoring
kube-state-metrics – deployments, pods, nodes
node-exporter: EC2 instances metrics – CPU, memory, network
In your case, to monitor memory i believe it will be enough 1 :)
I've read some pages about monitoring k8s, and I found kubernetes_sd_config (within prometheus), metrics-server (took the place of heapster) and kube-state-metrics. All of them could provides metrics, but what's the difference?
Does kubernetes_sd_config (within prometheus) provide all the data those I can get using metrics-server and kube-state-metrics?
Is kubernetes_sd_config just enough for monitoring?
Is metrics-server just for providing data (less than kubernetes_sd_config) to the internal components(such as hpa controller)?
Is kube-state-metrics just for the objects (pod, deployment...) in k8s?
what is their own target respectively?
1 Metrics-server is a cluster level component which periodically scrapes container CPU and memory usage metrics from all Kubernetes nodes served by Kubelet through Summary API.
The Kubelet exports a "summary" API that aggregates stats from all pods.
$ kubectl proxy &
Starting to serve on 127.0.0.1:8001
$ NODE=$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")
$ curl localhost:8001/api/v1/proxy/nodes/${NODE}:10255/stats/summary
Use-Cases:
Horizontal Pod Autoscaler.
kubectl top --help: command.
2 kube-state-metrics
is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it
Use-Cases
count the number of k8s Objects.
How many namespaces are there ?
sysdig-k8s-state-metrics provide the further Information.
3 Prometheus Node_Exporter − Gets the host level matrices and exposes them to Prometheus.
Use-Cases
User and Kernel Space level information.
Lastly, kubernetes_sd_config is the configuration file defines everything related to scraping targets.
You can decide in the config file what kind of information you want to gather and from whom.
I deployed Prometheus and Grafana into my cluster.
When I open the dashboards I don't get data for pod CPU usage.
When I check Prometheus UI, it shows pods 0/0 up, however I have many pods running in my cluster.
What could be the reason? I have node exporter running in all of nodes.
Am getting this for kube-state-metrics,
I0218 14:52:42.595711 1 builder.go:112] Active collectors: configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets
I0218 14:52:42.595735 1 main.go:208] Starting metrics server: 0.0.0.0:8080
Here is my Prometheus config file:
https://gist.github.com/karthikeayan/41ab3dc4ed0c344bbab89ebcb1d33d16
I'm able to hit and get data for:
http://localhost:8080/api/v1/nodes/<my_worker_node>/proxy/metrics/cadvisor
As it was mentioned by karthikeayan in comments:
ok, i found something interesting in the values.yaml comments, prometheus.io/scrape: Only scrape pods that have a value of true, when i remove this relabel_config in k8s configmap, i got the data in prometheus ui.. unfortunately k8s configmap doesn't have comments, i believe helm will remove the comments before deploying it.
And just for clarification:
kube-state-metrics vs. metrics-server
The metrics-server is a project that has been inspired by Heapster and is implemented to serve the goals of the Kubernetes Monitoring Pipeline. It is a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Summary API. The metrics are aggregated, stored in memory and served in Metrics API format. The metric-server stores the latest values only and is not responsible for forwarding metrics to third-party destinations.
kube-state-metrics is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it. And just like the metric-server it too is not responsibile for exporting its metrics anywhere.
Having kube-state-metrics as a separate project also enables access to these metrics from monitoring systems such as Prometheus.