I want to know the share of resource - cpu, memory- per kubernetes's pod.
And I want to know what the standard of share is
This is hard to do using kubectl only (or I don't know how). What we usually do is to use the kubelet metric-server to export all metric to prometheus. We then use Grafana to calculate those values. The following metrics should allow you to calculate your values:
CPU cores:
kube_node_status_allocatable_cpu_cores - available cores
kube_pod_container_resource_requests_cpu_cores - requested cores per container
container_cpu_usage_seconds_total - used cores per container
Memory:
kube_node_status_allocatable_memory_bytes - available memory
kube_pod_container_resource_requests_memory_bytes - requested memory by container
container_memory_usage_bytes - used memory by container
You can filter those by label (i.e. by pod name or namespace) and calculate all kinds of things based on them.
Try kubectl top and friends. More words because SO requires them.
You can use metrics server which is the Cluster-wide aggregator of resource usage data. It is the source of container resource metrics for Kubernetes built-in autoscaling pipelines. It collects resource metrics from Kubelets and exposes them in Kubernetes apiserver. You can also access the metrics API by kubectl top.
Another solution is using Prometheus as suggested by #jankantert.
Related
I have:
deployments of services A and B in k8s
Prometheus stack
I wanna scale service A when metric m1 of service B is changed.
Solutions which I found and not suitable more or less:
I can define HPA for service A with the following part of spec:
- type: Object
object:
metric:
name: m1
describedObject:
apiVersion: api/v1
kind: Pod
name: certain-pod-of-service-B
current:
value: 10k
Technically, it will work. But it's not suitable for dynamic nature of k8s.
Also I can't use pods metric (metrics: - type: Pods pods:) in HPA cause it will request m1 metric for pods of service A (which obviously doesn't have this)
Define custom metric in prometheus-adapter which query m1 metric from pods of service B. It's more suitable, but looks like workaround cause I already have a metric m1
The same for external metrics
I feel that I miss something cause it doesn't seem like a non realistic case :)
So, advise me please how to scale one service by metric of another in k8s?
I decided to provide a Community Wiki answer that may help other people facing a similar issue.
The Horizontal Pod Autoscaler is a Kubernetes feature that allows to scale applications based on one or more monitored metrics.
As we can find in the Horizontal Pod Autoscaler documentation:
The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).
There are three groups of metrics that we can use with the Horizontal Pod Autoscaler:
resource metrics: predefined resource usage metrics (CPU and
memory) of pods and nodes.
custom metrics: custom metrics associated with a Kubernetes
object.
external metrics: custom metrics not associated with a
Kubernetes object.
Any HPA target can be scaled based on the resource usage of the pods (or containers) in the scaling target. The CPU utilization metric is a resource metric, you can specify other resource metrics besides CPU (e.g. memory). This seems to be the easiest and most basic method of scaling, but we can use more specific metrics by using custom metrics or external metrics.
There is one major difference between custom metrics and external metrics (see: Custom and external metrics for autoscaling workloads):
Custom metrics and external metrics differ from each other:
A custom metric is reported from your application running in Kubernetes.
An external metric is reported from an application or service not running on your cluster, but whose performance impacts your Kubernetes application.
All in all, in my opinion it is okay to use custom metrics in the case above,
I did not find any other suitable way to accomplish this task.
I'm working on memory monitoring using Prometheus (prometheus-operator Helm chart).
While investigating values I've noticed that memory usage (container_memory_working_set_bytes ) is being scraped from two endpoints:
/metrics/cadvisor
/metrics/resource/v1alpha1 (/metrics/resource from kubernetes 1.18)
I've figured out how to disable one of the endpoints in the chart but I'd like to understand the purpose of both.
I understand that /metrics/cadvisor returns three values - pod's container (or more if a pod has multiple containers), some special container POD (is it some internal memory usage to run a POD service?) and a sum of all containers (then the result has empty label container="").
On the other hand /metrics/resource/v1alpha1 returns only memory usage of a pod's containers (without container="POD" and without sum of these container="")
Is /metrics/resource/v1alpha1 then planned to replace /metrics/cadvisor as a single source of metrics?
Seeing that both endpoints (both are enabled by default in prometheus-operator) return the same metrics any sum() queries can return values 2 as big as a real memory usage.
Appreciate any clarification in this subject!
Answer is partial
I understand that /metrics/cadvisor returns three values - pod's
container (or more if a pod has multiple containers), some special
container POD (is it some internal memory usage to run a POD service?)
and a sum of all containers (then the result has empty label
container="").
container_name=="POD" is the "pause" container for the pods. The pause container is a container which holds the network namespace for the pod. Kubernetes creates pause containers to acquire the respective pod’s IP address and set up the network namespace for all other containers that join that pod. This container is a part of whole ecosystem and it starts first in pods to configure PODs network in the first place prior to scheduling another pods. After pod has been started - there is nothing to do for pause container.
Pause container code for your reference: https://github.com/kubernetes/kubernetes/tree/master/build/pause
Example of pause containers:
docker ps |grep pause
k8s_POD_etcd-master-1_kube-system_ea5105896423fc919bf9bfc0ab339888_0
k8s_POD_kube-scheduler-master-1_kube-system_155707e0c19147c8dc5e997f089c0ad1_0
k8s_POD_kube-apiserver-master-1_kube-system_fe660a7e8840003352195a8c40a01ef8_0
k8s_POD_kube-controller-manager-master-1_kube-system_807045fe48b23a157f7fe1ef20001ba0_0
k8s_POD_kube-proxy-76g9l_kube-system_e2348a94-eb96-4630-86b2-1912a9ce3a0f_0
k8s_POD_kube-flannel-ds-amd64-76749_kube-system_bf441436-bca3-4b49-b6fb-9e031ef7513d_0
container_name!=="POD" It filters out metric streams for the pause container, not metadata generally. Most people, if they want to graph the containers in their pod, don't want to see resource usage for the pause container, as it doesn't do much. The name of the pause container is an implementation detail of some container runtimes, but doesn't apply to all, and isn't guaranteed to stick around.
Official (obsolete v1.14) page shows differences between cadvisor and metrics resource monitoring:
Kubelet
The Kubelet acts as a bridge between the Kubernetes master and
the nodes. It manages the pods and containers running on a machine.
Kubelet translates each pod into its constituent containers and
fetches individual container usage statistics from the container
runtime, through the container runtime interface. For the legacy
docker integration, it fetches this information from cAdvisor. It then
exposes the aggregated pod resource usage statistics through the
kubelet resource metrics api. This api is served at
/metrics/resource/v1alpha1 on the kubelet’s authenticated and
read-only ports.
cAdvisor
cAdvisor is an open source container resource usage and
performance analysis agent. It is purpose-built for containers and
supports Docker containers natively. In Kubernetes, cAdvisor is
integrated into the Kubelet binary. cAdvisor auto-discovers all
containers in the machine and collects CPU, memory, filesystem, and
network usage statistics. cAdvisor also provides the overall machine
usage by analyzing the ‘root’ container on the machine.
Also you should know that kubelet exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Those 3 metrics do not have same lifecycle.
As per helm prometheus values yaml - there are 3 options and you can disable what you dont need
## Enable scraping /metrics/cadvisor from kubelet's service
##
cAdvisor: true
## Enable scraping /metrics/probes from kubelet's service
##
probes: true
## Enable scraping /metrics/resource from kubelet's service
##
resource: true
# From kubernetes 1.18, /metrics/resource/v1alpha1 renamed to /metrics/resource
resourcePath: "/metrics/resource/v1alpha1"
My opinion /metrics/resource/ wont replace google's cadvisor. Just disable in your case what you dont need. It just depends on your needs. For example, I found an article Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles where 4 diff tools being used to monitor everything.
metrics-server – CPU, memory, file-descriptors, disks, etc of the cluster
cAdvisor – a Docker daemon metrics – containers monitoring
kube-state-metrics – deployments, pods, nodes
node-exporter: EC2 instances metrics – CPU, memory, network
In your case, to monitor memory i believe it will be enough 1 :)
I've read some pages about monitoring k8s, and I found kubernetes_sd_config (within prometheus), metrics-server (took the place of heapster) and kube-state-metrics. All of them could provides metrics, but what's the difference?
Does kubernetes_sd_config (within prometheus) provide all the data those I can get using metrics-server and kube-state-metrics?
Is kubernetes_sd_config just enough for monitoring?
Is metrics-server just for providing data (less than kubernetes_sd_config) to the internal components(such as hpa controller)?
Is kube-state-metrics just for the objects (pod, deployment...) in k8s?
what is their own target respectively?
1 Metrics-server is a cluster level component which periodically scrapes container CPU and memory usage metrics from all Kubernetes nodes served by Kubelet through Summary API.
The Kubelet exports a "summary" API that aggregates stats from all pods.
$ kubectl proxy &
Starting to serve on 127.0.0.1:8001
$ NODE=$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")
$ curl localhost:8001/api/v1/proxy/nodes/${NODE}:10255/stats/summary
Use-Cases:
Horizontal Pod Autoscaler.
kubectl top --help: command.
2 kube-state-metrics
is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it
Use-Cases
count the number of k8s Objects.
How many namespaces are there ?
sysdig-k8s-state-metrics provide the further Information.
3 Prometheus Node_Exporter − Gets the host level matrices and exposes them to Prometheus.
Use-Cases
User and Kernel Space level information.
Lastly, kubernetes_sd_config is the configuration file defines everything related to scraping targets.
You can decide in the config file what kind of information you want to gather and from whom.
I want to monitor the current/target CPU utilization at the deployment/HPA level using Prometheus. GCP Kubernetes monitoring has these metrics available on Stackdriver dashboard but could not find them how they are tracking it.
Following links contains the list of HPA metrics exposed, which does not have the required/target CPU utilization.
https://github.com/kubernetes/kube-state-metrics/blob/1dfe6681e9/docs/horizontalpodautoscaler-metrics.md
I think you can take a look at cAdvisor. Actually, cAdvisor is a part of kubelet service and represents itself as a monitoring agent for performance and resource usage by the containers within particular node. By default, cAdvisor exposes container statistics across Prometheus metrics which are available in /metrics endpoint for each container. I guess you can use container_cpu_load_average_10s metric in order to fetch CPU utilization per each container for relevant Pod/Deployment.
I defined my deployment resources
resources:
limits:
cpu: 900m
memory: 2500Mi
now on http://localhost:8001/api
how can I get the max usage of memory and cpu (in order to handle and define usages and resources well)?
Usually, you will need to implement some monitoring solution for your K8s cluster to store historical metrics.
If your Kubernetes deployment runs on GKE, you can use Stackdriver
for that and if you have opted for Stackdriver Premium, you will see your historical metrics there. If it's your own Kubernetes deployment, Prometheus/Grafana is a popular choice
By default the pods run with unbounded CPU and memory limits. This means that any pod in the system will be able to consume as much CPU and memory as is on the node that executes the pod.
Reference:
https://kubernetes.io/docs/tasks/administer-cluster/cpu-memory-limit/
Some tools like kubectl top can only get the resource usage of your pod in current time. You need a monitor service.
kube-state-metrics -> prometheus -> grafana-kubernetes-app is a popular solution to monitor metrics of self deployed k8s cluster.
kube-state-metrics can generates metrics about the state of k8s objects, such as node, pod, deployment.
you just need to trace metrics kube_pod_container_resource_requests_cpu_cores and kube_pod_container_resource_requests_memory_bytes as the document specified.
peometheus should collect metrics from kube-state-metrics every few seconds and generate time series data.
grafana-kubernetes-app charts them easily. You can find the maximum cpu usage of a given pod then.
if you are running on GKE
for now day, the kubernetes console has advanced and you can see all statistics on the console ui
go to https://console.cloud.google.com/kubernetes/list
under workloads choose the deployment
A little late to the party, but I created Kube Eagle for exactly this purpose: https://github.com/google-cloud-tools/kube-eagle