I have installed Heapster in my Kubernetes cluster.I can give resource usage from command line, for example kubectl top pods, and Kubernetes web panel.
I'm trying to get resource usage via web API from Heapster. Actually I'd like to get resource usage (e.g: ram and cpu) a node, pod or namespace from Web API.
There is a web api http://localhost:8001/swagger-2.0.0.json in Kubernetes but there isn't any API for resource usage or Heapster data.
Is there any way to get resource usage via web API in Kubernetes ?
thanks
Question has been answered in the above comment section.
In order to access the Node as well as Pod Metrics, It's better to use Metrics_server which is the successor of heapster.
The metrics server collects CPU and memory usage for nodes and pods by pooling data from Kubelet.
View nodes and pods metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods"
Related
I'm trying to find the sutiable expression to set alert that can calculate the Pods utlization that are reaching 80% for both memory and CPU on all the pods that are exisiting on namespace so i appreciate if can someone help me to achieve that
The Kubernetes ecosystem includes two complementary add-ons for aggregating and reporting valuable monitoring data from your cluster: Metrics Server and kube-state-metrics.
Metrics Server collects resource usage statistics from the kubelet on each node and provides aggregated metrics through the Metrics API. Metrics Server stores only near-real-time metrics in memory, so it is primarily valuable for spot checks of CPU or memory usage, or for periodic querying by a full-featured monitoring service that retains data over longer timespans.
kube-state-metrics is a service that makes cluster state information easily consumable. Whereas Metrics Server exposes metrics on resource usage by pods and nodes, kube-state-metrics listens to the Control Plane API server for data on the overall status of Kubernetes objects (nodes, pods, Deployments, etc.), as well as the resource limits and allocations for those objects. It then generates metrics from that data that are available through the Metrics API.
Once you have installed the same, you can use the following command to get the metrics
kubectl top pod <pod-name> -n <namespace> --containers
When trying to do HorizontalPodAutoscaling I'm getting (failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API) how can solve this problem.
As far as I understand before using hpa you have to install metrics-server. More in below docs and links.
Before you begin
This example requires a running Kubernetes cluster and kubectl, version 1.2 or later. metrics-server monitoring needs to be deployed in the cluster to provide metrics via the resource metrics API, as Horizontal Pod Autoscaler uses this API to collect metrics. The instructions for deploying this are on the GitHub repository of metrics-server, if you followed getting started on GCE guide, metrics-server monitoring will be turned-on by default.
Kubernetes metrics server
Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.
Additional links:
https://www.cloudtechnologyexperts.com/autoscaling-microservices-in-kubernetes-with-horizontal-pod-autoscaler/
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
https://www.youtube.com/watch?v=dALta9zQkfs
I've got some deployment on a basic k8s cluster withouth defining requests and limits.
Is there any way to check how much the pod is asking for memory and cpu?
Depending on whether the metrics-server is installed in your cluster, you can use:
kubectl top pod
kubectl top node
After installing the Metrics Server, you can query the Resource Metrics API directly for the resource usages of pods and nodes:
All nodes in the cluster:
kubectl get --raw=/apis/metrics.k8s.io/v1beta1/nodes
A specific node:
kubectl get --raw=/apis/metrics.k8s.io/v1beta1/nodes/{node}
All pods in the cluster:
kubectl get --raw=/apis/metrics.k8s.io/v1beta1/pods
All pods in a specific namespace:
kubectl get --raw=/apis/metrics.k8s.io/v1beta1/namespaces/{namespace}/pods
A specific pod:
kubectl get --raw=/apis/metrics.k8s.io/v1beta1/namespaces/{namespace}/pods/{pod}
The API returns you the absolute CPU and memory usages of the pods and nodes.
From this, you should be able to figure out how much resources each pod consumes and how much free resources are left on each node.
I've read some pages about monitoring k8s, and I found kubernetes_sd_config (within prometheus), metrics-server (took the place of heapster) and kube-state-metrics. All of them could provides metrics, but what's the difference?
Does kubernetes_sd_config (within prometheus) provide all the data those I can get using metrics-server and kube-state-metrics?
Is kubernetes_sd_config just enough for monitoring?
Is metrics-server just for providing data (less than kubernetes_sd_config) to the internal components(such as hpa controller)?
Is kube-state-metrics just for the objects (pod, deployment...) in k8s?
what is their own target respectively?
1 Metrics-server is a cluster level component which periodically scrapes container CPU and memory usage metrics from all Kubernetes nodes served by Kubelet through Summary API.
The Kubelet exports a "summary" API that aggregates stats from all pods.
$ kubectl proxy &
Starting to serve on 127.0.0.1:8001
$ NODE=$(kubectl get nodes -o=jsonpath="{.items[0].metadata.name}")
$ curl localhost:8001/api/v1/proxy/nodes/${NODE}:10255/stats/summary
Use-Cases:
Horizontal Pod Autoscaler.
kubectl top --help: command.
2 kube-state-metrics
is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it
Use-Cases
count the number of k8s Objects.
How many namespaces are there ?
sysdig-k8s-state-metrics provide the further Information.
3 Prometheus Node_Exporter − Gets the host level matrices and exposes them to Prometheus.
Use-Cases
User and Kernel Space level information.
Lastly, kubernetes_sd_config is the configuration file defines everything related to scraping targets.
You can decide in the config file what kind of information you want to gather and from whom.
There are three levels of metrics collection to consider in Kubernetes - Node, Pod and the Application that runs in the pod.
For Node and Application metrics I have solutions that work wonderfully, but I am stuck on pod metrics.
I have tried cAdvisor and Kube state metrics but none of them give me what I want. Kube state metrics only gives information that is already known like pod CPU limits and requests. cAdvisor doesn't insert pod labels to container names so I have no means of knowing which pod is misbehaving.
Given a pod, I'd like to know it's CPU, memory and storage usage both with respect to the pod itself and also with respect to the node it is scheduled on.
I am using prometheus to collect metrics via the prometheus operator CRD.
Can anyone help suggest an open source metrics exporter that would do the job I mentioned above?
The standard metric collector is Heapster. It comes preinstalled in many vendors like GKE also. With Heapster installed, you can just do kubectl top pods to see cpu/mem metrics on the client side. You can plug it with some sink to store the results for archival.
https://github.com/kubernetes/heapster