Are there any metrics I can use to know if pods are in the running state or errored-out, crashloopbackoff state etc in GKE Google Cloud?
Basically I want a metric I can export to Stackdriver that can tell if my jobs are running healthy pods or pods have errors and no pods are running( Evicted, crashloopbackoff etc. )
According to the official documentation Cloud Monitoring supports the following metric types from Google Kubernetes Engine:
Kubernetes metrics
I believe you can use for your case:
Related
When trying to do HorizontalPodAutoscaling I'm getting (failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API) how can solve this problem.
As far as I understand before using hpa you have to install metrics-server. More in below docs and links.
Before you begin
This example requires a running Kubernetes cluster and kubectl, version 1.2 or later. metrics-server monitoring needs to be deployed in the cluster to provide metrics via the resource metrics API, as Horizontal Pod Autoscaler uses this API to collect metrics. The instructions for deploying this are on the GitHub repository of metrics-server, if you followed getting started on GCE guide, metrics-server monitoring will be turned-on by default.
Kubernetes metrics server
Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.
Additional links:
https://www.cloudtechnologyexperts.com/autoscaling-microservices-in-kubernetes-with-horizontal-pod-autoscaler/
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
https://www.youtube.com/watch?v=dALta9zQkfs
Apparently the kubelet metrics regarding PVs were made unavailable from K8s v1.12
It used to be that you could scrape for:
kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_free
kubelet_volume_stats_inodes_used
kubelet_volume_stats_used_bytes
Our cluster is running v1.13.3 and we operate Kafka, Prometheus, InfluxDB and MongoDB all inside Kubernetes with PVCs., so getting usage statistics on these volumes is very important.
Has anyone found other ways to gather these metrics?
I've heard of possibility to load a sidecar container with a node-exporter on every container that uses PVCs. Can be do-able but I'd prefer a more elegant solution.
I deployed Prometheus and Grafana into my cluster.
When I open the dashboards I don't get data for pod CPU usage.
When I check Prometheus UI, it shows pods 0/0 up, however I have many pods running in my cluster.
What could be the reason? I have node exporter running in all of nodes.
Am getting this for kube-state-metrics,
I0218 14:52:42.595711 1 builder.go:112] Active collectors: configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets
I0218 14:52:42.595735 1 main.go:208] Starting metrics server: 0.0.0.0:8080
Here is my Prometheus config file:
https://gist.github.com/karthikeayan/41ab3dc4ed0c344bbab89ebcb1d33d16
I'm able to hit and get data for:
http://localhost:8080/api/v1/nodes/<my_worker_node>/proxy/metrics/cadvisor
As it was mentioned by karthikeayan in comments:
ok, i found something interesting in the values.yaml comments, prometheus.io/scrape: Only scrape pods that have a value of true, when i remove this relabel_config in k8s configmap, i got the data in prometheus ui.. unfortunately k8s configmap doesn't have comments, i believe helm will remove the comments before deploying it.
And just for clarification:
kube-state-metrics vs. metrics-server
The metrics-server is a project that has been inspired by Heapster and is implemented to serve the goals of the Kubernetes Monitoring Pipeline. It is a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Summary API. The metrics are aggregated, stored in memory and served in Metrics API format. The metric-server stores the latest values only and is not responsible for forwarding metrics to third-party destinations.
kube-state-metrics is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it. And just like the metric-server it too is not responsibile for exporting its metrics anywhere.
Having kube-state-metrics as a separate project also enables access to these metrics from monitoring systems such as Prometheus.
There are three levels of metrics collection to consider in Kubernetes - Node, Pod and the Application that runs in the pod.
For Node and Application metrics I have solutions that work wonderfully, but I am stuck on pod metrics.
I have tried cAdvisor and Kube state metrics but none of them give me what I want. Kube state metrics only gives information that is already known like pod CPU limits and requests. cAdvisor doesn't insert pod labels to container names so I have no means of knowing which pod is misbehaving.
Given a pod, I'd like to know it's CPU, memory and storage usage both with respect to the pod itself and also with respect to the node it is scheduled on.
I am using prometheus to collect metrics via the prometheus operator CRD.
Can anyone help suggest an open source metrics exporter that would do the job I mentioned above?
The standard metric collector is Heapster. It comes preinstalled in many vendors like GKE also. With Heapster installed, you can just do kubectl top pods to see cpu/mem metrics on the client side. You can plug it with some sink to store the results for archival.
https://github.com/kubernetes/heapster
I am running Kubernetes cluster 1.5.3 on IBM Bluemix, I would like to get the pod's resources utilization (memory and cpu) as raw data points. Is Kubernetes expose such API?
➜ bluemix git:(master) ✗ k cluster-info
Kubernetes master is running at https://x:x
Heapster is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
You can use heapster or kube-state-metrics to achieve this. In many kubernetes deployments heapsteris already installed. Both can be easily deployed in-cluster.