I'm looking to monitor the number of active Endpoints in NodePort and ClusterIP service. There are several cases when my pods restart or get destroyed. So its important to know if atleast one Endpoint is there to serve the incoming request.
Is there some metric for it in cAdvisor that I can expose via Prometheus? If not is there some way to track this?
I have no idea about cadvicer for monitoring endpoints but by using prometheus blackbox we can achieve this.
The Prometheus Blackbox exporter allows endpoints exploration over several protocols, such as HTTP(S), DNS, TCP, and ICMP. This exporter generates multiple metrics on your configured targets, like general endpoint status, response time, redirect information,Detecting endpoint failures, or certificate expiration dates.
Refer to this link for installation and to configure the setup.
Related
I want to use prometheus to scrape metrics from my distributed web service.
I have four kind of services setup with docker-compose or kubernetes.
Flask: 5000
Redis-Queue: 6379
Prometheus
Workers: Horizontally scaled based on system load. They get their working instructions over the redis Queue.
It is strait forward how to scrape metrics from Flask.
However, what is best-practise to get the metrics from the Workers? I cannot bind a port to them, because, I do not know, how many of them exist.
I was thinking about using a prometheus pushgateway. However, as I found out, this is not recommended.
the answer depends whether your workers lifetime is long or short
if a worker lives to execute a single task and then quits push gateway is the correct way to send metrics from the worker to Prometheus.
if a worker lives for at least two Prometheus scrape periods (which is configurable) you can definitely open a port on the worker and have Prometheus scrape metrics from a dedicated endpoint.
Prometheus's default scrape configuration comes with a scrape job that will scrape any pod with the following annotation:
prometheus.io/scrape: true
it also derives the scrape endpoint from the following annotations on the pod
prometheus.io/scheme: http
prometheus.io/path: /metrics
prometheus.io/port: 3000
so you can easily annotate worker pods with the above annotations to direct Prometheus to scrape metrics from them
We have components which use the Go library to write status to prometheus,
we are able to see the data in Prometheus UI,
we have components outside the K8S cluster which need to pull the data from
Prometheus , how can I expose this metrics? is there any components which I should use ?
You may want to check the Federation section of the Prometheus documents.
Federation allows a Prometheus server to scrape selected time series
from another Prometheus server. Commonly, it is used to either achieve scalable Prometheus monitoring setups or to pull related metrics from one service's Prometheus into another.
It would require to expose Prometheus service out of the cluster with Ingress or nodePort and configure the Center Prometheus to scrape metrics from the exposed service endpoint. You will have set also some proper authentication. Here`s an example of it.
Second way that comes to my mind is to use Kube-state-metrics
kube-state-metrics is a simple service that listens to the Kubernetes
API server and generates metrics about the state of the objects.
Metrics are exported on the HTTP endpoint and designed to be consumed either by Prometheus itself or by scraper that is compatible with Prometheus client endpoints. However this differ from the Metrics Server and generate metrics about the state of Kubernetes objects: node status, node capacity, number of desired replicas, pod status etc.
Having a cluster running on VMs on our private cloud and using MetalLB as ingress-controller we need to see the network traffic and HTTP codes returned from our applications to see in Grafana HTTP requests and traffic load the way you see it on AWS Load Balancers for example.
We have deployed Prometheus through the Helm deployment in all nodes so we can gather metrics from all the cluster but didn't find any metric containing the needed information. Tried looking the metrics in Prometheus about ingresses, proxy, http but there is nothing matching our need. Also tried some Grafana dashboards from the repository but nothing shows the metrics.
Thanks.
As I can see in below diagram I figure out in kubernetes we have two loadbalancer. One of them loadbalance between nodes and one of them loadbalance between pods.
If I use them both I have two loadbalancer.
Imagine some user want to connect to 10.32.0.5 the kubernetes send its request to node1(10.0.0.1) and after that send the request to pod (10.32.0.5) in nod3(10.0.0.3) but it is unuseful because the best route is to send request nod3(10.0.0.3) directly.
Why the NodePort is insufficient for load-balancing?
Why the NodePort is not LoadBalancer?(it LoadBalance between pods in different node but why we need another load balancer?)
note: I know that if I use NodePort and the node goes down it creates problem but I can say that I can use keepalived for it. The question is
why we need to loadbalance between nodes? keepalived attract all request to one IP.
Why we have two loadbalancer?
Wether you have two load-balancers depends on your setup.
In your example you have 3 nginx pods and 1 nginx service to access the pods. The service builds an abstraction layer, so you don't have to know how many pods there are and what IP addresses they have. You just have to talk to the service and it will loadbalance to one of the pods (docs).
It now depends on your setup how you access the service:
you might want to publish the service via NodePort. Then you can directly access the service on a node.
you might also publish it via LoadBalancer. This gives you another level of abstraction and the caller needs to know less about the actual setup of your cluster.
See docs for details.
I'd like to monitor my Kubernetes Service objects to ensure that they have > 0 Pods behind them in "Running" state.
However, to do this I would have to first group the Pods by service and then group them further by status.
I would also like to do this programatically (e.g. for each service in namespace ...)
There's already some code that does this in the Sensu kubernetes plugin: https://github.com/sensu-plugins/sensu-plugins-kubernetes/blob/master/bin/check-kube-service-available.rb but I haven't seen anything that shows how to do it with Prometheus.
Has anyone setup kubernetes service level health checks with Prometheus? If so, how did you group by service and then group by Pod status?
The examples I have seen for Prometheus service checks relied on the blackbox exporter:
The blackbox exporter will try a given URL on the service. If that succeeds, at least one pod is up and running.
See here for an example: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml in job kubernetes-service-endpoints
The URL to probe might be your liveness probe or something else. If your services don't talk HTTP, you can make the blackbox exporter test other protocols as well.