Having a cluster running on VMs on our private cloud and using MetalLB as ingress-controller we need to see the network traffic and HTTP codes returned from our applications to see in Grafana HTTP requests and traffic load the way you see it on AWS Load Balancers for example.
We have deployed Prometheus through the Helm deployment in all nodes so we can gather metrics from all the cluster but didn't find any metric containing the needed information. Tried looking the metrics in Prometheus about ingresses, proxy, http but there is nothing matching our need. Also tried some Grafana dashboards from the repository but nothing shows the metrics.
Thanks.
Related
In al the tutorials about Kubernetes cluster I have read I didn't see that they mention to 2 load balancers, but only one for the ingress pods.
However, in a proper production environment, should's we have 2 different load balancers?
to balance between the master nodes for requests to the ApiServer.
to balance between the Ingress podes to control the external traffic.
to balance between the master nodes for requests to the ApiServer.
For all production environments its advised to have load
balancer for API Server. This is the first step as part of K8S HA mode creation. More details are in k8s documentation
to balance between the Ingress podes to control the external traffic.
You are correct for this also it’s definitely required to handle external traffic. All the ingress services are created of LoadBalancer Type in their implementations.
Can anyone please help me understand the ingress traffic flow to a pod in kubernetes? Any web links or documents are much appreciated.
In my application there is a intermittent connection timed out so i want to understand how the traffic is flowing in to cluster and where do i need to enable tcpdump to understand what is happening when there is timeout.
Your question does not contain enough information to give you a detailed answer. There are different types of ingress controllers, and load balancers as well.
So, suppose:
you are using Azure Kubernetes Service
you are using Azure Load Balancer
you have two types of backend pods, each has its own dedicated service
you are using Nginx as ingress controller which is able to do LAYER 7 (OSI) load balancing
Nginx has also its own pods and a service sits in front of these pods. This service has a Service IP which is available only within the AKS cluster. Due to this, additionally you can use Azure Load Balancer (ALB) to make your backend pods available for the public. ALB is a layer 4 load balancer, which sends the incoming traffic to the worker nodes.
Kube-proxy is running on every worker nodes and able to recognize that the traffic from the ALB was destined to the Nginx service.
See the flow on the image below:
We have components which use the Go library to write status to prometheus,
we are able to see the data in Prometheus UI,
we have components outside the K8S cluster which need to pull the data from
Prometheus , how can I expose this metrics? is there any components which I should use ?
You may want to check the Federation section of the Prometheus documents.
Federation allows a Prometheus server to scrape selected time series
from another Prometheus server. Commonly, it is used to either achieve scalable Prometheus monitoring setups or to pull related metrics from one service's Prometheus into another.
It would require to expose Prometheus service out of the cluster with Ingress or nodePort and configure the Center Prometheus to scrape metrics from the exposed service endpoint. You will have set also some proper authentication. Here`s an example of it.
Second way that comes to my mind is to use Kube-state-metrics
kube-state-metrics is a simple service that listens to the Kubernetes
API server and generates metrics about the state of the objects.
Metrics are exported on the HTTP endpoint and designed to be consumed either by Prometheus itself or by scraper that is compatible with Prometheus client endpoints. However this differ from the Metrics Server and generate metrics about the state of Kubernetes objects: node status, node capacity, number of desired replicas, pod status etc.
My prometheus server is running on different server. Also I have another kubernetes cluster. So, I need monitoring kubernetes pod metrics using prometheus running on different servers.
To monitor external cluster I would take advantage of Prometheus federation topology.
In your Kubernetes cluster install node-exporter pods and configure Prometheus with short-term storage.
Expose the Prometheus service (you can follow this guide) outside of Kubernetes cluster, this can be done either by LB or a node port.
Configure the Prometheus server to scrape metrics from Kubernetes endpoints configuring them with correct tags and proper authentication.
I refer this doc.
I want to send data from my device and visualize it on grafana so, how to connect prometheus(deployed as a cluster in gcp) to GCP pubsub.
Prometheus is pull-based rather than push-based. So, whatever the metrics source is, it must expose the metrics in Prometheus format, and Prometheus will periodically query them with HTTP request.
If directly exposing the metrics is not possible, the metrics source can push the metrics to some intermediate component which exposes the metrics in Prometheus format so that Prometheus can query them.
It seems this is the approach taken by the document you're referring to. The metrics are submitted from the source via PubSub to a Metrics Telemetry Converter pod running in the Kubernetes cluster, which exposes them in Prometheus format.
You then have to configure Prometheus to scrape the metrics from this pod, as you would configure it for any other job.