I am trying to monitor my kubernetes cluster metrics using Prometheus and grafana. Here is the link which was i followed. Facing the issue with kubernetes-service-endpoints (2/3 up) in my Prometheus dashboard.
below is grafana dashboard which is used in this task.
I checked my Prometheus pod logs .It shows the errors like
Could anybody suggest how to get system services metrics in the above dashboard?
(or)
suggest me the any grafana dashboard name for monitoring the kubernetes cluster using Prometheus?
Check your prometheus.yaml it should have static configs as for prometheus,
static_configs:
- targets:
- localhost:9090
curl localhost:9090/metrics and make sure you're receiving metrics as output
For grafana dashboard, create an org -> prometheus as data source configure prometheus IP:PORT and click on test and confirm connectivity is available.
Open .json file and change the configs according your requirements and import and check.
Related
I am using GKE.
I have a cluster which is using stackdriver-adapter to get GCP metrics inside the cluster. I am using these metrics to create HPAs. This is working fine.
But now I need to create HPA on metrics which are provided by prometheus. I am trying to launch prometheus-adapter but it is failing because the API service has already been created by stackdriver-adapter. But if I delete the stackdriver my present HPAs will fail.
Can we have both prometheus-adapter and stackdriver-adpater running in the same cluster?
If no, I guess we need to send prometheus-metrics to stackdriver? But wouldn't that be slow?
As said in the comments:
Have a look at the documentation Using Prometheus, you'll find there how to install Prometheus and get external metrics. After that, follow the documentation Custom and external metrics for autoscaling workloads to configure HPA.
You can configure a sidecar to the Prometheus server that will send the data from the Prometheus to the Stackdriver. From this point you will be able to use the Prometheus metrics as External metrics when configuring the HPA.
You will need to check following requirements before "installing" the collector:
You must be running a compatible Prometheus server and have configured it to monitor the applications in your cluster. To learn how to install Prometheus on your cluster, refer to the Prometheus Getting Started guide.
You must have configured your cluster to use Cloud Operations for GKE. For instructions, see Installing Cloud Operations for GKE.
You must have the Kubernetes Engine Cluster Admin role for your cluster. For more information, see GKE roles.
You must ensure that your service account has the proper permissions. For more information, see Use Least Privilege Service Accounts for your Nodes.
-- Cloud.google.com: Stackdriver: Solutions: GKE: Prometheus: Before you begin
For testing purposes of installing Prometheus and configuring the data transfer to the Stackdriver, I used the script from:
Github.com: Stackdriver: Stackdriver-prometheus-sidecar
Steps:
download the repository:
$ git clone https://github.com/Stackdriver/stackdriver-prometheus-sidecar.git
set the following environment variables (values are examples):
export KUBE_NAMESPACE="prometheus"
export KUBE_CLUSTER="gke-prometheus"
export GCP_REGION="europe-west3-c"
export GCP_PROJECT="awesome-project-12345"
export SIDECAR_IMAGE_TAG="0.8.0"
SIDECAR_IMAGE_TAG can be found here:
Gcr.io: Stackdriver-prometheus: Stackdriver prometheus sidecar
run the script:
kube/full/deploy.sh
After successfully spawning Prometheus with a Stackdriver sidecar you should be able to see the metrics in the Cloud Console:
GCP Cloud Console (Web UI) -> Monitoring -> Metrics Explorer
Example:
From this point you can follow the guide for configuring HPA and set your External metric as the source for autoscaling your Deployment/Statefulset:
Cloud.google.com: Kubernetes Engine: Tutorials: Autoscaling metrics
Additional resources:
Kubernetes.io: Horizontal Pod Autoscaler
Cloud.google.com: Custom and external metrics for autoscaling workloads
Having a cluster running on VMs on our private cloud and using MetalLB as ingress-controller we need to see the network traffic and HTTP codes returned from our applications to see in Grafana HTTP requests and traffic load the way you see it on AWS Load Balancers for example.
We have deployed Prometheus through the Helm deployment in all nodes so we can gather metrics from all the cluster but didn't find any metric containing the needed information. Tried looking the metrics in Prometheus about ingresses, proxy, http but there is nothing matching our need. Also tried some Grafana dashboards from the repository but nothing shows the metrics.
Thanks.
I had been trying to implement Kubernetes HPA using Metrics from Kafka-exporter. Hpa supports Prometheus, so we tried writing the metrics to prometheus instance. From there, we are unclear on the steps to do. Is there an article where it will explain in details ?
I followed https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07
for same in GCP and we used stack driver, and the implementation worked like a charm. But, we are struggling in on-premise setup, as stack driver needs to be replaced by Prometheus
In order to scale based on custom metrics, Kubernetes needs to query an API for metrics to check for those metrics. That API needs to implement the custom metrics interface.
So for Prometheus, you need to setup an API that exposes Prometheus metrics through the custom metrics API. Luckily, there already is an adapter.
When I implemented Kubernetes HPA using Metrics from Kafka-exporter I had a few setbacks which I solved doing the following:
I deployed the kafka-exporter container as a sidecar to the pods I
wanted to scale. I found that the HPA scales the pod it gets the
metrics from.
I used annotations to make Prometheus scrape the metrics from the pods with exporter.
Then I verified that the kafka-exporter metrics are getting to Prometheus. If it's not there you can't advance further.
I deployed prometheus adapter using its helm chart. The adapter will "translate" Prometheus's metrics into custom Metrics
Api, which will make it visible to HPA.
I made sure that the metrics are visible in k8s by executing kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 from one of
the master nodes.
I created an hpa with the matching metric name.
Here is a complete guide explaining how to implement Kubernetes HPA using Metrics from Kafka-exporter
Please comment if you have more questions
I am running Service monitors to gather metrics from pods. Then with the help of the Prometheus operator, I am using serviceMonitorSelector to catch those metrics in Prometheus. I see those metrics in Prometheus being collected.
Now, I am trying to export those custom metrics from Prometheus to AWS Cloudwatch. Does anyone have any idea how to do that? The end result is to set and alerting system with the help of Zenoss on Cloudwatch.
You have set up something like the prometheus-to-cloudwatch. You can run it in Kubernetes or on any server then make it scrape the same target that Prometheus is scraping. (prometheus-to-cloudwatch scrapes metrics from exporters or as a Prometheus client and not from the Prometheus server)
Then whatever you scrape will show up as metrics in Cloudwatch and then you can set alerts on those. For Zenoss you can use the AWS ZenPack and read the metrics from CloudWatch.
The Kubernetes Prometheus Operator automatically scrapes the services in your Kubernetes cluster and dynamically scrapes them as they get created, you will probably have check what targets are being scraped by Prometheus dynamically to configure what to scrape with prometheus-to-cloudwatch (Or you could build another operator; a prometheus-to-cloudwatch operator, but that will take time/work)
(There isn't such as thing as a scraper of the Prometheus server to CloudWatch either)
I have Grafana and Prometheus set up on my k8s cluster. Both were installed thru helm using https://github.com/helm/charts/tree/master/stable.
Both Grafana and Prometheus are set up thru k8s nginx ingress with my domian addresses.
When I try to set up the Prometheus as a data source in Grafana I get HTTP Error Bad Gateway. In the chrome console in Grafana page I see:
http://grafana.domain.com/api/datasources/proxy/1/api/v1/query?query=1%2B1&time=1554043210.447
Grafana version: Grafana v6.0.0 (commit: 34a9a62)
Grafana data source settings for Prometheus:
URL: https://prometheus.mydomain.com:9090
Access: Server(Default)
Auth:
Basic & TLS Client Auth
What might be wrong and how to debug/fix it?
In Grafana data source settings for prometheus database add prometheus service dns and service port. Like below
<prometheus service name>. Namespace. Svc. Cluster. Local:9090
if you are run Grafana and Prometheus on docker on your local machine and this will do for the datasource settings
Add host as {host.docker.internal} : {port}
example - http://{host.docker.internal}:9090
When you are trying to add a data source, like Prometheus it is a little bit confusing because they are asking for the http, but you have to put your IP adress.
You just have to open CMD and write ipconfig /all and then look at the IPv4 Direction and then you will have your IP.
So the last part is going to Prometheus and in the URL you have to put:
http://{your_IP}:9090