My Opensearch app runs in Kubernetes.
I found this OpenSearch exporter plugin:
https://github.com/aparo/opensearch-prometheus-exporter
to export metrics to prometheus.
Is there any option to integrate this solution to kubernetes?
Yes, after you install plugins, You need Prometheus running on Kubernetes.
I recommend you install Prometheus-operator, which creates Prometheus service for you.
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md
It includes the "ServiceMonitor" kind, which defines what "pod" under a "service" you would like to monitor.
After creating the "ServerMonitor" point to your Opensearch pod correctly, Prometheus will scape your metrics.
Related
I have deployed the istio service mesh on the GKE cluster using base & istiod helm charts using this documents in the istio-system namespace.
I have deployed Prometheus, grafana & alert-manager using kube-prometheus-stack helm chart.
Every pod of this workload is working fine; I didn't see any error. Somehow I didn't get any metrics in Prometheus UI related to istio workload. Because of that, I didn't see any network graph in kiali dashboard.
Can anyone help me resolve this issue?
Istio expects Prometheus to discover which pods are exposing metrics through the use of the Kubernetes annotations prometheus.io/scrape, prometheus.io/port, and prometheus.io/path.
The Prometheus community has decided that those annotations, while popular, are insufficiently useful to be enabled by default. Because of this the kube-prometheus-stack helm chart does not discover pods using those annotations.
To get your installation of Prometheus to scrape your Istio metrics you need to either configure Istio to expose metrics in a way that your installation of Prometheus expects (you'll have to check the Prometheus configuration for that, I do not know what it does by default) or add a Prometheus scrape job which will do discovery using the above annotations.
Details about how to integrate Prometheus with Istio are available here and an example Prometheus configuration file is available here.
Need to add additionalScrapConfigs for istio in kube-prometheus-stack helm chart values.yaml.
prometheus:
prometheusSpec:
additionalScrapeConfigs:
- {{ add your scrap config for istio }}
I am using GKE.
I have a cluster which is using stackdriver-adapter to get GCP metrics inside the cluster. I am using these metrics to create HPAs. This is working fine.
But now I need to create HPA on metrics which are provided by prometheus. I am trying to launch prometheus-adapter but it is failing because the API service has already been created by stackdriver-adapter. But if I delete the stackdriver my present HPAs will fail.
Can we have both prometheus-adapter and stackdriver-adpater running in the same cluster?
If no, I guess we need to send prometheus-metrics to stackdriver? But wouldn't that be slow?
As said in the comments:
Have a look at the documentation Using Prometheus, you'll find there how to install Prometheus and get external metrics. After that, follow the documentation Custom and external metrics for autoscaling workloads to configure HPA.
You can configure a sidecar to the Prometheus server that will send the data from the Prometheus to the Stackdriver. From this point you will be able to use the Prometheus metrics as External metrics when configuring the HPA.
You will need to check following requirements before "installing" the collector:
You must be running a compatible Prometheus server and have configured it to monitor the applications in your cluster. To learn how to install Prometheus on your cluster, refer to the Prometheus Getting Started guide.
You must have configured your cluster to use Cloud Operations for GKE. For instructions, see Installing Cloud Operations for GKE.
You must have the Kubernetes Engine Cluster Admin role for your cluster. For more information, see GKE roles.
You must ensure that your service account has the proper permissions. For more information, see Use Least Privilege Service Accounts for your Nodes.
-- Cloud.google.com: Stackdriver: Solutions: GKE: Prometheus: Before you begin
For testing purposes of installing Prometheus and configuring the data transfer to the Stackdriver, I used the script from:
Github.com: Stackdriver: Stackdriver-prometheus-sidecar
Steps:
download the repository:
$ git clone https://github.com/Stackdriver/stackdriver-prometheus-sidecar.git
set the following environment variables (values are examples):
export KUBE_NAMESPACE="prometheus"
export KUBE_CLUSTER="gke-prometheus"
export GCP_REGION="europe-west3-c"
export GCP_PROJECT="awesome-project-12345"
export SIDECAR_IMAGE_TAG="0.8.0"
SIDECAR_IMAGE_TAG can be found here:
Gcr.io: Stackdriver-prometheus: Stackdriver prometheus sidecar
run the script:
kube/full/deploy.sh
After successfully spawning Prometheus with a Stackdriver sidecar you should be able to see the metrics in the Cloud Console:
GCP Cloud Console (Web UI) -> Monitoring -> Metrics Explorer
Example:
From this point you can follow the guide for configuring HPA and set your External metric as the source for autoscaling your Deployment/Statefulset:
Cloud.google.com: Kubernetes Engine: Tutorials: Autoscaling metrics
Additional resources:
Kubernetes.io: Horizontal Pod Autoscaler
Cloud.google.com: Custom and external metrics for autoscaling workloads
I have setup Prometheus / Grafana in Kubernetes using stable/prometheus-operator helm chart. I have setup RabbitMQ exporter helm chart as well. I have to setup the alerts for RabbitMQ which is running on k8s. I can't see any target for RabbitMQ in Prometheus. RabbitMQ is not showing up in targets so that I can monitor metrics. It's critical.
Target in the RabbitMQ exporter can be set by passing the arguments to the helmchart. We have to just set the url and password of RabbitMQ in helmchart using --set.
I want to configure istio in such a way that it does not uses Prometheus or Grafana which come by default with it. I want to use my existing Prometheus and Grafana which is already deplyoed in cluster. Any help will be appreciated.
You need to configure your existing prometheus with a scrape configuration.For prometheus config you can use this ConfigMap. For grafana you need to configure your prometheus as datasource and you can use this configMap for that. You can generate the ConfigMaps using helm template and use.
Check this guide
I already have some services in my k8s cluster and want to mantain them separately. Examples:
grafana with custom dashboards and custom dockerfile
prometheus-operator instead of basic prometheus
jaeger pointing to elasticsearch as internal storage
certmanager in my own namespace (also I use it for nginx-ingress legacy routing)
Is it possible to use existing instances instead of creating istio-specific ones? Can istio communicate with them or it's hardcoded?
Yes - it is possible to use external services with istio. You can disable grafana and prometheus just by setting proper flags in values.yaml of istio helm chart (grafana.enabled=false, etc).
You can check kyma-project project to see how istio is integrated with prometheus-operator, grafana deployment with custom dashboards, and custom jaeger deployment. From your list only certmanager is missing.
Kubernetes provides quite a big variety of Networking and Load Balancing features from the box. However, the idea to simplify and extend the functionality of Istio sidecars is a good choice as they are used for automatic injection into the Pods in order to proxy the traffic between internal Kubernetes services.
You can implement sidecars manually or automatically. If you choose the manual way, make sure to add the appropriate parameter under Pod's annotation field:
annotations:
sidecar.istio.io/inject: "true"
Automatic sidecar injection requires Mutating Webhook admission controller, available since Kubernetes version 1.9 released, therefore sidecars can be integrated for Pod's creation process as well.
Get yourself familiar with this Article to shed light on using different monitoring and traffic management tools in Istio.