Can anybody let me know how to config
kubernetes pods to send alert to slack channel ?
Thanks in Advance
Rishabh Gupta
Kubernetes dosn't provide out of the box slack integration.
There are few projects that you can use:
https://hub.kubeapps.com/charts/stable/kube-slack - runs on Kubernetes, watches for evnets and sends pod failures notifications to Slac
https://hub.kubeapps.com/charts/stable/kubewatch - similar project. depending on configuration can be quiet noisy
If you need more complex motoring you can use Prometheus and it's alert manager https://prometheus.io/docs/alerting/notification_examples/
I would suggest go with Prometheus operator, as this project is cloud-native and give good hold on managing the alerts.
Handles alerts with alert manager
We can add data source with Grafana
Write our own expressions for pod and microservices monitoring
Works on scraping over http/s request
we can easily expose the Applicaton metrics
https://github.com/coreos/kube-prometheus
Related
Kubernetes. Prometheus. A test application that can send metrics or a simple phrase to an HTTP request.
The end goal is to see graphs in Grafana. I want to make sure that everything works up to the current setup moment.
Now I want to see the metrics by URL, to see that the ServiceMonitor is working correctly, that they reach the Graphana. But so far I haven't been able to find a good enough troubleshooting guide.
I assume it's okay not to show everyone your metrics. But I want to make sure that this is normal behavior and which way to google in case I want to allow it.
This is a completely intended behavior when trying to use Grafana to visualize data from Managed Service for Prometheus. Metrics will be available via http request when trying to browse Grafana, as during the deployment we port-forward the frontend service to our local machine. As it is not a good practice to leave our metrics available to everyone for some security reasons, Grafana port-forwards the frontend service to our local machine.
Google Cloud APIs all require authentication using OAuth2. However, Grafana doesn't support OAuth2 authentication for Prometheus data sources. To use Grafana with Managed Service for Prometheus, you must use the Prometheus UI as an authentication proxy.
You can refer to these documentations to know more about Grafana & How-to-Deploy-Grafana.
We are trying to figure out which monitoring options will be suitable for our environment.
We have two clusters in GCP and we installed Istio (with Helm) in both of them. We are also using Workload Identity and Stackdriver Monitoring.
Now, we would like to create dashboards (or charts) for kubernetes objects (such as, deployments, containers, cronjobs, services, etc.) and want to set alerts on them. So can anyone suggest free monitoring options to achieve these all? We don't want to go with any third party paid software.
Thank you in advance.
If you are using GCP GKE then default stack driver logging & monitoring is best option.
It's free if you are using GCP service and using stack driver monitoring you can monitoring and creat respective dashboards as per need.
For alerts, you can use the GCP Uptime check option available in monitoring itself which sends the email. For call alerts, you may have to use some custom or third applications.
You can read more at : https://cloud.google.com/monitoring/docs
Uptime checks : https://cloud.google.com/monitoring/uptime-checks
I have two Kubernetes clusters representing dev and staging environments.
Separately, I am also deploying a custom DevOps dashboard which will be used to monitor these two clusters. On this dashboard I will need to show information such as:
RAM/HD Space/CPU usage of each deployed Pod in each environment
Pod health (as in if it has too many container restarts etc)
Pod uptime
All these stats have to be at a cluster level and also per namespace, preferably. As in, if I query a for a particular namespace, I have to get all the resource usages of that namespace.
So the webservice layer of my dashboard will send a service request to the master node of my respective cluster in order to fetch this information.
Another thing I need is to implement real time notifications in my DevOps dashboard. Every time a container fails, I need to catch that event and notify relevant personnel.
I have been reading around and two things that pop up a lot are Prometheus and Metric Server. Do I need both or will one do? I set up Prometheus on a local cluster but I can't find any endpoints it exposes which could be called by my dashboard service. I'm also trying to set up Prometheus AlertManager but so far it hasn't worked as expected. Trying to fix it now. Just wanted to check if these technologies have the capabilities to meet my requirements.
Thanks!
I don't know why you are considering your own custom monitoring system. Prometheus operator provides all the functionality that you mentioned.
You will end up only with your own grafana dashboard with all required information.
If you need custom notification you can set it up in Alertmanager creating correct prometheusrules.monitoring.coreos.com, you can find a lot of preconfigured prometheusrules in kubernetes-mixin
.
Using labels and namespaces in Alertmanager you can setup a correct route to notify person responsible for a given deployment.
Do I need both or will one do?, yes, you need both - Prometheus collects and aggregates metric when Metrick server exposes metrics from your cluster node for your Prometheus to scrape it.
If you have problems with Prometheus, Alertmanger and so on consider using helm chart as entrypoint.
Prometheus + Grafana are a pretty standard setup.
Installing kube-prometheus or prometheus-operator via helm will give you
Grafana, Alertmanager, node-exporter and kube-state-metrics by default and all be setup for kubernetes metrics.
Configure alertmanager to do something with the alerts. SMTP is usually the first thing setup but I would recommend some sort of event manager if this is a service people need to rely on.
Although a dashboard isn't part of your requirements, this will inform how you can connect into prometheus as a data source. There is docco on adding prometheus data source for grafana.
There are a number of prebuilt charts available to add to Grafana. There are some charts to visualise alertmanager too.
Your external service won't be querying the metrics directly with prometheus, in will be querying the collected data in prometheus stored inside your cluster. To access the API externally you will need to setup an external path to the prometheus service. This can be configured via an ingress controller in the helm deployment:
prometheus.ingress.enabled: true
You can do the same for the alertmanager API and grafana if needed.
alertmanager.ingress.enabled: true
grafana.ingress.enabled: true
You could use Grafana outside the cluster as your dashboard via the same prometheus ingress if it proves useful.
I have a mule application which mostly does HTTP requests, which is logging as plain text. I want to push these logs as metrics to Prometheus. Since this is a legacy application it would take a substantial amount of time to change code and push metrics directly into Prometheus storage.
Idea is to show Prometheus metrics in Grafana Dashboard.
Is there any intermediate tool that converts plain text to metrics?
Anything that helps with this requirement.
FYI- We have Nagios and Splunk which is doing this task as of now, we are looking to move our solution to Prometheus and Grafana
In situations like these you can use tools like https://github.com/fstab/grok_exporter to convert logs into metrics.
We're migrating to Kubernetes and many of our services are using Hystrix which expose a stream of server sent event data consumed by Turbine and visualized by a Hystrix Dashboard. I'd like to implement a service discovery plugin to Turbine which auto discovers our Hystix streams running on Kubernetes.
Would it be a good idea to use labels for this? I.e. define a label that includes the path to the hystrix stream for each pod using Hystrix?
If labels are not a good idea, what would be?
I had the same problem and I created a little project to address it.
essentially I implemented a hystric instance discovery class to work with kubernetes services.
You can find my project here.
This projects is in its infancy so bugs are likely. Feedback it welcome.
Thanks
Raffaele
How about making Hystrix a Kubernetes service?Kubernetes has two suggested ways to discover services: https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/services.md#discovering-services