This question already has answers here:
How to expose kafka metrics to /actuator/metrics with spring boot 2
(4 answers)
Closed 2 months ago.
hi i have a consumer with multiple listeners with concurrency as 3. Each listener consume one topic. I'm trying to get the consumer lag of the containers in prometheus metric endpint. I checked the default metrics and only the listener related success and failure count and sum are available. Is there any option that i can get consumer lag exposed as a prometheus metric ?
EDIT
I'm using spring-kafka and with that in the documentation it says i can simple get the listener and broker related metrics (https://docs.spring.io/spring-kafka/reference/html/#monitoring-listener-performance). What i did was call the prometheus endpoint like myUrl/prometheus and i was able to see the listener logs.
So is there a way to view consumer lag like that?
As far as I know, Spring does not provide an easy way to do this (see Monitoring documentation for more details).
You can use some of the existing solutions to eliminate dependency on your consumers implementations. For example:
Kafka Lag Exporter.
It provides a metrics like kafka_consumergroup_group_lag with labels: cluster_name, group, topic, partition, member_host, consumer_id, client_id.
It is easy to set up and can run anywhere, but it provides features to run easily on Kubernetes clusters.
Kafka Exporter
It provides more different kafka metrics. For the lag, it has an example of the kafka_consumergroup_lag metric with the labels: consumergroup, partition, topic
Related
I deployed a kafka cluster with consumer and producer clients on Kubernetes using Strimzi operator. I used the following strimzi deployment file ,
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml
I am using Kafka exporter to monitor consumer related metrics (Messages in/out per second per topic, lag by consumer group, offsets etc..). However, I am interested in configuring Prometheus to scrape the kafka_exporter metric "kafka_consumergroup_members" for later display on Grafana. What additional configuration shall I add to the strimzi Prometheus configuration file (https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/prometheus-rules.yaml) or any other deployment file (e.g., https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml) so that "kafka_consumergroup_members" from the kafka_exporter metric is scraped.
The Kafka Exporter is a separate tool which provides additional metrics not provided by Kafka itself. It is not configurable in what metrics does it offer - you can only limit for which topics / consumer groups it will show the metrics.
So all metrics supported by Kafka Exporter are published on its metrics endpoint and when Prometheus scapes them it should scrape all of them. So if you have the other KAfka Exporter metrics in your Prometheus, you should already have this one as well (you actuall need to have some active consumer grups for it to show up).
I'm new to Kafka. During study to kafka, I think monitoring consumer's lag is needed. When I search from google and docs, I found few ways.
Kafka - Prometheus - graphana
kafka - burrow - someDB - graphana
kafka - burrow_stat?(I can't understand what it is..)
kafka - datadog
what I want to ask is
document says that burrow is for monitoring, can I visualize like graph(dashboard)?
without other tools like graphana or kibana or datadog??
I just trying to get less pipeline steps. What should be the simple way to visualize consumer's lag?
If you are doing the setup in an organisation, datadog or prometheus is probably the way to go. You can capture other Kafka related metrics as well. These agents also have integrations with many other tools beside Kafka and will be a good common choice for monitoring.
If you are just doing it for personal POC type of a project and you just want to view the lag, I find CMAK very useful (https://github.com/yahoo/CMAK). This does not have historical data, but provides a good current visual state of Kafka cluster including lag.
For cluster wide metrics you can use kafka_exporter (https://github.com/danielqsj/kafka_exporter) which exposes some very useful cluster metrics(including consumer lag) and is easy to integrate with prometheus and visualize using grafana.
Burrow is extremely effective and specialised in monitoring consumer lag.Burrow is good at caliberating consumer offset and more importantly validate if the lag is malicious or not. It has integrations with pagerduty so that the alerts are pushed to the necessary parties.
https://community.cloudera.com/t5/Community-Articles/Monitoring-Kafka-with-Burrow-Part-1/ta-p/245987
What burrow has:
Non-threshold based lag monitoring algorithm capable to evaluate potential slow downs.
Integration with pagerduty
Exporters for prometheus, AppD etc for historical metrics
Pluggable UI
If you are looking for quick solution you can deploy burrow followed by the burrow front end https://github.com/GeneralMills/BurrowUI
On my Kafka cluster,I am able to view and monitor certain Mbean JMX metrics like RequestsPerSec etc. However I only see a very few of the metrics mentioned in the Apache Kafka documentation on my JConsole. Is there a way to enable others. Especially is there a way to enable a few specific ones explicitly.
Kafka has different metric sets on particular components, and some versions of Kafka have more/less metrics than others
The brokers, producers and consumers each have different JMX metrics
There's no way to disable/enable MBeans
How can I measure the rate of Kafka producers and consumers in a Java application? Does Kafka broker provides any performance metrics?
Kafka itself exposes lots of performance metrics. At my employer, we use Prometheus to ingest + store those metrics, and a Grafana frontend for graphs + dashboards. We also instrument our apps, including our Java apps with the Prometheus library, and expose+scrape custom metrics to help us understand all aspects of our data pipelines and performance.
I try to export Kafka metrics per JMX to Prometheus and display them with Grafana, but I´m struggling to get the Consumer metrics (to be more precise this one:
kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+) )
Everytime I try to fetch this Mbean, it doesn´t even show up. I read all the time that I have to "look into the client", or "I´m looking in the broker metrics, but I need the consumer metrics", but nobody does explain how to do this, so I´m asking you guys if you could help me. Is there some kind of configuration, or special JMX Port to get Consumer metrics or something like that?
The pattern for my config file to look for MBeans:
- pattern : kafka.consumer<type=(.+), name=(.+), client-id=(.+)><>(Count|Value)
name: kafka_consumer_$1_$2
Labels:
clientId: "$3"
Also, i need to fetch the Metrics with JMX, because i dont have access to the Kafka server.
I´m using this project as an example: https://github.com/rama-nallamilli/kafka-prometheus-monitoring
The following two things are possible:
A. May be given client already disconnected from Kafka
B. May be this metric is not present on broker. It might be visible in the JVM application which is running the consumer code. I am not sure but here is how you can check:
Restart your consumer application with JMX enabled
Use visual vm to connect to the above jvm
It should show all the available JMX metrics.
If the metrics contain metrics of your choice then you were looking at wrong place (broker). If not then I am wrong.
I do not have exact configuration but 1 mistake that i can point out in your configuration is that, name can not be the matching pattern for consumer metrics.
Try to remove the pattern with this:
- pattern : kafka.consumer<type=(.+), client-id=(.+)><>(Count|Value)
For more reference you can check the
Apache kafka docs
I am also having the problem for creating a generic pattern for consumer and producer.
Will post here as soon as i get this figured out.
#xBoLLo