I deployed a kafka cluster with consumer and producer clients on Kubernetes using Strimzi operator. I used the following strimzi deployment file ,
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml
I am using Kafka exporter to monitor consumer related metrics (Messages in/out per second per topic, lag by consumer group, offsets etc..). However, I am interested in configuring Prometheus to scrape the kafka_exporter metric "kafka_consumergroup_members" for later display on Grafana. What additional configuration shall I add to the strimzi Prometheus configuration file (https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/prometheus-rules.yaml) or any other deployment file (e.g., https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml) so that "kafka_consumergroup_members" from the kafka_exporter metric is scraped.
The Kafka Exporter is a separate tool which provides additional metrics not provided by Kafka itself. It is not configurable in what metrics does it offer - you can only limit for which topics / consumer groups it will show the metrics.
So all metrics supported by Kafka Exporter are published on its metrics endpoint and when Prometheus scapes them it should scrape all of them. So if you have the other KAfka Exporter metrics in your Prometheus, you should already have this one as well (you actuall need to have some active consumer grups for it to show up).
Related
I want to monitor kafka topic message in prometheus. I will be using kafka connect for the same, but I want to understand how to get the message content details in prometheus tsdb.
You would need to use PushGateway since Prometheus scrapes endpoints, and has no Kafka consumer API. Similarly, Kafka Connect Sinks don't tend to populate any internal metrics server, other than their native JMX server. In other words, JMX Exporter won't let you "see Kafka messages".
There are HTTP Kafka Connect sinks that exist, and you should try using them to send data to PushGateway API.
However, this is a poor use-case for Prometheus.
InfluxDB, for example, can be populated by a Telegraf Kafka consumer (or Kafka Connect), or Elasticsearch by LogStash (or Kafka Connect).
Then Grafana can use Influx or Elasticseach as a datasource to view events... Or it can also use a regular RDBMS
How to push JMX metrics using JMX exporter from Kafka consumer to prometheus without making any change in kafka server side
Kafka server is not responsible for exposing consumer client metrics, and the JMX exporter is pulled, not pushed, depending on how you've configured the Prometheus server metrics scraper
As for how to get consumer metrics - add the JMX exporter to the JVM args when you run the consumer, just like any other Java client you may be monitoring, then modify your scraper to discover the consumer endpoint(s)
If you literally want to push data to Prometheus, that would require the PushGateway
On my Kafka cluster,I am able to view and monitor certain Mbean JMX metrics like RequestsPerSec etc. However I only see a very few of the metrics mentioned in the Apache Kafka documentation on my JConsole. Is there a way to enable others. Especially is there a way to enable a few specific ones explicitly.
Kafka has different metric sets on particular components, and some versions of Kafka have more/less metrics than others
The brokers, producers and consumers each have different JMX metrics
There's no way to disable/enable MBeans
We have Spring Boot applications deployed on OKD (The Origin Community Distribution of Kubernetes that powers Red Hat OpenShift). Without much tweaking by devops team, we got in prometheus scraped kafka consumer metrics from kubernetes-service-endpoints exporter job, as well as some producer metrics, but only for kafka connect api, not for standard kafka producer api. This is I guess a configuration for that job:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
What is needed to change in scrape config in order to collect what's been missing?
This issue with micrometer is the source of the problem.
So, we could add jmx exporter, or wait for the issue resolution.
I am working on Kafka --> Prometheus --> Grafana pipeline. I have java application which send message inside a kafka topic. But in prometheus it shows only the message count of topic. I am running an instance of JMX Exporter when I run Kafka.
export JMX_YAML=/home/kafka_2.12-2.3.0/prometheus/kafka-0-8-2.yml
export JMX_JAR=/home/kafka_2.12-2.3.0/prometheus/jmx_prometheus_javaagent-0.6.jar
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:$JMX_JAR=7076:$JMX_YAML"
bin/kafka-server-start.sh config/server.properties
But I need to read the topic data in prometheus. Is there any direct Kafka to Prometheus importer?
I have heard about "Kafka Connect framework"? How to configure it inside prometheus?
Prometheus doesn't run Kafka Connect; you would have to configure that separately.
Also, Prometheus is pulled based, so you at the very least would have to use PushGateway, assuming a Kafka Connector did exist.
If you just want to ultimately display data in Grafana, there are existing connectors for Elasticsearch, Influx, Cassandra, and most JDBC databases
Telegraf or Logstash could be used as alternatives to Kafka Connect, as well, or you can write your own consumer.