I want to monitor kafka topic message in prometheus. I will be using kafka connect for the same, but I want to understand how to get the message content details in prometheus tsdb.
You would need to use PushGateway since Prometheus scrapes endpoints, and has no Kafka consumer API. Similarly, Kafka Connect Sinks don't tend to populate any internal metrics server, other than their native JMX server. In other words, JMX Exporter won't let you "see Kafka messages".
There are HTTP Kafka Connect sinks that exist, and you should try using them to send data to PushGateway API.
However, this is a poor use-case for Prometheus.
InfluxDB, for example, can be populated by a Telegraf Kafka consumer (or Kafka Connect), or Elasticsearch by LogStash (or Kafka Connect).
Then Grafana can use Influx or Elasticseach as a datasource to view events... Or it can also use a regular RDBMS
Related
I deployed a kafka cluster with consumer and producer clients on Kubernetes using Strimzi operator. I used the following strimzi deployment file ,
https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml
I am using Kafka exporter to monitor consumer related metrics (Messages in/out per second per topic, lag by consumer group, offsets etc..). However, I am interested in configuring Prometheus to scrape the kafka_exporter metric "kafka_consumergroup_members" for later display on Grafana. What additional configuration shall I add to the strimzi Prometheus configuration file (https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/prometheus-rules.yaml) or any other deployment file (e.g., https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml) so that "kafka_consumergroup_members" from the kafka_exporter metric is scraped.
The Kafka Exporter is a separate tool which provides additional metrics not provided by Kafka itself. It is not configurable in what metrics does it offer - you can only limit for which topics / consumer groups it will show the metrics.
So all metrics supported by Kafka Exporter are published on its metrics endpoint and when Prometheus scapes them it should scrape all of them. So if you have the other KAfka Exporter metrics in your Prometheus, you should already have this one as well (you actuall need to have some active consumer grups for it to show up).
How to push JMX metrics using JMX exporter from Kafka consumer to prometheus without making any change in kafka server side
Kafka server is not responsible for exposing consumer client metrics, and the JMX exporter is pulled, not pushed, depending on how you've configured the Prometheus server metrics scraper
As for how to get consumer metrics - add the JMX exporter to the JVM args when you run the consumer, just like any other Java client you may be monitoring, then modify your scraper to discover the consumer endpoint(s)
If you literally want to push data to Prometheus, that would require the PushGateway
I am working on Kafka --> Prometheus --> Grafana pipeline. I have java application which send message inside a kafka topic. But in prometheus it shows only the message count of topic. I am running an instance of JMX Exporter when I run Kafka.
export JMX_YAML=/home/kafka_2.12-2.3.0/prometheus/kafka-0-8-2.yml
export JMX_JAR=/home/kafka_2.12-2.3.0/prometheus/jmx_prometheus_javaagent-0.6.jar
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:$JMX_JAR=7076:$JMX_YAML"
bin/kafka-server-start.sh config/server.properties
But I need to read the topic data in prometheus. Is there any direct Kafka to Prometheus importer?
I have heard about "Kafka Connect framework"? How to configure it inside prometheus?
Prometheus doesn't run Kafka Connect; you would have to configure that separately.
Also, Prometheus is pulled based, so you at the very least would have to use PushGateway, assuming a Kafka Connector did exist.
If you just want to ultimately display data in Grafana, there are existing connectors for Elasticsearch, Influx, Cassandra, and most JDBC databases
Telegraf or Logstash could be used as alternatives to Kafka Connect, as well, or you can write your own consumer.
We are going to replace our Activemq broker with Kafka broker. In activemq we could send a message to it so that it sends us its queues' metrics.
But I can't find a way in kafka to ask for its topic metrics. Can someone help me?
Kafka metrics are of 3 types.
Kafka Server (Broker) metrics
Producer metrics
Consumer metrics
You can collect Kafka metrics using JMX (https://github.com/jmxtrans/jmxtrans).
For more information on collecting the metrics and monitoring those via a dashboard, read the following link, https://softwaremill.com/monitoring-apache-kafka-with-influxdb-grafana/
Is there a way to send data from influxDB to kafka? Also, the kafka topic has an avro schema defined. So I wanted to know if there was a way to send data into kafka from InfluxDB keeping this in mind as well.
There appears to be a few options for this:
Telegraf daemon by Influx there is also an Output plugin.
Kafka consumer for InfluxDB (written in Python)
You could also write your own using the InfluxDB API.