We are going to replace our Activemq broker with Kafka broker. In activemq we could send a message to it so that it sends us its queues' metrics.
But I can't find a way in kafka to ask for its topic metrics. Can someone help me?
Kafka metrics are of 3 types.
Kafka Server (Broker) metrics
Producer metrics
Consumer metrics
You can collect Kafka metrics using JMX (https://github.com/jmxtrans/jmxtrans).
For more information on collecting the metrics and monitoring those via a dashboard, read the following link, https://softwaremill.com/monitoring-apache-kafka-with-influxdb-grafana/
Related
I want to monitor kafka topic message in prometheus. I will be using kafka connect for the same, but I want to understand how to get the message content details in prometheus tsdb.
You would need to use PushGateway since Prometheus scrapes endpoints, and has no Kafka consumer API. Similarly, Kafka Connect Sinks don't tend to populate any internal metrics server, other than their native JMX server. In other words, JMX Exporter won't let you "see Kafka messages".
There are HTTP Kafka Connect sinks that exist, and you should try using them to send data to PushGateway API.
However, this is a poor use-case for Prometheus.
InfluxDB, for example, can be populated by a Telegraf Kafka consumer (or Kafka Connect), or Elasticsearch by LogStash (or Kafka Connect).
Then Grafana can use Influx or Elasticseach as a datasource to view events... Or it can also use a regular RDBMS
Basically I want to send messages from a MQTT(mosquito) broker to a knative event source(kafka) . In case of a simple kafka broker I could use the confluent's kafkaconnect but in this case it's a knative event source rather than a broker. The problems lies with conversion to cloud events.
Since you have a current MQTT broker which can read/write to Kafka, you might the the Kafka source to convert the Kafka messages to CloudEvents and send them to your service.
If you're asking about how to connect the MQTT broker with Kafka, I'd suggest either finding or writing an MQTT source or using something outside the Knative ecosystem.
Is there a way to get number of messages that arrived on a Kafka topic in a day?
Looking for a solution that can fetch the number of messages arrived on a topic for a particular day.
Ps- we have confluent enterprise and also using prometheus and grafana for metrics.
I'm not familiar with confluent enterprise but we are collecting Kafka metrics using Burrow to Prometheus and using this metric to show incoming message rate:
sum(rate(**kafka_burrow_topic_partition_offset**{cluster="Kafka1",topic="myTopic"}[5m]))
read this https://www.datadoghq.com/blog/collecting-kafka-performance-metrics/#monitor-consumer-health-with-burrow
From Kafka Monitoring doc:
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
I want to monitor Kafka broker health with the help of Kafka Consumer JMX metrics. I am running a Kafka Consumer application which generates the JMX metrics. Is there any metrics which can tell us the Kafka broker health?
I know, we can monitor the broker health with the help of Broker metrics but I want to monitor with the help Consumer JMX metrics.
Please let me know if it is possible.
Thanks
Using JMX server for monitoring Kafka metrics I want to get all network IO for each broker(node). Using MBeans kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec I can get network IO just when some data are produced to or consume from a broker, but we know there is some network IO between brokers for replication, metadata, connecting to Zookeeper and so on. In my Kafka cluster, each node network IO is about 6kb, while no data is consumed or produced. Is there any Kafka metrics to monitor network IO aside from data produced or consumed?
Under kafka.network:type=RequestMetrics,name=RequestsPerSec you find counters for all the request types including Fetch and FetchFollower issued even when there is no produce/consume traffic to the cluster.
You can check the produced or consumed rate either through enabling jmx at producer and consumer or at broker , both are possible.
In broker there are several metrics for n/w and request rates , for example
BrokerTopicMetrics.topic.{topic}.BytesInPerSec
BrokerTopicMetrics.topic.{topic}.BytesOutPerSec
You can check the jmx metrics exposed in the below kafka doc, although this is not exhaustive , if you want to see all the metrics ,you can enable the jmx on broker/producer/consumer and check through VisualVM or any other tool
https://docs.confluent.io/current/kafka/monitoring.html