Using JMX server for monitoring Kafka metrics I want to get all network IO for each broker(node). Using MBeans kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec and kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec I can get network IO just when some data are produced to or consume from a broker, but we know there is some network IO between brokers for replication, metadata, connecting to Zookeeper and so on. In my Kafka cluster, each node network IO is about 6kb, while no data is consumed or produced. Is there any Kafka metrics to monitor network IO aside from data produced or consumed?
Under kafka.network:type=RequestMetrics,name=RequestsPerSec you find counters for all the request types including Fetch and FetchFollower issued even when there is no produce/consume traffic to the cluster.
You can check the produced or consumed rate either through enabling jmx at producer and consumer or at broker , both are possible.
In broker there are several metrics for n/w and request rates , for example
BrokerTopicMetrics.topic.{topic}.BytesInPerSec
BrokerTopicMetrics.topic.{topic}.BytesOutPerSec
You can check the jmx metrics exposed in the below kafka doc, although this is not exhaustive , if you want to see all the metrics ,you can enable the jmx on broker/producer/consumer and check through VisualVM or any other tool
https://docs.confluent.io/current/kafka/monitoring.html
Related
I want to know whether Kafka broker is up or not? I have enabled JMX in Kafka but couldn't find any Mbean name that can provide me the status of Kafka. Any idea?
There are multiple things to check health of a cluster, both of which are individual Mbeans and will need to be aggregated over the entire cluster
Is there only one controller
Are there no out-of-sync replicas
You may also want to externally port check the brokers from the environments you want to produce and consume from
I have a Kakfa broker running, which I am monitoring with JMX.
This broker is a docker container running as a process started with kafka-server-start.sh JMX port 9999 is exposed as and used as an environment variables.
When I connect to the JMX port and try to list all the domains, I get the following;
kafka
kafka.cluster
kafka.controller
kafka.coordinator.group
kafka.coordinator.transaction
kafka.log
kafka.network
kafka.server
kafka.utils
I dont see kafka.producer which is understandable because the producer for this Kafka broker are N numbers of different applications, but at this point I am confused.
How do I get the kafka.producer metrics as well.
Do I have to expose the kafka.producer metrics in each of N application that is acting as producer OR is there some configuration that start gathering kafka.producer metrics on the broker only.
What is the correct way of doing this. Please help.
Yes you are correct , to capture the producer JMX metrics , you need to enable JMX in all the processes which are running the kafka producer instance.
It might be helpful to rephrase producing as writing over an unreliable network in this context.
From this perspective, the most reasonable place to measure writing characteristics seems to be the client itself (i.e. in each "application" as you call it).
If messages between the producer and the broker are lost, you can still send stats to a local "metric store" for example (e.g. you could see a "spike" in record-retry-rate or some other relevant metric).
Additionally, pairing Kafka producer metrics with additional, local metrics might be extremely useful (JVM stats, detailed business metrics and so on). Keep in mind, that the client will almost definitely run on a different machine in a production environment, and might be affected by different factors, than the broker itself.
If you intend to monitor your client application (which will most likely happen anyway), then I'd simply do it there (i.e. the standard way).
We are going to replace our Activemq broker with Kafka broker. In activemq we could send a message to it so that it sends us its queues' metrics.
But I can't find a way in kafka to ask for its topic metrics. Can someone help me?
Kafka metrics are of 3 types.
Kafka Server (Broker) metrics
Producer metrics
Consumer metrics
You can collect Kafka metrics using JMX (https://github.com/jmxtrans/jmxtrans).
For more information on collecting the metrics and monitoring those via a dashboard, read the following link, https://softwaremill.com/monitoring-apache-kafka-with-influxdb-grafana/
Is there a way to monitor my kafka cluster using nagios? any working plugin, api or whatever to check: broker status, partition status, memory status, current offset and all valuable metrics from my cluster?
We are using Nagios to monitor Kafka JMX metrics (we use JMXeval, but you can use any of your favorite JMX monitoring script for Nagios) where we can find many useful metrics like memory, lag, number of offline partition, and so on.
I can highly recommend you to read this article about Kafka monitoring, where you can find many useful tips what you can monitor - https://blog.serverdensity.com/how-to-monitor-kafka/
Because JMX is by default disabled, you need enable it first. You can follow instruction on Enable JMX on Kafka Brokers
Beside some basic monitoring metrics like CPU , memory and network usage. Is there anyway that I can actually monitor the running Kafka application, such as number of messages in/out, stream throughput, stream size ...?
Thank you.
Kafka offers various metrics reporting in both the server and the client. See the Monitoring document for details.