Kafka Monitoring: request latencies from JMX - apache-kafka

Question

We want to monitor Kafka and have two specific requirements: use headless tools and store performance metrics in a CSV file. Following Gwen Shapira series [1] I am leaning towards request latencies and kafka.tools.JmxTool to start with.
Setup: Kafka 0.11, exposed JMX, headless metric collection tools
Q: what JMX beans provide metrics as presented on [2], likely per Broker: “request queue”, “request local”, “response remote”, “response queue”, “response send”?
[1] slides
https://www.slideshare.net/ConfluentInc/metrics-are-not-enough-monitoring-apache-kafka-and-streaming-applications/
[2] desired Kafka metrics

Answer 1

After some exploring, here is the full set to open Kafka JMX at port 3999 and gather the request metrics:
1 Open Kafka JMX port at 3999:
update bin/kafka-run-class.sh:
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=0.0.0.0 -Djava.net.preferIPv4Stack=true"
fi
# JMX port to use
if [ $JMX_PORT ]; then
KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=${JMX_PORT} -Dcom.sun.management.jmxremote.rmi.port=${JMX_PORT} "
fi
update bin/kafka-server-start.sh:
if [ -z "$JMX_PORT" ]; then
export JMX_PORT=3999
fi
2 bash script to gather Kafka metrics and publish them to the journalctl:
#!/bin/bash
PIPE=/tmp/kafka-monitoring-temp.out
mkfifo $PIPE
# Start logging to journal
systemd-cat -t 'kafka-monitoring' < $PIPE &
sleep_pid=$(
sleep 9999d > $PIPE & # keep pipe open
echo $! # but allow us to close it later...
)
arrayName=( "kafka.network:type=RequestMetrics,name=RequestQueueTimeMs,request=Fetch"
"kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Fetch"
"kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Fetch"
"kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=Fetch"
"kafka.network:type=RequestMetrics,name=ResponseSendTimeMs,request=Fetch"
)
for name in "${arrayName[#]}"; do
timeout 1s /ust/lib/kafka/bin/kafka-run-class.sh kafka.tools.JmxTool --object-name "${name}" --jmx-url service:jmx:rmi:///jndi/rmi://127.0.0.1:3999/jmxrmi --reporting-interval 1100 | tee $PIPE
done
kill $sleep_pid
rm $PIPE

Answer 2

There is a good chapter on Monitoring in the "Kafka: The Definitive Guide" (pdf is freely available from Confluent's site). The book shows following Request-related metrics:

Answer 3

Here they are in addition to some description of what they mean. I hope it helps.
Request Queue: time that request waits at the request queue
kafka.network:type=RequestChannel,name=RequestQueueSizeMs
Request Local: time processing the request at the leader
kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Produce
Response Remote: time waiting for the followers to process the request
kafka.network:type=RequestMetrics,name=RemoteTimeMs,request=Produce
Response Queue: time that the request waits at the response queue
kafka.network:type=RequestMetrics,name=ResponseQueueTimeMs,request=Produce
Response Send: time to send the response
kafka.network:type=RequestMetrics,name=ResponseSendTimeMs