I am trying to monitor the consumer lag for old consumer groups ( which uses zookeeper) using JMX.
This is how I have enabled JMX.
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=`hostname -f` -Djava.net.preferIPv4Stack=true"
if [ "x$ISKAFKASERVER" == "xtrue" ]; then
export JMX_PORT=9887
And I am able to see multiple been when I connect using JConsole. However I don't see kafka.consumer
Seems like you enabled jmx on kafka brokers. You need to enable jmx on the instances where consumer is running for getting consumer metrics
Related
I have a ubuntu machine that is having kafka and zookeepr installed in it, I am using spring boot for making the consumer and producer, locally the process works, however, when the deploy the producer and consumer jar to another machine it doesn't work
Kafka defaults to only listen locally.
You need to set these in Kafka's server.properties
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<external-ip>:9092
https://www.confluent.io/blog/kafka-listeners-explained/
Then, obviously, don't use localhost:9092 in your remote client code.
You should never need Zookeeper connection details. Besides, as of Kafka 3.3.1, Zookeeper isn't required at all.
How can I enable Kafka source connector idempotency feature?
I know in confluent we can override producer configs by producer.* properties in the worker configuration, but how about Kafka itself? is it the same?
After setting these configs where can I see applied configs for my connect worker?
Confluent doesn't modify the base Kafka Connect properties.
For configuration of the producers used by Kafka source tasks and the consumers used by Kafka sink tasks, the same parameters can be used but need to be prefixed with producer. and consumer. respectively
Starting with 2.3.0, client configuration overrides can be configured individually per connector by using the prefixes producer.override. and consumer.override. for Kafka sources or Kafka sinks respectively
https://kafka.apache.org/documentation/#connect_running
However, Kafka Connect sources aren't idenpotent - KAFKA-7077 & KIP-308
After setting these configs where can I see applied configs for my connect worker
In the logs, it should show the ProducerConfig or ConsumerConfig when the tasks start
I need to write an API to display consumer metrics in java. I thought of using Kafka admin client. Is there any way we can retrieve consumer metrics information
I checked admin client code. I have Kafka consumer class. I have consumer metrics method, but it is not holding the information.
Kafka consumer can report a lot of metrics through JMX, you just need to add a few Java properties to your consumer, either in the code or through command-line. For example, for console consumer:
$ KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9398 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" kafka-console-consumer ...
I am using confluent HDFS sink connector and would like to know how to get consumer properties to expose through either JMX or REST API.
I checked the following two properties, however, I don't know how to expose metrics to jmx port
connect-standalone.properties
consumer.properties
Set JMX_PORT when you launch Kafka Connect. e.g.
export JMX_PORT=4242
./bin/connect-distributed ./etc/kafka/connect-distributed.properties
You can then connect to JMX using JConsole, JMXTerm, etc.
Had the same issue a few days back - this stackoverflow link has one way how the issue was resolved. It was able to expose the metrics using jmx exporter, which got scraped by Prometheus.
I enabled JMX on the kafka brokers on port 8081. When I view the MBean properties in jConsole, I only see the following for kafka.consumer-
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=ReplicaFetcherThread-2-413
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=ReplicaFetcherThread-0-413
But none of the other ones that are identified in here under Kafka Consumer Metrics are emitted by JMX.
Kafka Version # 0.8.2.1
I am specifically interested in -
kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)
Any thoughts?
The JMX PORT you are listening is the broker port. But the Mbean of kafka.consumer: is the consumer jvm metrics. So if you have another JVM that is consume a topic, you can see kafka.consumer Mbeans.
ConsumerLag is an overloaded term in Kafka, it can stand for:
Consumer's metric: Calculated difference between a consumer's current log offset and a producer’s current log offset. You can find it under JMX bean, if you're using Java/Scala based consumer (e.g. pykafka consumer doesn't export metrics):
kafka v0.8.2.x:
kafka.consumer:type= ConsumerFetcherManager, name=MaxLag, clientId=([-.\w]+)
kafka v0.9+:
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+)
Consumer lag used to be stored in ZooKeeper (Kafka <= v0.8), newer versions of Kafka have special topic __consumer_offsets that stores each consumer's lag. There are tools (e.g. kafka-manager) that can compute lag by consuming messages from this topic and calculating lag. In kafka-manager you have to enable this feature for each cluster:
[ ] Poll consumer information (Not recommended for large # of consumers)
Broker's metric: Represent the offset differences between partition leaders and their followers. You can find this metric under JMX bean:
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)
This may help to find it for 0.8, but I am currently running a Kafka 0.10 broker and consumer. When using a console consumer, I pointed jconsole to this consumer and found on the MBeans TAB: kafka.consumer-> consumer-fetcher-manager-metric -> consumer-1 -> Attributes -> records-max-lag.