I need to write an API to display consumer metrics in java. I thought of using Kafka admin client. Is there any way we can retrieve consumer metrics information
I checked admin client code. I have Kafka consumer class. I have consumer metrics method, but it is not holding the information.
Kafka consumer can report a lot of metrics through JMX, you just need to add a few Java properties to your consumer, either in the code or through command-line. For example, for console consumer:
$ KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9398 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" kafka-console-consumer ...
Related
I want to monitor kafka topic message in prometheus. I will be using kafka connect for the same, but I want to understand how to get the message content details in prometheus tsdb.
You would need to use PushGateway since Prometheus scrapes endpoints, and has no Kafka consumer API. Similarly, Kafka Connect Sinks don't tend to populate any internal metrics server, other than their native JMX server. In other words, JMX Exporter won't let you "see Kafka messages".
There are HTTP Kafka Connect sinks that exist, and you should try using them to send data to PushGateway API.
However, this is a poor use-case for Prometheus.
InfluxDB, for example, can be populated by a Telegraf Kafka consumer (or Kafka Connect), or Elasticsearch by LogStash (or Kafka Connect).
Then Grafana can use Influx or Elasticseach as a datasource to view events... Or it can also use a regular RDBMS
We are currently on HDF (Hortonworks Dataflow) 3.3.1 which bundles Kafka 2.0.0 and are trying to use Kafka Connect in distributed mode to launch a Google Cloud PubSub Sink connector.
We are planning on sending back some metadata into a Kafka Topic and need to integrate a Kafka producer into the flush() function of the Sink task java code.
Would this have a negative impact on the process where Kafka Connect commits back the offsets to Kafka (as we would be adding a overhead of running a Kafka producer before the flush).
Also, how does Kafka Connect get the Bootstrap servers list from the configuration when it is not specified in the Connector Properties for either the sink or the source? I need to use the same Bootstrap server list to start the producer.
Currently I am changing the config for the sink connector, adding bootstrap server list as a property and parsing it in the Java code for the connector. I would like to use bootstrap server list from the Kafka Connect worker properties if that is possible.
Kindly help on this.
Thanks in advance.
need to integrate a Kafka producer into the flush() function of the Sink task java code
There is no producer instance exposed in the SinkTask API...
Would this have a negative impact on the process where Kafka Connect commits back the offsets to Kafka (as we would be adding a overhead of running a Kafka producer before the flush).
I mean, you can add whatever code you want. As far as negative impacts go, that's up to you to benchmark on your own infrastructure. Obviously adding more blocking code makes the other processes slower overall
how does Kafka Connect get the Bootstrap servers list from the configuration when it is not specified in the Connector Properties for either the sink or the source?
Sinks and sources are not workers. Look at connect-distributed.properties
I would like to use bootstrap server list from the Kafka Connect worker properties if that is possible
It's not possible. Adding extra properties to the sink/source configs are the only way. (Feel free to make a Kafka JIRA requesting such a feature of exposing the worker configs, though)
I am looking for a kafka rest api which will list the same details as
kafka-consumer-groups.sh - - describe
Would return, basically i am trying to get the details of offset of each partition and its lag for a particular consumer group
There's several tools outside of Kafka to do this
Remora https://github.com/zalando-incubator/remora
LinkedIn Burrow
Various Prometheus Kafka lag exporters
I am trying to monitor the consumer lag for old consumer groups ( which uses zookeeper) using JMX.
This is how I have enabled JMX.
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=`hostname -f` -Djava.net.preferIPv4Stack=true"
if [ "x$ISKAFKASERVER" == "xtrue" ]; then
export JMX_PORT=9887
And I am able to see multiple been when I connect using JConsole. However I don't see kafka.consumer
Seems like you enabled jmx on kafka brokers. You need to enable jmx on the instances where consumer is running for getting consumer metrics
I have a Kafka cluster running on Mesos. I'm trying to increase number of partitions on a topic. That usually works with bin/kafka-topics.sh --alter command. Is this option exposed via dcos cli or kafka-mesos rest API? From what I see its not exposed.
If not, what is the best way to access kafka's cli within mesos installation?
Right now I use dcos cli to get broker IP and then in an adhoc way get to
/var/lib/mesos/slave/slaves/11aaafce-f12f-4aa8-9e5c-200b2a657225-S3/frameworks/11aaafce-f12f-4aa8-9e5c-200b2a657225-0001/executors/broker-1-7cf26bed-aa40-464b-b146-49b45b7800c7/runs/849ba6fb-b99e-4194-b90b-8c9b2bfabd7c/kafka_2.10-0.9.0.0/bin/kafka-console-consumer.sh
Is there a more direct way?
We've just released a new version of the Kafka framework with DC/OS 1.7. The new version supports changing the partition count via dcos kafka [--name=frameworkname] topic partitions <topicname> <count>.
See also: Service documentation ("Alter Topic Partition Count" reference)