Is there a way to get number of messages that arrived on a Kafka topic in a day?
Looking for a solution that can fetch the number of messages arrived on a topic for a particular day.
Ps- we have confluent enterprise and also using prometheus and grafana for metrics.
I'm not familiar with confluent enterprise but we are collecting Kafka metrics using Burrow to Prometheus and using this metric to show incoming message rate:
sum(rate(**kafka_burrow_topic_partition_offset**{cluster="Kafka1",topic="myTopic"}[5m]))
read this https://www.datadoghq.com/blog/collecting-kafka-performance-metrics/#monitor-consumer-health-with-burrow
From Kafka Monitoring doc:
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
Related
I created an application which use Kafka. So how can I find how many MB/sec my consumers reads ?
My topics have only one partition
Are you using the Java Kafka consumer API? If yes, some JMX metrics are exposed by it and some more specific to consumer fetching. You can see them here:
https://kafka.apache.org/documentation/#consumer_fetch_monitoring
I am trying to read all messages in the topic of Kafka. I am using confluent cloud service so don't run Kafka in my localhost. I set the configurations as: 'enable.auto.commit': 'True','auto.offset.reset': 'earliest', 'default.topic.config': {'auto.offset.reset': 'smallest'}. However it gives me no message, or if I send message from producer at the same time it gives only that message not all offset messages.
How can I read all offset messages in python?
I didn't use confluent cloud service, If you want to consume all offsets, there are a few things you should pay attention to.
set new consumer groupId that does not consume any data
set 'auto.offset.rese=earliest' or 'auto.offset.reset=smallest' , you need to refer to the kafka version
pay attention to the automatic expiration date of the Topic
I am looking for a kafka rest api which will list the same details as
kafka-consumer-groups.sh - - describe
Would return, basically i am trying to get the details of offset of each partition and its lag for a particular consumer group
There's several tools outside of Kafka to do this
Remora https://github.com/zalando-incubator/remora
LinkedIn Burrow
Various Prometheus Kafka lag exporters
I am trying to write an ingestion application on GCP by using Apache Beam.I should write it in a streaming way to read data from Kafka or pubsub topics and then ingest to datasource.
while it seems straight forward to write it with pubsub and apache beam but my question is what would happen if my ingestion fails or to be restarted and if it again reads all data from the start of pubsub topic or like kafka it can read from latest committed offsets in the topic?
Pub/sub messages are persisted until they are delivered and acknowledge by the subscribers which receives pending messages from its subscription. Once the message is acknowledge, it's removed from the subscription's queue.
For more information regarding the message flow, check this document
Hope it helps.
We are going to replace our Activemq broker with Kafka broker. In activemq we could send a message to it so that it sends us its queues' metrics.
But I can't find a way in kafka to ask for its topic metrics. Can someone help me?
Kafka metrics are of 3 types.
Kafka Server (Broker) metrics
Producer metrics
Consumer metrics
You can collect Kafka metrics using JMX (https://github.com/jmxtrans/jmxtrans).
For more information on collecting the metrics and monitoring those via a dashboard, read the following link, https://softwaremill.com/monitoring-apache-kafka-with-influxdb-grafana/