I am trying to do this without using Confluent Control Center, since I do not have a license.
I am able to see the Kafka Broker metrics by using dcos task metrics details <broker-id> and see that all of these are already exposed on my DCOS Prometheus instance.
However, I do not see any consumer/producer metrics available on Prometheus, despite having some producers/consumer tasks on dcos.
Is there a process I can follow to expose kafka prodcuer/consumer metrics on dcos? I tried the following https://github.com/ibm-cloud-architecture/refarch-eda/blob/master/docs/kafka/monitoring.md .
But from my understanding we cannot use JMX on a Kafka instance hosted on DCOS (yet) (soruce: https://jira.mesosphere.com/browse/DCOS_OSS-3632?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel)
Any ideas?
You would have to add Prometheus JMX exporters to each of your Kafka Java processes, and then you would need to have the Prometheus server be able to scrape those. You would do this by downloading that JAR in each of the processes (containers?), then editing the KAFKA_OPTS environment varible to include the -javaagent option
AFAIK, this does not require setting up a remotely accessible JMX port.
Note: Control Center doesn't monitor JMX values. It uses Kafka MetricsReporters and Interceptors. Use of these interfaces, if you chose to write your own, or find others, doesn't require Control Center at all.
Related
I am deploying a kafka cluster on Kubernetes using Strimzi kafka operator. I need to be able to query Kafka JMX MBEANS remotely through HTTP/REST using Jolokia (Jolokia is an agent that converts and exposes JMX MBEAN measurements for querying over HTTP…).
AFAIK, Strimzi documentation does not provide any hint on how to attach Jolokia to kafka brokers. Hence can you please provide a hint on what kind of modifications to the deployment files (strimzi operator and/or cluster deployment files) so that Jolokia is attached to the brokers/Zookeeper instances.
Last I checked, Strimzi offers Prometheus JMX Exporter already, not Jolokia. Prometheus also offers Mbeans over HTTP. https://strimzi.io/docs/operators/latest/overview.html#metrics-overview_str
But the concept is the same, and the fact you're using Strimzi doesn't really matter, since the process is the same regardless of how Kafka is running - you need that JVM agent added into KAFKA_OPTS environment variable. You might want to use a custom docker image that has the Jolokia agent available
We have Spring Boot applications deployed on OKD (The Origin Community Distribution of Kubernetes that powers Red Hat OpenShift). Without much tweaking by devops team, we got in prometheus scraped kafka consumer metrics from kubernetes-service-endpoints exporter job, as well as some producer metrics, but only for kafka connect api, not for standard kafka producer api. This is I guess a configuration for that job:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
What is needed to change in scrape config in order to collect what's been missing?
This issue with micrometer is the source of the problem.
So, we could add jmx exporter, or wait for the issue resolution.
I deployed confluent on a Single cloud on a Linux VM and I'm planning to configure interceptors for the Consumers.
But how do I configure Control Center in order to access its Web-Interface?
Deployed Confluent Control Center Enterprise Edition.
Not very clear what issues you're having.
Edit the server.properties file. Add the bootstrap servers, Zookeeper quorum, and optionally the connect cluster.
With that, you should at least be able to see the topics in the cluster
Run the control-center binary with the server property file as a parameter.
See additional setup steps required for the clients to get monitoring to work at the documentation
I have been working on Prometheus for the last one year, with various exporters like Redis, MongoDB, ElasticSearch, Nginx, etc. Recently I started working on JMX exporter for Kafka and Cassandra which is a 3rd party exporter. I have created a "Kafka port forwarder(KPF)" instance that runs JMX exporter, with kafka.yaml file. If anyone of you might be knowing, JMX has opened a remote port for Kafka at :10054. We have multiple kafka servers with 10054 remote port. We have tweaked the kafka.yaml to scrape metrics from one of the Kafka server over :7071, so we provide :7071 to Prometheus config file. But now we need to do the same for, say 50 Kafka servers.
So I wanted to know if there is a way we can provide port range for a job-name or target-group in Prometheus config file. I know it is complicated. I am providing a diagram to simplify my architecture. Kafka-port-forwarding
Also, we need to open two ports per Kafka server (the other is for Zookeeper). This may also be used for any other service not just Kafka.
This is not possible, you must list each target individually in your static_configs/file_sd_configs file. This could be auto-generated by the templating of your configuration management system.
We are using kafka cluster and we want monitor this cluster.
Current our approach:
collect jmx metrics (telegraf jolokia plugin or jmxtrans)
store into influx
render via grafana
But we faced with problem of static configuration of kafka broker's list.
We can configure only static list via Jmxtrans or Telegraf jolokia plugin.
We would like dynamic list (example configure only kafka cluster zookeeper connection for getting brokers list in each iteration of collecting metrics) for case scaling up/down of kafka.
May be do exist another instruments for monitoring dynamic kafka cluster?
My finally solution next:
Custom bash input plugin (because i don't want build own telegraf with one custom go input plugin and I know not very well go language now:) )
In bash plugin zkCli is used for all kafka nodes discovery
Next bash script post bulk jolokia requet to each kafka node, aggregates and format to influx plugin.