I am deploying a kafka cluster on Kubernetes using Strimzi kafka operator. I need to be able to query Kafka JMX MBEANS remotely through HTTP/REST using Jolokia (Jolokia is an agent that converts and exposes JMX MBEAN measurements for querying over HTTP…).
AFAIK, Strimzi documentation does not provide any hint on how to attach Jolokia to kafka brokers. Hence can you please provide a hint on what kind of modifications to the deployment files (strimzi operator and/or cluster deployment files) so that Jolokia is attached to the brokers/Zookeeper instances.
Last I checked, Strimzi offers Prometheus JMX Exporter already, not Jolokia. Prometheus also offers Mbeans over HTTP. https://strimzi.io/docs/operators/latest/overview.html#metrics-overview_str
But the concept is the same, and the fact you're using Strimzi doesn't really matter, since the process is the same regardless of how Kafka is running - you need that JVM agent added into KAFKA_OPTS environment variable. You might want to use a custom docker image that has the Jolokia agent available
Related
I am building a SvelteKit application with Kafka Support. I have created a kafka.js file and tested the kafka support with a local kafka setup, which is successful. Now, When I replaced the topic name with a topic that is running in the kafka of our Kubernetes cluster, am not seeing any response.
How do I test this connection that is establishing between Kafka in Kubernetes cluster and the JS web application ? Any hints could be much helpful. Doing just console logs are not helpful so far because kafka itself not getting hit.
Any two pods deployed in the same namespace can communicate using local service names. So, do the brokers have a Service resource?
For example, assuming you are in namespace: default, and you have kafka-svc, then you'd setup bootstrap.servers: kafka-svc.svc.cluster.local.
You also need to configure Kafka's advertised.listeners. Related blog - https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/
But this requires NodeJS (or other language) backend, and not SvelteKit UI (which cannot connect to backend TCP server, only use some HTTP bridge). Your HTTP options would include some custom server, or Confluent REST Proxy, Strimzi Kafka Bridge, etc. But you were already told this.
We have Spring Boot applications deployed on OKD (The Origin Community Distribution of Kubernetes that powers Red Hat OpenShift). Without much tweaking by devops team, we got in prometheus scraped kafka consumer metrics from kubernetes-service-endpoints exporter job, as well as some producer metrics, but only for kafka connect api, not for standard kafka producer api. This is I guess a configuration for that job:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
What is needed to change in scrape config in order to collect what's been missing?
This issue with micrometer is the source of the problem.
So, we could add jmx exporter, or wait for the issue resolution.
I am trying to do this without using Confluent Control Center, since I do not have a license.
I am able to see the Kafka Broker metrics by using dcos task metrics details <broker-id> and see that all of these are already exposed on my DCOS Prometheus instance.
However, I do not see any consumer/producer metrics available on Prometheus, despite having some producers/consumer tasks on dcos.
Is there a process I can follow to expose kafka prodcuer/consumer metrics on dcos? I tried the following https://github.com/ibm-cloud-architecture/refarch-eda/blob/master/docs/kafka/monitoring.md .
But from my understanding we cannot use JMX on a Kafka instance hosted on DCOS (yet) (soruce: https://jira.mesosphere.com/browse/DCOS_OSS-3632?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel)
Any ideas?
You would have to add Prometheus JMX exporters to each of your Kafka Java processes, and then you would need to have the Prometheus server be able to scrape those. You would do this by downloading that JAR in each of the processes (containers?), then editing the KAFKA_OPTS environment varible to include the -javaagent option
AFAIK, this does not require setting up a remotely accessible JMX port.
Note: Control Center doesn't monitor JMX values. It uses Kafka MetricsReporters and Interceptors. Use of these interfaces, if you chose to write your own, or find others, doesn't require Control Center at all.
I am looking at using Kafka but the documentation states that I need to set up Zookeeper. I already have a service discovery set up, I am using Consul. I don't want to have to look after Zookeeper as well.
Can you use Consul instead of Zookeeper to run Kafka? If so is there any documentation on how to do this anywhere?
Apache Kafka currently supports only Apache Zookeeper. It doesn't have any out of the box support for Consul.
This long-standing Kafka issue tracks adding Zookeeper alternatives. It may never happen, so in the meantime, there's the option of running a Zookeeper proxy like zetcd or parkeeper in front of etcd or Consul.
Kafka only uses Zookeeper for broker and topic discovery. Short of adding Consul support to the code base yourself, there is no other alternatives.
We are using kafka cluster and we want monitor this cluster.
Current our approach:
collect jmx metrics (telegraf jolokia plugin or jmxtrans)
store into influx
render via grafana
But we faced with problem of static configuration of kafka broker's list.
We can configure only static list via Jmxtrans or Telegraf jolokia plugin.
We would like dynamic list (example configure only kafka cluster zookeeper connection for getting brokers list in each iteration of collecting metrics) for case scaling up/down of kafka.
May be do exist another instruments for monitoring dynamic kafka cluster?
My finally solution next:
Custom bash input plugin (because i don't want build own telegraf with one custom go input plugin and I know not very well go language now:) )
In bash plugin zkCli is used for all kafka nodes discovery
Next bash script post bulk jolokia requet to each kafka node, aggregates and format to influx plugin.