I am looking at the docs of Apache Kafka.
The consumer connects to the Kafka by using the IP address/port of zookeepers.
Is it possible to use the IP address/port of broker?
Yes, when using the Simple consumer API you get to manage consumption directly from the brokers. See usage example here
Of course you can.
The consumer connects to the Kafka by using the IP address/port of zookeepers. What you point is High-level API
And connecting to broker directly points Low-level API
Maybe this can help you
Related
Basically I want to send messages from a MQTT(mosquito) broker to a knative event source(kafka) . In case of a simple kafka broker I could use the confluent's kafkaconnect but in this case it's a knative event source rather than a broker. The problems lies with conversion to cloud events.
Since you have a current MQTT broker which can read/write to Kafka, you might the the Kafka source to convert the Kafka messages to CloudEvents and send them to your service.
If you're asking about how to connect the MQTT broker with Kafka, I'd suggest either finding or writing an MQTT source or using something outside the Knative ecosystem.
Please find the screen shot, what we are trying to achieve
Our deployment servers can't connect to internet directly without proxy. Hence needed a way to send messages to outside organization kafka cluster. Please note that we do not want to use kafka rest.
Connecting to Kafka is very complex and doesn't support this scenario; when you first connect to the bootstrap servers; they send the actual connections the client needs to make, based on the broker properties (advertised listeners).
My pipeline is:
Kerberized Kafka --> Logstash (hosted on a different server) --> Splunk.
Can I replace the Logstash component with Kafka Connect?
Could you point me to a resource/guide where I can use kerberized Kafka as a source for my Kafka connect (which is hosted separately)?
From the documentation, what I understood is that if Kafka Connect is hosted on the same cluster as that of Kafka, that's quite possible. But I don't have that option right now, as our Kafka cluster is multi-tenant and hence not approved for additional processes on the cluster.
Kerberos keytabs aren't commonly machine/JVM specific, so yes, Kafka Connect should be able to be configured very similarly to Logstash since both are JVM processes using native Kafka protocol.
You shouldn't run Connect on the brokers anyway
If you can't add Kafka Connect to an existing Kafka cluster, you will have to spin up a separate Kafka Connect (Cluster or standalone).
I've written about it here: enter link description here
How to proxy Apache Kafka producer requests on the Kafka broker, and redirect to a separate Kafka cluster?
In my specific case, it's not possible to update the clients that write to this cluster. This means, it's not feasible to do the following:
Updating the bootstrap broker configuration in the client
Rewrite the client code to support Confluent REST Proxy
Therefore, I'm looking for proxy that would work on the Kafka Protocol
Here are some potential options I've discovered so far:
Kafka-proxy
EnvoyProxy
Does anyone have any experience with the tools above (or alternative tools) that would allow me to redirect a binary TCP Kafka request to a separate Kafka cluster?
Since late 2021, Envoy proxy does that for Produce requests (with kafka-mesh-filter), however it's still a work in progress.
In general your Proxy will need to understand the Kafka protocol and maintain necessary connections to upstream Kafka cluster(s).
I am trying to create a tool that can kill kafka producers and consumers randomly in order to simulate production environment. Is there a way to find the active producers and consumers in a kafka cluster? I want to exactly know which thread in which host is acting as the producer or the consumer?
I started by getting the I.P. of at least one kafka broker from the HDP cluster.
I checked the open connections on the kafka broker on its specified port (by default it is 6667) and retrieved the IP addresses of the connected machines.
Using their IP Addresses, I found out the processes that are connected to the kafka broker.
Thus, I figured out the machines that are kafka producers and consumers.