we have 3 Kafka brokers in a cluster running in our private VPN. our client wants to send messages to our Kafka cluster. our admin has created public-IP for only one Kafka broker out of 3. client cluster is able to communicate with our VPN through this public-IP(checked using Wireshark), but we are not receiving any messages.
Whether we need to create public IPs for the remaining 2 Kafka brokers also?
Can anyone suggest what configs should we change in order to work?
Clients are required to communicate directly with leader brokers of the partitions, so all brokers need to advertise some resolvable address for themselves through the VPN
I think you're using the VPN term wrong. You probably mean VPC or some kind of private network.
As mentioned by OneCricketeer you must assign public address to every broker. But also you must set advertised.listeners property or KAFKA_ADVERTISED_LISTENERS to the public IP address or public DNS name pointing to that public address otherwise your client will not be able to connect.
More details: https://www.confluent.io/blog/kafka-listeners-explained/
Related
I have setup kakfa on ec2 instance. I have assigned elastic ip address to that instance. I am able to start the zoo keeper and kafka and create topics. I am not able to connect to broker from my local machine. When i searched, I understood tgar I need to configure listener and advertised host name in the server properties file. I tried enterung the public elasticip address but its not working.
Where am I going wrong and what values do I need to configure. I want a basic single node sigle broker kafka setup on ec2.
You need to configure your advertised.listeners per https://rmoff.net/2018/08/02/kafka-listeners-explained/
I have this question regarding configuring kafka listeners properties correctly -
listeners and advertised.listeners.
In my config I am setting below props:
listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://u-kafkatst-kafkadev-5.sd.xxx.com:9092
The clients connect using u-kafkatst-kafkadev-5.sd.xxx.com:9092. Do I need to have the same value in listener and advertised.listeners. Here u-kafkatst-kafkadev-5.sd.xxx.com is a dns record that points to the host where kafka broker is running.
What are the situations where I would want to keep them same and different?
Thanks!
The advertised.listeners property is important if you are doing anything other than connecting to a broker directly on the same network. If you are using Docker, Kubernetes, IaaS (AWS, GCP, etc) then you need to expose the external address for the client to know where to connect to.
This article explains it all in depth.
I have a cluster inside a VPN which contains a server with private IP. I'm trying to set up a Kafka communication between an external server to my private server. My approach is to set an IP table where a public IP is pointing my private IP. Also, I opened the port 9092 and 9093 to make it reachable from outside. Now I am available to connect successfully to my server with the public IP from the external server.
telnet <public_ip> 9092
Connected to <public_ip>
My kafka broker is under a cloudera cluster and I created it with Cloudera Manager. The configuration is the following:
kafka.properties:
listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.host.name:
<public_ip>
Using this broker configuration the comunication works perfectly inside the cluster either using the public_ip or private_ip of the kafka broker host.
What I see now is that I have a working broker that can be used with a public_ip and a external server that is able to reach the public_ip and it's required ports. But when I try to connect to the broker from a external server, I have the following error:
NO BROKERS AVAILABLE
There's no more information of the error. On my external server I have the kafka python package where I configure the producer as:
"bootstrap_servers": ["<publi_ip>:9092"]
on a existing TOPIC of my kafka broker.
Especifications:
private host
cloudera: CDH 5.12.0
kafka: kafka 2.2.0-1.2.2.0
zookeeper: Zookeeper 3.4.5
external host
kafka Python package: kafka-python==1.4.2
The problem is very similar to this post. But in this case he uses a forwarded port with public ip. Is any possibility to do it with ip tables? Anyone has managed to do it on a cloudera cluster?
Thank you in advance.
The question isn't specific to Cloudera or Python. And I don't think Cloudera Manager has some setting that'll set this up for you.
advertised.listeners will have to be a publicly resolvable address that can be used to access each broker individually by clients (e.g two brokers cannot have the same listener setting and be used from a port forward from the public address to the internal address)
Your setup is very similar to Kafka running in Docker or Cloud providers such as AWS, in that you're interacting over two networks, so refer to this blog for more information
Also, unless you setup some other firewall settings to prevent random access, don't expose brokers in the plaintext protocol
In a PoC cloudera cluster provided with kafka(1 broker) I'm trying to send messages to the cluster with an external producer through a public IP. My cluster nodes are hosted in Azure and all of them are reachable from my local machine where the external producer is running.
Following this great article https://rmoff.net/2018/08/02/kafka-listeners-explained/ I figured out the kafka listeners config I need as this broker public IP is only reachable from my local machine and inside the cluster they can only communicate using their internal hostnames:
listeners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://internal-broker-node-hostname:19092,EXTERNAL://public-broker-ip:9092
inter.broker.listener.name=INTERNAL
With this configuration when I try to start the broker I get this error:
org.apache.kafka.common.config.ConfigException: Only one of inter.broker.listener.name and security.inter.broker.protocol should be set.
Which makes sense as in Kafka doc for this property inter.broker.listener.name they say:
Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocol. It is an error to set this and security.inter.broker.protocol properties at the same time.
The problem I'm facing is that I'm unable to unset security.inter.broker.protocol through the cloudera manager as I only have the option of switching options in a radio button.
How can I unset that property?
Internal and external are just strings
So, you could set PLAINTEXT for CDH and just redefine the external listener as being advertised
listeners=PLAINTEXT://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
advertised.listeners=PLAINTEXT://internal-broker-node-hostname:19092,EXTERNAL://public-broker-ip:9092
You'll also want to make sure your external broker IPs are static
I have 3 Kafka brokers running in a isolated network region, my client can not connect them directly, so I have to use a VIP(virtual ip) to connect the brokers.
For example:
my brokers' IP are: 10.5.1.5, 10.5.1.6, 10.5.1.7,
my VIPs' ip are: 200.100.1.5, 200.100.1.6, 200.100.1.7, they one to one paired.
So when I indicate the bootstrap list as 200.100.1.5, the cluster response me the mixed VIPs and Broker ips, such as: 10.5.1.5, 10.5.1.6, 200.100.1.5, 200.100.1.6 ..., then the connection failed, because my program can not reach broker's ip, only can reach VIPs.
My current configuration as following, it responses both IP and VIP:
listeners=INTERNAL://:9092,EXTERNAL_PLAINTEXT://:8080
advertised.listeners=EXTERNAL_PLAINTEXT://200.100.1.5:8080,INTERNAL://10.5.1.5:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
inter.broker.listener.name=INTERNAL
How can I let Kafka only response the VIP list please.
I've got the answer, it could be the following:
advertised.listeners=PLAINTEXT://200.100.1.5:8080
listeners=PLAINTEXT://10.5.1.5:9092
And remove the listener.security and inter.broker.
You can use the broker setting called advertised.listeners to tell your brokers to include a different IP/hostname in their response to clients.
advertised.listeners:
Listeners to publish to ZooKeeper for clients to use, if different
than the listeners config property. In IaaS environments, this may
need to be different from the interface to which the broker binds. If
this is not set, the value for listeners will be used. Unlike
listeners it is not valid to advertise the 0.0.0.0 meta-address.
In your example, for the first broker you can have:
advertised.listeners=PLAINTEXT://200.100.1.5:9092
listeners=PLAINTEXT://10.5.1.5:9092