I have setup kakfa on ec2 instance. I have assigned elastic ip address to that instance. I am able to start the zoo keeper and kafka and create topics. I am not able to connect to broker from my local machine. When i searched, I understood tgar I need to configure listener and advertised host name in the server properties file. I tried enterung the public elasticip address but its not working.
Where am I going wrong and what values do I need to configure. I want a basic single node sigle broker kafka setup on ec2.
You need to configure your advertised.listeners per https://rmoff.net/2018/08/02/kafka-listeners-explained/
Related
I have created an multi-node Azure Databricks cluster inside a VNET & I have created a multi-node Kafka HDInsight cluster inside different VNET. I have peered this 2 VNETs. After peering, my 2 machines are able to ping each other.
I am trying to dump messages to Kafka topic from Databricks cluster using Spark Structured Streaming & I am getting socket timeout error.
Upon research, I found that in Kafka we need to setup listeners & advertised.listeners in server.properties file.
In my scenario, what should I put the values for listeners & advertised.listeners? Would be very helpful if anyone can suggest me what all changes I need to make in server.properties file.
You need to create a listener for the host/IP on which your client machine (where Spark is running) can connect to your broker.
See https://rmoff.net/2018/08/02/kafka-listeners-explained/
I have this question regarding configuring kafka listeners properties correctly -
listeners and advertised.listeners.
In my config I am setting below props:
listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://u-kafkatst-kafkadev-5.sd.xxx.com:9092
The clients connect using u-kafkatst-kafkadev-5.sd.xxx.com:9092. Do I need to have the same value in listener and advertised.listeners. Here u-kafkatst-kafkadev-5.sd.xxx.com is a dns record that points to the host where kafka broker is running.
What are the situations where I would want to keep them same and different?
Thanks!
The advertised.listeners property is important if you are doing anything other than connecting to a broker directly on the same network. If you are using Docker, Kubernetes, IaaS (AWS, GCP, etc) then you need to expose the external address for the client to know where to connect to.
This article explains it all in depth.
I'm not sure if this should be a Druid or Zookeeper question.
Druid uses Zookeeper to discover Druid nodes. In my setup, the Druid nodes are AWS autoscale instances and get hostnames that don't resolve via DNS. So when Druid asks ZK, it gets back:
spanky-asg-druid-master-10-0-10-218.sandbox-foo.net:8081
Is there a way to either configure ZK to return an IP instead of a hostname or for Druid to ask ZK for an IP?
If you set the druid.host=1.2.3.4 (using the right IP, of course) in the runtime properties of the Druid processes, then they will announce in ZK using IPs instead of hostnames.
I have a cluster inside a VPN which contains a server with private IP. I'm trying to set up a Kafka communication between an external server to my private server. My approach is to set an IP table where a public IP is pointing my private IP. Also, I opened the port 9092 and 9093 to make it reachable from outside. Now I am available to connect successfully to my server with the public IP from the external server.
telnet <public_ip> 9092
Connected to <public_ip>
My kafka broker is under a cloudera cluster and I created it with Cloudera Manager. The configuration is the following:
kafka.properties:
listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.host.name:
<public_ip>
Using this broker configuration the comunication works perfectly inside the cluster either using the public_ip or private_ip of the kafka broker host.
What I see now is that I have a working broker that can be used with a public_ip and a external server that is able to reach the public_ip and it's required ports. But when I try to connect to the broker from a external server, I have the following error:
NO BROKERS AVAILABLE
There's no more information of the error. On my external server I have the kafka python package where I configure the producer as:
"bootstrap_servers": ["<publi_ip>:9092"]
on a existing TOPIC of my kafka broker.
Especifications:
private host
cloudera: CDH 5.12.0
kafka: kafka 2.2.0-1.2.2.0
zookeeper: Zookeeper 3.4.5
external host
kafka Python package: kafka-python==1.4.2
The problem is very similar to this post. But in this case he uses a forwarded port with public ip. Is any possibility to do it with ip tables? Anyone has managed to do it on a cloudera cluster?
Thank you in advance.
The question isn't specific to Cloudera or Python. And I don't think Cloudera Manager has some setting that'll set this up for you.
advertised.listeners will have to be a publicly resolvable address that can be used to access each broker individually by clients (e.g two brokers cannot have the same listener setting and be used from a port forward from the public address to the internal address)
Your setup is very similar to Kafka running in Docker or Cloud providers such as AWS, in that you're interacting over two networks, so refer to this blog for more information
Also, unless you setup some other firewall settings to prevent random access, don't expose brokers in the plaintext protocol
In a PoC cloudera cluster provided with kafka(1 broker) I'm trying to send messages to the cluster with an external producer through a public IP. My cluster nodes are hosted in Azure and all of them are reachable from my local machine where the external producer is running.
Following this great article https://rmoff.net/2018/08/02/kafka-listeners-explained/ I figured out the kafka listeners config I need as this broker public IP is only reachable from my local machine and inside the cluster they can only communicate using their internal hostnames:
listeners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://internal-broker-node-hostname:19092,EXTERNAL://public-broker-ip:9092
inter.broker.listener.name=INTERNAL
With this configuration when I try to start the broker I get this error:
org.apache.kafka.common.config.ConfigException: Only one of inter.broker.listener.name and security.inter.broker.protocol should be set.
Which makes sense as in Kafka doc for this property inter.broker.listener.name they say:
Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocol. It is an error to set this and security.inter.broker.protocol properties at the same time.
The problem I'm facing is that I'm unable to unset security.inter.broker.protocol through the cloudera manager as I only have the option of switching options in a radio button.
How can I unset that property?
Internal and external are just strings
So, you could set PLAINTEXT for CDH and just redefine the external listener as being advertised
listeners=PLAINTEXT://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
advertised.listeners=PLAINTEXT://internal-broker-node-hostname:19092,EXTERNAL://public-broker-ip:9092
You'll also want to make sure your external broker IPs are static