I'm not able to connect to zk of other kafka cluster nodes. To debug the issue, have tried connecting to other zk cluster ip on port 2181 with telnet, but have observed that its not able to connect/reach.
i.e.
From ip1 box :
telnet <ip2> 2181
is not working/reachable.
whereas, if i try to telnet to same box ip for port 2181 its working fine & is connected.
i.e.
From IP1 box, telnet 2181 is working fine.
How can i debug this issue & check the root cause?
Also, please confirm if confluent zk uses telnet internally for connectivity to zk cluster ?
There is no such thing as "Confluent ZK".
Zookeeper does not use Telnet to verify connectivity between nodes. It has its own protocols, which are at the JVM level, using Java TCP Sockets.
Related
Update
I'm updating this post to reflect the current configuration (and also, following #OneCricketeer response, more info)
According to this 2018 blog (which everyone seems to refer to) I am running Kafka (in a Docker Compose stack) with this configuration:
KAFKA_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER:PLAINTEXT,LOCAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
the Kafka broker should listen on both ports 29092 (used "within" the Docker network) as well as port 9092 (used by clients running on the host).
The problem is that it seems that the Kafka broker only responds on port 29092 when started like this:
image: confluentinc/cp-kafka:7.0.1
hostname: kafka
container_name: kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
(environment as above)
Using kafkacat this is what I get with the configuration above:
└─( nc -vz localhost 9092
Connection to localhost 9092 port [tcp/*] succeeded!
└─( kafkacat -b localhost:9092 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure
Opening and connection on port 29092 works "as expected" only I would have to hack around /etc/hosts to make kafka0 point back to 127.0.0.1, which, as pointed out is A Really Bad Idea (hence the whole point of this question).
The diagrams and the text in that Confluent blog lead me to believe that either I'm missing something in the configuration of the container, or things have changed since 2018.
So I finally solved the riddle, thanks in part to this question.
The TL;DR version is that the host part of the listeners configuration has a different (opposite) meaning than it has for advertised.listeners: in the latter it means the host the client ought to talk to, in the former, the NIC address(es) the server should bind to.
In other words, changing the value of the host in listeners to be 0.0.0.0 (all available NICs) makes everything work:
KAFKA_LISTENERS: DOCKER://0.0.0.0:29092,LOCAL://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER:PLAINTEXT,LOCAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
from the host machine (outside the container):
└─( kafkacat -b localhost:9092 -L
Metadata for all topics (from broker 1: localhost:9092/1):
1 brokers:
broker 1 at localhost:9092 (controller)
4 topics:
topic "devices" with 1 partitions:
...
while from within the Docker network, it all works as before.
For example:
{
"#timestamp":"2021-12-25T00:05:27Z",
"#service":"benthos",
"component":"benthos.output",
"level":"INFO",
"message":"Sending Kafka messages to addresses: [kafka0:29092]"
}
and clients running on the host machine can connect to the Kafka cluster:
from pykafka import KafkaClient
client = KafkaClient(hosts='localhost:9092')
print(f"Available topics: {client.topics}")
# Produce or Consume data here
The full code sample is available here.
Worth noting that, AFAICT, the original blog post was giving an incorrect configuration for the listeners.
Never modify /etc/hosts to "fix" your listeners.
Nothing has changed until Kafka completely removes the need for Zookeeper.
lead me to believe that either I'm missing something in the configuration of the container
There are default listeners, but ports in Docker Compose has nothing to do with the actual broker server configuration. There are not two ports listening for external traffic. Only one for each internal and external.
Note that you are returned kafka0, as per your advertised setting, which is not the hostname you've set in your compose file, so your internal clients will have errors resolving hostname kafka0...
seems that Kafka only listens on port 29092 when started like this
It listens on both just fine.
$ nc -vz localhost 9092
Connection to localhost port 9092 [tcp/XmlIpcRegSvc] succeeded!
$ nc -vz localhost 29092
Connection to localhost port 29092 [tcp/*] succeeded!
The kafka protocol doesn't respond with the "bob" internal listener outside of the Docker network, no.
How many?
As many as necessary for your network. For example, I've seen up to four for internal replication using SASL, external clients using SSL, PLAINTEXT while we've migrated those clients to SSL, then later added SASL_SSL for more authorization controls.
I currently can produce and consume messages on my local machine. I want to change it so that 1 machine has zookeeper, the broker, and the producer and another machine has the consumer. What server properties do I need to adjust to make this work?
In server-properties, I tried uncommenting listeners=PLAINTEXT://:9092, setting advertised.listeners=PLAINTEXT://< machineA's static IP >:9092, as well as changing zookeeper.connect=localhost:2181 to zookeeper.connect=< Machine A's static IP >:9092.
However, none of above have worked so far. Any guidance in the right direction would be very helpful.
setting advertised.listeners=PLAINTEXT://< machineA's static IP >:9092, as well as changing zookeeper.connect=localhost:2181 to zookeeper.connect=< Machine A's static IP >:9092
The zookeeper.connect should point to zookeeper port and not the kafka broker port. So, it should be
zookeeper.connect=<Machine A's static IP>:2181
Specifies the ZooKeeper connection string in the form hostname:port
where host and port are the host and port of a ZooKeeper server
Broker configurations reference
Also, ensure that your firewall isn't restricting access to your consumer on the broker port (9092). If it is linux and you are using iptables, you may want to allow that port.
iptables -A INPUT -p tcp --dport 9092 -j ACCEPT
I have a cluster inside a VPN which contains a server with private IP. I'm trying to set up a Kafka communication between an external server to my private server. My approach is to set an IP table where a public IP is pointing my private IP. Also, I opened the port 9092 and 9093 to make it reachable from outside. Now I am available to connect successfully to my server with the public IP from the external server.
telnet <public_ip> 9092
Connected to <public_ip>
My kafka broker is under a cloudera cluster and I created it with Cloudera Manager. The configuration is the following:
kafka.properties:
listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.listeners=PLAINTEXT://<private_ip>:9092,SSL://<private_ip>:9093
advertised.host.name:
<public_ip>
Using this broker configuration the comunication works perfectly inside the cluster either using the public_ip or private_ip of the kafka broker host.
What I see now is that I have a working broker that can be used with a public_ip and a external server that is able to reach the public_ip and it's required ports. But when I try to connect to the broker from a external server, I have the following error:
NO BROKERS AVAILABLE
There's no more information of the error. On my external server I have the kafka python package where I configure the producer as:
"bootstrap_servers": ["<publi_ip>:9092"]
on a existing TOPIC of my kafka broker.
Especifications:
private host
cloudera: CDH 5.12.0
kafka: kafka 2.2.0-1.2.2.0
zookeeper: Zookeeper 3.4.5
external host
kafka Python package: kafka-python==1.4.2
The problem is very similar to this post. But in this case he uses a forwarded port with public ip. Is any possibility to do it with ip tables? Anyone has managed to do it on a cloudera cluster?
Thank you in advance.
The question isn't specific to Cloudera or Python. And I don't think Cloudera Manager has some setting that'll set this up for you.
advertised.listeners will have to be a publicly resolvable address that can be used to access each broker individually by clients (e.g two brokers cannot have the same listener setting and be used from a port forward from the public address to the internal address)
Your setup is very similar to Kafka running in Docker or Cloud providers such as AWS, in that you're interacting over two networks, so refer to this blog for more information
Also, unless you setup some other firewall settings to prevent random access, don't expose brokers in the plaintext protocol
We are trying to consume from a Kafka Cluster using the Java Client. The Cluster is a behind a Jump host and hence the only way to access is through a SSH Tunnel. But we are not able read because once the consumer fetches metadata it uses the original hosts to connect to brokers. Can this behaviour be overridden? Can we ask Kafka Client to not use the metadata?
Not as far as I know.
The trick I used when I needed to do something similar was:
setup a virtual interface for each Kafka broker
open a tunnel to each broker so that broker n is bound to virtual interface n
configure your /etc/hosts file so that the advertised hostname of broker n is resolved to the ip of the virtual interface n.
Es.
Kafka brokers:
broker1 (advertised as broker1.mykafkacluster)
broker2 (advertised as broker2.mykafkacluster)
Virtual interfaces:
veth1 (192.168.1.1)
veth2 (192.168.1.2)
Tunnels:
broker1: ssh -L 192.168.1.1:9092:broker1.mykafkacluster:9092 jumphost
broker2: ssh -L 192.168.1.2:9092:broker1.mykafkacluster:9092 jumphost
/etc/hosts:
192.168.1.1 broker1.mykafkacluster
192.168.1.2 broker2.mykafkacluster
If you configure your system like this you should be able reach all the brokers in your Kafka cluster.
Note: if you configured your Kafka brokers to advertise an ip address instead of a hostname the procedure can still work but you need to configure the virtual interfaces with the same ip address that the broker advertises.
You don't actually have to add virtual interfaces to acces the brokers via SSH tunnel if they advertise a hostname. It's enough to add a hosts entry in /etc/hosts of your client and bind the tunnel to the added name.
Assuming broker.kafkacluster is the advertised.hostname of your broker:
/etc/hosts:
127.0.2.1 broker.kafkacluster
Tunnel:
ssh -L broker.kafkacluster:9092:broker.kafkacluster:9092 <brokerhostip/name>
Try sshuttle like this:
sshuttle -r user#host broker-1-ip:port broker-2-ip:port broker-3-ip:port
Of course, the list of broker depends on advertised listeners broker setting.
Absolutely best solution for me was to use kafkatunnel (https://github.com/simple-machines/kafka-tunnel). Worked like a charm.
Changing the /etc/hosts file is NOT the right way.
Quoting Confluent blog post:
I saw a Stack Overflow answer suggesting to just update my hosts file…isn’t that easier?
This is nothing more than a hack to work around a misconfiguration instead of actually fixing it.
You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise, they’ll try to connect to the internal host address—and if that’s not reachable, then problems ensue.
Confluent blog post
Additionally you can have a look at this Pull Request on GitHub where I wrote an integration test to connect to Kafka via SSH. It should be easy to understand even if you don't know Golang.
There you have a full client and server example (see TestSSH). The test is bringing up actual Docker containers and it runs assertions against them.
TL;DR I had to configure the KAFKA_ADVERTISED_LISTENERS when connecting over SSH so that the host advertised by each broker would be one reachable from the SSH host. This is because the client connects to the SSH host first and then from there it connects to a Kafka broker. So the host in the advertised.listeners must be reachable from the SSH server.
I am in a bit of a bind (pun intended).
I have a ubuntu server running kafka & zookeeper. This server has both ipv4 and ipv6 protocols installed.
In the server.properties file, I specified the host.name and advertised.host.name as my public ipv4 address.
When I start zookeeper and kafka, kafka still binds to ipv6 port.
$ netstat -l -t | grep 9092
tcp6 0 0 ferozed-linux3.mydomain:9092 [::]:* LISTEN
The client machine from which I am running a producer in a MapReduce job, is an IPv4 machine. Due to the fact that the server is binding to an IPv6 interface, the client is unable to connect to it.
Any idea how I can fix this?
Starting from Andrey's answer, you can solve it just by adding an environment variable
export KAFKA_OPTS="-Djava.net.preferIPv4Stack=True"
If I understand you correctly, you need the following:
System.setProperty ("java.net.preferIPv4Stack", "true");
in the code, or
-Djava.net.preferIPv4Stack = True
in the web server startup script worked.
Did you try it?
Add this is kafka/bin/Kafka-run-class.sh
KAFKA_OPTS="-Djava.net.preferIPv4Stack=True"
And restart Kafka broker.