Linux producer to windows consumer failed to resolve - apache-kafka

I'm running zookeeper, kafka servers along with a consumer client on a windows machine. When I use a simple producer program in python or c++ in a different machine also running windows, everything works normal. However, when I use the same producer program in a linux machine it does not work, it accuses failed to resolve . All computers are connected on the same network. I have tried to set the listeners and advertised.listeners in the kafka server.properties in several different manners, but with the same result.
Machine running windows with servers and consumer:
IP:192.168.0.149
hostname:something
second Machine running linux or windows (dual boot) with the producer program:
IP:192.168.0.150
hostame:somethingelse
some tried configurations at server.properties:
1
listeners=PLAINTEXT:\0.0.0.0:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
2
listeners=PLAINTEXT:\localhost:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
3
#listeners=
#advertised.listeners=
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
4
listeners=PLAINTEXT:\192.168.0.149:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows if consumer passes the ip instead of localhost, does no work on linux

Listeners should always stay as 0.0.0.0. This is the bind address to accept all external connections
If you want to run Kafka on windows, you need to deal with the windows firewall. Try disabling it while testing connections. It would probably be best that you switch the broker to run on Linux.
The advertised listeners should only be the hostname if both systems are able to resolve each other by DNS name. Otherwise, save yourself the extra network hop and use the IP as the advertised listener. If you're getting "failed to resolve", seems like a DNS issue, and ping windows-host shouldn't work either
To debug your connection, I would suggest using kafkacat
kafkacat -b kafka-host:9092 -L

Related

unable to connect to kafka broker (via zookeeper) using Conduktor client

Able to connect successfully to local kafka broker/cluster running locally (dockerized) using Conduktor, but when trying to connect to Kafka cluster running on Unix VM, getting below error.
Error:
"The broker [...] is reachable but Kafka can't connect. Ensure you have access to the advertised listeners of the the brokers and the proper authorization"
Appreciate any assistance.
running locally (dockerized)
When running in docker, you need to ensure that the ports are accessible from outside of your container. To verify this, try doing a telnet <ip> <port> and check if you are able to connect.
Since the error message says, the broker is reachable, I suppose you would be able to successfully telnet to the broker.
Next, check your broker config called advertised.listeners. Here you need to mention your IP:Port combination where IP is what you will be giving in your client program i.e. Conduktor.
An example for that would be
advertised.listeners=PLAINTEXT://1.2.3.4:9092
and then restart your broker and reconnect. If you are using ssl then you need to provide some extra configuration. See Configuring Kafka brokers for more.
Try to add in /etc/hosts (Unix-like) or C:\Windows\System32\drivers\etc\hosts (windows-like) the Kafka server in such manner kafka_server_ip kafka_server_name_in_dns (e.g. 10.10.0.1 kafka).

Kafka: Send Messages to kerberized cluster from Windows machine

I am working on a scenario where my kafka producer code will reside on a windows machine & my kafka borker is kerberized in a unix machine. Both of the machines are within the same domain. The firewall ports are open between windows & unix machine.
Need help in getting to know how to proceed on this problem.
I went through following questions:
How to Access kafka brokers secured by kerberos from Eclispe running on windows 7
Connect to Kafka on Unix from Windows with Kerberos
However, none of them have been concluded successfully.

Is there a way to start a Zookeeper server using my static ip instead of localhost

I've started learning some big data tools for a new project, and right now I'm on Kafka and Zookeeper.
I have them both install on my local machine, and I can start them up and start producing and consuming messages just fine. Now, I want to try it having two machines, one with a kafka broker, zookeepr and a producer, and the other with a consumer. Lets call them Machine A and Machine B.
Machine A has runs the Zookeeper server, the broker and a producer. Machine B runs a consumer. From what I think I understand, I should be able to setup the consumer to listen to a topic from the producer on Machine A, using Zookeeper. Since both machines are on the same network (i.e. my local home network), I thought I could change the kafka broker server.properties to use my static ip address for Machine A, and then have the consumer on Machine B connect to it.
My problem, is that zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181 so when my broker tries to connect to it using my static ip address (i.e 192.168.x.x), it times out. I have looked all over for a solution, but I cannot find anything that tells me how to configure the Zookeeper sever to start on a different ip address.
Maybe my understanding of these technologies is simply wrong, but I thought this would be a fairly simple thing to do. Does anyone know any way to resolve this? Or else if I'm doing it completely wrong, what is the correct approach
zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181
Well, that is the bind address.
You need to also (preferably) have a static IP for Zookeeper, then set zookeeper.connect within the server.properties file of Kafka to reach to that other machine's external address.
From the Zookeeper configuration file, you would make sure you have the myid file and have a line in the property file that looks like this (without the double brackets)
server.{{ myid }}={{ ip_address }}:2888:3888
You wouldn't find this in the Kafka documentation, but it is in the Zookeeper documentation
However, if Kafka and Zookeeper are on the same machine, this isn't necessary.
Your external consumer should be setting bootstrap.servers property and the Kafka IP address(es) w/ port 9092.
Your problem might me related instead to the advertised.listeners setting within Kafka.
For example, start with listeners=PLAINTEXT://:9092
As of Zookeeper 3.3.0 (see Advanced Configuration):
clientPortAddress : New in 3.3.0: the address (ipv4, ipv6 or hostname)
to listen for client connections; that is, the address that clients
attempt to connect to. This is optional, by default we bind in such a
way that any connection to the clientPort for any
address/interface/nic on the server will be accepted
So you could use:
clientPortAddress=127.0.0.1

Unable to connect broker - kafka Tool

I am facing below error message when i was trying to connect and see the topic/consumer details of one of my kafka clusters we have.
we have 3 brokers in the cluster which I able to see but the topic and its partitions.
Note : I have kafka 1.0 and kafka tool version is 2.0.1
I had the same issue on my MacBook Pro. The tool was using "tshepo-mbp" as the hostname which it could not resolve. To get it to work I added 127.0.0.1 tshepo-mbp to the /etc/hosts file.
kafka tool is most likely using the hostname to connect to the broker and cannot reach it. You maybe connecting to the zookeeper host by IP address but make sure you can connect/ping the host name of the broker from the machine running the kafka tool.
If you cannot ping the broker either fix the network issues or as a workaround edit the host file on your client to let it know how to reach the broker by its name
This issue occurs if you have not set listeners and advertised.listeners property in server.properties file.
For Ex:
config/server.properties
...
listeners=PLAINTEXT://:9092
...
advertised.listeners=PLAINTEXT://<public-ip/host-name>:9092
...
To fix this issue, we need to change the server.properties file.
$ vim /usr/local/etc/kafka/server.properties
Here update the listeners value from
listeners=PLAINTEXT://:9092
to
listeners=PLAINTEXT://localhost:9092
source:https://medium.com/#Ankitthakur/apache-kafka-installation-on-mac-using-homebrew-a367cdefd273
For better visibility (even already commented the same in early days thread)
In my case, I got to know when I used Kafkatool from my local machine, tool tris to find out Kafka broker port which was blocked from my cluster admins for my local machine, that is the reason I was not able to connect.
Resolution:
Either ask the admin to open the port for intranet if they can, if they can not you can use tunnelling for your testing purpose or time being for your port.
Hope this would help a few.

Produce Messages in Kafka in HDF3.0.2 from local machine

I have Hortonworks DataFlow (HDF3.0.2) running in VmWare on my mac. Kafka broker is running on port 6667 and the IP address of sandbox is 172.17.0.2
In java program, running locally on my mac, I have bootstrap server configured as below:
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "172.17.0.2:6667");
Java program just hangs cannot produce message in kafka topic. I have tried disabling the firewall, added entry in /etc/hosts as:
172.17.0.2 sandbox-hdf.hortonworks.com
and changed the bootstrap servers config entry to use sandbox-hdf.hortonworks.com, but no luck.
telnet command to 172.17.0.2 6667 hangs as well and it gives connection timeout error.
Any help to produce message in Kafka running in HDF 3.0.2 from outside of Vmware is highly appreciated. Please let me know if I am missing anything.
Thanks for your time and help.