I have Hortonworks DataFlow (HDF3.0.2) running in VmWare on my mac. Kafka broker is running on port 6667 and the IP address of sandbox is 172.17.0.2
In java program, running locally on my mac, I have bootstrap server configured as below:
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "172.17.0.2:6667");
Java program just hangs cannot produce message in kafka topic. I have tried disabling the firewall, added entry in /etc/hosts as:
172.17.0.2 sandbox-hdf.hortonworks.com
and changed the bootstrap servers config entry to use sandbox-hdf.hortonworks.com, but no luck.
telnet command to 172.17.0.2 6667 hangs as well and it gives connection timeout error.
Any help to produce message in Kafka running in HDF 3.0.2 from outside of Vmware is highly appreciated. Please let me know if I am missing anything.
Thanks for your time and help.
Related
sorry but i am racking my brains to figure this out.
My kafka broker says waiting to connect to zookeper and then shuts down (time out).I checked broker config / port did netstat etc also confirmed zookeper settings. But then if i restart my ubuntu kernel (where kafka broker is running) all is well. Zookeeper is on Linux mint. Please can someone advice where what log files can give me more detail - i did the usual netstat and lsof no luck.
its windows 10 for some reason it is not recognizing my partitioned ip.
Fixed it was a config item on my end.
Able to connect successfully to local kafka broker/cluster running locally (dockerized) using Conduktor, but when trying to connect to Kafka cluster running on Unix VM, getting below error.
Error:
"The broker [...] is reachable but Kafka can't connect. Ensure you have access to the advertised listeners of the the brokers and the proper authorization"
Appreciate any assistance.
running locally (dockerized)
When running in docker, you need to ensure that the ports are accessible from outside of your container. To verify this, try doing a telnet <ip> <port> and check if you are able to connect.
Since the error message says, the broker is reachable, I suppose you would be able to successfully telnet to the broker.
Next, check your broker config called advertised.listeners. Here you need to mention your IP:Port combination where IP is what you will be giving in your client program i.e. Conduktor.
An example for that would be
advertised.listeners=PLAINTEXT://1.2.3.4:9092
and then restart your broker and reconnect. If you are using ssl then you need to provide some extra configuration. See Configuring Kafka brokers for more.
Try to add in /etc/hosts (Unix-like) or C:\Windows\System32\drivers\etc\hosts (windows-like) the Kafka server in such manner kafka_server_ip kafka_server_name_in_dns (e.g. 10.10.0.1 kafka).
I am currently working with a simple project to query the messages from Apache Kafka topic using Apache Drill. And now I am encountering an error when running the Apache Drill cluster when running this command.
sqlline.bat -u "jdbc:drill:zk=localhost:2181"
And the error that I encountered is:
No active Drillbit endpoint found from ZooKeeper. Check connection parameters
I am using the single cluster instance of ZooKeeper that came from Apache Kafka.
Can anyone help me with this problem? Is it ok to use the Zookeeper from Apache Kafka installation with Drill?
sqlline.bat -u "jdbc:drill:zk=localhost:2181" command only connects to running DrillBit. If you have Drill running in distributed mode, please replace localhost with the correct IP address of the node, where Zookeeper is running and update port if needed.
If you want to start Drill in embedded mode, you may try running drill-embedded.bat or sqlline.bat -u "jdbc:drill:zk=local" command.
For more details please refer to https://drill.apache.org/docs/starting-drill-on-windows/.
I'm running zookeeper, kafka servers along with a consumer client on a windows machine. When I use a simple producer program in python or c++ in a different machine also running windows, everything works normal. However, when I use the same producer program in a linux machine it does not work, it accuses failed to resolve . All computers are connected on the same network. I have tried to set the listeners and advertised.listeners in the kafka server.properties in several different manners, but with the same result.
Machine running windows with servers and consumer:
IP:192.168.0.149
hostname:something
second Machine running linux or windows (dual boot) with the producer program:
IP:192.168.0.150
hostame:somethingelse
some tried configurations at server.properties:
1
listeners=PLAINTEXT:\0.0.0.0:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
2
listeners=PLAINTEXT:\localhost:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
3
#listeners=
#advertised.listeners=
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
4
listeners=PLAINTEXT:\192.168.0.149:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows if consumer passes the ip instead of localhost, does no work on linux
Listeners should always stay as 0.0.0.0. This is the bind address to accept all external connections
If you want to run Kafka on windows, you need to deal with the windows firewall. Try disabling it while testing connections. It would probably be best that you switch the broker to run on Linux.
The advertised listeners should only be the hostname if both systems are able to resolve each other by DNS name. Otherwise, save yourself the extra network hop and use the IP as the advertised listener. If you're getting "failed to resolve", seems like a DNS issue, and ping windows-host shouldn't work either
To debug your connection, I would suggest using kafkacat
kafkacat -b kafka-host:9092 -L
I am unable to connect to kafka broker (http://127.0.0.1:6667) of Hortonworks Sandbox 2.6 from my host machine. Every time I connect, it is saying the site is not reachable. What I am doing wrong.
FYI, I have enabled port forwarding for 6667 for my sandbox.
have you checked this: https://de.hortonworks.com/tutorial/sandbox-port-forwarding-guide/section/1/#add-ports-to-the-docker-script
Essentially you also need to enable port forwarding for the Docker Sandbox, just Host -> VM wont suffice, you need Host -> VM -> Docker container.
Helped in my case for Sandbox HDP 2.6+
Kafka is not an HTTP server; your browser will never be able to reach that address.
You will need to download Kafka CLI tools or SSH into the machine to connect to Kafka.