Kafka Manager not able to connect to brokers - apache-kafka

Question

I have 3 brokers (brker1, broker2 and broker3), my kafka manger is able connect to broker2 but not able to connect with remaining two brokers (broker1 and broker3). I restarted kafka-manager, broker1 and broker3 as well but still not connecting to these 2 brokers.
Note :- kafka is running on 3 brokers
This log I got from kafka-manage
[error] k.m.a.c.BrokerViewCacheActor - Failed to get broker metrics for BrokerIdentity(3,broker3,9092,9999,false)
java.rmi.ConnectException: Connection refused to host: IP; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.8.0_121]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_121]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_121]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_121]
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) ~[na:1.8.0_121]
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) ~[na:1.8.0_121]
at com.sun.proxy.$Proxy6.newClient(Unknown Source) ~[na:na]
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) ~[na:1.8.0_121]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_121]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_121]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_121]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_121]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_121]
at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_121]
at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_121]
at java.net.Socket.<init>(Socket.java:211) ~[na:1.8.0_121]
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_121]
[info] k.m.a.KafkaManagerActor - Updating internal state...
I am using kafka-manager-1.3.2.1
3 brokers(broker1, broker2 and broker3) are running
this is the log I am getting from broker1 (kafka-manager not able to connect to this broker)
[2018-05-08 07:10:11,217] INFO [GroupCoordinator 1]: Loading group metadata for topic-1 with generation 27 (kafka.coordinator.GroupCoordinator)
[2018-05-08 07:10:11,217] INFO [GroupCoordinator 1]: Loading group metadata for topic-2 with generation 3 (kafka.coordinator.GroupCoordinator)
[2018-05-08 07:10:11,217] INFO [Group Metadata Manager on Broker 1]: Finished loading offsets from [__consumer_offsets,2] in 28 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-08 07:10:11,218] INFO [Group Metadata Manager on Broker 1]: Loading offsets and group metadata from [__consumer_offsets,16] (kafka.coordinator.GroupMetadataManager)
[2018-05-08 07:10:11,224] INFO [Group Metadata Manager on Broker 1]: Finished loading offsets from [__consumer_offsets,16] in 6 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-08 07:17:53,774] INFO [Group Metadata Manager on Broker 1]: Removed 0 expired offsets in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)

Answer 1

The error "Connection refused" could indicate there is no process listening on that port, or there is a firewall on that machine blocking that port.
From the machine where kafka-manager runs, can you do nc broker3 9092 and nc broker3 9999? If these two commands "hang" it means it's not just a Kafka Manager problem.
I realize this is an old question, I just came across it by chance as I was looking for something else.

Answer 2

It seems you need to publish JMX PORT/HOST so that Kafka-Manager would be able to discover brokers, those are controlled by below params in broker starter script.
Djava.rmi.server.hostname=${ip}
JMX_PORT=9997
For more details follow: https://github.com/yahoo/kafka-manager/issues/214