zookeeper not able to connect to two server - apache-kafka

Question

I have two zookeeper/kafka servers.
say,
10.10.1.9
10.10.1.10
When i start my zokeeper with configuration:
On 10.10.1.9,
dataDir=/var/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=10.10.1.9:2888:3888
server.2=10.10.1.10:2888:3888
I get the following warning on 10.10.1.9:
[2016-05-23 07:26:56,047] WARN Cannot open channel to 2 at election address /10.10.1.10:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
Meaning to say that zookeeper is not able to connect to 10.10.1.10 server.
ON 10.10.1.10 zookeeper logs, i get the following logs:
[2016-05-23 06:01:48,513] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:23:00,400] INFO Accepted socket connection from /10.10.1.9:44808 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:23:05,537] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -720899 (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:23:05,538] INFO Closed socket connection for client /10.10.1.9:44808 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:24:43,180] INFO Accepted socket connection from /10.10.1.9:44821 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:24:45,189] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -1179651 (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:24:45,189] INFO Closed socket connection for client /10.10.1.9:44821 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:47:47,166] INFO Accepted socket connection from /10.10.1.9:45015 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:47:51,080] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
[2016-05-23 07:47:51,091] INFO Closed socket connection for client /10.10.1.9:45015 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:56:05,693] INFO Accepted socket connection from /10.10.1.9:45085 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:56:11,359] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
[2016-05-23 07:56:11,360] INFO Closed socket connection for client /10.10.1.9:45085 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 08:01:22,066] INFO Accepted socket connection from /10.10.1.9:45127 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 08:01:26,015] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
Anybody encountered such issue? Any idea?
Can you list down all the port permissions or any thing else to be done for the two zookeeper servers to be able to communicate.
Thanks in advance guys!!