Apache ZooKeeper Cluster loses connectivity after leader election - apache-zookeeper

Question

I am running a ZooKeeper Cluster with five nodes, each of which has the following configuration (plus the correct quorum information at the end):
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/usr/local/zookeeper/data
clientPort=2181
minSessionTimeout=4000
maxSessionTimeout=40000
4lw.commands.whitelist=*
The cluster works fine at first. For testing purposes, I then kill the instance of ZooKeeper that is currently the leader (each ZooKeeper server is running in the foreground of a screen session). Then, a leader election is triggered, as expected, and another node is elected leader.
However, when then trying to send get/set/create requests to th cluster in a Java client (the process that was running fine initially), as well as interfacing with the cluster via zkCli.sh just gives the state of the client as CONNECTING forever.
At this time, I have conducted the obvious troubleshooting steps such as
echo stat | nc localhost 2181 -- On any one of the servers that are still running this will just give an indication of all being well, such as
Clients:
/10.0.0.1:35264[1](queued=0,recved=2,sent=1)
/10.0.0.2:49230[0](queued=0,recved=1,sent=0)
/127.0.0.1:34162[0](queued=0,recved=1,sent=0)
/10.0.0.3:49530[0](queued=0,recved=1,sent=0)
/10.0.0.1:35250[0](queued=0,recved=1,sent=0)
/10.0.0.4:35406[1](queued=0,recved=2,sent=1)
/10.0.0.2:49304[1](queued=0,recved=1,sent=1)
Latency min/avg/max: 0/0/0
Received: 484
Sent: 140
Connections: 7
Outstanding: 343
Zxid: 0x200000000
Mode: leader
Node count: 15
Proposal sizes last/min/max: 32/32/75
echo ruok | nc localhost 2181 just ouputs iamok
The following is the contents (abridged) of the log file on the server which is elected leader after I inject the failure:
2019-08-12 12:25:36,799 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower#69] - FOLLOWING - LEADER ELECTION TOOK - 23 MS
2019-08-12 12:25:37,004 [myid:4] - WARN [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner#282] - Unexpected exception, tries=0, remaining init limit=9798, connecting to /10.0.0.10:2888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:233)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:262)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:77)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1271)
2019-08-12 12:25:39,148 [myid:4] - WARN [NIOWorkerThread-4:NIOServerCnxn#370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-08-12 12:25:39,174 [myid:4] - INFO [NIOWorkerThread-5:FourLetterCommands#234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2019-08-12 12:25:39,175 [myid:4] - INFO [NIOWorkerThread-5:FourLetterCommands#235] - The list of enabled four letter word commands is : [[wchs, stat, wchp, dirs, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, telnet close, crst, cons]]
2019-08-12 12:25:39,175 [myid:4] - INFO [NIOWorkerThread-5:NIOServerCnxn#518] - Processing stat command from /127.0.0.1:59614
2019-08-12 12:25:39,858 [myid:4] - WARN [NIOWorkerThread-6:NIOServerCnxn#370] - Exception causing close of session 0x0: ZooKeeperServer not running
2019-08-12 12:25:39,906 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner#391] - Getting a diff from the leader 0x0
2019-08-12 12:25:39,913 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner#546] - Learner received NEWLEADER message
2019-08-12 12:25:40,253 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Learner#529] - Learner received UPTODATE message
2019-08-12 12:25:40,266 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor#256] - Configuring CommitProcessor with 16 worker threads.
2019-08-12 12:25:40,536 [myid:4] - WARN [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower#125] - Got zxid 0x100000001 expected 0x1
2019-08-12 12:25:40,536 [myid:4] - INFO [SyncThread:4:FileTxnLog#216] - Creating new log file: log.100000001
2019-08-12 12:25:43,617 [myid:4] - INFO [NIOWorkerThread-7:NIOServerCnxn#518] - Processing stat command from /127.0.0.1:59638
2019-08-12 12:25:43,619 [myid:4] - INFO [NIOWorkerThread-7:StatCommand#53] - Stat command output
2019-08-12 12:26:06,907 [myid:4] - WARN [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower#96] - Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:158)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:92)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1271)
2019-08-12 12:26:06,908 [myid:4] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker#1176] - Connection broken for id 5, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
2019-08-12 12:26:06,909 [myid:4] - WARN [RecvWorker:5:QuorumCnxManager$RecvWorker#1179] - Interrupting SendWorker
2019-08-12 12:26:06,909 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):Follower#201] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:201)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1275)
2019-08-12 12:26:06,910 [myid:4] - WARN [SendWorker:5:QuorumCnxManager$SendWorker#1092] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
2019-08-12 12:26:06,910 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):LearnerZooKeeperServer#165] - Shutting down
2019-08-12 12:26:06,911 [myid:4] - WARN [SendWorker:5:QuorumCnxManager$SendWorker#1102] - Send worker leaving thread id 5 my id = 4
2019-08-12 12:26:06,911 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperServer#558] - shutting down
2019-08-12 12:26:06,912 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FollowerRequestProcessor#139] - Shutting down
2019-08-12 12:26:06,912 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):CommitProcessor#362] - Shutting down
2019-08-12 12:26:06,912 [myid:4] - INFO [FollowerRequestProcessor:4:FollowerRequestProcessor#110] - FollowerRequestProcessor exited loop!
2019-08-12 12:26:06,914 [myid:4] - INFO [CommitProcessor:4:CommitProcessor#195] - CommitProcessor exited loop!
2019-08-12 12:26:06,917 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FinalRequestProcessor#514] - shutdown of request processor complete
2019-08-12 12:26:06,922 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):SyncRequestProcessor#191] - Shutting down
2019-08-12 12:26:06,923 [myid:4] - INFO [SyncThread:4:SyncRequestProcessor#169] - SyncRequestProcessor exited!
2019-08-12 12:26:06,923 [myid:4] - WARN [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer#1318] - PeerState set to LOOKING
2019-08-12 12:26:06,924 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):QuorumPeer#1193] - LOOKING
2019-08-12 12:26:06,925 [myid:4] - INFO [QuorumPeer[myid=4](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection#885] - New election. My id = 4, proposed zxid=0x100000024
2019-08-12 12:26:07,129 [myid:4] - WARN [WorkerSender[myid=4]:QuorumCnxManager#677] - Cannot open channel to 5 at election address /10.0.0.10:3888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:648)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:705)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:618)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)
at java.lang.Thread.run(Thread.java:748)
2019-08-12 12:26:07,337 [myid:4] - INFO [WorkerSender[myid=4]:QuorumCnxManager#430] - Have smaller server identifier, so dropping the connection: (5, 4)
2019-08-12 12:26:07,337 [myid:4] - INFO [WorkerReceiver[myid=4]:FastLeaderElection#679] - Notification: 2 (message format version), 3 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-08-12 12:26:07,338 [myid:4] - INFO [WorkerReceiver[myid=4]:FastLeaderElection#679] - Notification: 2 (message format version), 4 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 4 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-08-12 12:26:07,338 [myid:4] - INFO [WorkerReceiver[myid=4]:FastLeaderElection#679] - Notification: 2 (message format version), 1 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-08-12 12:26:07,338 [myid:4] - INFO [WorkerReceiver[myid=4]:FastLeaderElection#679] - Notification: 2 (message format version), 4 (n.leader), 0x100000024 (n.zxid), 0x2 (n.round), LOOKING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2019-08-12 12:26:07,342 [myid:4] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener#888] - Received connection request /172.17.0.6:46520
2019-08-12 12:26:07,345 [myid:4] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener#888] - Received connection request /172.17.0.6:46522
2019-08-12 12:26:07,345 [myid:4] - WARN [SendWorker:4:QuorumCnxManager$SendWorker#1092] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
2019-08-12 12:44:00,079 [myid:4] - WARN [RecvWorker:2:QuorumCnxManager$RecvWorker#1176] - Connection broken for id 2, my id = 4, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
2019-08-12 12:44:43,382 [myid:4] - INFO [SessionTracker:QuorumZooKeeperServer#157] - Submitting global closeSession request for session 0x4003ead876c0025
2019-08-12 12:44:45,381 [myid:4] - INFO [SessionTracker:ZooKeeperServer#398] - Expiring session 0x1003ead86e90025, timeout of 40000ms exceeded
2019-08-12 12:44:45,381 [myid:4] - INFO [SessionTracker:QuorumZooKeeperServer#157] - Submitting global closeSession request for session 0x1003ead86e90025
2019-08-12 12:45:00,485 [myid:4] - INFO [/0.0.0.0:3888:QuorumCnxManager$Listener#888] - Received connection request /10.0.0.7:42596
2019-08-12 12:45:00,485 [myid:4] - WARN [RecvWorker:4:QuorumCnxManager$RecvWorker#1176] - Connection broken for id 4, my id = 4, error =
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1161)
2019-08-12 12:45:00,485 [myid:4] - WARN [SendWorker:2:QuorumCnxManager$SendWorker#1092] - Interrupted while waiting for message on queue
I would be extremely grateful for any help as to why the cluster doesn't seem to interconnect (or at least won't serve requests anymore) after the leader election takes place, or any advice what to look at for debugging purposes.