After few successful updates to solr, throws SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper - apache-zookeeper

Question

I am using solr version 7.3.1 and using it with 3 external zookeeper nodes.
Below is my zookeeper config for one of the node, all have similar config:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/<user>/zookeeper/zookdata/zk2/
clientPort=2182
server.1=localhost:2666:3666
server.2=localhost:2667:3667
server.3=localhost:2668:3668
Then, I am using these 3 nodes to start solr. In my application I am using localhost:2182,localhost:2183 to connect to solr, using below code.
List<String> zkHosts = Arrays.asList(solrZkHostPort.split(","));
CloudSolrClient.Builder builder = new CloudSolrClient.Builder(zkHosts, Optional.empty());
solrClient = builder.build();
I am using multiple spark executors to update document to solr. It works fine for some 1100-1300 records update after that update fails with below exception:
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper localhost:2182,localhost:2183 within 10000 ms
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:120)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:110)
at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:285)
at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:155)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:828)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at com.package.SomeApplicationClass
... 16 more
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper localhost:2182,localhost:2183 within 10000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:232)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
... 26 more
I get below exception too, not sure though if this has any significance:
18/09/20 12:45:40 WARN ClientCnxn: Session 0x0 for server localhost/127.0.0.1:2183, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
any idea what could be the problem? Do I need to change zookeeper config or they way I create solr client needs to change?

Answer 1

Noob mistake.
My spark job was creating more then 1000 spark executors and for each executor, solr client was getting created and I was not closing the solrClient. closed solrclient after executor completed.