Zookeeper and Solr - unable to start the cluster - apache-zookeeper

Question

I've inherited a platform which runs Zookeeper and Solr. The main problem at first was the zoo.cfg and Zookeeper had old setup so DNS named resolution wasn't working.
I've already fixed. I also fixed the issue of the myid value in /var/lib/zookeeper thanks to this thread (someone left a string value there...)
Zookeeper - three nodes and nothing but errors
Now, the output of logs is different:
2017-08-28 14:24:19,368 [myid:3] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager#400] - Cannot open channel to 2 at election address zookeeper-test-2/10.240.102.89:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer#149] - Resolved hostname: zookeeper-test-2 to address: zookeeper-test-2/10.240.102.89
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 400
and If I run this script from one EC2 inside the VPC I can see is almost working:
+
+ SOLR
+ Online Nodes: 2
+ solrcloud-test-2 [ Connection to 8983 succeeded ]
+ solrcloud-test-1 [ Connection to 8983 succeeded ]
-------------
+ ELB not responding properly. HTTP response: 503
+
+ ZOOKEEPER
+ Online Instances: 3
+ zookeeper-test-3 [ Connection to 2181 succeeded ]
+ zookeeper-test-2 [ Connection to 2181 succeeded ]
+ zookeeper-test-1 [ Connection to 2181 succeeded ]
+ Minimal Configured: 3
+ Cluster Status: UP
as you can see even the Solr Ec2 seems to be online the ELB doesn't detect the path of the Solr.
I checked the parameters of how Tomcat was bringing up the Solr and I detected again a misconfiguration related to the hostnames so I fixed and restarted Tomcat, however my ELB health check still does not detect the url.
This check is setup as
HTTP:8983/solr/
I do a netstat I can see the port up and listening
[root#solrcloud-test-2 ~]# netstat -lnp |grep 8983
tcp 0 0 :::8983 :::* LISTEN 3338/java
Moreover another thing I noticed is even I restarted tomcat the logs of serviced are freezed in the past 25th of August!
ERROR - 2017-08-25 17:10:25.369; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: java.net.UnknownHostException: zookeeper-1: unknown error
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:139)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:207)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:152)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:189)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:136)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: zookeeper-1: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:41)
at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:114)
... 22 more
INFO - 2017-08-25 17:10:25.370; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
ERROR - 2017-08-25 17:10:25.525; org.apache.solr.core.CoreContainer; CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=650576553
But in the other hand the catalina.out
INFO: Deploying web application archive /var/lib/tomcat7/webapps/solr.war
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/solr] startup failed due to previous errors
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /var/lib/tomcat7/webapps/solr.war has finished in 3,447 ms
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8983"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3579 ms
So maybe my problem here is I don't know how exactly restart the Solr component? Is really my first work with this software so apologies if my questions are totally noob :)
UPDATE:
I found new info in the logs. Now I think I'm closer to the root issue. However I don't understand the problem, some files missing?
Aug 29, 2017 9:12:11 AM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter SolrRequestFilter
java.lang.NoClassDefFoundError: Failed to initialize Apache Solr: Could not find necessary SLF4j logging jars. If using Jetty, the SLF4j logging jars need to go in the jetty lib/ext directory. For other containers, the corresponding directory should be used. For more information, see: http://wiki.apache.org/solr/SolrLogging
Thanks

Answer 1

Wow, apologies people. Maybe I shot the question very fast. The setup was completely misconfigured so basically to sum up, Zookeeper cluster was partially running because service was up and running but it was lack of service. So Solr was unable to connect to them. Once I was able to bring up Zookeeper to good status Solr could connect to it.
thanks anyway