unregistering the JDBC driver? - eclipse

I'm running tomcat under/inside of eclipse while developing a web application. The web app is using hsqldb in embedded mode, via hibernate and guice. Things seem to be working fine, except when I stop tomcat. Eclipse has a green start button and a red stop button for Tomcat. When I click on the stop button it doesn't immediately stop it nicely like it did before I added hibernate and hsqldb to the mix. Now it waits a few seconds and then eclipse gives me a dialog box about not being able to stop tomcat and to click OK to force it to terminate.
Does anyone know what I need to do to fix this? I found some other responses saying to put the hsqldb jar file in tomcat's lib directory but I was wondering there is anything I could do that's a little less drastic.
Here's what's in the error output from tomcat (in the eclipse console window):
Jan 31, 2017 7:04:11 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
WARNING: The web application [basic] registered the JDBC driver [org.hsqldb.jdbc.JDBCDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 31, 2017 7:04:11 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [basic] appears to have started a thread named [HSQLDB Timer #276f0355] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
java.lang.Thread.run(Thread.java:745)
Jan 31, 2017 7:04:11 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [basic] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Jan 31, 2017 7:04:11 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [basic] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1d4fc7e8]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(PersistenceContext[entityKeys=[],collectionKeys=[]];ActionQueue[insertions=ExecutableList{size=0} updates=ExecutableList{size=0} deletions=ExecutableList{size=0} orphanRemovals=ExecutableList{size=0} collectionCreations=ExecutableList{size=0} collectionRemovals=ExecutableList{size=0} collectionUpdates=ExecutableList{size=0} collectionQueuedOps=ExecutableList{size=0} unresolvedInsertDependencies=null])]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

I have a stupid answer but I'll show you what I did.
#JulienR: I already had the shutdown=true in my persistence.xml file for the javax.persistence.jdbc.url value.
I created a ServletContextListener and added it to my web.xml file; here's the code. The first part that uses the EntityManager to issue the SHUTDOWN command is my code. The code for the ClassLoader and driver eyeballing I got off of here. So with this it's no longer complaining about HSQLDB but I'm still getting a WARNING about two threads that weren't stopped (and I have to wait for eclipse to time out waiting for it). I've appended the relevant lines from the log.
public class BasicServletContextListener implements ServletContextListener {
private final transient Logger log =
LoggerFactory.getLogger(BasicServletContextListener.class);
// private final Provider<EntityManager> entityManagerProvider;
#Override
public void contextDestroyed(ServletContextEvent event) {
this.log.debug("contextEvent: {}", event.toString());
final EntityManagerFactory entityManagerFactory =
Persistence.createEntityManagerFactory("ilmp");
final EntityManager entityManager =
entityManagerFactory.createEntityManager();
entityManager.getTransaction().begin();
final Query query =
entityManager.createNativeQuery("SHUTDOWN COMPACT;");
this.log.debug("query: {}", query.executeUpdate());
entityManager.getTransaction().commit();
entityManager.close();
// Now deregister JDBC drivers in this context's ClassLoader:
// Get the webapp's ClassLoader
final ClassLoader cl = Thread.currentThread().getContextClassLoader();
// Loop through all drivers
final Enumeration<Driver> drivers = DriverManager.getDrivers();
while (drivers.hasMoreElements()) {
final Driver driver = drivers.nextElement();
if (driver.getClass().getClassLoader() == cl) {
// This driver was registered by the webapp's
// ClassLoader, so deregister it:
try {
this.log.info("Deregistering JDBC driver {}", driver);
DriverManager.deregisterDriver(driver);
}
catch (final SQLException ex) {
this.log.error("Error deregistering JDBC driver {}", driver,
ex);
}
}
else {
// driver was not registered by the webapp's
// ClassLoader and may be in use elsewhere
this.log.trace(
"Not deregistering JDBC driver {} as it does not belong to this webapp's ClassLoader",
driver);
}
}
}
#Override
public void contextInitialized(ServletContextEvent event) {
this.log.debug("contextEvent: {}", event.toString());
}
}
>
INFO: 2017-Feb-01 19:05:36.023 [localhost-startStop-2] org.hibernate.hql.internal.QueryTranslatorFactoryInitiator.initiateService.47: HHH000397: Using ASTQueryTranslatorFactory
Hibernate: SHUTDOWN COMPACT;
INFO: 2017-Feb-01 19:05:36.226 [localhost-startStop-2] sun.reflect.NativeMethodAccessorImpl.invoke0.-2: Database closed
INFO: 2017-Feb-01 19:05:36.273 [localhost-startStop-2] sun.reflect.NativeMethodAccessorImpl.invoke0.-2: open start - state not modified
INFO: 2017-Feb-01 19:05:36.351 [localhost-startStop-2] sun.reflect.NativeMethodAccessorImpl.invoke0.-2: Database closed
DEBUG: 2017-Feb-01 19:05:36.460 [localhost-startStop-2] com.objecteffects.basic.persist.BasicServletContextListener.contextDestroyed.42: query: 0
INFO: 2017-Feb-01 19:05:36.460 [localhost-startStop-2] com.objecteffects.basic.persist.BasicServletContextListener.contextDestroyed.59: Deregistering JDBC driver org.hsqldb.jdbc.JDBCDriver#3cd32e8d
Feb 01, 2017 7:05:36 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [basic] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Feb 01, 2017 7:05:36 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [basic] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Feb 01, 2017 7:05:36 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [basic] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1bc6eba0]) and a value of type [org.hibernate.internal.SessionImpl] (value [SessionImpl(PersistenceContext[entityKeys=[EntityKey[com.objecteffects.basic.persist.TumblrSecretsEntity#1], EntityKey[com.objecteffects.basic.persist.TumblrSecretsEntity#2]],collectionKeys=[]];ActionQueue[insertions=ExecutableList{size=0} updates=ExecutableList{size=0} deletions=ExecutableList{size=0} orphanRemovals=ExecutableList{size=0} collectionCreations=ExecutableList{size=0} collectionRemovals=ExecutableList{size=0} collectionUpdates=ExecutableList{size=0} collectionQueuedOps=ExecutableList{size=0} unresolvedInsertDependencies=null])]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Feb 01, 2017 7:05:36 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8080"]
Feb 01, 2017 7:05:36 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-nio-8009"]
Feb 01, 2017 7:05:36 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-8080"]
Feb 01, 2017 7:05:36 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-nio-8009"]
The stupid answer is that I looked at my setup for something I wrote several years ago (I'm retired and just dinking around trying to write something for myself) and adding the c3p0 config lines I'd used previously to this persistence.xml makes it shut down without the delay, although I still get a bunch of warnings about zombie threads. Here are the relevant lines (still commented out).
<!-- <property -->
<!-- name="hibernate.c3p0.min_size" -->
<!-- value="5" /> -->
<!-- <property -->
<!-- name="hibernate.c3p0.max_size" -->
<!-- value="20" /> -->
<!-- <property -->
<!-- name="hibernate.c3p0.timeout" -->
<!-- value="1800" /> -->
<!-- <property -->
<!-- name="hibernate.c3p0.max_statements" -->
<!-- value="50" /> -->

Related

Tomcat v9.0 server failed to start -- with a twist

First off, I did notice that there are multiple similar questions on this topic in SO. (e.g: Tomcat v9.0 Server at localhost failed to start in Eclipse). The error message and console messages are quite similar. I have tried the suggestions from that question as well as from a few others - but to no avail.
Secondly, the twist is that this works perfectly fine from a different eclipse workspace on the same laptop. The two eclipse projects are virtually identical (copy/paste). I'm hoping someone can help me figure out as to why this is so. I am more than willing to provide additional information (code, screenshots etc.) - but I don't know where to start.
A little bit of history/ context:
My web-projects suddenly started running into this problem (not sure what I changed) and after being unable to resolve it, I un-installed and re-installed eclipse as well as tomcat on my laptop. I also started with a fresh eclipse work-space. The projects in the new work-space work fine. Those in the earlier work space still show the same error.
It would indeed be very helpful to understand as to what is causing this.
update: I'm adding the error message and the console logs as indicated #Tim Biegeleison in the comment below.
Error message:
Console logs:
Dec 13, 2020 7:50:06 AM org.apache.catalina.startup.Catalina start
SEVERE: The required Server component failed to start so Tomcat is unable to start.
org.apache.catalina.LifecycleException: A child container failed during start
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:928)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
Caused by: java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: A child container failed during start
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
... 13 more
Caused by: org.apache.catalina.LifecycleException: A child container failed during start
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:928)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 13 more
Caused by: org.apache.tomcat.util.MultiThrowable: 5 wrapped Throwables: [org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#62727399]][org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#130dca52]][org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#2654635]][org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#2371aaca]][org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#5b529706]]
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:920)
... 21 more
Dec 13, 2020 7:50:06 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-8080"]
Dec 13, 2020 7:50:06 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Catalina]
Dec 13, 2020 7:50:06 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-8080"]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.catalina.loader.WebappClassLoaderBase (file:/C:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/lib/catalina.jar) to field java.io.ObjectStreamClass$Caches.localDescs
WARNING: Please consider reporting this to the maintainers of org.apache.catalina.loader.WebappClassLoaderBase
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Tomcat 7 randomly crashes with ClassNotFoundException

I have a Scala application running in a Tomcat7 that lately has started randomly crashing due to ClassNotFoundException errors for standard classes such as scala.collection.immutable.ListSet even when the server has been running perfectly fine for some time. The errors started getting more frequent and now we're getting several crashes a day on some servers
This is a sample of the crash log we're getting in the catalina.out log file.
Dec 25, 2018 8:50:47 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Dec 25, 2018 8:50:47 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6938 ms
Uncaught error from thread [TPDispatcher-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[TPDispatcher]
java.lang.NoClassDefFoundError: scala/collection/immutable/ListSet$
at scala.collection.immutable.HashSet$HashSet1.updated0(HashSet.scala:145)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet.$plus(HashSet.scala:56)
at scala.collection.immutable.HashSet.$plus(HashSet.scala:33)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:24)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:22)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at com.mycompany.lib.actors.BadgeUpdater.updateBadge(BadgeUpdater.scala:23)
at com.mycompany.lib.actors.BadgeUpdater$$anonfun$receive$1.applyOrElse(BadgeUpdater.scala:15)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at com.twinpush.tp_dispatcher.lib.actors.BadgeUpdater.aroundReceive(BadgeUpdater.scala:11)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: scala.collection.immutable.ListSet$
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1859)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1702)
... 33 more
Dec 26, 2018 2:15:31 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Dec 26, 2018 2:15:31 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
I've tried recompiling several times, with no luck. I have no idea of what may cause ClassNotFoundException on a server that's already running, so any hint will be highly appreciated.
Environment info:
Scala version: 2.10.4
Server version: Apache Tomcat/7.0.68 (Ubuntu)
Server built: Oct 30 2018 12:54:52 UTC
Server number: 7.0.68.0
OS Name: Linux
OS Version: 4.4.0-141-generic
Architecture: amd64
JVM Version: 1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12
JVM Vendor: Oracle Corporation
Turns out I had a library that was leaking file descriptors and at some point the JVM was unable to open class files (is that even possible?). Upgrading the library fixed the leak and the JVM stopped crashing.

Zookeeper and Solr - unable to start the cluster

I've inherited a platform which runs Zookeeper and Solr. The main problem at first was the zoo.cfg and Zookeeper had old setup so DNS named resolution wasn't working.
I've already fixed. I also fixed the issue of the myid value in /var/lib/zookeeper thanks to this thread (someone left a string value there...)
Zookeeper - three nodes and nothing but errors
Now, the output of logs is different:
2017-08-28 14:24:19,368 [myid:3] - WARN [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager#400] - Cannot open channel to 2 at election address zookeeper-test-2/10.240.102.89:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer#149] - Resolved hostname: zookeeper-test-2 to address: zookeeper-test-2/10.240.102.89
2017-08-28 14:24:19,368 [myid:3] - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection#852] - Notification time out: 400
and If I run this script from one EC2 inside the VPC I can see is almost working:
+
+ SOLR
+ Online Nodes: 2
+ solrcloud-test-2 [ Connection to 8983 succeeded ]
+ solrcloud-test-1 [ Connection to 8983 succeeded ]
-------------
+ ELB not responding properly. HTTP response: 503
+
+ ZOOKEEPER
+ Online Instances: 3
+ zookeeper-test-3 [ Connection to 2181 succeeded ]
+ zookeeper-test-2 [ Connection to 2181 succeeded ]
+ zookeeper-test-1 [ Connection to 2181 succeeded ]
+ Minimal Configured: 3
+ Cluster Status: UP
as you can see even the Solr Ec2 seems to be online the ELB doesn't detect the path of the Solr.
I checked the parameters of how Tomcat was bringing up the Solr and I detected again a misconfiguration related to the hostnames so I fixed and restarted Tomcat, however my ELB health check still does not detect the url.
This check is setup as
HTTP:8983/solr/
I do a netstat I can see the port up and listening
[root#solrcloud-test-2 ~]# netstat -lnp |grep 8983
tcp 0 0 :::8983 :::* LISTEN 3338/java
Moreover another thing I noticed is even I restarted tomcat the logs of serviced are freezed in the past 25th of August!
ERROR - 2017-08-25 17:10:25.369; org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: java.net.UnknownHostException: zookeeper-1: unknown error
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:139)
at org.apache.solr.cloud.ZkController.<init>(ZkController.java:207)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:152)
at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:189)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:136)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5633)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: zookeeper-1: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:41)
at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:114)
... 22 more
INFO - 2017-08-25 17:10:25.370; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
ERROR - 2017-08-25 17:10:25.525; org.apache.solr.core.CoreContainer; CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=650576553
But in the other hand the catalina.out
INFO: Deploying web application archive /var/lib/tomcat7/webapps/solr.war
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more Filters failed to start. Full details will be found in the appropriate container log file
Aug 29, 2017 8:39:26 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/solr] startup failed due to previous errors
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /var/lib/tomcat7/webapps/solr.war has finished in 3,447 ms
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8983"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8443"]
Aug 29, 2017 8:39:26 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Aug 29, 2017 8:39:26 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3579 ms
So maybe my problem here is I don't know how exactly restart the Solr component? Is really my first work with this software so apologies if my questions are totally noob :)
UPDATE:
I found new info in the logs. Now I think I'm closer to the root issue. However I don't understand the problem, some files missing?
Aug 29, 2017 9:12:11 AM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter SolrRequestFilter
java.lang.NoClassDefFoundError: Failed to initialize Apache Solr: Could not find necessary SLF4j logging jars. If using Jetty, the SLF4j logging jars need to go in the jetty lib/ext directory. For other containers, the corresponding directory should be used. For more information, see: http://wiki.apache.org/solr/SolrLogging
Thanks
Wow, apologies people. Maybe I shot the question very fast. The setup was completely misconfigured so basically to sum up, Zookeeper cluster was partially running because service was up and running but it was lack of service. So Solr was unable to connect to them. Once I was able to bring up Zookeeper to good status Solr could connect to it.
thanks anyway

Jetty slow startup... hibernate issue?

I'm running Jetty 8 with Eclipse. After a Java update this morning, JettyLauncher is now taking 6 minutes to start up, as opposed to 10 seconds before. Several of the delays seem to deal with Hibernate (I don't remember seeing too much about Hibernate when Jetty was running at a normal speed). Could anyone confirm or deny that hibernate is causing an issue with the JettyLauncher? Or otherwise pinpoint where this delay is coming from?
The Jetty launch delays the most during these lines in the console output:
5347 [main] INFO org.hibernate.validator.util.Version - Hibernate Validator 4.2.0.Final
...
74022 [main] INFO org.hibernate.impl.SessionFactoryObjectFactory - Not binding factory to JNDI, no JNDI name configured
starting guide stats cache service
97089 [main] INFO org.springframework.beans.factory.config.PropertiesFactoryBean - Loading properties file from ServletContext resource [/WEB-INF/properties/build.properties]
118670 [main] INFO org.springframework.orm.hibernate3.HibernateTransactionManager - Using DataSource [org.springframework.jdbc.datasource.DriverManagerDataSource#16bc6df1] of Hibernate SessionFactory for HibernateTransactionManager
CacheReportsJob.cacheGuideStats()
119649 [main] INFO org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 119133 ms
119661 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - started o.e.j.w.WebAppContext{/,file:/C:/Users/MBeatty/Documents/Tortoise%20SVN/Beek%20Checkout%20v2/trunk/server/}
Oct 17, 2013 11:54:22 AM com.sun.faces.config.ConfigureListener contextInitialized
INFO: Initializing Mojarra 2.1.3 (FCS b02) for context ''
Oct 17, 2013 11:54:59 AM com.sun.faces.spi.InjectionProviderFactory createInstance
INFO: JSF1048: PostConstruct/PreDestroy annotations present. ManagedBeans methods marked with these annotations will have said annotations processed.
Oct 17, 2013 11:55:43 AM org.primefaces.webapp.PostConstructApplicationEventListener processEvent
INFO: Running on PrimeFaces 3.2-SNAPSHOT
Oct 17, 2013 11:55:43 AM com.sun.faces.lifecycle.ELResolverInitPhaseListener populateFacesELResolverForJsp
INFO: JSF1027: [null] The ELResolvers for JSF were not registered with the JSP container.

Deploying grails 2.0.3 application to cloudfoundry without cloud foundry grails-plugin?

I'm able to deploy my grails 2.0.3 app with the cloud foundry grails plugin and cf-push to the cloud and everything works fine.
Now i want to use the sts/eclipse cloud foundry extension to deploy with eclipse (not with grails command cf-push!) and see the server instances in server view and so on.
I create a new cloudfoundry instance with the eclipse plugin and it deploys but aborts with error message:
May 14, 2012 10:39:21 AM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-19036
May 14, 2012 10:39:21 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 377 ms
May 14, 2012 10:39:21 AM org.apache.catalina.realm.JAASRealm setContainer
INFO: Set JAAS app name Catalina
May 14, 2012 10:39:21 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 14, 2012 10:39:21 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.35
May 14, 2012 10:39:21 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Configuring Spring Security Core ...
... finished configuring Spring Security Core
May 14, 2012 10:39:32 AM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
May 14, 2012 10:39:32 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [] startup failed due to previous errors
May 14, 2012 10:39:32 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [net.sf.ehcache.CacheManager#1edb587] but has failed to stop it. This is very likely to create a memory leak.
May 14, 2012 10:39:32 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [projectBee.Requestmap.data] but has failed to stop it. This is very likely to create a memory leak.
May 14, 2012 10:39:32 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [projectBee.Role.data] but has failed to stop it. This is very likely to create a memory leak.
May 14, 2012 10:39:32 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [org.hibernate.cache.UpdateTimestampsCache.data] but has failed to stop it. This is very likely to create a memory leak.
May 14, 2012 10:39:32 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [org.hibernate.cache.StandardQueryCache.data] but has failed to stop it. This is very likely to create a memory leak.
Stopping Tomcat because the context stopped.
Is here somebody who knows a solution for this problem? Thank you!
You need to look at the real log files which will have the stacktrace and detailed error messages. Use the vmc commandline client or STS. There is a /logs folder which probably doesn't have what you want, but also a /tomcat/logs which probably does. Also check out stacktrace.log.