All connection pools fail to initialize when one connection pool fails to initialize - scala

Question

Is it a bug?
I am using Playframework 2.6.6 with scala 2.12.3. I have 3 databases defined in application.conf ( the settings for the hikaricp connection pools are all over the map because I experimented with them):
db {
mentions {
driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/mentions"
password="***"
username="play"
hikaricp {
maximumPoolSize = 2
minimumIdle=2
}
}
postgres {
driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/postgres"
password="***"
username="postgres"
hikaricp {
maximumPoolSize = 9
minimumIdle=9
}
}
sqlserver {
driver="net.sourceforge.jtds.jdbc.Driver"
url="jdbc:jtds:sqlserver://10.211.55.5:1433/db"
username="sa"
password="***"
hikaricp {
maximumPoolSize=4
minimumIdle=4
//connectionTestQuery="select 1"
}
}
}
With this configuration the Hikaricp connection pool fails to initialize the sql server connection pool and actually none of the other two connection pools are initialized. The fix for the sql server connection error is to set the connectionTestQuery. That's not the point of this post. I don't understand why the other two connections fail when this one fails. In my mind the other two connection pools should still function.
Here are the error messages from the console:
[info] application - Creating Pool for datasource 'sqlserver'
[error] c.z.h.p.PoolBase - HikariPool-1 - Failed to execute isValid() for connection, configure connection test query (null).
[info] application - Creating Pool for datasource 'sqlserver'
[error] c.z.h.p.PoolBase - HikariPool-2 - Failed to execute isValid() for connection, configure connection test query (null).
[info] application - Creating Pool for datasource 'sqlserver'
[error] c.z.h.p.PoolBase - HikariPool-3 - Failed to execute isValid() for connection, configure connection test query (null).
[error] application -
! #761fipg95 - Internal server error, for (GET) [/] ->
play.api.UnexpectedException: Unexpected exception[CreationException: Unable to create injector, see the following errors:
1) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=sqlserver)) to ProviderTarget(play.api.db.NamedDatabaseProvider#65c05a8b)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
2) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=mentions)) to ProviderTarget(play.api.db.NamedDatabaseProvider#700c1d35)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
3) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=postgres)) to ProviderTarget(play.api.db.NamedDatabaseProvider#1f4f0047)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
3 errors]
at play.core.server.DevServerStart$$anon$1.reload(DevServerStart.scala:186)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:124)
at play.core.server.AkkaHttpServer.modelConversion(AkkaHttpServer.scala:183)
at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:189)
at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$3(AkkaHttpServer.scala:106)
at akka.stream.impl.fusing.MapAsync$$anon$23.onPush(Ops.scala:1172)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=sqlserver)) to ProviderTarget(play.api.db.NamedDatabaseProvider#65c05a8b)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
2) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=mentions)) to ProviderTarget(play.api.db.NamedDatabaseProvider#700c1d35)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
3) Error injecting method, java.lang.AbstractMethodError
at com.google.inject.util.Providers$GuicifiedProviderWithDependencies.initialize(Providers.java:149)
at play.api.db.DBModule$$anonfun$$lessinit$greater$1.$anonfun$new$1(DBModule.scala:25):
Binding(interface play.api.db.Database qualified with QualifierInstance(#play.db.NamedDatabase(value=postgres)) to ProviderTarget(play.api.db.NamedDatabaseProvider#1f4f0047)) (via modules: com.google.inject.util.Modules$OverrideModule -> play.api.inject.guice.GuiceableModuleConversions$$anon$1)
3 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:84)
at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:185)
at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:137)
at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
at play.core.server.DevServerStart$$anon$1.$anonfun$reload$3(DevServerStart.scala:174)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
Caused by: java.lang.AbstractMethodError: null
at net.sourceforge.jtds.jdbc.JtdsConnection.isValid(JtdsConnection.java:2833)
at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:456)
at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:423)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:381)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:205)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:448)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:519)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:113)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at play.api.db.HikariCPConnectionPool.$anonfun$create$1(HikariCPModule.scala:51)
An update: I still think that architecturally, if one connection pool doesn't work, the other connections pools should not be affected. Today I shutdown my windows vm running sql server and I had only my postgresql server running. The page I tested connected only to the postgresql dbs and it still failed because the sql server connect pool didn't work. This time it failed with a configuration error java.sql.SQLException - Login Timed out.

Answer 1

After doing some debugging in 2.6.6 & 2.6.7, I discovered the following:
java.lang.AbstractMethodError inherits java.lang.IncompatibleClassChangeError which inherits java.lang.LinkageError. java.lang.LinkageError is considered fatal according to the NonFatal object:
object NonFatal {
/**
* Returns true if the provided `Throwable` is to be considered non-fatal, or false if it is to be considered fatal
*/
def apply(t: Throwable): Boolean = t match {
// VirtualMachineError includes OutOfMemoryError and other fatal errors
case _: VirtualMachineError | _: ThreadDeath | _: InterruptedException | _: LinkageError | _: ControlThrowable => false
case _ => true
}
/**
* Returns Some(t) if NonFatal(t) == true, otherwise None
*/
def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else None
}
So the Try calls sprinkled all over the code don't catch this exception.
The exception ends up being handled by the guice layers. The call goes back to the guice Initializer.injectAll method which (among other objects) injects the NamedDatabaseProvider objects for each database connection configured in the application.conf file (the binding to the NamedDatabaseProvider is wired in the the DBModule). The NamedDatabaseProvider objects depend on DBApi whose initialization fails every time due to the AbstractMethodError exception which is considered fatal.
Ultimately the problem is a combination of circumstances: 1. the exception throws in the Hikaricp code is fatal, 2. The DBApi code wants to initialize all connection pools and it doesn't handle fatal errors.
I don't consider this a bug. Perhaps the Hickaricp dudes could have thrown an UnsupportedOperationException or something similar.
For others interested I found this: https://github.com/njlg/playdb.