Slick: java.util.concurrent.RejectedExecutionException in Quartz Job - scala

Question

I have Play 2.5 App that uses Slick. The play app has few Quartz job scheduled and runs within the app. The Quartz Job uses slick to insert/update data in the database. The Quartz Job fails with the below exception.
[error] o.q.c.ErrorLogger - Job (8a3a3ec7f96a2d1aceb2dc96c5dddaed.becdc8372f54d501222a2ca94c264ff0 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception.
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2#5fae8abe rejected from java.util.concurrent.ThreadPoolExecutor#39c55f20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 268]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
at slick.backend.DatabaseComponent$DatabaseDef$class.runSynchronousDatabaseAction(DatabaseComponent.scala:230)
at slick.jdbc.JdbcBackend$DatabaseDef.runSynchronousDatabaseAction(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInContext(DatabaseComponent.scala:207)
at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInternal(DatabaseComponent.scala:75)
at slick.jdbc.JdbcBackend$DatabaseDef.runInternal(JdbcBackend.scala:38)
[error] o.q.c.ErrorLogger - Job (8a3a3ec7f96a2d1aceb2dc96c5dddaed.becdc8372f54d501222a2ca94c264ff0 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception.
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2#5fae8abe rejected from java.util.concurrent.ThreadPoolExecutor#39c55f20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 268]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
at slick.backend.DatabaseComponent$DatabaseDef$class.runSynchronousDatabaseAction(DatabaseComponent.scala:230)
at slick.jdbc.JdbcBackend$DatabaseDef.runSynchronousDatabaseAction(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInContext(DatabaseComponent.scala:207)
at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInternal(DatabaseComponent.scala:75)
at slick.jdbc.JdbcBackend$DatabaseDef.runInternal(JdbcBackend.scala:38)
This failure doesnt happen when slick calls are made outside of the Quartz job but only when inside the Quartz job. Some of the initial few Quartz job succeeds but then after few runs the above error is thrown.
what is the causing this issue and why the stacktrace says the pool size is zero as shown below?
Caused by: java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2#5fae8abe rejected from java.util.concurrent.ThreadPoolExecutor#39c55f20[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 268]
Below is the Slick and Quartz settings
slick.dbs.default.driver="slick.driver.MySQLDriver$"
slick.dbs.default.db.driver="com.mysql.jdbc.Driver"
slick.dbs.default.db.url="jdbc:mysql://mysql-server:3306/database"
slick.dbs.default.db.user=username
slick.dbs.default.db.password=password
slick.dbs.default.db.queueSize = 10000
slick.dbs.default.db.connectionTimeout=5s
// Quartz Settings
scheduler = {
org.quartz.scheduler.instanceId = AUTO
org.quartz.threadPool.class = "org.quartz.simpl.SimpleThreadPool"
org.quartz.threadPool.threadCount = 100
org.quartz.threadPool.threadPriority = 5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread = true
org.quartz.dataSource.quartzDataSource.driver=com.mysql.jdbc.Driver
org.quartz.dataSource.quartzDataSource.URL="jdbc:mysql://mysql-server:3306/database"
org.quartz.dataSource.quartzDataSource.user=username
org.quartz.dataSource.quartzDataSource.password=password
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = "org.quartz.impl.jdbcjobstore.JobStoreTX"
org.quartz.jobStore.driverDelegateClass = "org.quartz.impl.jdbcjobstore.StdJDBCDelegate"
org.quartz.jobStore.dataSource = quartzDataSource
org.quartz.jobStore.tablePrefix = QRTZ_
org.quartz.jobStore.isClustered = true
org.quartz.jobStore.useProperties = true
org.quartz.dataSource.quartzDataSource.maxConnections=3
}