I am using akka-persistence-jdbc plugin for Akka Persistence in some parts of application and slick directly in another parts.
After migrating the the hottest parts from direct slick usage to akka-persistence HikariCP started to throw exceptions:
WARN com.zaxxer.hikari.pool.LeakTask Connection leak detection triggered for connection org.postgresql.jdbc.PgConnection#3a3c8d5d, stack trace follows
java.lang.Exception: Apparent connection leak detected
That's not a problem of slow SQL queries. Explain analyze shows that PostgreSQL executes them in ~1 millisecond. But connection waits for something and not being released for minutes. Some more details on that are here.
akka-persistence-jdbc uses streaming for writes and reads, could it be a slick bug or something is wrong in the way akka-persistence-jdbc does such operations?
I am using slick 3.1.1, HikariCP 2.3.7, PostgreSQL 9.4 with max_connections: 120.
Related
I have a Spring Boot application (v2.1.5) that uses JPA (Hibernate) to connect to a Postgres DB. Spring Boot uses HikariCP for connection pooling.
In my production environment, I see the following query executed every few seconds regardless of DB activity (almost as if they are some kind of health check?):
SET application_name = 'PostgreSQL JDBC Driver'
I am struggling to get to the bottom of why these queries are performed so often and if they can be avoided because in my local environment the above statement is only executed when a query to the DB is performed. I still do not understand why, but it is less frequent and different behaviour compared to production.
Are these queries necessary? Can they be avoided?
Thanks.
UPDATE:
Here is a screenshot of the queries received by the DB to which the Spring boot app is connecting using HikariCP. The time is shown as "Just now" because all of the query shown are only ~0.5 seconds apart and all within the "current minute".
This seems to be performed by the Hikari Connection Pool. See Default HikariCP connection pool starting Spring Boot application and it's answers.
I wouldn't bother about it since, it is not performing a "extremely high number" of these operations, but only every few seconds, possibly whenever a connection is handed out by or returned to the pool.
If it really bothers you, you could look into various places for disabling it.
The last comment here suggests that setting the connection property assumeMinServerVersion to 9.0 or higher might help.
Since this is probably triggered by the HikariConnectionPool it might be configurable there, by configuring the behaviour when starting, lending and returning a connection.
I have a WildFly which hosts an app which is calling some long running SQL queries (say queries or SP calls which take 10-20 mins or more).
Previously this WildFly was pointing to SQL Server 2008, now to Postgres 11.
Previously when I killed/rebooted WildFly, I had noticed that pretty quickly (if not instantaneously) the long-running SP/query calls that were triggered from the Java code (running in WildFly) were being killed too.
Now... with Postgres I noticed that these long running queries remain there and keep running in Postgres even after the app server has been shut down.
What is causing this? Who was killing them in SQL Server (the server-side itself or the JDBC driver or ...)? Is there some setting/parameter in Postgres which controls this behavior i.e. what the DB server will do with the query provided that the client who triggered the query has been shut down.
EDIT: We do a graceful WildFly shutdown by sending a command to WF to shutdown itself. Still the behavior seems different between SQL Server and Postgres.
If shutting down the application server causes JDBC calls that terminate the database session, this should not happen. If it doesn't close the JDBC connection properly, I'd call that a bug in the application server. If it does, but the queries on the backend are not canceled, I'd call that a bug in the JDBC driver.
Anyway, a workaround is to set tcp_keepalives_idle to a low value so that the server detects dead TCP connections quickly and terminates the query.
I'm getting the message below in Play for Scala, what does this mean and what could be the reason? Is this related to Slick or to JDBC (I'm using both)?
[warn] c.z.h.p.HikariPool - HikariPool-7 - Unusual system clock change
detected, soft-evicting connections from pool.
Possible bug in HikariCP
There was some issues in HikariCP that cause this error:
https://github.com/brettwooldridge/HikariCP/issues/559
So be sure yo you use 2.4.4 version or newer
Possible time shifting
HikariCP will log a warning for large forward leaps. Large forward leaps often occur on laptops that go into sleep mode or VMs that are suspended and resumed.
There is similar question:
HikariPool-1 - Unusual system clock change detected, soft-evicting connections from pool
The only thing I would add is that NTP synchronization also could have bugs: Clock drift even though NTPD running
After upgrading to Titan 1.0.0 I started to see the following exceptions under load, using Cassandra (2.2.6) as the storage backend:
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)[:1.8.0_102]
at java.lang.Thread.sleep(Thread.java:340)[:1.8.0_102]
at
java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)[:1.8.0_102]
at
com.thinkaurelius.titan.diskstorage.util.time.TimestampProviders.sleepPast(TimestampProviders.java:138)
at
com.thinkaurelius.titan.diskstorage.common.DistributedStoreManager.sleepAfterWrite(DistributedStoreManager.java:222)
... 66 more
Can this be fixed through configuration?
While there are several configuration items available around timestamps, I did not find any that strikes me as relevant to the timestamp provider itself.
You should check your Cassandra logs. I have found that Titan under load starts to throw these types of errors as well as Timeout errors when Cassandra starts its compaction process.
Grep for the keyword "GC" in /var/log/cassandra/system.log monitor your disk usage using dstat. If you see "GC" often then you under going heavy compaction and this bogs down titan.
To get around this you can try to optimise how you load your data into titan so as to not cause compaction to often.
The following are just things we tried that worked for our case:
Avoid deletions. Deletions trigger tombstoning which leads to compaction.
Increase the size of your JVM. One of the things which causes compaction to run is when you start to run out of memory so this makes it less likely to run.
You can try to use different compaction strategies. Each one is optimised for a different use case.
Currently have a project that is running Play + Play-Slick and we have issues in regards to running tests in parallel. The problem is that for every single Specification, it creates its own instance of a HikariCP connection pool which means that the tests end up failing with
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already
You can tell this is happening because you get a lot of log messages that are saying
[info] - com.zaxxer.hikari.HikariDataSource - HikariCP pool db is starting.
Since its quite a large test suite (with many specifications), using the
override def is = args(sequential = true) ^ super.is
Isn't ideal, because that will only work for the specifications that are a subclass of a that specification (however we have many specifications that are in seperate packages that have no common super type).
Is there some way to configure specs2 to use a global connection pool, so that only a single HikariCP connection pool is used for the entire testing phase