I am trying to create Hikari connection pool with Postgres database.
Hikari version is 3.4.1 and postgres driver version is 42.2.5
My connection properties are as follows.
mdb.hcp.driverClassName=org.postgresql.Driver
mdb.hcp.username=user1
mdb.hcp.jdbcUrl=jdbc:postgresql://localhost:2001/db1?ssl=false
mdb.hcp.password=pass
mdb.hcp.maximumPoolSize=1
mdb.contractSchema=schema1
I am getting below exception.
06-12-2019 17:40:08.181 [main] INFO com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Driver does not support get/set network timeout for connections. (Method org.postgresql.jdbc4.Jdbc4Connection.getNetworkTimeout() is not yet implemented.)
06-12-2019 17:40:08.649 [main] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Default transaction isolation level detection failed (This ResultSet is closed.).
06-12-2019 17:40:08.656 [main] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Error thrown while acquiring connection from data source
org.postgresql.util.PSQLException: This ResultSet is closed.
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.checkClosed(AbstractJdbc2ResultSet.java:2852)
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.setFetchSize(AbstractJdbc2ResultSet.java:1875)
at org.postgresql.jdbc4.Jdbc4Statement.createResultSet(Jdbc4Statement.java:37)
at org.postgresql.jdbc2.AbstractJdbc2Statement$StatementResultHandler.handleResultRows(AbstractJdbc2Statement.java:221)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1853)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLUpdate(AbstractJdbc2Connection.java:382)
at org.postgresql.jdbc2.AbstractJdbc2Connection.getTransactionIsolation(AbstractJdbc2Connection.java:904)
at com.zaxxer.hikari.pool.PoolBase.checkDefaultIsolation(PoolBase.java:471)
at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:434)
at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:402)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:355)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:473)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:562)
at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115)
at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81)
Please help!
Try to switch to a newer postgres driver version https://mvnrepository.com/artifact/org.postgresql/postgresql
Related
I have Spring Boot application running on Kubernetes connected to Mongo database (deployed with Mongo Operator)
Sometimes - I can't see a pattern so far - the connection between Spring application & MongoDB breaks. After some time connection is restored but this may cause an error to the user.
Logs from application:
12:00:32.606 INFO org.mongodb.driver.connection : Closed connection [connectionId{localValue:3, serverValue:17292}] to kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017 because there was a socket exception raised by this connection.
12:00:32.607 INFO org.mongodb.driver.cluster : No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketReadException: Prematurely reached end of stream}}]}. Waiting for 30000 ms before timing out
12:00:32.612 INFO org.mongodb.driver.cluster : Exception in monitor thread while connecting to server kompas2mongo-0.kompas2mongo-svc.dev.svc.cluster.local:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188)
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:589)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
at java.base/java.net.Socket.connect(Socket.java:648)
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107)
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
... 4 common frames omitted
12:00:42.808 INFO org.mongodb.driver.cluster : Cluster description not yet available. Waiting for 30000 ms before timing out
12:01:02.612 ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/api] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Prematurely reached end of stream; nested exception is com.mongodb.MongoSocketReadException: Prematurely reached end of stream] with root cause
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
Logs from MongoDB:
2022-08-21T12:00:33.583Z INFO controllers/replica_set_controller.go:132 Reconciling MongoDB {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.583Z DEBUG controllers/replica_set_controller.go:134 Validating MongoDB.Spec {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.583Z DEBUG controllers/replica_set_controller.go:143 Ensuring the service exists {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.600Z INFO controllers/replica_set_controller.go:390 The service already exists... moving forward: services "kompas2mongo-svc" already exists {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.600Z INFO controllers/replica_set_controller.go:308 Creating/Updating AutomationConfig {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.628Z INFO agent/agent_readiness.go:52 All 1 Agents have reached Goal state {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.628Z INFO controllers/replica_set_controller.go:288 Creating/Updating StatefulSet {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.639Z DEBUG controllers/replica_set_controller.go:298 Ensuring StatefulSet is ready, with type: RollingUpdate {"ReplicaSet": "dev/kompas2mongo"}
2022-08-21T12:00:33.639Z INFO controllers/mongodb_status_options.go:110 ReplicaSet is not yet ready, retrying in 10 seconds
2022-08-21T12:00:34.612Z INFO controllers/replica_set_controller.go:132 Reconciling MongoDB {"ReplicaSet": "dev/kompas2mongo"}
Mongo is configured as a Replica Set and application.yaml looks like this:
spring.data.mongodb:
host: ${MONGO_HOST}
database: ${MONGO_DATABASE}
username: ${MONGO_USERNAME}
password: ${MONGO_PASSWORD}
In documentation there is different connection string for replica set. Could this be the issue? Or do you have other ideas?
My app is running fine locally when testing with Postgres. But I am trying to connect to my RDS database which has Status:'Available' and Publicly acessible: 'Yes'.
RDS environment image
When updating the properties file with Amazon RDS endpoint in the datasource URL I get an error when running the application. The username and password are also correct in the properties file.
Application.properties file image
The error on Intellij console is:
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:313) ~[postgresql-42.3.1.jar:42.3.1]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.3.1.jar:42.3.1]
at org.postgresql.Driver.makeConnection(Driver.java:466) ~[postgresql-42.3.1.jar:42.3.1]
at org.postgresql.Driver.connect(Driver.java:265) ~[postgresql-42.3.1.jar:42.3.1]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-4.0.3.jar:na]
at
... 55 common frames omitted
2022-01-18 23:26:55.667 WARN 18380 --- [ restartedMain] o.h.e.j.e.i.JdbcEnvironmentInitiator : HHH000342: Could not obtain connection to query metadata
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:313) ~[postgresql-42.3.1.jar:42.3.1]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.3.1.jar:42.3.1]
2022-01-18 23:26:55.667 INFO 18380 --- [ restartedMain] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect
Hibernate:
drop table if exists employee cascade
2022-01-18 23:26:56.026 INFO 18380 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2022-01-18 23:27:07.043 ERROR 18380 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: The connection attempt failed.
at
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-01-18 23:27:07.074 ERROR 18380 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to open JDBC Connection for DDL execution
at
Process finished with exit code 0
Any suggestions would be greatly appreciated.
Note that spring is not complaining about wrong username/password. It cannot even connect to the database to check if the username/password is correct.
That indicates a network issue (i.e. there is no network connectivity from your machine to the RDS instance).
Please check if your security group allows access to the given port (5432) from your IP
I want to run a self managed Thingsboard on AWS (t2.micro).
I have installed Thingsboard CE on a t2.micro AWS instance running Ubuntu 20.04 sever.
I followed the aws setup and Ubuntu install guides(postgresql + built in queue service).
I also set up haproxy using this guide.
I was able to successfully log in to my Thingsboard. I only changed the passwords and checked the basic functionalities, but didn't create any new dashboards or made any modifications.
After this I left the computer on, running Thingsboard. Next day I could not reach Thingsboard and although the AWS instance was running I could not ssh into it anymore. After stopping and starting(reboot didn't work) the instance everything was ok (could ssh and Thingsboard was reachable).
I can reproduce this failure just by leaving the instance on, it seems that after serveral hours (5-8 hrs) Thingsboard(or something else not sure) fails which freezes the whole computer.
I have checked two things:
I checked CPU utiization on AWS monitoring.
It seems that after some hours there is a big jump in CPU load and then it drops back to almost zero. While Thingsboard is running, it is constant.See printscreen from AWS monitoring
I checked the Thingsboard logs (in /var/log/thingsboard):
There are some errors, but unfortunately most of the things are not enough for me guess what could be a problem with the fresh installation. Here are some lines from the log:
2021-11-12 00:21:59,626 [http-nio-0.0.0.0-8080-exec-13] INFO o.a.coyote.http11.Http11Processor - Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name
[0x160x030x010x00{0x010x000x00w0x030x030x170xb80xb80xe50xef0x000xb50x0a&0x930x020x00:0xde0xd70xa00xab0xb
70x8bU0xc00x92r0x9330x10O0x8c<o0xf70xf90x000x000x1a0xc0/0xc0+0xc00x110xc00x070xc00x130xc00x090xc00x140xc00x0a0x000x050x00/0x0050xc00x120x000x0a0x010x000x0040x000x050x000x050x010x000
x000x000x000x000x0a0x000x080x000x060x000x170x000x180x000x190x000x0b0x000x020x010x000x000x0d0x000x100x000x0e0x040x010x040x030x020x010x020x030x040x010x050x010x060x010xff0x010x000x010x00...].
HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:417)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-11-12 00:22:01,486 [sql-queue-2-ts-4-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#4393afd0 (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-2-ts latest-8-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#75b9496b (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-0-ts latest-6-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#31849eec (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-0-ts-2-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#725fafe3 (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
Some more:
2021-11-12 00:23:46,205 [sql-log-1-thread-1] INFO o.t.s.dao.sql.TbSqlBlockingQueue - Queue-2 [TS Latest] queueSize [9] totalAdded [0] totalSaved [0] totalFailed [0]
2021-11-12 00:23:47,741 [sql-queue-0-ts-2-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-2-ts-4-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-2-ts latest-8-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-0-ts latest-6-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:48,022 [sql-queue-0-ts-2-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 634223ms.
2021-11-12 00:23:48,058 [sql-queue-0-ts-2-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,022 [sql-queue-0-ts latest-6-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 634223ms.
2021-11-12 00:23:48,059 [sql-queue-0-ts latest-6-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,022 [sql-queue-2-ts latest-8-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 624177ms.
2021-11-12 00:23:48,059 [sql-queue-2-ts latest-8-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,023 [sql-queue-2-ts-4-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 627819ms.
2021-11-12 00:23:48,059 [sql-queue-2-ts-4-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
At the last:
2021-11-12 00:33:10,919 [sql-queue-0-ts latest-6-thread-1] ERROR o.t.s.dao.sql.TbSqlBlockingQueue - [TS Latest] Failed to save 1 entities
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:448)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:574)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:361)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
at org.thingsboard.server.dao.sqlts.insert.latest.psql.PsqlLatestInsertTsRepository$$EnhancerBySpringCGLIB$$381b448c.saveOrUpdate(<generated>)
at org.thingsboard.server.dao.sqlts.SqlTimeseriesLatestDao.lambda$init$3(SqlTimeseriesLatestDao.java:133)
at org.thingsboard.server.dao.sql.TbSqlBlockingQueue.lambda$init$2(TbSqlBlockingQueue.java:71)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:276)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:284)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:184)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:402)
... 16 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 634223ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:695)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 23 common frames omitted
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:877)
at org.postgresql.jdbc.PgConnection.setNetworkTimeout(PgConnection.java:1610)
at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:560)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:173)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
... 28 common frames omitted
What is interesting that the timestaps on the CPU load going to max don't precisely correlate with the error messages in the log.
I apologise for the long error messages, but right now I don't know what could be the root cause.
I haven't tried to reinstall the whole computer yet.
My question would be, how should I proceed? Does anyone have ever faced similar issues? What logs/services/etc. should I check to grasp the root cause?
Should I try using a machine with more resources? Should I try other database and queue service?
In the current form this Thingsboard instance is not stable even for tests.
Edit: Sorry I could not format properly the first part of the error code.
Edit2: First link was wrong.
Here are some points:
It looks like the operating system run out of memory and become unresponsive. To fix the issue try to manage Java heap memory
For 4Gb instance, this Java heap limit JAVA_OPTS="$JAVA_OPTS -Xms1024M -Xmx1024M" may be useful, because Java uses some non-heap memory as well, PostgreSQL and others require some memory to run.
t2 instances on AWS may slow down the whole thing by CPU throttling. Instances like c6 or m5 are better options by performance.
In-memory queues may result in out-of-memory issues and data loss in case of a heavy message rate or some processing congestion due to the third party. Consider using Kafka to make your installation much stable and solid.
After I increased the RAM to 4GB (from 1GB) the ThingsBoard server is up without problem. There are no sporadic freezes anyomore. As there were no other provable suggestions for the problem and now my system works without a problem I consider the question answered.
I am running a Spring boot Java application using default HikariCP as data source:
Hikari version - 3.4.5
Spring Boot version - 2.4.5 running in AWS EKS
JDBI version - 3.9.1
DB - AWS RDS Postgres
My Spring application becomes unresponsive as it loses all the connections in the Hikari Pool. This does not happen deterministically, but once it starts happening, the only recovery possible is through restart.
Hikari Config:
The pool keeps losing the connections and fails to connect back, leaving total=0 connections.
06-23 08:14:33.242 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
06-23 08:15:07.189 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
06-23 08:15:07.190 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Fill pool skipped, pool is at sufficient level.
06-23 08:15:32.785 [HikariPool-1 connection closer] DEBUG PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#58f866df: (connection was evicted)
06-23 08:15:41.278 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Pool stats (total=9, active=1, idle=8, waiting=0)
06-23 08:15:49.809 [HikariPool-1 connection adder] DEBUG PoolBase - HikariPool-1 - Failed to create/setup connection: Connection attempt timed out.
06-23 08:15:49.809 [HikariPool-1 connection adder] DEBUG HikariPool - HikariPool-1 - Cannot acquire connection from data source
org.postgresql.util.PSQLException: Connection attempt timed out.
at org.postgresql.Driver$ConnectThread.getResult(Driver.java:422)
at org.postgresql.Driver.connect(Driver.java:267)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:725)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:711)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
06-23 08:15:49.810 [HikariPool-1 connection adder] DEBUG HikariPool - HikariPool-1 - Connection add failed, sleeping with backoff: 250ms
06-23 08:15:49.816 [HikariPool-1 connection closer] DEBUG PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc.PgConnection#7303422e: (connection was evicted)
(After more of these)
06-23 10:23:21.731 [ForkJoinPool-1-worker-7] DEBUG HikariPool - HikariPool-1 - Add connection elided, waiting 2, queue 10
06-23 10:23:55.883 [ForkJoinPool-1-worker-7] DEBUG HikariPool - HikariPool-1 - Timeout failure stats (total=0, active=0, idle=0, waiting=0)
06-23 10:23:55.883 [pool-3-thread-4] DEBUG HikariPool - HikariPool-1 - Timeout failure stats (total=0, active=0, idle=0, waiting=0)
06-23 10:23:55.883 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Pool stats (total=0, active=0, idle=0, waiting=0)
06-23 10:23:55.883 [HikariPool-1 housekeeper] DEBUG HikariPool - HikariPool-1 - Fill pool skipped, pool is at sufficient level.
FWIW, I've found the logs for leak detection, but they are followed by unleaked logs. The leak threshold is set to 20 seconds.
06-23 08:07:04.506 [HikariPool-1 housekeeper] WARN ProxyLeakTask - Connection leak detection triggered for org.postgresql.jdbc.PgConnection#58f866df on thread ForkJoinPool-1-worker-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:223)
at com.sun.proxy.$Proxy119.getAutoCommit(Unknown Source)
at org.jdbi.v3.core.transaction.LocalTransactionHandler.isInTransaction(LocalTransactionHandler.java:118)
at org.jdbi.v3.core.Handle.(Handle.java:76)
at org.jdbi.v3.core.Jdbi.open(Jdbi.java:309)
(Application specific stacktrace)
...
06-23 08:07:04.527 [ForkJoinPool-1-worker-1] INFO ProxyLeakTask - Previously reported leaked connection org.postgresql.jdbc.PgConnection#58f866df on thread ForkJoinPool-1-worker-1 was returned to the pool (unleaked)
When I redeploy the application, it starts up perfectly, and other instances of this application work correctly, so I have ruled out issues on network and RDS side.
My questions:
Is connection leaks the source of problem? As they are unleaked, shouldn't Hikari be able to reclaim the connections and create new ones? Is there a way to confirm the root cause?
The leaks don't happen often, but when they do, they start happening for several requests until the app becomes unresponsive. Is there a way to find out why the requests start taking more than 20 seconds?
What config changes can help with more diagnosis and further, to fix the problem?
Thanks.
If you don't close connection, hikari can't know you need/want to close it
You must find where you aren't closing connection properly in code
You can reduce the leaks by setting a lower maxLifetime to return the connection to pool sooner
This property controls the maximum lifetime of a connection in the pool. An in-use connection will never be retired, only when it is closed will it then be removed. On a connection-by-connection basis, minor negative attenuation is applied to avoid mass-extinction in the pool. We strongly recommend setting this value, and it should be several seconds shorter than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. The minimum allowed value is 30000ms (30 seconds). Default: 1800000 (30 minutes)
Follow Hikari's configuration tips for PostgreSQL
You can add monitor tools/code, but you must fix the problem
I am running a maven spring boot application which fetches data from postgreSQL. But I am getting such type of error when I run the project using the command
mvn spring-boot:run
->
2019-05-29 14:37:03.750 WARN 18368 --- [ main] unknown.jul.logger : IOException occurred while connecting to mc:19745
java.net.SocketTimeoutException: connect timed out
2019-05-29 14:36:52.241 ERROR 18368 --- [ main] org.postgresql.Driver : Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
Caused by: java.net.SocketTimeoutException: connect timed out
2019-05-29 14:36:53.245 ERROR 18368 --- [ main]
com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: The connection attempt failed.
I have mentioned only the main errors.
The application.properties file is:
spring.main.banner.mode=/offlinekyc
# Oracle settings
spring.datasource.url=jdbc:postgresql://mc:19745/post
spring.datasource.username=postgre
spring.datasource.password=123
spring.jpa.properties.hibernate.default_schema=oms_owner
spring.thymeleaf.cache=false
spring.jpa.properties.hibernate.dialect= org.hibernate.dialect.PostgreSQLDialect
http.mappers.json-pretty-print=true
spring.jpa.properties.hibernate.jdbc.lob.non_contextual_creation=true
hibernate.discriminator.ignore_explicit_for_joined=false
# create and drop tables and sequences, loads import.sql
spring.jpa.hibernate.ddl-auto= update
spring.datasource.driverClassName=org.postgresql.Driver
# HikariCP settings
# spring.datasource.hikari.*
spring.datasource.hikari.connection-timeout=60000
spring.datasource.hikari.maximum-pool-size=5
#hibernate.dialect=org.hibernate.dialect.PostgreSQL9Dialect
#hibernate.default_schema=oms_owner
#hibernate.show_sql=true
#hibernate.format_sql=true
#hibernate.jdbc.lob.non_contextual_creation=true
spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults=false
# Because detection is disabled you have to set correct dialect by hand.
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL9Dialect
Hibernate.properties file also have the same matter.
Here my post database is in mc:19745 server. I am little confused regarding localhost and server address.
Can anyone help in this matter?