PostgreSQL on cloudfoundry throws PSQLException: FATAL: terminating connection due to administrator command - postgresql

I have a grails application successfully running in cloudfoundry with H2 as database. I now want to switch to postgresql. Everything works as expected when I run the application locally against postgreSQL.
On the cloudfoundry instance I get an exception during one rather long running analysis, which I perform as a background task (using grails executor plugin). Within this asynchronous task
def future = executorService.submit({
return analysisService.analyzeProject(model, project)
})
I get the following exception from the database:
2012-11-26 10:27:38,319 [pool-2-thread-1] ERROR interceptor.TransactionInterceptor - Application exception overridden by rollback exception
org.springframework.dao.DataAccessResourceFailureException: Hibernate operation: could not execute query; SQL [select this_.id as id8_0_, this_.version as version8_0_, this_.language as language8_0_, this_.url as url8_0_ from sonar_adapter_configuration this_]; FATAL: terminating connection due to administrator command; nested exception is org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at myapp.adapters.sonar.SonarAdapterService.loadSonarConfig(SonarAdapterService.groovy:184)
at myapp.adapters.sonar.SonarAdapterService.determineArtefactSizes(SonarAdapterService.groovy:145)
at myapp.project.AnalysisService.analyzeProject(AnalysisService.groovy:46)
at myapp.project.ProjectController$_analyzeProject_closure2.doCall(ProjectController.groovy:69)
at grails.plugin.executor.PersistenceContextRunnableWrapper$_run_closure1.doCall(PersistenceContextRunnableWrapper.groovy:34)
at grails.plugin.executor.PersistenceContextWrapper.wrap(PersistenceContextWrapper.groovy:35)
at grails.plugin.executor.PersistenceContextRunnableWrapper.run(PersistenceContextRunnableWrapper.groovy:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
... 13 more
Any ideas what might cause this?

Chances are you are hitting the configured timeout for long running connections (which can't be changed if I'm not mistaken, as it is here to protect the server from ill formed queries that would consume all cpu/io)
What you may want to try is to slice your long running query into smaller queries and sum up the results (if it's a select)

Related

heroku springboot quratz prostgesql error Caused by: org.postgresql.util.PSQLException: ERROR: relation "qrtz_locks" does not exist

when trying to deploy springboot quartz postgresesql in heroku faceing below error,
with this same configuration the app connects from my ide to postgrese without any issue,
any help would be greatly appreciated.
## QuartzProperties
spring.quartz.job-store-type=jdbc
spring.quartz.properties.org.quartz.threadPool.threadCount=5
spring.quartz.jdbc.initialize-schema=never
spring.quartz.jdbc.schema=pw
org.springframework.context.ApplicationContextException: Failed to start bean 'quartzScheduler'; nested exception is org.springframework.scheduling.SchedulingException: Could not start Quartz Scheduler; nested exception is org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ERROR: current transaction is aborted, commands ignored until end of transaction block [See nested exception: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block]]
Caused by: org.postgresql.util.PSQLException: ERROR: relation "qrtz_locks" does not exist

Springboot HikariCP

I using springboot with HikariCP, but after a while my app crash and I got the error:
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
...
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
....
Caused by: java.sql.SQLTransientConnectionException: HikariPool-6 - Connection is not available, request timed out after 30000ms.
This is my aplication.properties
spring.datasource.type=com.zaxxer.hikari.HikariDataSource
spring.datasource.url=jdbc:postgresql://localhost:5432/db_dnaso
#spring.datasource.url=jdbc:postgresql://172.16.1.10:5432/db_dnaso
spring.datasource.username=postgres
spring.datasource.password=dna44100
spring.datasource.driver-class-name=org.postgresql.Driver
So I have a lot of save, find and anothers access to DB, how can I visualize how method are blocking my connection?
tks
It looks like your your database server is running out of connection. Default value for property maximumPoolSize in Hikari is 10. That means it will try to create 10 connection on server startup and it will not start if not able to acquire 10 connection or may fail if your db server pool size having less connection in the pool as you are creating using Hikari configuration.
If you are able to start Spring Boot server and then facing this issue then try enabling leakDetectionThreshold and check which connection is taking more time and not returning to Hikari pool .
spring:
datasource:
hikari:
leak-detection-threshold: 2000
Enable the leakDetectionThreshold, set to something like 1 minute (60000ms). It is likely that you have a connection leak somewhere ... a connection is borrowed but never closed (returned).
In springboot, you can set spring.datasource.hikari.leak-detection-threshold=10000 (in milliseconds) in application.properties.
And the default value for maximumPoolSize is 10. You can change it with spring.datasource.hikari.maximum-pool-size=xx.

MongoDB: PLAIN method not suported

I'm getting this message when I'm trying to deploy my war artifact. My app is using hibernate ogm and it's trying to build persistence context when it's deployed. The message I'm getting is:
org.hibernate.service.spi.ServiceException: OGM000071: Unable to start datatore provider Caused by: org.hibernate.service.spi.ServiceException: OGM000071: Unable to start datatore provider Caused by: org.hibernate.HibernateException: OGM001214: Unable to connect to MongoDB instance: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongo:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=PLAIN, userName='living', source='lvdb', password=, mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException: Command failed with error 2: 'Unsupported mechanism PLAIN' on server mongo:27017. The full response is { \"supportedMechanisms\" : [\"MONGODB-CR\", \"MONGODB-X509\", \"SCRAM-SHA-1\"], \"ok\" : 0.0, \"errmsg\" : \"Unsupported mechanism PLAIN\", \"code\" : 2, \"codeName\" : \"BadValue\" }}}] Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongo:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=PLAIN, userName='living', source='lvdb', password=, mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException: Command failed with error 2: 'Unsupported mechanism PLAIN' on server mongo:27017. The full response is { \"supportedMechanisms\" : [\"MONGODB-CR\", \"MONGODB-X509\", \"SCRAM-SHA-1\"], \"ok\" : 0.0, \"errmsg\" : \"Unsupported mechanism PLAIN\", \"code\" : 2, \"codeName\" : \"BadValue\" }}}]"}}
What do I need to do in order to use the other mechanisms?
You'd have to specificy the property hibernate.ogm.mongodb.authentication_mechanism (see here).
Which is the authentication mechanism you'd want to use? We may not support it yet, but if you let us know about your preference, we can try and look into making it work quickly.
For MongoDB, default supported mechanism is 'SCRAM-SHA-1'. So, while connecting, you need to specify this mechanism. I saw the same error when connecting to MongoDB through spring-data. While I didn't make any special configuration for mongo, first and obvious option for me to try was PLAIN mechanism for authentication. Finally, resolved it using SCRAM-SHA-1.
For Spring-data, code looked something like this:
MongoCredential mongoCredential = MongoCredential.createScramSha1Credential(mongoUser, mongoDB, mongoPass.toCharArray());
Hope this helps.

How to log connection info with Mybatis

When i have a error on a database connection, i need to log it (with Database name and username that tried to connect to the db, don't need to log the password), I use Sprig-Mybatis on my aplication.
Any ideas how to do that?
Best Regards
There is no other special way to do as a simple try/catch in java will get you the expection.
For example a simple program where I am calling a mybatis mapper method:
try{
List result = mapper.myBatisMapperMethod();//this is responsible to set up
//a connection and run a query
}
Suppose the above call fails due to invalid authentication say the below where my password is wrong connection
exception that I usually get in mybatis:
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'applicationUsername'.)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user 'applicationUsername'.)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1225)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 65 more
Caused by:
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'applicationUsername'.
what I do is have a simple catch to above try you can have an sql exception catched but here I use simply Exception.
catch(Exception e){
//Suppose you are using Log4j You can simply
logger.error("This is my error log message"+e.getMessage());
//The above will log any kind of exception including the authentication fail or you
can specifically log a particular kind of exception.Java wont log failed password it just logs the user who tried unless you explicitly want to log it.
}

Twiddle connecting to remote jboss for JMX based operation

We have a couple of jboss instances running on different machines. We needed a script to invoke a JMX console operation using twiddle utility available with JBoss.
For localhost the command is working fine but when we try to access the remote via:
twiddle.sh -s <servername>:<http port> -uadmin -p<password> serverinfo -c
but for remote servers we are facing following exception:
05:25:34,205 ERROR [Twiddle] Exec failed
org.jboss.util.NestedRuntimeException: - nested throwable: (javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table])
at org.jboss.console.twiddle.Twiddle$1.getServer(Twiddle.java:144)
at org.jboss.console.twiddle.command.MBeanServerCommand.getMBeanServer(MBeanServerCommand.java:59)
at org.jboss.console.twiddle.command.MBeanServerCommand.queryMBeans(MBeanServerCommand.java:66)
at org.jboss.console.twiddle.command.InvokeCommand.execute(InvokeCommand.java:274)
at org.jboss.console.twiddle.Twiddle.main(Twiddle.java:306)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
at org.jboss.console.twiddle.Twiddle.createMBeanServerConnection(Twiddle.java:251)
at org.jboss.console.twiddle.Twiddle.connect(Twiddle.java:269)
at org.jboss.console.twiddle.Twiddle.access$300(Twiddle.java:63)
at org.jboss.console.twiddle.Twiddle$1.getServer(Twiddle.java:140)
... 4 more
Caused by: java.rmi.NoSuchObjectException: no such object in table
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:675)
... 10 more
Any help?
We weren't able to find a solution for this and finally managed to do so using SSH only.