We have created a Spring based application which is interacting with MongoDB as back end. We are using MongoTemplate to initiate connections.While running the application , we are experiencing connection time outs with below error,
com.mongodb.DBPortPool$ConnectionWaitTimeOut: Connection wait timeout after 1500 ms
We have changed below parameters, but no luck.
connections-per-host
connect-timeout
max-wait-time
Our observation is that , whenever we experience the timeouts , nubmer of open connections remains the same all the time even before and after the timeouts.
Can you please help me to pin point an issue ?
Related
We are facing issue with windows service, its hanging just after opening the oracle connection and while creating the oracle command.
Before killing the service, when we analyzed the database logs, it showed the inactive session (not closed) and it’s matching the ‘DB last call time’ with the ‘service hung time’. That particular db session keeps waiting for application to instruct the next steps and stays idle, until we kill the service.
Most importantly this issue happens only sometimes without any pattern.
We have pool size set to 50 and connection timeout to 10min at application.
Types of oracle commands used:
using (OracleCommand cmd = oraCon.CreateCommand())
_cmd = new OracleCommand(“sqlText”, _connection)
Some of our applications are facing issues with the connection pool. I run one of them. A JEE application on Payara 4.1 which uses PostgreSQL 9.5.8.
I have as good as no problems when running the application localy with local db instance. When running on the remote environment I have seen issues happening every 10 minutes that the application was unresponsive (well, it actually responded everything with HTTP status 503). Guessing it was related to opening connections taking long, we have set the parameter idleTimeoutInSeconds="0" in jdbc-resource. Now we have the same issues about 4 times a day which is an improvement, but - well - neighbour systems are still complaining.
We usually run with 5 steady connection allowing maximum connections of 30. Our application usually uses 1 up to 2 to handle traffic. With TCP dump I have seen, that at a certain point in time the connection pool tries to open many connections (the pool realizes the connections it holds have been closed by the DB without any information like TCP FIN, opening each connections takes about 1 second). During this time of about 30 seconds not all requests can be safely queued and some 503 happen.
Locally everything is fine. Opening a connection takes ~50ms and everyone is happy. Our postgres team is not helping at all and I am stuck with a problem. As I don't see any improvement possibility with the connection pool in JEE, I have radical ideas going in the direction of:
Refreshing the connections myself. All the time. Constantly. (Which would be hard to implement in JEE where I can not simply look into the connection pool and tell each connection to be refreshed just in case).
Replacing the not-helping-at-all JEE implementation of connection pool with something that works better. (Future generations of developers maintaining our app will hate me...)
Replacing the DB with something managed by myself. (Even dumber idea)
Does anyone:
Has any idea how I could perform 1 or 2 above?
Has any other ideas what could help?
Here my current JDBC resource definition if needed:
<jdbc-resource poolName="<poolName>" jndiName="<jndiName>" isConnectionValidationRequired="true"
connectionValidationMethod="table" validationTableName="version()" maxPoolSize="30"
validateAtmostOncePeriodInSeconds="30" statementTimeoutInSeconds="30" isTimerPool="true" steadyPoolSize="5"
idleTimeoutInSeconds="0" connectionCreationRetryAttempts="100000" connectionCreationRetryIntervalInSeconds="30"
maxWaitTimeInMillis="2000">
I have a play app on heroku connecting to a postgres instance with play-slick. Around 30% of the time when I deploy a new application I get this in my logs:
java.sql.SQLTransientConnectionException: db - Connection is not available, request timed out after 1007ms.
When I restart the application it will usually start again, though sometimes it takes a few tries.
Any advice for what I can do to debug this?
Most likely, there is a period of time where both the old app and the new app are trying to get connections to the database, which means you have double you max allowed connections active.
There are two solutions:
Upgrade your database plan to allow for more connection
Reduce your max db connections by half
play-slick uses HikariCP to pool connections, so you can probably configure your max connections with maximumPoolSize.
I believe I've figured out what the issue was. I used the default heroku play Procfile which contains -Ddb.default.url=${DATABASE_URL} and additionally had the slick db url specified in my conf. Removing the former solved the problem.
I am sending a soap request using soapUI to fetch data from oracle 10g db. Soap UI successfully displays response when the db fetches results within 30 seconds.
But the real problem is when the db response exceeds 30 seconds, soapUI displays following error message :
Fault occurred while processing.
I have tried the below 3 scenarios :
1) Increased the socket timeout to 1200000
2) Increased the timeout values in tomcat server config file (/conf/server.xml)
3) Checked for any Null Pointer exception and found none.
Please help me to get success message in soapUI. Thanks in advance.
There are many components between the SoapUI adapter and the database engine. Most of these will have a configurable timeout.
The listener on the database server will pass the query to the database engine - and the database engine itself will have some protection against long-running queries. It's quite likely that the database is killing queries that run over 30 seconds.
You can prove this by capturing a query from your application and trying the same query directly in the database administration tool. This will tell you why the query fails (if it fails.)
JDBC calls a component listening on the database server - this flow itself will have a timeout, which you can set at the JDBC level somewhere in your environment.
Brief Background:
We have a cloud based Warehouse Management System that uses Glassfish to dish out the java interface. The Warehouse Management System consists of a Dashboard and a mobile application - both of which talk constantly with the Glassfish server (using a web browser).
Issue:
Recently our PostgreSQL database server HDD failed. After restoring from a backup and moving the database to an Amazon Web Service Server, idle connections seem to be dropping out. This causes the entire Warehouse Management System to fail. Restarting the Glassfish server seems to fix the issue until the idle connection causes it to fail again.
It happens around 3-4 times per day after approx 20mins of idle activity i.e. our customer's lunch breaks, after hours etc..
Question:
Is there a setting that I'm missing in the postgresql.conf file? What else could be causing this?
Attachments:
I've attached a screenshot containing the output of running 'select * from pg_stat_activity;' and also the postgresql.conf file.
select * from pg_stat_activity
postgresql.conf
Log:
postgresql-8.4-main.log shows this occasionally, although it doesn't seem to be when it cuts out.
2015-10-19 07:51:41 NZDT [9971-1] postgres#customerName LOG: unexpected EOF on client connection
glassfish server.log is riddled with these lines:
[#|2015-10-19T07:46:49.715+1300|SEVERE|glassfish3.1.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=25;_ThreadName=Thread-2;|WebModule[/pns-CustomerName]Received InterruptedException on request thread
[#|2015-10-20T09:34:42.351+1300|WARNING|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-8080(2).|
[#|2015-10-20T07:33:55.414+1300|WARNING|glassfish3.1.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=14;_ThreadName=Thread-2;|Response Error during finishResponse java.lang.NullPointerException
Thanks in advance