PostgreSQL Exception: “An I/O error occured while sending to the backend” - postgresql

I run two web app in a machine and one DB in another machine.(They use the same DB)
One can run very well,But another one always down after about 4 hours.
Here is error information:
Error 2014-11-03 13:31:05,902 [http-bio-8080-exec-7] ERROR spi.SqlExceptionHelper - An I/O error occured while sending to the backend.
| Error 2014-11-03 13:31:05,904 [http-bio-8080-exec-7] ERROR spi.SqlExceptionHelper - This connection has been closed.
Postgresql logs:
2014-10-26 23:41:31 CDT WARNING: pgstat wait timeout
2014-10-27 01:13:48 CDT WARNING: pgstat wait timeout
2014-10-27 03:55:46 CDT LOG: could not receive data from client: Connection timed out
2014-10-27 03:55:46 CDT LOG: unexpected EOF on client connection
Who caused this problem, app or database? or net?

Reason:
At this point it was clear that the TCP connection that was sitting idle was already broken, but our app still assumed it to be open. By idle connections, I mean connections in the pool that aren’t in active use at the moment by the application.
After some search, I came to the conclusion that the network firewall between my app and the database is dropping the idle/stale connections after 1 hour. It seemed to be a common problem that many people have faced.
Solution:
In grails, you can set this in DataSource.groovy.
environments {
development {
dataSource {
//configure DBCP
properties {
maxActive = 50
maxIdle = 25
minIdle = 1
initialSize = 1
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 60000
numTestsPerEvictionRun = 3
maxWait = 10000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
validationQuery = "SELECT 1"
}
}
}
}

Related

Program gets stuck while acquiring data from MongoDB

I'm trying to develop a server/client program, the server is based on Netty. When the server gets data from a netty channel, I will query data from mongo. However, the channel will get stuck when it opened MongoDB's connection.
My code is simple is:
Query query = new Query(Criteria.where("account_id").is(accountId).and("account_type").is(accountType));
return seqKeyConvert.toProtoBuf(mongoTemplate.findOne(query, SeqKeyMongoEntity.class, collectionName));
And the server's log will be like this:
[nioEventLoopGroup-3-1] org.mongodb.driver.connection - Opened connection[connectionId{localValue:7, serverValue:7530577}] to 100.75.53.176:5521
[nioEventLoopGroup-3-6] org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:7530585}] to 100.75.53.176:5521
[nioEventLoopGroup-3-8] org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:7530584}] to 100.75.53.176:5521
[nioEventLoopGroup-3-7] org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:7530586}] to 100.75.53.176:5521
The MongoDB configuration is like:
spring:
data:
mongodb:
manager:
address: mongodb://100.75.53.176:5521,100.75.53.177:5521
min-connections-per-host: 2
max-connections-per-host: 100
threads-allowed-to-block-for-connection-multiplier: 10
server-selection-timeout: 30000
max-wait-time: 60000
max-connection-idel-time: 28800000
max-connection-life-time: 0
connect-timeout: 30000
socket-timeout: 0
socket-keep-alive: false
ssl-enabled: false
ssl-invalid-host-name-allowed: false
always-use-m-beans: false
heartbeat-socket-timeout: 20000
heartbeat-connect-timeout: 20000
min-heartbeat-frequency: 500
heartbeat-frequency: 10000
local-threshold: 15
The channel will be stuck here and without any error throws. I have tried to test whether the MongoDB is ok and if I had set too few MongoDB connection number limit but without any finding. Any advice? Thank you.
I use InitialzationBean to set some variables that use MongoDB. It turns out this problem could be solved if I replace InitialzationBean with CommandLineRunner. Temporarily, I don't if some kind of race condition happens.

Spring Boot 1.5.3 Creating more connection than specified in application.properties

I am working on a project where i have dual datasource configured. On testing i have limit the no of max-active connections to five but when i checked on database, i found that application create around 25+ connections.
Code Sample
# Number of ms to wait before throwing an exception if no connection is available.
spring.datasource.tomcat.max-wait=1000
# Maximum number of active connections that can be allocated from this pool at the same time.
spring.datasource.tomcat.max-active=1
spring.datasource.tomcat.max-idle=1
spring.datasource.tomcat.min-idle=1
spring.datasource.tomcat.initial-size=1
# Validate the connection before borrowing it from the pool.
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.test-while-idle = true
spring.datasource.tomcat.validation-query = true
spring.datasource.tomcat.time-between-eviction-runs-millis = 360000
spring.rdatasource.tomcat.max-wait=1000
# Maximum number of active connections that can be allocated from this pool at the same time.
spring.rdatasource.tomcat.max-active=1
spring.rdatasource.tomcat.max-idle=1
spring.rdatasource.tomcat.min-idle=1
spring.rdatasource.tomcat.initial-size=1
# Validate the connection before borrowing it from the pool.
spring.rdatasource.tomcat.test-on-borrow=true
spring.rdatasource.tomcat.test-while-idle= true
spring.rdatasource.tomcat.validation-query = true
spring.rdatasource.tomcat.time-between-eviction-runs-millis = 360000
above connection is working fine, but exceeding no of connection to database. User which i am using is limited to 10 connection.
when i hit request to application than i am getting
query wait timeout error with unable to create initial pool size.
I am using tomcat connection pooling
Please provide me the solution so application will run with 10 connections limit which is set at database.

Spymemcached issue with 2 nodes configured

I am using Ketama algorithm's spymemcached for my project. I do have two memcached servers running as part of HA (high availability) and my configurations are
hibernate.cache.use_second_level_cache=true
hibernate.cache.use_query_cache=true
hibernate.cache.region.factory_class=kr.pe.kwonnam.hibernate4memcached.Hibernate4MemcachedRegionFactory
hibernate.cache.default_cache_concurrency_strategy=NONSTRICT_READ_WRITE
hibernate.cache.region_prefix=myProjectCache
hibernate.cache.use_structured_entries=false
h4m.adapter.class=kr.pe.kwonnam.hibernate4memcached.spymemcached.SpyMemcachedAdapter
h4m.timestamper.class=kr.pe.kwonnam.hibernate4memcached.timestamper.HibernateCacheTimestamperMemcachedImpl
h4m.adapter.spymemcached.hosts=host1:11211,host2:11211
h4m.adapter.spymemcached.hashalgorithm=KETAMA_HASH
h4m.adapter.spymemcached.operation.timeout.millis=5000
h4m.adapter.spymemcached.transcoder=kr.pe.kwonnam.hibernate4memcached.spymemcached.KryoTranscoder
h4m.adapter.spymemcached.cachekey.prefix=myProject
h4m.adapter.spymemcached.kryotranscoder.compression.threashold.bytes=20000
# 10 minutes
h4m.expiry.seconds=600
# a day
h4m.expiry.seconds.validatorCache.org.hibernate.cache.spi.UpdateTimestampsCache=86400
# 1 hour
h4m.expiry.seconds.validatorCache.org.hibernate.cache.internal.StandardQueryCache=3600
# 30 minutes
h4m.expiry.seconds.myProjectCache.database1=1800
h4m.expiry.seconds.myProjectCache.database2=1800
Configurations are followed as per the link below :
SpyMemcachedAdapter
Both nodes host1 and host2 are reachable, up and running.
Issue :
As part of testing HA , when I bring down one memcached (host1) my application is connecting to host2 but only after trying to connect host1 (which will be timedout - as host1 is made down) for every request. Which will result in too much of time taken
Below is the exception thrown for every request
2017-07-07 17:27:31.915 [SimpleAsyncTaskExecutor-6] ERROR u.c.o.sProcessor - TransId:004579 - Exception occurred while processing request :Timeout waiting for value: waited 5,000 ms. Node status: Connection Status { /host1:11211 active: false, authed: true, last read: 247,290 ms ago /host2:11211 active: true, authed: true, last read: 5 ms ago }
2017-07-07 17:28:54.666 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to failure to connect to {QA sa=/host1:11211, #Rops=0, #Wops=214, #iq=0, topRop=null, topWop=Cmd: 5 Opaque: 341143 Key: myProject.myProjectCache.databse1# Amount: 0 Default: 1499444604639 Exp: 2592000, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:677)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:436)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:1446)
2017-07-07 17:28:54.666 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=/host1:11211, #Rops=0, #Wops=214, #iq=0, topRop=null, topWop=Cmd: 5 Opaque: 341143 Key: myProject.myProjectCache.databse1# Amount: 0 Default: 1499444604639 Exp: 2592000, toWrite=0, interested=0}, attempt 14.
2017-07-07 17:28:54.841 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for myProject.myProjectCache.databse1#-1:my.co.org.myProject.dao.entity.databse1.tablexyz#14744.
Am using memcached for the first time, not sure whether this is the behavior of spymemcached? Or am I missing something in my configurations? Or
by changing timeout configurations will it resolve time taken to process the request?
Any suggestions/help much appreciated.
If you are using DefaultConnectionFactory which uses OOTB ConnectionFactoryBuilder then the reconnect will happen after failed operation count has reached timeoutExceptionThreshold (in version 2.7 of spymemcached) is initialized to 998. So if you create your own ConnectionFactory and change the timeoutExceptionThreshold to lower value then you should see the automatic recovery.
Hope this helps.

Failed connection with sphinx

I have a problem starting and working with sphinx.
I was able to run indexer --all, but now I want to search it, and I keep getting this error when I run searchd --status.
WARNING: failed to connect to 127.0.0.1:9312: Connection refused
WARNING: failed to connect to 0.0.0.0:9306: Connection refused
FATAL: failed to connect to daemon: please specify listen with sphinx protocol in your config file
sphinx query() returns false, and I guess that's related to connection problem.
Here's the part of my .conf file.
searchd
{
listen = 127.0.0.1:9312
listen = 9306:sphinx
listen = 2471:mysql41
log = /var/log/sphinx/searchd.log
query_log = /var/log/sphinx/query.log
max_matches = 1000
read_timeout = 5
max_children = 30
pid_file = /var/run/sphinx/searchd.pid
seamless_rotate = 1
preopen_indexes = 1
unlink_old = 1
workers = threads # for RT to work
binlog_path = /var/lib/sphinx
}
What am I missing in configuration of listening ports?
As noted in comments, indicates searchd daemon not actully running.
Can try using searchd to start the daemon (and later searchd --stop), which can show errors you might not see with using service/init.d starting.
(because if the log file itself is not functional, there is nowhere for errors to go :)

Memcached - Connection refused

I tried a simple test with memcached from jelastic and always getting the exception "COnnection refused"... But the URL ist correct. Is some add
MemcachedClient c = new MemcachedClient(
new InetSocketAddress("memcached-myexample.jelastic.dogado.eu", 11211));
c.set("someKey", 3600, user);
User cachedUser = (User) c.get("someKey");
Here is the exception:
2014-01-02 00:07:41.820 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=memcached-myexample.jelastic.dogado.eu/92.51.168.106:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-01-02 00:07:41.833 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for someKey.
2014-01-02 00:07:41.835 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for someKey.
2014-01-02 00:07:41.858 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl#2dc1482f
2014-01-02 00:07:41.859 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to failure to connect to {QA sa=memcached-myexample.jelastic.dogado.eu/92.51.168.106:11211, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd: set Key: someKey Flags: 1 Exp: 3600 Data Length: 149, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:629)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:409)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:1334)
I would try to telnet to your memcached cluster in order to rule out a firewall issue. You can do that with the following command.
telnet memcached-myexample.jelastic.dogado.eu 11211
If that doesn't work then you have network issues. If this is the case I would first check to see if you have a firewall up.
Add int portNum = 11211; at the first, and try again.
int portNum = 11211;
MemcachedClient c = new MemcachedClient(
new InetSocketAddress("memcached-myexample.jelastic.dogado.eu", portNum));
// Store a value (async) for one hour
c.set("someKey", 3600, someObject);
// Retrieve a value (synchronously).
Object myObject=c.get("someKey");
Thanks but the error was on a firewall rule from the provider. So not my failure.
Check /etc/memcached.conf file and update the server IP address from which you want to access the cache.