We are using PostgreSQL Crane plan, and got a lot of log like this
app postgres - - [5-1] ... LOG: could not receive data from client: Connection reset by peer
We are using about 50 dynos.
Is PostgreSQL running out of connections with bunch of dynos?
Can someone help me explain this case?
Thanks
From what I've found the cause for the errors is the client not disconnecting at the end of the session, or a new connection not being created.
New connection solving the problem:
Postgres error on Heroku with Resque
Explicit disconnection solving the problem:
https://github.com/resque/resque/issues/367 (comment #2)
There's a Heroku FAQ entry on this: Understanding Heroku Postgres Log Statements and Common Errors: could not receive data from client: Connection reset by peer.
Although this log is emitted from postgres, the cause for the error has nothing to do with the database itself. Your application happened to crash while connected to postgres, and did not clean up its connection to the database. Postgres noticed that the client (your application) disappeared without ending the connection properly, and logged a message saying so.
If you are not seeing your application’s backtrace, you may need to ensure that you are, in fact, logging to stdout (instead of a file) and that you have stdout sync’d.
Related
Heroku deleted my rails 5 app's database. I created a new DB, they recovered the old one and copied it to the new one, then attached it. But my site won't load. Heroku support hasn't tried to help beyond giving links to troubleshooting I've done many times. I've been trying every possible fix I can find for a week.
In the error message, I noticed the host that is listed for my database in the settings does not match the host in the timeout error.
MY HOST (per heroku settings): ec2-18-2XX-36-213.compute-1.amazonaws.com
The host in the logs: ec2-5X-83-25-2X7.compute-1.amazonaws.com
(I replaced a couple numbers with X b/c Idk if that's sensitive information)
Is that why the connection won't work? If so, how do I fix it?
FATAL -- : [465fe0b0-f387-4b4a-b215-779dbb8dbf7e] PG::ConnectionBad (connection to server at "ec2-5X-83-25-2X7.compute-1.amazonaws.com" (5X.83.25.2X7), port 5432 failed: Connection timed out
Turns out, all I needed to do was go into my new database's settings, copy the DATABASE_URL, go into the config vars of my app, and paste over the old DATABASE_URL.
So many wasted hours because heroku tech support couldn't articulate that. Took 5 seconds, then site loaded immediately. Didn't need to push anything, commit anything, change any code, etc
I have a Heroku Postgres DB(free tier) that's connected to my backend API for testing purposes. Today I tried accessing the database and I kept getting an error "too many connections for Role 'role'". Please note that I've not connected to this API today and for some reason, all 20 connections have been used up.
I can't even connect to the DB through PgAdmin to even try and kill some of the connections as I get the same error there.
Any help please?
I have several applications running on tomcat with a local PostgresSQL database. And tomcat occasionally reports the following errors:
org.postgresql.util.PSQLException: Connection attempt timed out.
I am able to connect to the database using other tools such as DBeaver. And this problem looks only happens when several applications are connecting to the database. So I want to know how to troubleshoot this issue. Is there any log in PostgresSQL that I can check?
PostgreSQL does have logging. It is very configurable, and we can't tell you how you have it configured. Common locations are /var/log/postgresql/, and in PGDATA/log/. However, a connection timeout will probably not be in the postgresql log as it probably never achieved contact with the postgresql server to start with.
when I run start.bat file of tomcat on linux with postgres database. It gives following error after changed in postgres_hba.conf file.
C3P0PooledConnectionPoolManager[identityToken->2u13c99u1fd86im176rze7|779becbd]-HelperThread-#1--WARN -com.mchange.v2.resourcepool.BasicResourcePool:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#1de8b573 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
The error message is fairly clear. You have tried to open more database connections than you have configured in postgresql.conf. Even the shortest search of your logs and the manuals would have made this clear.
I am getting FATAL: the database system is shutting down errors while creating PostgreSQL JDBC connections to a PostgreSQL 9.2 server. The specific exception path I'm getting from JDBC is here:
Caused by: org.postgresql.util.PSQLException: FATAL: the database system is shutting down
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
From various log files (from PostgreSQL, from our management layer, from the app using PostgreSQL), I do not see any database shutdown actually happening (other connections are created to PostgreSQL as usual, no shutdown was initiated from our management layer, etc.), however in the PostgreSQL server logs I do see the error message with the same timestamp:
2014-06-16 12:30:00.736 GMT LOG: connection received: host=127.0.0.1 port=38530
2014-06-16 12:30:00.737 GMT FATAL: the database system is shutting down
Researching online, I learned that this error message is used whenever PostgreSQL shuts down connections.
Why would PostgreSQL refuse to give me a new connection? Could this be caused by some sort of resource contention? How can I get more information about the error from PostgreSQL itself?
This issue turned out to be caused by a very bad misuse of PostgreSQL where our server had booted two PostgreSQL instances on the same data directory (it had deleted postmaster.pid and used a new port number, so the normal safeguards against this behavior were bypassed) which explains why the logs did not hold any useful information -- they were being overwritten by the PostgreSQL instance which wasn't refusing connections. The actual issue was caused by a complex interaction between the competing PostgreSQL instances, and I hope nobody else runs into this issue as well!
We encountered this problem when the server was restarted. In this case the reboot did not initiate the connection with PostgreSql. Just check the status: "/etc/init.d/postgresql status" if the return is: "no server running" just start: "/etc/init.d/postgresql start"
Maybe there is not enough space on hard drive.
SOLUTION:
brew remove postgresql
Temporary rename the lock files: mv /tmp/.s.PGSQL.5432.lock /tmp/BK.s.PGSQL.5432.lockand mv /tmp/.s.PGSQL.5432 /tmp/BK.s.PGSQL.5432
brew install postgresql
Enjoy!