I'm using JDBC to run queries against my PostgreSQL DB. When the queries take too long I want them to die. I set socketTimeout (as shown below), which does successfully kill the connection, but if I look at pg_stat_activity on the PostgreSQL server, the query often continues to run (for up to a few minutes).
My code looks a little like this:
Properties props = new Properties();
props.setProperty("user", datasource.getUsername());
props.setProperty("password", datasource.getPassword());
props.setProperty("socketTimeout", ""+(timeoutSeconds));
props.setProperty("loginTimeout", "10");
conn = datasource.getDbType().getDriver().connect(url, props);
conn.setReadOnly(true);
conn.setAutoCommit(false);
PreparedStatement statement = conn.prepareStatement(sql);
return statement.executeQuery();
My assumption is that socketTimeout literally just kills the connection, but not the running process on the postgres server. I.e., this timeout is driver-side not PostgreSQL-side.
Is there a way to set a timeout (on a per-statement level) that will actually make PostgreSQL kill the query itself after a given timeout?
Note: I have found one solution so far, which is to explictly cancel the query after a given timeout (i.e. using statement.cancel()). This works, but isn't ideal, since if my server dies, then nobody will kill the running queries.
You should set statement_timeout.
Related
I'm using SQLAlchemy inside a Lambda with RDS Proxy (PostgreSQL).
This configuration works, but when I'm invoking about 100 lambdas at the same time some lambdas receive a timeout (3 seconds) even when I'm increasing the timeout for 30 seconds I still get a timeout.
After investigating where this timeout comes from, I concluded that the SQLAlchemy hangs when it tries to get a connection to the database.
I thought the RDS proxy could easily handle 100+ concurrent connections.
Here is my engine configuration:
db_url = f"postgresql://{db_username}:{secret}#{db_hostname}:{db_port}/{db_name}"
engine = create_engine(
db_url,
echo=True,
echo_pool="debug",
poolclass=NullPool)
Usually, when SQLAlchemy creates a connection it logs the following:
DEBUG sqlalchemy.pool.impl.NullPool Created new connection <connection object at 0x7f074ebe5580; dsn: 'user=user password=xxx dbname=dbname host=rds-default.{rds-proxy-id}.{region}.rds.amazonaws.com port=port', closed: 0>
But sometimes it will just hang and won't log anything and eventually timeout.
Any help would be appreciated
I have several clients (FreeRadius servers) that connect to a single central Pgbouncer.
When I utilise one of these FreeRadius servers I can see that several database connections are created by Pgbouncer.
select *
from pg_stat_activity
where datname = 'master_db';
When I utilise the same freeradius server again, pgbouncer isn't re-using the existing open connections but keeps creating new ones. Once I reach over 30 connections the whole database comes to an halt.
PGbouncer.ini
server_idle_timeout = 10
max_client_conn = 500
default_pool_size = 30
postgresql.conf: (Postgres 13)
max_connections = 150
Based on my research Pgbouncer is supposed to allocate a single database connection to a client (from the default_pool_size) and then create as many internal connections the client needs (up to max_client_conn).
But what I observe here is the opposite. What am I missing, please?
UPDATE:
The solution Laurenz suggested works but throws this error, when using asyncpg behind the scenes:
NOTE: pgbouncer with pool_mode set to "transaction" or "statement" does not support prepared statements properly. You have two options: * if you are using pgbouncer for connection pooling to a single server, switch to the connection pool functionality provided by asyncpg, it is a much better option for this purpose; * if you have no option of avoiding the use of pgbouncer, then you can set statement_cache_size to 0 when creating the asyncpg connection object.
You will need to use the session option for POOL_MODE so that it is able to maintain connection sessions opened by asyncpg because of the asynchronous nature
You should the following in your
if using pgbouncer.ini file
[pgbouncer]
pool_mode = session
...
...
...
or if using env variable
POOL_MODE=session
extra source: https://magicstack.github.io/asyncpg/current/faq.html#why-am-i-getting-prepared-statement-errors
If you are getting intermittent prepared statement
"asyncpg_stmt_xx" does not exist or prepared statement
“asyncpg_stmt_xx” already exists errors, you are most likely not
connecting to the PostgreSQL server directly, but via pgbouncer.
pgbouncer, when in the "transaction" or "statement" pooling mode, does
not support prepared statements. You have several options:
if you are using pgbouncer only to reduce the cost of new connections
(as opposed to using pgbouncer for connection pooling from a large
number of clients in the interest of better scalability), switch to
the connection pool functionality provided by asyncpg, it is a much
better option for this purpose;
disable automatic use of prepared statements by passing
statement_cache_size=0 to asyncpg.connect() and asyncpg.create_pool()
(and, obviously, avoid the use of Connection.prepare());
switch pgbouncer’s pool_mode to session.
I am using a Hikari data source in my application and database kills all the connections which are idle for more than 15 mins. So i wanted to set a connection test query. How do I know when this query is fired and how can I log when this query is fired?
I have got a resources release problem.
PostgreSQL runs on a server 1.
Go lang service runs on a server 2 in a Docker container.
There is an ssh tunnel Docker container for a connection between the database and the service. It does not know anything except ssh.
Docker is in swarm mode.
The service (2) connects to the database via golang database/sql library. I call sql.Open(), driverName = "postgres". Then everything is ok. In some time, may be in 30 minutes, (*DB) Query() returns an error read: connection reset by peer. If I call (*DB) Ping() previously, Ping() does not return an error, but the next call Query() does.
If I call Query() again in some time, a new connection is created. I can see it in a select * from pg_stat_activity; query in the database (state = idle). But the previous connection has not been removed.
So I call (*DB) Close(), create a new DB object and call sql.Open().
Close closes the database, releasing any open resources.
(https://golang.org/pkg/database/sql/#Conn.Close)
But after the Close call I can still see the connection in a select * from pg_stat_activity; query in the database (state = idle).
I see the bad connection and the new one.
As the result there is a resources leak.
What is the correct way to handle a read: connection reset by peer error? Why have I got this error?
Assuming that no statements to close the connection are made before my script ends and no exception is encountered before closing the connection, does the database's connection stay open?
I'm connecting to the database programmatically via Python Psycopg2 and via Java JDBC4 driver.
Not entirely sure what you want exactly, but let's try:
You can see the connections that exist at any time with PGAdmin or this SQL command
SELECT * FROM pg_stat_activity;
It should be fairly simple to spot when - for your specific use case - the connection closes.
If an SQL query is running at the time you close a connection, I think it will run to completion, ie the backend serving it will remain alive, even if the connection is closed from the client side.