I'm using SQLAlchemy inside a Lambda with RDS Proxy (PostgreSQL).
This configuration works, but when I'm invoking about 100 lambdas at the same time some lambdas receive a timeout (3 seconds) even when I'm increasing the timeout for 30 seconds I still get a timeout.
After investigating where this timeout comes from, I concluded that the SQLAlchemy hangs when it tries to get a connection to the database.
I thought the RDS proxy could easily handle 100+ concurrent connections.
Here is my engine configuration:
db_url = f"postgresql://{db_username}:{secret}#{db_hostname}:{db_port}/{db_name}"
engine = create_engine(
db_url,
echo=True,
echo_pool="debug",
poolclass=NullPool)
Usually, when SQLAlchemy creates a connection it logs the following:
DEBUG sqlalchemy.pool.impl.NullPool Created new connection <connection object at 0x7f074ebe5580; dsn: 'user=user password=xxx dbname=dbname host=rds-default.{rds-proxy-id}.{region}.rds.amazonaws.com port=port', closed: 0>
But sometimes it will just hang and won't log anything and eventually timeout.
Any help would be appreciated
Related
I have several clients (FreeRadius servers) that connect to a single central Pgbouncer.
When I utilise one of these FreeRadius servers I can see that several database connections are created by Pgbouncer.
select *
from pg_stat_activity
where datname = 'master_db';
When I utilise the same freeradius server again, pgbouncer isn't re-using the existing open connections but keeps creating new ones. Once I reach over 30 connections the whole database comes to an halt.
PGbouncer.ini
server_idle_timeout = 10
max_client_conn = 500
default_pool_size = 30
postgresql.conf: (Postgres 13)
max_connections = 150
Based on my research Pgbouncer is supposed to allocate a single database connection to a client (from the default_pool_size) and then create as many internal connections the client needs (up to max_client_conn).
But what I observe here is the opposite. What am I missing, please?
UPDATE:
The solution Laurenz suggested works but throws this error, when using asyncpg behind the scenes:
NOTE: pgbouncer with pool_mode set to "transaction" or "statement" does not support prepared statements properly. You have two options: * if you are using pgbouncer for connection pooling to a single server, switch to the connection pool functionality provided by asyncpg, it is a much better option for this purpose; * if you have no option of avoiding the use of pgbouncer, then you can set statement_cache_size to 0 when creating the asyncpg connection object.
You will need to use the session option for POOL_MODE so that it is able to maintain connection sessions opened by asyncpg because of the asynchronous nature
You should the following in your
if using pgbouncer.ini file
[pgbouncer]
pool_mode = session
...
...
...
or if using env variable
POOL_MODE=session
extra source: https://magicstack.github.io/asyncpg/current/faq.html#why-am-i-getting-prepared-statement-errors
If you are getting intermittent prepared statement
"asyncpg_stmt_xx" does not exist or prepared statement
“asyncpg_stmt_xx” already exists errors, you are most likely not
connecting to the PostgreSQL server directly, but via pgbouncer.
pgbouncer, when in the "transaction" or "statement" pooling mode, does
not support prepared statements. You have several options:
if you are using pgbouncer only to reduce the cost of new connections
(as opposed to using pgbouncer for connection pooling from a large
number of clients in the interest of better scalability), switch to
the connection pool functionality provided by asyncpg, it is a much
better option for this purpose;
disable automatic use of prepared statements by passing
statement_cache_size=0 to asyncpg.connect() and asyncpg.create_pool()
(and, obviously, avoid the use of Connection.prepare());
switch pgbouncer’s pool_mode to session.
I am using AWS DocumentDB as my database in my application which is developed in node.js which is MongoDB compatible. I have used MongoClient to connect to my database. My application executes but it makes approx 1000 DB connections for each execution and then it fails with an error:
MongoServerSelectionError: connection <monitor> to xxx.xx.xx.xxx:27017 closed.
I tried to use client.close() as well to close the connections then it gave that the connection pool is closed. I believe that issue is due to so many database connections.
How can I release the connections in MongoDB? Any help is appreciated.
It appears from your question that what is happening is that you are defining a new MongoClient object each time you invoke your function resulting in the driver creating a new database connection with each function call. Instead, define the client connection outside the AWS Lambda handler function and reuse the connection with each function call.
Flask with SQLAlchemy
Flask==0.10.1
SQLAlchemy==1.0.8
In production after a lot of usages (connection) we are getting this error. After that restarting the server helps, what will be the permanent solution
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\
Similarly, we are creating a scoped session, we are doing session.close()
I tried Null Pool which doesn't help
Any ideas on that?
Relevant finding
Is it odd that my SQLAlchemy MySQL connection always ends up sleeping?
Handle mysql restart in SQLAlchemy
We need to handle it in SQLAlchemy Dealing Disconnect
https://docs.sqlalchemy.org/en/latest/core/pooling.html#dealing-with-disconnects
After a period of inactivity, my go web service is getting a net.OpError with message read tcp x.x.x.x:52086->x.x.x.x:24414: read: connection reset by peer when executing the first postgres sql query. After the error, the subsequent requests will work fine.
The postgres database is hosted with compose.com which has haproxy in front of the postgres db. My go web app is using standard sql and sqlx.
I've tried running a ticker invoking db.Ping() every 15 minutes, but this hasn't fixed the issue.
Why is the go standard sql lib not handling these connection drops?
Since no one wrote that explicity. The solution to this problem is setting db.SetConnMaxLifetime(time.Minute). I tried it and it works. Connection reset occurs often on AWS where there is inactivity limit set to 350 seconds, after that TCP RST is returned.
I am facing the below issue
ERROR: canceling statement due to user request
inconsistently after I enabled query timeout for xa datasource in my xxx-ds.xml. I have added the following in my ds xml file.
<query-timeout>180</query-timeout>
The query timeout is set for 180 seconds, which means any sql query
that takes more than 180 seconds will get cancelled from application
server side.
But the issue I am facing is inconsistent and the queries gets timed out now and then without taking exactly 180 seconds.
We are using connection pooling also.
While searching stackoverflow found this question, which discusses about the possible causes for this issue while using connection pooling.
The solution suggested there was to set statement_timeout setting in postgresql.conf file. But it is bit difficult for me to enable statement_timeout setting in my database environment as the DB server is shared by multiple applications.
I would like to have a solution to terminate timed out queries from client side effectively and consistently while using connection pooling. I am using
JBoss 4.2.2-GA
postgresql 9.2 (64 bit)
java 1.7
postgresql-9.2-1002.jdbc4.jar
It looks like the issue is with postgresql driver 9.2. When I upgraded to 9.3, issue is fixed.