Flask SQLAlchemy: application crashed on postgres server reboot - postgresql

I have a standard setup of flask_sqlalchemy and an AWS RDS with postgresql.
I initialize the DB adapter the standard way:
from flask_sqlalchemy import SQLAlchemy
from flask import Flask
app = Flask(__name__)
db = SQLAlchemy()
db.init_app(app)
class Example(db.Model):
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
username = db.Column(db.String(256))
If I reboot the RDS server and a SELECT starts during reboot I get:
psycopg2.OperationalError: terminating connection due to administrator
command SSL connection has been closed unexpectedly
The above exception was the direct cause of the following exception:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError)
terminating connection due to administrator command SSL connection has
been closed unexpectedly
That I understand. The DB is unavailable.
BUT: After that the application cannot recover from the situation because:
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid
transaction is rolled back
The above exception was the direct cause of the following exception: sqlalchemy.exc.StatementError:
(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid
transaction is rolled back
I understand the application throws exceptions during reboot.
But what puzzles me that it never recovers.
There is no write transaction to the DB! I don't understand what rollback the exception is describing.
Edit:
This is how all commits are implemented:
try:
db.session.commit()
except Exception as e:
db.session.rollback()
But there is no commit triggering the exception. Only SELECTS.

Do this and it should fix it for you session.rollback().
If you want to find this error, wrap it into a try/except-block and perform a session.rollback().
EDIT:
So I test some of the stuff and I believe the reason for the error lies within the try/except block. Ideally the doc says this is how you should be implementing inserts:
Create the Python object
Add it to the session
Commit the session
e.g.
>>> db.session.add(User(name="Flask", email="example#example.com"))
>>> db.session.commit()
Likewise for deletes:
>>> db.session.delete(me)
>>> db.session.commit()
For querying you do not need to commit exclusively, the query attribute in Flask-SQLAlchemy does it for you on the Model class.
so for selecting you just do:
users = User.query.all()
Which in your case is:
something = Example.query.all()
One more thing, can you please clarify or give more info on why you are implementing commits like this:
try:
db.session.commit()
except Exception as e:
db.session.rollback()
PS: I might have misread your last edit, my bad!

Related

FastAPI + Postgres + SQLAlchemy - Fatal: remaining connection slots are reserved

I am using FastAPI as a full stack with Jinja2 template.
The main problem is SQLalchemy and postgres
Here's example of main page
async def read_posts(request: Request, page: int = 1, page_size: int = 12, db: Session = Depends(get_db)):
posts = db.query(models.Post).offset(start).limit(end).all()
return templates.TemplateResponse("index.html", {"request": request, "posts": posts})
I just have a blog with a lot of posts, and speed of loading page is very slow, I think I somehow wrongly build queries to the database, but can't find what I did wrong, it is very simple app.
But the main problem is that website is not able to withstand a load, there's statistics from one of the services to check the load
LOAD STATS
here's the logs of error when there is load
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (::1), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections
i find out that is connection leak, but i can't find the source of the problem. I spent 2 days to find the problem and got nothing
I found out the answer
in FastApi you connect to database like this
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
SQLALCHEMY_DATABASE_URL = "sqlite:///./my.db"
engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
# Dependency
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
Most important part is get_db(), so you call this command wherever you access the database and think when your query is ended it will finally close, but it won't happen.
When you return posts to template, as in the example above, there is still connection to database and because of it there's connection overflow.
It won't happen if you use JsonResponse for example, but with TemplateResponse database will still connected and working

Kotlin error connecting to Postgres : org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail

I'm trying to connect my mobile app to the Postgresql server(using PgAdmin) but it's not working and give this error. I've tried using a lot of form of url in my code but it won't do.
val url = "jdbc:postgresql://localhost/postgres"
val user = ""
val password = "password"
Class.forName("org.postgresql.Driver")
val db: Connection = DriverManager.getConnection(url, user, password)
db.close();
I think my url is wrong. How do I fix that ?
PgAdmin DB :
Error :
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.codemobiles.project_eva/com.codemobiles.project_eva.FeedActivity}: org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
Caused by: org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
THANKS!!!
Sol : I quit using this jdbc and use the api from server instead.
I got the same error Driver version 42.2.23, when I try with ItelliJ everything's is fine!

pymongo - MongoClient retryWrites=false is not working

I'm currently working on a simple python CRUD script to check MongoDB out. Turns out I'm liking it a lot, but I have found myself being unable to work with MongoDB transactions. Everytime I try to start a transaction an exception is thrown saying:
This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.
And, eventhough I've already added that option to my connection string:
self._client = MongoClient('mongodb://localhost/?retryWrites=false')
self._db = self._client.workouts
self._collection = self._db.workouts
That error is still popping up when running the following lines of code:
with self._client.start_session() as s:
with s.start_transaction():
self._collection.delete_one({'_id': id}, session=s)
next = self._collection.find_one({'_id': next_id}, session=s)
return next
What can I do?
I'm running python 3.7.3, pymongo 3.9.0 and MongoDB 4.0.12.

postgresql pg_database_size throwing exception on random times

We are using Azure database for PostgreSQL ( Service ) for creating DB for each user when user register to the application ( less than 25 users databases right now ).
For reporting purpose we need information which each user's DB size.
To retrieve database size we have a Postgres function which fires the following query
SELECT pg_database.datname , pg_database_size(pg_database.datname) FROM
pg_database
We execute this function every hour throw azure function but at random time Postgres throw exceptions
Exception: Npgsql.PostgresException (0x80004005): 58P01: could not read directory "base/16452": No such file or directory at...
Exception remain same at most of the time with different directory or file location
Sometimes it also throws the exception
Exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException
Working on the solution at the MSDN forums here.

Flask-sqlalchemy losing connection after restarting of DB server

I use flask-sqlalchemy in my application. DB is postgresql 9.3.
I have simple init of db, model and view:
from config import *
from flask import Flask, request, render_template
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://%s:%s#%s/%s' % (DB_USER, DB_PASSWORD, HOST, DB_NAME)
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
login = db.Column(db.String(255), unique=True, index=True, nullable=False)
db.create_all()
db.session.commit()
#app.route('/users/')
def users():
users = User.query.all()
return '1'
And all works fine. But when happens DB server restarting (sudo service postgresql restart), on first request to the /users/ I obtain sqlalchemy.exc.OperationalError:
OperationalError: (psycopg2.OperationalError) terminating connection due to administrator command
SSL connection has been closed unexpectedly
[SQL: ....
Is there any way to renew connection inside view, or setup flask-sqlalchemy in another way for renew connection automatically?
UPDATE.
I ended up with using clear SQLAlchemy, declaring engine, metadata and db_session for every view, where I critically need it.
It is not solution of question, just a 'hack'.
So question is open. I am sure, It will be nice to find solution for this :)
The SQLAlchemy documentation explains that the default behaviour is to handle disconnects optimistically. Did you try another request - the connection should have re-established itself ? I've just tested this with a Flask/Postgres/Windows project and it works.
In a typical web application using an ORM Session, the above condition would correspond to a single request failing with a 500 error, then the web application continuing normally beyond that. Hence the approach is “optimistic” in that frequent database restarts are not anticipated.
If you want the connection state to be checked prior to a connection attempt you need to write code that handles disconnects pessimistically. The following example code is provided at the documentation:
from sqlalchemy import exc
from sqlalchemy import event
from sqlalchemy.pool import Pool
#event.listens_for(Pool, "checkout")
def ping_connection(dbapi_connection, connection_record, connection_proxy):
cursor = dbapi_connection.cursor()
try:
cursor.execute("SELECT 1")
except:
# optional - dispose the whole pool
# instead of invalidating one at a time
# connection_proxy._pool.dispose()
# raise DisconnectionError - pool will try
# connecting again up to three times before raising.
raise exc.DisconnectionError()
cursor.close()
Here's some screenshots of the event being caught in PyCharm's debugger:
Windows 7 (Postgres 9.4, Flask 0.10.1, SQLAlchemy 1.0.11, Flask-SQLAlchemy 2.1 and psycopg 2.6.1)
On first db request
After db restart
Ubuntu 14.04 (Postgres 9.4, Flask 0.10.1, SQLAlchemy 1.0.8, Flask-SQLAlchemy 2.0 and psycopg 2.5.5)
On first db request
After db restart
In plain SQLAlchemy you can add the pool_pre_ping=True kwarg when calling the create_engine function to fix this issue.
When using Flask-SQLAlchemy you can use the same argument, but you need to pass it as a dict in the engine_options kwarg:
app.db = SQLAlchemy(app, engine_options={"pool_pre_ping": True})