Facing issues with sqlalchemy+postgresql session management - postgresql

Question

I am using sqlalchemy with PostgreSQL and the Pyramid web framework. Here is my models.py:
engine = create_engine(db_url, pool_recycle=3600,
isolation_level="READ UNCOMMITTED")
Session = scoped_session(sessionmaker(bind=engine))
session = Session()
Base = declarative_base()
Base.metadata.bind = engine
class _BaseMixin(object):
def save(self):
session.add(self)
session.commit()
def delete(self):
session.delete(self)
session.commit()
I am inheriting both the Base and _BaseMixin for my models. For example:
class MyModel(Base, _BaseMixin):
__tablename__ = 'MY_MODELS'
id = Column(Integer, primary_key=True, autoincrement=True)
The reason is that I would like to do something like
m = MyModel()
m.save()
I am facing weird issues with session all the time. Sample error messages include
InvalidRequestError: This session is in 'prepared' state; no further SQL can be emitted within this transaction.
InvalidRequestError: A transaction is already begun. Use subtransactions=True to allow subtransactions.
All I want to do is to commit what I have in the memory into the DB. But intermittently SQLAlchemy throws errors like described above and fails to commit.
Is there something wrong with my approach?

Answer 1

tl;dr The problems is that you're sharing one Session object between threads. It fails, because Session object is not thread-safe itself.
What happens?
You create a Session object, which is bound to current thread (let's call it Thread-1). Then you closure it inside your _BaseMixin. Incoming requests are handled in the different threads (let's call them Thread-2 and Thread-3). When request is handling, you call model.save(), which uses Session object created in Thread-1 from Thread-2 or Thread-3. Multiple requests can run concurrently, which together with thread-unsafe Session object gives you totally indeterministic behaviour.
How to handle?
When using scoped_session(), each time you create new object with Session() it will be bound to current thread. Furthermore if there is a session bound to current thread it will return you existing session instead of creating new one.
So you can move session = Session() from the module level to your save() and delete() methods. It will ensure, that you are always using session from current thread.
class _BaseMixin(object):
def save(self):
session = Session()
session.add(self)
session.commit()
def delete(self):
session = Session()
session.delete(self)
session.commit()
But it looks like duplication and also doesn't make sense to create Session object (it will always return the same object inside current thread). So SA provides you ability to implicitly access session for current thread. It produces clearer code.
class _BaseMixin(object):
def save(self):
Session.add(self)
Session.commit()
def delete(self):
Session.delete(self)
Session.commit()
Please, also note that for normal applications you never want to create Session objects explicitly. But want to use implicit access to thread-local session by using Session.some_method().
Further reading
Contextual/Thread-local Sessions.
When do I construct a Session, when do I commit it, and when do I close it?.