Default value doesn't work in SQLAlchemy + PostgreSQL + aiopg + psycopg2 - postgresql

I've found an unexpected behavior in SQLAlchemy. I'm using the following versions:
SQLAlchemy (0.9.8)
PostgreSQL (9.3.5)
psycopg2 (2.5.4)
aiopg (0.5.1)
This is the table definition for the example:
import asyncio
from aiopg.sa import create_engine
from sqlalchemy import (
MetaData,
Column,
Integer,
Table,
String,
)
metadata = MetaData()
users = Table('users', metadata,
Column('id_user', Integer, primary_key=True, nullable=False),
Column('name', String(20), unique=True),
Column('age', Integer, nullable=False, default=0),
)
Now if I try to execute a simple insert to the table just populating the id_user and name, the column age should be auto-generated right? Lets see...
#asyncio.coroutine
def go():
engine = yield from create_engine('postgresql://USER#localhost/DB')
data = {'id_user':1, 'name':'Jimmy' }
stmt = users.insert(values=data, inline=False)
with (yield from engine) as conn:
result = yield from conn.execute(stmt)
loop = asyncio.get_event_loop()
loop.run_until_complete(go())
This is the resulting statement with the corresponding error:
INSERT INTO users (id_user, name, age) VALUES (1, 'Jimmy', null);
psycopg2.IntegrityError: null value in column "age" violates not-null constraint
I didn't provide the age column, so where is that age = null value coming from? I was expecting something like this:
INSERT INTO users (id_user, name) VALUES (1, 'Jimmy');
Or if the default flag actually works should be:
INSERT INTO users (id_user, name, Age) VALUES (1, 'Jimmy', 0);
Could you put some light on this?

This issue has been confirmed has an aiopg bug. Seems like at the moment it's ignoring the default argument on data manipulation.
I've fixed the issue using server_default instead:
users = Table('users', metadata,
Column('id_user', Integer, primary_key=True, nullable=False),
Column('name', String(20), unique=True),
Column('age', Integer, nullable=False, server_default='0'))

I think you need to use inline=True in your insert. This turns off 'pre-execution'.
Docs are a bit cryptic on what exactly this 'pre-execution' entails, but they mentions default parameters:
:param inline:
if True, SQL defaults present on :class:`.Column` objects via
the ``default`` keyword will be compiled 'inline' into the statement
and not pre-executed. This means that their values will not
be available in the dictionary returned from
:meth:`.ResultProxy.last_updated_params`.
This piece of docstring is from Update class, but they have a shared behavior with Insert.
Besides, that's the only way they test it:
https://github.com/zzzeek/sqlalchemy/blob/rel_0_9/test/sql/test_insert.py#L385

Related

Insert UUID to PostgreSQL table using PDO

I am using UUID's in PostgreSQL as Primary Key for my tables. I am trying to insert the UUID using PDO and prepared statement. However, I am not able to bind the value using bindValue. These are the steps I am trying to follow:
$sql = "INSERT INTO customers.customers(customer_id, first_name, last_name, email_address)";
$sql .= " VALUES(":customer_id", ":first_name", ":last_name", ":email_address")";
$this->stmt = prepare($sql);
$this->stmt = bindValue(":customer_id", $customer_uuid); //*** WHAT PDO value to use Here??????
$this->stmt = bindValue(":first_name", $first_name, PDO::PARAM_STR);
$this->stmt = bindValue(":last_name", $last_name, PDO::PARAM_STR);
$this->stmt = bindValue(":email_address", "$email_address, PDO::PARAM_STR);
$this->stmt->execute();
As can be seen here, each "bindValue" statement required to have a PDO parameter type value. I have not been able to find what value can be used for a UUID type column. I tried using the PDO::PARAM_STR, but this creates a Data Type error when inserting the UUID int he column as the column type for customer_id is UUID.
Any suggestions from the community here?
I tried converting the UUID to string and then back to UUID, but it did not work. From here, I don't know what else I can try.

Use UUID in Doobie SQL update

I have the following simple (cut down for brevity) Postgres table:
create table users(
id uuid NOT NULL,
year_of_birth smallint NOT NULL
);
Within a test I have seeded data.
When I run the following SQL update to correct a year_of_birth the error implies that I'm not providing the necessary UUID correctly.
The Doobie SQL I run is:
val id: String = "6ee7a37c-6f58-4c14-a66c-c17083adff81"
val correctYear: Int = 1980
sql"update users set year_of_birth = $correctYear where id = $id".update.run
I have tried both with and without quotes around the given $id e.g. the other version is:
sql"update users set year_of_birth = $correctYear where id = '$id'".update.run
The error upon running the above is:
org.postgresql.util.PSQLException: ERROR: operator does not exist: uuid = character varying
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Both comments provided viable solutions.
a_horse_with_no_name suggested the use of cast which works though the SQL becomes no so nice when compared to the other solution.
AminMal suggested the use of available Doobie implicits which can handle a UUID within SQL and thus avoid a cast.
So I changed my code to the following:
import doobie.postgres.implicits._
val id: UUID = UUID.fromString("6ee7a37c-6f58-4c14-a66c-c17083adff81")
sql"update users set year_of_birth = $correctYear where id = $id".update.run
So I'd like to mark this question as resolved because of the comment provided by AminMal

Race condition when manually setting the id in sqlalchemy

I want that all my object have a unique id that is set by PostgreSQL with a (serial) and another id that depends to the first one.
When creating an object if I set the second id after saving it, I'll have an INSERT and an UPDATE on the table, what is not really the best.
So to have only one INSERT I fetch the id from the PostgreSQL sequence and set the id with it instead of letting PostgreSQL do it at INSERT stage.
I'm pretty new on SQLAlchemy and want to be sure that this way of doing is race condition proof.
Thanks for you thoughts on this idea
class MyModel:
def __init__(self, session, **data):
"""
Base constructor for almost all model classes, performing common tasks
"""
cls = type(self)
if session:
"""To avoid having an UPDATE right after the INSERT we manually fetch
the next available id using a postgresl internal
SELECT nextval(pg_get_serial_sequence('events', 'id'));
To do that we need the table's name and the sequence
column's name, by chance we use the same name in all our
model
"""
table_name = cls.__tablename__
qry = f"SELECT nextval(pg_get_serial_sequence('{table_name}', 'id'))"
rs = session.execute(qry)
# TODO : find a non ugly way to to that
for row in rs:
next_id = row[0]
# manually set the object id
self.id = next_id
# set the external_id before saving the object in the database
self.ex_id = cls.ex_id_prefix + self.id
session.add(self)
session.flush([self])
If you are targetting Postgresql 12 or later, you can use a generated column. SQLAlchemy's Computed column type will create such a column, and we can pass an SQL expression to compute the value.
The model would look like this:
class MyModel(Base):
__tablename__ = 't68225046'
ex_id_prefix = 'prefix_'
id = sa.Column(sa.Integer, primary_key=True)
ex_id = sa.Column(sa.String,
sa.Computed(sa.text(":p || id::varchar").bindparams(p=ex_id_prefix)))
producing this DDL
CREATE TABLE t68225046 (
id SERIAL NOT NULL,
ex_id VARCHAR GENERATED ALWAYS AS ('prefix_' || id::varchar) STORED,
PRIMARY KEY (id)
)
and a single insert statement
2021-09-19 ... INFO sqlalchemy.engine.Engine BEGIN (implicit)
2021-09-19 ... INFO sqlalchemy.engine.Engine INSERT INTO t68225046 DEFAULT VALUES RETURNING t68225046.id
2021-09-19 ... INFO sqlalchemy.engine.Engine [generated in 0.00014s] {}
2021-09-19 ... INFO sqlalchemy.engine.Engine COMMIT
For earlier releases of Postgresql, or if you don't need to store the value in the database, you could simulate it with a hybrid property.
import sqlalchemy as sa
from sqlalchemy import orm
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.sql import cast
Base = orm.declarative_base()
class MyModel(Base):
__tablename__ = 't68225046'
ex_id_prefix = 'prefix_'
id = sa.Column(sa.Integer, primary_key=True)
#hybrid_property
def ex_id(self):
return self.ex_id_prefix + str(self.id)
#ex_id.expression
def ex_id(cls):
# See https://stackoverflow.com/a/54487891/5320906
return cls.ex_id_prefix + cast(cls.id, sa.String)

Spring JPA globally_quoted_identifiers incorrectly quoting column type TEXT

I'm using the property globally_quoted_identifiers to deal with issues of reserved keywords being used for column names in an application I'm maintaining. However I've just encountered a bug, where a create table statement is being generated like so...
create table `MyTable` (`id` bigint not null auto_increment, `body` `TEXT`...
The create statement fails in MySQL, because TEXT should not be quoted.
I'm not sure why this is happening. It doesn't do it to other column types, bigint, varchar etc.
Is there something else I need to do, to have JPA correctly handle the MySQL TEXT column type?
Update: This is the data class which demonstrates the issue...
#Entity
data class MyTable(
#Id
#GeneratedValue(strategy= GenerationType.IDENTITY)
val id: Long? = null,
#Column(columnDefinition = "TEXT")
var body: String? = null
)
This will result in the above table create SQL above, when globally_quoted_identifiers is enabled.
Can you try with below config,
hibernate.globally_quoted_identifiers = true
hibernate.globally_quoted_identifiers_skip_column_definitions = true

How do I prevent sql alchemy from inserting the None value to field?

The Alembic migration script :
def upgrade():
uuid_gen = saexp.text("UUID GENERATE V1MC()")
op.create_table(
'foo',
sa.Column('uuid', UUID, primary_key=True, server_default=uuid_gen),
sa.Column(
'inserted',
sa.DateTime(timezone=True),
server_default=sa.text("not null now()"))
sa.Column('data', sa.Text)
)
This is my Base class for SQL Alchemy:
Class Foo(Base):
__tablename__ = 'foo'
inserted = Column(TIMESTAMP)
uuid = Column(UUID, primary_key=True)
data = Column(TEXT)
It has a static mehtod for insert :
#staticmethod
def insert(session, jsondata):
foo = Foo()
foo.data = jsondata['data']
if 'inserted' in jsondata:
foo.inserted = jsondata['inserted']
if 'uuid' in jsondata:
foo.uuid = jsondata['uuid']
session.add(foo)
return foo
the purpose of the 2 if's are to simplify testing. this way i can "inject" a uuid and inserted date, to get predictible data for my tests
When trying to insert data
foo = Foo()
foo.insert(session, {"data": "foo bar baz"})
session.commit()
I get an IntegrityError :
[SQL: 'INSERT INTO foo (inserted, data) VALUES (%(inserted)s, %(data)s) RETURNING foo.uuid'] [parameters: {'data': 'foo bar baz', 'inserted': None}]
wich seem normal to me because the insert violates the "not-null" constraint in the postgres database.
How do I prevent sql alchemy from inserting the None value to the inserted field ?
While playing and testing around, I found that if the "inserted" column is defined as primary key , sql alchemy does not include the field in the insert statement.
def upgrade():
uuid_gen = saexp.text("UUID GENERATE V1MC()")
op.create_table(
'foo',
sa.Column('uuid', UUID, primary_key=True, server_default=uuid_gen),
sa.Column(
'inserted',
primary_key=True,
sa.DateTime(timezone=True),
server_default=sa.text("not null now()"))
sa.Column('data', sa.Text)
)
But this is not what I want.
The primary problem is the server_default which is missing in the inserted member in class Foo. It's only present in the alembic script. Note that the alembic definitions are only used when running the migrations. They do not affect the application. For this reason, it's a good idea to copy the exact same definitions from the alembic script to your application (or vice-versa).
Because no value is defined in the model definition, sqlalchemy seems to set this to None when the class is instantiated. This will then be sent to the DB which will complain. To fix this, either set default or server_default on the model definition (the class inheriting from Base).
Some additional notes/questions:
Where does UUID GENERATE V1MC() come from? The official docs look different. I replaced it with func.uuid_generate_v1mc().
The server_default value in your case contains not null which is incorrect. You should set nullable=False on you column attribute (see below).
alembic script
# revision identifiers, used by Alembic.
revision = THIS_IS_DIFFERENT_ON_EACH_INSTANCE! # '1b7e145f2138'
down_revision = None
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID
def upgrade():
op.create_table(
'foo',
sa.Column('uuid', UUID, primary_key=True,
server_default=sa.func.uuid_generate_v1mc()),
sa.Column(
'inserted',
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()")),
sa.Column('data', sa.Text)
)
def downgrade():
op.drop_table('foo')
tester.py
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, create_engine, func
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.dialects.postgresql import (
TEXT,
TIMESTAMP,
UUID,
)
engine = create_engine('postgresql://michel#/michel')
Session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = declarative_base()
class Foo(Base):
__tablename__ = 'foo'
inserted = Column(TIMESTAMP, nullable=False,
server_default=func.now())
uuid = Column(UUID, primary_key=True,
server_default=func.uuid_generate_v1mc()),
data = Column(TEXT)
#staticmethod
def insert(session, jsondata):
foo = Foo()
foo.data = jsondata['data']
if 'inserted' in jsondata:
foo.inserted = jsondata['inserted']
if 'uuid' in jsondata:
foo.uuid = jsondata['uuid']
session.add(foo)
return foo
if __name__ == '__main__':
session = Session()
Foo.insert(session, {"data": "foo bar baz"})
session.commit()
session.close()
output after execution
[9:43:54] michel#BBS-nexus [1 background job(s)]
/home/users/michel/tmp› psql -c "select * from foo"
uuid | inserted | data
--------------------------------------+-------------------------------+-------------
71f5fd32-0602-11e6-aebb-27be4bbac26e | 2016-04-19 09:43:45.297191+02 | foo bar baz
(1 row)