how to use composite data types (e.g. geomval) in SQLAlchemy? - postgresql

I'm trying to replicate a nested raw PostGreSQL/PostGIS raster query using SQLAlchemy(0.8)/GeoAlchemy2(0.2.1) and can't figure how to access the components of a geomval data type. It's a compound data type that returns a 'geom' and a 'val'.
Here is the raw query that works:
SELECT (dap).val, (dap).geom
FROM (SELECT ST_DumpAsPolygons(rast) as dap FROM my_raster_table) thing
And the SQLAlchemy query I'm currently working with:
import geoalchemy2 as ga2
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
metadata = MetaData()
my_raster_table = Table('my_raster_table', metadata,
Column('rid', Integer),
Column('rast', ga2.Raster))
engine = create_engine(my_conn_str)
session = sessionmaker(engine)()
q = session.query(ga2.func.ST_DumpAsPolygons(my_raster_table.c.rast).label('dap'))
And then I'd like to access that in a subquery, something like this:
q2 = session.query(ga2.func.ST_Area(q.subquery().c.dap.geom))
But that syntax doesn't work, or I wouldn't be posting this question ;). Anyone have ideas? Thanks!

The solution ended up being fairly simple:
First, define a custom GeomvalType, inheriting geoalchemy2's CompositeType and specifying a typemap specific to geomval:
class GeomvalType(ga2.types.CompositeType):
typemap = {'geom':ga2.Geometry('MULTIPOLYGON'),'val':Float}
Next, use type_coerce to cast the ST_DumpAsPolygons result to the GeomvalType in the initial query:
q = session.query(type_coerce(ga2.func.ST_DumpAsPolygons(my_raster_table.c.rast), GeomvalType()).label('dap'))
Finally, access it (successfully!) from the subquery as I was trying to before:
q2 = session.query(ga2.func.ST_Area(q.subquery().c.dap.geom))

Related

[Flask][SQLAlchemy] Implementing a database column for a Multiselect: How to use ArrayOfEnum with Enum python classes in flask-sqlalchemy?

I did a bunch of research but I did not get through a sophisticated example helping me out to use ArrayOfEnum() to store a class-specific Enum in a PostgreSQL database. I do use Flask, Flask-SQLAlchemy, as well SQLAlchemy.
My example originates by a WTForm using a MultiSelect which I want to store in the database:
genres = SelectMultipleField(
'genres', validators=[DataRequired()], choices=[
('Alternative', 'Alternative'),
('Blues', 'Blues'),
...
])
I would like to define an enum in python like:
class Genres(enum.Enum):
alternative ='Alternative'
blues ='Blues'
classical = 'Classical'
And use it as an authorized data type in the model, like this:
class Artist(db.Model):
__tablename__ = 'Artist'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
city = db.Column(db.String(120))
genres = db.Column(pg.ArrayOfEnum(Genres))
My general setup is: requirements.txt:
alembic==1.7.7
Babel==2.9.0
click==8.1.2
colorama==0.4.4
Flask==2.1.1
Flask-Migrate==3.1.0
Flask-Moment==0.11.0
Flask-SQLAlchemy==2.4.4
Flask-WTF==0.14.3
greenlet==1.1.2
importlib-metadata==4.11.3
itsdangerous==2.1.2
Jinja2==3.0.3
Mako==1.2.0
MarkupSafe==2.1.1
postgres==4.0
psycopg2-binary==2.9.3
psycopg2-pool==1.1
python-dateutil==2.6.0
pytz==2022.1
six==1.16.0
SQLAlchemy==1.4.35
Werkzeug==2.0.0
WTForms==3.0.1
zipp==3.8.0
In this question I do specifically ask for the solution using the ENUM with ARRAY, but I'm also open to use other solutions that might fit.

Using multiple POSTGRES databases and schemas with the same Flask-SQLAlchemy model

I'm going to be very specific here, because similar questions have been asked, but none of the solutions work for this problem.
I'm working on a project that has four postgres databases, but let's say for the sake of simplicity there are 2. Namely, A & B
A,B represent two geographical locations, but the tables and schemas in the database are identical.
Sample model:
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
db = SQLAlchemy()
Base = declarative_base()
class FRARecord(Base):
__tablename__ = 'tb_fra_credentials'
recnr = Column(db.Integer, primary_key = True)
fra_code = Column(db.Integer)
fra_first_name = Column(db.String)
This model is replicated in both databases, but with different schemas, so to make it work in A, I need to do:
__table_args__ = {'schema' : 'A_schema'}
I'd like to use a single content provider that is given the database to access, but has identical methods:
class ContentProvider():
def __init__(self, database):
self.database = database
def get_fra_list():
logging.debug("Fetching fra list")
fra_list = db.session.query(FRARecord.fra_code)
Two problems are, how do I decide what db to point to and how do I not replicate the model code for different schemas (this is a postgres specific problem)
Here's what I've tried so far:
1) I've made separate files for each of the models and inherited them, so:
class FRARecordA(FRARecord):
__table_args__ = {'schema' : 'A_schema'}
This doesn't seem to work, because I get the error:
"Can't place __table_args__ on an inherited class with no table."
Meaning that I can't set that argument after the db.Model (in its parent) was already declared
2) So I tried to do the same with multiple inheritance,
class FRARecord():
recnr = Column(db.Integer, primary_key = True)
fra_code = Column(db.Integer)
fra_first_name = Column(db.String)
class FRARecordA(Base, FRARecord):
__tablename__ = 'tb_fra_credentials'
__table_args__ = {'schema' : 'A_schema'}
but got the predictable error:
"CompileError: Cannot compile Column object until its 'name' is assigned."
Obviously I can't move the Column objects to the FRARecordA model without having to repeat them for B as well (and there are actually 4 databases and a lot more models).
3) Finally, I'm considering doing some sort of sharding (which seems to be the correct approach), but I can't find an example of how I'd go about this. My feeling is that I'd just use a single object like this:
class FRARecord(Base):
__tablename__ = 'tb_fra_credentials'
#declared_attr
def __table_args__(cls):
#something where I go through the values in bind keys like
for key, value in self.db.app.config['SQLALCHEMY_BINDS'].iteritems():
# Return based on current session maybe? And then have different sessions in the content provider?
recnr = Column(db.Integer, primary_key = True)
fra_code = Column(db.Integer)
fra_first_name = Column(db.String)
Just to be clear, my intention for accessing the different databases was as follows:
app.config['SQLALCHEMY_DATABASE_URI']='postgresql://%(user)s:\
%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_A
app.config['SQLALCHEMY_BINDS']={'B':'postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_B,
'C':'postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_C,
'D':'postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_D
}
Where the POSTGRES dictionaries contained all the keys to connect to the data
I assumed with the inherited objects, I'd just connect to the correct one like this (so the sqlalchemy query would automatically know):
class FRARecordB(FRARecord):
__bind_key__ = 'B'
__table_args__ = {'schema' : 'B_schema'}
Finally found a solution to this.
Essentially, I didn't create new classes for each database, I just used different database connections for each.
This method on its own is pretty common, the tricky part (which I couldn't find examples of) was handling schema differences. I ended up doing this:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
Session = sessionmaker()
class ContentProvider():
db = None
connection = None
session = None
def __init__(self, center):
if center == A:
self.db = create_engine('postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_A, echo=echo, pool_threadlocal=True)
self.connection = self.db.connect()
# It's not very clean, but this was the extra step. You could also set specific connection params if you have multiple schemas
self.connection.execute('set search_path=A_schema')
elif center == B:
self.db = create_engine('postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES_B, echo=echo, pool_threadlocal=True)
self.connection = self.db.connect()
self.connection.execute('set search_path=B_schema')
def get_fra_list(self):
logging.debug("Fetching fra list")
fra_list = self.session.query(FRARecord.fra_code)
return fra_list

Sqlalchemy + Postgres: synthetic/artificial id mixin with sequence

I've found the mixin pattern to be really handy for staying DRY, but I am having trouble with sequences. Note, I'm using postgres.
We use alembic migrations, and I'd really like the --autogeneration to work with this sequence, though I understand this might not be possible right now. However, it looks like setting up the sequence without an ORM identifier, prevents the sequence from being dropped later if I wanted to perform a downgrade.
Through googling, I found some explanation on how to properly setup a sequence. Essentially: separate the id and its sequence.
Current Code looks a bit like this:
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declared_attr
class AutoIdMixin(object):
"""Generates an synthetic identifier primary key.
"""
# See: http://docs.sqlalchemy.org/en/latest/core/defaults.html#associating-a-sequence-as-the-server-side-default
#declared_attr
def id_seq(cls):
bases = cls.__bases__
Base = bases[0]
sequence_prefix = 'seq'
schema = cls._schema_name
sequence_id = '_'.join((sequence_prefix, schema, cls.__tablename__, 'id'))
sequence = sa.Sequence(sequence_id, 1, 1, metadata=Base.metadata)
return sequence
#declared_attr
def id(cls):
column_id = sa.Column(sa.types.Integer, cls.id_seq.next_value(), primary_key=True)
return column_id
With the code above, I end up with a non-helpful error:
AttributeError: Neither 'next_value' object nor 'Comparator' object has an attribute '_set_parent_with_dispatch'
In an RTM moment, it looks like I missed a keyword: server_default.
#declared_attr
def id(cls):
sequence = cls.id_seq
column_id = sa.Column(sa.types.Integer, server_default=sequence.next_value(), primary_key=True)
return column_id

Spyne model for existing Database structure

I have an issue with defining model in spyne to generate several levels "in" SOAP11.
I used example at first, but my task is to generate service for tables already existing, so I got stuck and try to understand wheter to seek in Spyne properties or Sqlalchemy.
To be precise, i'll take example from site and show what i'm trying to reach:
class Permission(TableModel):
__tablename__ = 'permission'
id = UnsignedInteger32(pk=True)
application = Unicode(values=('usermgr', 'accountmgr'))
operation = Unicode(values=('read', 'modify', 'delete'))
perm_user_id = integer
last field is the FK for user table, but its name is different from user_id
class User(TableModel):
__tablename__ = 'user'
id = UnsignedInteger32(pk=True)
user_name = Unicode(32, min_len=4, pattern='[a-z0-9.]+', unique=True)
full_name = Unicode(64, pattern='\w+( \w+)+')
email = Unicode(64, pattern=r'[a-z0-9._%+-]+#[a-z0-9.-]+\.[A-Z]{2,4}')
last_pos = Point(2, index='gist')
permissions = Array(Permission).store_as('table')
--- SQL generated tries to add "WHEN user.id = permission.user_id" but I need another field (perm_user_id) to be filtered
Help me to define class to get correct inner tags.. actually it'll be about 3 more classes deep.
Thanx in Advance, Yury
Your answer is correct. Just as an alternative for simple tables, you can omit column definitions and let sqlalchemy's reflection engine figure it out.
meta = TableModel.Attributes.sqla_metadata
meta.reflect()
class User(TableModel):
__table__ = meta.tables['user']
The User class will be reconstructed using as much information as possible from the table columns and their types.
Found it myself, sorry to disturb anyone,
from spyne.model.complex import table
Permissions= Array(permission).customize(store_as=table(right='perm_user_id'))

Scala case class copy only with some parameters at runtime?

I'm using Play Framework and client can send only some fields to update in database. Then I need to do something like this:
g.copy(
partnumber = jGood.partnumber,
cost = jGood.cost
)
So, most of the fields I will have in jGood will be None and only some of them will be Some. Now how can I filter all those None fields and make a copy of class only with Some fields?
Consider this:
g.copy(
partnumber = jGood.partnumber.orElse(g.partnumber),
cost = jGood.cost.orElse(g.cost)
)