I'm following the sqlalchemy documentation for reflecting database tables using automap: http://docs.sqlalchemy.org/en/latest/orm/extensions/automap.html#generating-mappings-from-an-existing-metadata.
When I don't specific a schema, and Postgres uses the default public schema, this works as expected, and I find the names of my tables:
>>> m = MetaData()
>>> b = automap_base(bind=engine, metadata=m)
>>> b.prepare(engine, reflect=True)
>>> b.classes.keys()
['ads', 'spatial_ref_sys', 'income']
But when I specific an explicit schema, I don't have access to the tables in Base.classes anymore.
>>> m = MetaData(schema='geography')
>>> b = automap_base(bind=engine, metadata=m)
>>> b.prepare(engine, reflect=True)
>>> b.classes.keys()
[]
The MetaData reflected correctly though:
>>> b.metadata.tables
immutabledict({geography.usa_cbsa_centroids': Table('usa_cbsa_centroids', MetaData(bind=Engine(postgresql://asteroids:***#localhost:5432/asteroids)), Column('GEOID', VARCHAR(length=5), table=<u
sa_cbsa_centroids>, nullable=False), ...})
Note that the tables and columns are only known at runtime.
The answer is that database tables in SQLAlchemy require a primary key, and my table didn't have one. There is additional information on this page: http://docs.sqlalchemy.org/en/latest/faq/ormconfiguration.html#how-do-i-map-a-table-that-has-no-primary-key.
The SQLAlchemy ORM, in order to map to a particular table, needs there
to be at least one column denoted as a primary key column;
multiple-column, i.e. composite, primary keys are of course entirely
feasible as well. These columns do not need to be actually known to
the database as primary key columns, though it’s a good idea that they
are. It’s only necessary that the columns behave as a primary key
does, e.g. as a unique and not nullable identifier for a row.
Thanks to Michael Bayer for answering this on the sqlalchemy mailing list: https://groups.google.com/forum/#!topic/sqlalchemy/8F2tPkpR4bE
Related
I would like to obtain from database B, by querying its database tables, the original schemas and table names of its foreign tables (i.e. those in the original database A where they actually reside, as their name can be changed when created with the available fdw).
Any hint greatly appreciated!
Currently I am only able to find columns names and datatypes of the foreign tables (in database B) ...
I've got PostgreSQL DB with multiple schemas and tables in that schemas. Every row in table have PRIMARY UUID like "Ref_Key" => "41bf3b1e-91f0-491c-a6bd-c48a17e7c252"
Is it possible to find row only by it UUID, without specifying schema and table?
No, that is not possible. You can only query tables that explicitly appear in the FROM clause of a SELECT statement.
I just started using timescaleDB with postgresql. I have a database named storage_db which contains a table named day_ahead_prices.
After installing timescaledb, I was following Migrate from the same postgresql database to migrate my storage_db into a timescaledb.
When I did (indexes included):
CREATE TABLE tsdb_day_ahead_prices (LIKE day_ahead_prices INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES);
select create_hypertable('tsdb_day_ahead_prices', 'date_time');
It gave me the following error:
ERROR: cannot create a unique index without the column "date_time" (used in partitioning)
But when I did (indexed excluded):
CREATE TABLE tsdb_day_ahead_prices (LIKE day_ahead_prices INCLUDING DEFAULTS INCLUDING CONSTRAINTS EXCLUDING INDEXES);
select create_hypertable('tsdb_day_ahead_prices', 'date_time');
It was successful. Following which, I did
select create_hypertable('tsdb_day_ahead_prices', 'date_time');
and it gave me the following output:
create_hypertable
------------------------------------
(3,public,tsdb_day_ahead_prices,t)
(1 row)
I am a bit new to this so can anyone please explain to me what is the difference between both of them and why was I getting an error in the first case?
P.S.:
My day_ahead_prices looks as follows:
id | country_code | values | date_time
----+--------------+---------+----------------------------
1 | LU | 100.503 | 2020-04-11 14:04:30.461605
2 | LU | 100.503 | 2020-04-11 14:18:39.600574
3 | DE | 106.68 | 2020-04-11 15:59:10.223965
Edit 1:
I created the day_ahead_prices table in python using flask and flask_sqlalchemy and the code is:
class day_ahead_prices(db.Model):
__tablename__ = "day_ahead_prices"
id = db.Column(db.Integer, primary_key=True)
country_code = db.Column(avail_cc_enum, nullable=False)
values = db.Column(db.Float(precision=2), nullable=False)
date_time = db.Column(db.DateTime, default=datetime.now(tz=tz), nullable=False)
def __init__(self, country_code, values):
self.country_code = country_code
self.values = values
When executing CREATE TABLE tsdb_day_ahead_prices (LIKE day_ahead_prices INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES); you're telling the database to create the tsdb_day_ahead_prices table using the day_ahead_prices as a template (same columns, same types for those columns), but you're also telling it to include the default values, constraints and indexes that you have defined on the original table, and apply/create the same for your new table.
Then you are executing the timescaledb command that makes the tsdb_day_ahead_prices table
a hypertable. A hypertable is an abstraction that hides away the partitioning of the physical
table. (https://www.timescale.com/products/how-it-works). You are telling
TimescaleDB to make the tsdb_day_ahead_prices a hypertable using the date_time column as a partitioning key.
When creating hypertables, one constraing that TimescaleDB imposes is that the partitioning column (in your case 'date_time') must be included in any unique indexes (and Primary Keys) for that table. (https://docs.timescale.com/latest/using-timescaledb/schema-management#indexing-best-practices)
The first error you get cannot create a unique index without the column "date_time" is exactly because of this. You copied the primary key definition on the id column. So the primary key is preventing
the table to be a hypertable.
The second time, you created the tsdb_day_ahead_prices table but you didn't copy
the indexes from the original table, so the primary key is not defined (which is really a unique index). So the creation of the hypertable was successfull.
The output you get from the create_hypertable function tells you that you have a new hypertable, in the public schema, the name of the hypertable, and the internal id that timescaledb uses for it.
So now you can use the tsdb_day_ahead_prices as normal, and timescaledb underneath will make sure the data goes into the proper partitions/chunks
Does the id need to be unique for this table?
If you're going to be keeping time-series data
then each row may not really be unique for each id, but may be uniquely identified by the id at a given time.
You can create a separate table for the items that you're identifying
items(id PRIMARY KEY, country_code) and have the hypertable be
day_ahead_prices(time, value, item_id REFERENCES items(id))
Since Firebird 3, I can't modify a column type.
Before I use this kind of update:
update RDB$RELATION_FIELDS set
RDB$FIELD_SOURCE = 'MYTEXT'
where (RDB$FIELD_NAME = 'JXML') and
(RDB$RELATION_NAME = 'XMLTABLE')
because I get ISC error 335545030 ("UPDATE operation is not allowed for system table RDB$RELATION_FIELDS").
Maybe there is another way in Firebird 3?
Firebird 3 no longer allows direct updates to the system tables, as that was a way to potentially corrupt a database. See also System Tables are Now Read-only in the release notes. You will need to use DDL statements to do the modification.
It looks like you want to change the data type of a column to a domain. You will need to use alter table ... alter column ... for that. Specifically you will need to do:
alter table XMLTABLE
alter column JXML type MYTEXT;
This does come with some restrictions:
Changing the Data Type of a Column: the TYPE Keyword
The keyword TYPE changes the data type of an existing column to
another, allowable type. A type change that might result in data loss
will be disallowed. As an example, the number of characters in the new
type for a CHAR or VARCHAR column cannot be smaller than the existing
specification for it.
If the column was declared as an array, no change to its type or its
number of dimensions is permitted.
The data type of a column that is involved in a foreign key, primary
key or unique constraint cannot be changed at all.
This statement has been available since before Firebird 1 (InterBase 6.0).
Firebird 2.5 manual, chapter Data Definition (DDL) Statement, section TABLE:
ALTER TABLE tabname ALTER COLUMN colname TYPE typename
I have some T-SQL (SQL Server 2008) that I inherited and am trying to find out why some of queries are running really slow. In the Actual Execution Plan I have three clustered index scans which are costing me 19%, 21% and 26%, so this seems to be the source of my problem.
The contents of the fields are usually numeric (but some job numbers have an alpha prefix)
The database design (vendor supplied) is pretty poor. The max length of a job number in their application is 12 chars, but in the tables that are joined it is defined as varchar(50) in some places and varchar(15) in others. My parameter is a varchar(12), but I get same thing if I change it to a varchar(50)
The node contains this:
Predicate: [Live_Costing].[dbo].[TSTrans].[JobNo] as [sts1].[JobNo]=CONVERT_IMPLICIT(varchar(50),[#JobNo],0)
sts1 is a derived table, but the table it pulls jobno from is a varchar(50)
I don't understand why it's doing an implicit conversion between 2 varchars. Is it just because they are different lengths?
I'm fairly new to the execution plan
Is there an easy way to figure out which node in the exec plan relates to which part of the query?
Is the predicate, the join clause?
Regards
Mark
Some variables can have collation: enter link description here
Regardless you need to verify your collations, which can be specified at server, DB, table, and column level.
First, check your collation between tempdb and the vendor supplied database. It should match. If it doesn't, it will tend to do implicit conversions.
Assuming you cannot modify the vendor supplied code base, one or more of the following should help you:
1) Predefine your temp tables and specify the same collation for the key field as in the db in use, rather than tempdb.
2) Provide collations when doing string comparisons.
3) Specify collation for key values if using "select into" with a temp table
4) Make sure your collations on your tables and columns match your database collation (VERY important if you imported only specific tables from a vendor into an existing database.)
If you can change the vendor supplied code base, I would suggest reviewing the cost for making all of your char keys the same length and NOT varchar. Varchar has an overhead of 10. The caveat is that if you create a fixed length character field not null, it will be padded to the right (unavoidable).
Ideally, you would have int keys, and only use varchar fields for user interaction/lookup:
create table Products(ProductID int not null identity(1,1) primary key clustered, ProductNumber varchar(50) not null)
alter table Products add constraint uckProducts_ProductNumber unique(ProductNumber)
Then do all joins on ProductID, rather than ProductNumber. Just filter on ProductNumber.
would be perfectly fine.