How to avoid DROP DEFAULT statements with Doctrine 2 Migrations diff on first run? - postgresql

I had an existing PostgreSQL database with a table created like this:
CREATE TABLE product (id SERIAL PRIMARY KEY, name VARCHAR(100) DEFAULT NULL)
This table is described in a YML Doctrine2 file within a Symfony2 project:
Acme\DemoBundle\Entity\Product:
type: entity
table: product
fields:
id:
id: true
type: integer
nullable: false
generator:
strategy: SEQUENCE
name:
type: string
length: 100
nullable: true
When I run for the first time the Doctrine Migrations diff task, I should get a versioning file with no data in the up and down methods. But what I get instead is this :
// ...
class Version20120807125808 extends AbstractMigration
{
public function up(Schema $schema)
{
// this up() migration is autogenerated, please modify it to your needs
$this->abortIf($this->connection->getDatabasePlatform()->getName() != "postgresql");
$this->addSql("ALTER TABLE product ALTER id DROP DEFAULT");
}
public function down(Schema $schema)
{
// this down() migration is autogenerated, please modify it to your needs
$this->abortIf($this->connection->getDatabasePlatform()->getName() != "postgresql");
$this->addSql("CREATE SEQUENCE product_id_seq");
$this->addSql("SELECT setval('product_id_seq', (SELECT MAX(id) FROM product))");
$this->addSql("ALTER TABLE product ALTER id SET DEFAULT nextval('product_id_seq')");
}
}
Why are differences detected? How can I avoid this? I tried several sequence strategies with no success.

A little update on this question.
Using Doctrine 2.4, the solution is to use the IDENTITY generator strategy :
Acme\DemoBundle\Entity\Product:
type: entity
table: product
id:
type: integer
generator:
strategy: IDENTITY
fields:
name:
type: string
length: 100
nullable: true
To avoid DROP DEFAULT on fields that have a default value in the database, the default option on the field is the way to go. Of course this can be done with lifecycle callbacks, but it's necessary to keep the default value in the database if this database is used by other apps.
For a "DEFAULT NOW()" like default value, the solution is the following one:
Acme\DemoBundle\Entity\Product:
type: entity
table: product
id:
type: integer
generator:
strategy: IDENTITY
fields:
creation_date:
type: datetime
nullable: false
options:
default: CURRENT_TIMESTAMP

Doctrine 2.0 does not support the SQL DEFAULT keyword, and will always try to drop a postgres default value.
I have found no solution to this problem, I just let doctrine handle the sequences itself.

This is a opened bug registered here :
http://www.doctrine-project.org/jira/browse/DBAL-903

Related

typeorm relation in embedded column

I'm working with NestJs, Typeorm and Postgresql.
I'm trying to use ManyToOne relation in embedded entity. I need to load foreign key column in node environment, so added one additional column(createdById column below). This makes problem.
Here is my code.
A.ts
#Entity()
export class A {
#PrimaryGeneratedColumn()
id!: number;
#Column(() => Embed, { prefix: false })
embed!: Embed;
#CreateDateColumn({ name: 'created_at' })
createdAt!: Date;
}
Embed.ts
export class Embed {
#Column()
x!: number;
#Column()
y!: number;
#ManyToOne(() => B)
#JoinColumn({ name: 'created_by_id' })
createdBy?: B;
#Column({ name: 'created_by_id' })
createdById!: number;
}
B.ts
#Entity()
export class B {
#PrimaryGeneratedColumn()
id!: number;
#CreateDateColumn({ name: 'created_at' })
createdAt!: Date;
}
When I run the app with option TYPEORM_SYNCHRONIZE=true and TYPEORM_LOGGING=true, I get error messages like query failed: CREATE TABLE "a" ("id" SERIAL NOT NULL, "created_at" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(), "created_by_id" integer NOT NULL, "created_by_id" integer NOT NULL, "x" integer NOT NULL, "y" integer NOT NULL, CONSTRAINT "PK_684f21444e543375e4c2e6f27fe" PRIMARY KEY ("id")), Message: column \"created_by_id\" specified more than once.. Typeorm trying to create created_by_id column twice. (I applied custom NamingStrategy so that column of embedded entity's name to be snake_case)
If I place createdBy and createdById column to A directly, then it makes no error. Is it problem of typeorm version? Or any other solutions?
package version:
"dependencies": {
"#nestjs/typeorm": "7.1.0,
"typeorm": "0.2.31",
}
run with docker container,
node image: 16.14.2-alpine3.15,
postgres image: mdillon/postgis:11-alpine
TypeORM docs say you don't need to decorate embed class with #Entity().
See Name class in https://orkhan.gitbook.io/typeorm/docs/embedded-entities

Is it possible to work with different schemas in Fluent with PostgreSQL?

I need to create tables with different number of fields for different users. I want to create a schema for each user, and in it a custom set of tables. I can do this with direct database queries. This will create a categories table in the schema with the given name.:
func create(in schema: String, on db: Database) async throws {
let query: SQLQueryString = """
CREATE TABLE IF NOT EXISTS \(raw: schema).categories (
id uuid PRIMARY KEY,
name text NOT NULL,
...
);
"""
if let sql = db as? SQLDatabase {
try await sql.raw(query).run()
}
}
But I would like to do this with Fluent. (the my_schema schema and the necessary rights to it were previously created in the database) But the following code creates the "my_schema.categories" table in the public schema:
func create(in schema: String, on db: Database) async throws {
try await db.schema("\(schema).categories")
.id()
.field("name", .string, .required)
...
.create()
}
Is it possible to work with different schemas in Fluent and how to query tables from different schemas? I will be grateful for any ideas.
Yes, this was recently introduced to Fluent. Update to the latest version and then you can add a new static property on your model:
final class MyModel: Model {
static let schema = "table_name"
static let space = "schema_name"
// ...
}
See the docs for more details.
Hopefully I am understanding your exact request correctly. If so, using multiple schemas is pretty simple.
(1) First in your configuration define your schema connections, making sure to specify DatabaseID and defaults:
app.databases.use(.postgres(connection_info), as: .psql, isDefault: true)
app.databases.use(.postgres(connection_info), as: .otherDB, isDefault: false)
(2) Run your create table migrations specifying the database schema to add the table to:
app.migrations.add(MigrateCategories())
app.migrations.add(MigrateSomeOtherTable(), to: .otherDB)
(3) Query tables from different database schemas:
categories.create(on: req.db())
someOtherTable.create(on: req.db(.otherDB))
Now you've got two databases
.psql is default and has one table called categories
.otherDB has one table called someOtherTable
Use the DatabaseID to indicate where you want to create new tables and where you want to run queries.

Sequelize upsert throws unique violation for schema with nullable unique column

I'm facing an unique constraint violation issue when doing an upsert, because the UPDATE query built by sequelize ignores the partial index constraint defined by the model (unless it doesn't matter). I'm new to node+sequelize so I might be missing something obvious, but I went through all the potential places for finding the appropriate answers, inclusive of the sequelize code, but I'm not able to find the answer I'm looking for. Really appreciate your help!
My current versions:
"pg": "7.9.0",
"sequelize": "5.21.3"
I have a model that consists of a primary key: id and two other unique indexes of which one of them is a nullable field.
module.exports.Entities = sequelize.define('entities', {
id: {type: Sequelize.UUID, defaultValue: Sequelize.UUIDV4, allowNull: false, primaryKey: true},
cId: {type: Sequelize.STRING, allowNull: false},
pId: {type: Sequelize.UUID, allowNull: false},
eKey: {type: Sequelize.INTEGER, allowNull: true}
}, {
indexes: [
{
name: 'unique_c_id_p_id',
fields: ['c_id', 'p_id'],
unique: true
},
{
name: 'unique_e_key',
fields: ['e_key'],
unique: true,
where: {
eKey: {
[Op.not]: null
}
}
}
]
})
and the table itself looks like below:
CREATE TABLE public.entities (
id UUID DEFAULT uuid_generate_v4 (),
c_id UUID NOT NULL,
p_id UUID NOT NULL,
e_key INTEGER DEFAULT NULL,
CONSTRAINT ENTITY_SERVICE_PKEY PRIMARY KEY (id),
CONSTRAINT unique_c_id_p_id UNIQUE (c_id, p_id)
);
CREATE UNIQUE INDEX unique_e_key ON public.entities (e_key) WHERE e_key IS NOT NULL;
The upsert method call looks like:
module.exports.upsert = async (Model, values) => Model.upsert(values, {returning: true})
I pass the above Entities model, and the below value as arguments to this function.
{
"id"="3169d4e2-8e2d-451e-8be0-40c0b28e2aa9",
"c_id"="00000000-0000-0000-0000-000000000000",
"p_id"="78bce392-4a15-4a8a-986b-c9398787345f",
"e_key"= null
}
Issue: SequelizeUniqueConstraintError
Sequelize tries to do an insert followed by an update query when we attempt to update an existing record using the upsert method.
The insert query shows a conflict, since the record exists already, and sequelize upsert call proceeds on to invoke the update query.
However, the query that it builds to UPDATE looks something like below:
"SQL statement UPDATE entities SET id='3169d4e2-8e2d-451e-8be0-40c0b28e2aa9',c_id='00000000-0000-0000-0000-000000000000',p_id='78bce392-4a15-4a8a-986b-c9398787345f',e_key=NULL
WHERE (id = '3169d4e2-8e2d-451e-8be0-40c0b28e2aa9'
OR e_key IS NULL
OR (c_id = '00000000-0000-0000-0000-000000000000' AND p_id = '78bce392-4a15-4a8a-986b-c9398787345f'))
RETURNING id\nPL/pgSQL function pg_temp_5.sequelize_upsert() line 1 at SQL statement"
Now, I do understand the reason why it's throwing the unique constraint violation, since in the above query's WHERE clause sequelize calls OR e_key IS NULL since e_key = null and that could potentially return more than 1 record, and the SET is trying to update the same value for all those records that were returned thereby violating the primaryKey constraints, unique constraints etc.
What I would like to understand is that:
Why does sequelize not exclude the e_key unique constraint based on the partial index defined given that it picks the WHERE clause attributes based on the constraints defined in the Model & it's indexes?
Is there anything that I could do to get past this issue?
Or, am I missing something obvious that I could fix and try?
Really appreciate you taking your time to read and respond. Thanks!

Entity Framework change Key Type

I have created a model with various values but stupidly used a GUID for my key, I am currently attempting to change that to an Int but am getting an error when I do so.
I have run the enable migration command:
Enable-Migrations -Force -ContextTypeName project.Models.MyContext
This creates the migration I would expect but when I run:
Update-Database -Force
The error I'm getting is:
Operand type clash: uniqueidentifier is incompatible with int
I don't care about the data currently contained within the database since it is just using a SQL Server Express database just now, but I would prefer to find a way to migrate this instead of just having to drop the DB altogether, what's the best way to do this?
I have already got
Database.SetInitializer(new DropCreateDatabaseIfModelChanges<MyContext>());
in Global.asax.
I would expect that the generated migration is using AlterColumn to try and change the type of the field from guid to int. This is not possible, so you'll need to modify the generated migration yourself:
Assuming your table is dbo.People and the key is called Id, you probably have this at the moment:
DropPrimaryKey("dbo.People");
AlterColumn("dbo.People", "Id", c => c.Int(nullable: false, identity: true));
AddPrimaryKey("dbo.People", "Id");
Change it to:
DropPrimaryKey("dbo.People");
DropColumn("dbo.People", "Id");
AddColumn("dbo.People", "Id", c => c.Int(nullable: false, identity: true));
AddPrimaryKey("dbo.People", "Id");
Note that if you've got this key referenced elsewhere, this technique will not work if you've got any data present, as the keys are all being regenerated.
Update for EF Core generated migrations:
dotnet : System.Data.SqlClient.SqlException: Operand type clash: int is incompatible with uniqueidentifier
change
migrationBuilder.AlterColumn<Guid>(
name: "VATID",
schema: "catalogue",
table: "Products",
nullable: false,
oldClrType: typeof(int));
into
migrationBuilder.DropColumn(
name: "VATID",
schema: "catalogue",
table: "Products");
migrationBuilder.AddColumn<Guid>(
name: "VATID",
schema: "catalogue",
table: "Products",
nullable: false);
Of course, this will destroy your data for the certain column. But they obviously cannot be converted into GUID.
I am trying to add something to the selected answer since I don't have enough reputation to add comment there :
Please also add code to drop and recreate indexes if any else it will fail to drop column. e.g.
DropPrimaryKey("dbo.People");
DropIndex("IX_...") // If exists before
DropColumn("dbo.People", "Id");
AddColumn("dbo.People", "Id", c => c.Int(nullable: false,
identity: true));
AddPrimaryKey("dbo.People", "Id");
CreateIndex("IX_...", unique:(true/false)) // If existed before
The AlterColumn will Drop and Add keys (primary,foreign etc) but will not touch indexes. I have faced this.

Sails.js and pluralized relational database table names

How do I configure Sails.js / Waterline to default to pluralized relational database table names that correspond to singular models (same as Rails)?
(E.g. A model called 'Person' should default to a PostgreSQL table called 'people'.)
Just add the tableName: 'people' property to the model:
// Person.js
module.exports = {
tableName: 'people',
attributes: {
id: 'integer',
name: 'string'
}
};
There does not appear to be a global setting in Sails.js that pluralizes database table names automatically for models with singular names.
You can put this in your blueprints.js or local.js file:
blueprints: { // if in your local.js wrap in this object
pluralize: true
}
It won't get it right every time, so the tableName property is still useful for odd cases, but for most pluralized terms it will work.