I'm facing an unique constraint violation issue when doing an upsert, because the UPDATE query built by sequelize ignores the partial index constraint defined by the model (unless it doesn't matter). I'm new to node+sequelize so I might be missing something obvious, but I went through all the potential places for finding the appropriate answers, inclusive of the sequelize code, but I'm not able to find the answer I'm looking for. Really appreciate your help!
My current versions:
"pg": "7.9.0",
"sequelize": "5.21.3"
I have a model that consists of a primary key: id and two other unique indexes of which one of them is a nullable field.
module.exports.Entities = sequelize.define('entities', {
id: {type: Sequelize.UUID, defaultValue: Sequelize.UUIDV4, allowNull: false, primaryKey: true},
cId: {type: Sequelize.STRING, allowNull: false},
pId: {type: Sequelize.UUID, allowNull: false},
eKey: {type: Sequelize.INTEGER, allowNull: true}
}, {
indexes: [
{
name: 'unique_c_id_p_id',
fields: ['c_id', 'p_id'],
unique: true
},
{
name: 'unique_e_key',
fields: ['e_key'],
unique: true,
where: {
eKey: {
[Op.not]: null
}
}
}
]
})
and the table itself looks like below:
CREATE TABLE public.entities (
id UUID DEFAULT uuid_generate_v4 (),
c_id UUID NOT NULL,
p_id UUID NOT NULL,
e_key INTEGER DEFAULT NULL,
CONSTRAINT ENTITY_SERVICE_PKEY PRIMARY KEY (id),
CONSTRAINT unique_c_id_p_id UNIQUE (c_id, p_id)
);
CREATE UNIQUE INDEX unique_e_key ON public.entities (e_key) WHERE e_key IS NOT NULL;
The upsert method call looks like:
module.exports.upsert = async (Model, values) => Model.upsert(values, {returning: true})
I pass the above Entities model, and the below value as arguments to this function.
{
"id"="3169d4e2-8e2d-451e-8be0-40c0b28e2aa9",
"c_id"="00000000-0000-0000-0000-000000000000",
"p_id"="78bce392-4a15-4a8a-986b-c9398787345f",
"e_key"= null
}
Issue: SequelizeUniqueConstraintError
Sequelize tries to do an insert followed by an update query when we attempt to update an existing record using the upsert method.
The insert query shows a conflict, since the record exists already, and sequelize upsert call proceeds on to invoke the update query.
However, the query that it builds to UPDATE looks something like below:
"SQL statement UPDATE entities SET id='3169d4e2-8e2d-451e-8be0-40c0b28e2aa9',c_id='00000000-0000-0000-0000-000000000000',p_id='78bce392-4a15-4a8a-986b-c9398787345f',e_key=NULL
WHERE (id = '3169d4e2-8e2d-451e-8be0-40c0b28e2aa9'
OR e_key IS NULL
OR (c_id = '00000000-0000-0000-0000-000000000000' AND p_id = '78bce392-4a15-4a8a-986b-c9398787345f'))
RETURNING id\nPL/pgSQL function pg_temp_5.sequelize_upsert() line 1 at SQL statement"
Now, I do understand the reason why it's throwing the unique constraint violation, since in the above query's WHERE clause sequelize calls OR e_key IS NULL since e_key = null and that could potentially return more than 1 record, and the SET is trying to update the same value for all those records that were returned thereby violating the primaryKey constraints, unique constraints etc.
What I would like to understand is that:
Why does sequelize not exclude the e_key unique constraint based on the partial index defined given that it picks the WHERE clause attributes based on the constraints defined in the Model & it's indexes?
Is there anything that I could do to get past this issue?
Or, am I missing something obvious that I could fix and try?
Really appreciate you taking your time to read and respond. Thanks!
Related
I am using knex.js with postgresql
So I have a table with a nullable foreign key.
shortened version:
exports.up = function (knex, Promise) {
return knex.schema.createTable('Note', function (table) {
table.string('id').primary()
table
.string('sourceId')
.references('id')
.inTable('Source')
.onDelete('SET NULL')
.index()
})
}
exports.down = function (knex, Promise) {
return knex.schema.dropTable('Note')
}
I am able to create a Note with or without a sourceId. However, if I create a Note with a sourceId and then update it to set the sourceId to NULL, the update does not work. I do not get an error message, but the foreign key is not removed.
For example if I create a Note with:
{
id: '123',
sourceId: '456'
}
and then try to update it:
const result = await Note.query().updateAndFetchById(id, {
id: '123',
sourceId: null
})
The result I get is :
Note {
id: '123',
sourceId: '456'
}
I have no problem if I try to update other nullable values to null (as long as they are not foreign keys) and I can update the sourceId to a different source's id.
If I try to update a not nullable foreign key to null, I get an error. But in the above case, I get no error. It just doesn't update.
Any idea what might be going on here?
I have a js object with explicit id property. Like this:
const data = {
user_id: null,
user_email: faker.internet.email()
};
The value of user_id is null and there is a users table using user_id as its primary key.
I want to insert this data correctly and I hope knex can obey the primary key increment rule.
Here is my code:
async function insert(user: any) {
return await knex('users')
.insert(user)
.returning('*');
}
When I try to insert this data, got an error:
error: null value in column "user_id" violates not-null constraint
How can I solve this?
You can set the value of user_id to undefined. Then it will insert the data and
obey the primary key increment rule.
const data = {
user_id: undefined,
user_email: faker.internet.email()
};
Check the inserted row in users table. The value of user_id is 1
I had a table.
I added a new column.
Even though I had set default value in sequelize model, those columns still ended up empty.
So I get error Unhandled rejection SequelizeDatabaseError: column "col_name" contains null values
How do you populate new column with default values upon creation so not null constraint is not broken.
You can update the values first:
update t
set col_name = ?
where col_name is null;
Then add the not null constraint.
Even though I had set default value in sequelize model
I suspect there is a discrepancy between the migration and model. To insert the column with a migration containing a default value use defaultValue in your migration.
The following is a working example:
module.exports = {
up: (queryInterface, Sequelize) => {
return queryInterface.addColumn('tests', 'new_column', {
defaultValue: 'test',
type: Sequelize.STRING
})
}
}
Before running sequelize db:migrate
After sequelize db:migrate:
The documentation for the options object in addColumn is hard to find, it's listed for a different method
I managed to create a model/schema and insert geo-Points into Postgis using sequelize. Since I have a lot of points (>100K) I am longing back to the old way I used to import, using ogr2ogr (gdal), which was much faster (almost instant instead of >20 minutes). As I would like to continue to work with sequelize after the initial import I still want sequelize to create the model/schema upfront, but then do import with ogr2ogr, and then continue CRUD with sequelize.
Here I found this fragment “[….] One way to get around this is to create your table structures before hand and use OGR2OGR to just load the data.” Which gave me the idea that this could work for Postgres/Postgis as well.
Sequelize creates timestamp columns for createdAt and updatedAt, which I like. But when I use ogr2ogr I get “null value in column "createdAt" violates not-null constraint” as loginfo.
Based on this slightly similar issue I tried to add a createdAt column by adding an -sql option:
ogr2ogr -f PostgreSQL PG:"host='0.0.0.0' user='root' port='5432' dbname='db' password='pw'" /home/user/geojsonImportfile.json -nln "DataPoints" -a_srs EPSG:4326 -sql "SELECT url, customfield, wkb_geometry, CURRENT_TIMESTAMP as createdAt FROM '/home/usr/geojsonImportfile.json'" -dialect 'PostgreSQL'
The error I get when running this is:
ERROR 1: SQL Expression Parsing Error: syntax error, unexpected end of string, expecting '.'. Occurred around :
home/user/geojsonImportfile0.json'
Besides my lack of SQL knowledge I am not sure if this can work at all.
How can I solve this, i.e. make the import with ogr2ogr but keep the timestamp columns?
When you create a table with sequelize.define, createdAt and updatedAt columns are created automatically as timestamp with time zone NOT NULL.
But you can rule not-null constraint in your sequelize definition script:
const Mytable = sequelize.define('mytable', {
id: {type: Sequelize.INTEGER, primaryKey: true},
createdAt: {type: Sequelize.DATE, validate: {notNull:false}}
});
Then table is created like :
CREATE TABLE mytables
(
id integer NOT NULL,
"createdAt" timestamp with time zone,
"updatedAt" timestamp with time zone NOT NULL,
CONSTRAINT mytables_pkey PRIMARY KEY (id)
)
http://docs.sequelizejs.com/manual/tutorial/models-definition.html#configuration
#JGH following your suggestion it makes sense to have a default timestamp. I can already set this up using sequelize, as discussed here:
var user = sequelize.define('user', {
createdAt: {
type: DataTypes.DATE,
field: 'beginTime',
defaultValue: sequelize.literal('NOW()')
}
}, {
timestamps: true,
});
I am trying to add a many to many relationship through an explicitly created junction table using Sequelize and Postgresql.
The tables on either side of the relationship are associated like this:
Shop.belongsToMany(models.user, {through: 'visits' })
User.belongsToMany(models.shop, {through: 'visits' })
And the visits junction table primary key is defined like this:
id: {
type: DataTypes.INTEGER,
primaryKey: true,
autoIncrement: true // Automatically gets converted to SERIAL for postgres
}
When I try and insert into visits I get the following error:
ERROR: duplicate key value violates unique constraint "visits_shopId_userId_key"
DETAIL: Key ("shopId", "userId")=(1, 12) already exists.
After doing a pg_dump, I have tried to remove the composite key constraint by adding constraint: false to the models, but I still get the error.
(I have dropped the tables and resynced several times during the debugging process)
After digging around the Sequelize issues, it turns out that removing the constraint on the N:M composite key is an easy fix.
The through key can take an object with the unique: false property:
Shop.belongsToMany(models.user, {
through: {
model: 'visits',
unique: false
},
constraints: false
});