I am migrating data from MSSQL.
I created the database in PostgreSQL via npgsql generated migration. I moved the data across and now when the code tries to insert a value I am getting
'duplicate key value violates unique constraint'
The npgsql tries to insert a column with Id 1..how ever the table already has Id over a thousand.
Npgsql.EntityFrameworkCore.PostgreSQL is 2.2.3 (latest)
In my context builder, I have
modelBuilder.ForNpgsqlUseIdentityColumns();
In which direction should I dig to resolve such an issue?
The code runs fine if the database is empty and doesn't have any imported data
Thank you
The values inserted during the migration contained the primary key value, so the sequence behind the column wasn't incremented and is kept at 1. A normal insert - without specifying the PK value - calls the sequence, get the 1, which already exists in the table.
To fix it, you can bump the sequence to the current max value.
SELECT setval(
pg_get_serial_sequence('myschema.mytable','mycolumn'),
max(mycolumn))
FROM myschema.mytable;
If you already know the sequence name, you can shorten it to
SELECT setval('my_sequence_name', max(mycolumn))
FROM myschema.mytable;
Related
[Issue resolved. See Answer below.]
I have just encountered a series of “duplicate key value violates unique constraint” errors with a system that has been working well for months. And I cannot determine why they occurred.
Here is the error:
org.springframework.dao.DuplicateKeyException: PreparedStatementCallback;
SQL [
INSERT INTO transaction_item
(transaction_group_id, transaction_type, start_time, end_time) VALUES
(?, ?::transaction_type_enum, ?, ?)
];
ERROR: duplicate key value violates unique constraint "transaction_item_pkey"
Detail: Key (transaction_id)=(67109) already exists.;
Here is the definition of the relevant SEQUENCE and TABLE:
CREATE SEQUENCE transaction_id_seq AS bigint;
CREATE TABLE transaction_item (
transaction_id bigint PRIMARY KEY DEFAULT NEXTVAL('transaction_id_seq'),
transaction_group_id bigint NOT NULL,
transaction_type transaction_type_enum NOT NULL,
start_time timestamp NOT NULL,
end_time timestamp NOT NULL
);
And here is the only SQL statement used for inserting to that table:
INSERT INTO transaction_item
(transaction_group_id, transaction_type, start_time, end_time) VALUES
(:transaction_group_id, :transaction_type::transaction_type_enum, :start_time, :end_time)
As you can see, I’m not explicitly trying to set the value of transaction_id. I've defined a default value for the column definition, and using that to fetch a value formthe SEQUENCE.
I have been under the impression that the above approach is safe, even for use in high-concurrency situation. A SEQUENCE should never return the same value twice, right?
I’d really appreciate some help to understand why this has occurred, and how to fix it. Thank you!
I found the cause of this issue.
A few months ago (during development of this system) an issue was discovered that meant it was necessary to purge any existing test data from the database. I did this using DELETE FROM statements for all TABLES and ALTER ... RESTART statements for all SEQUENCES. These statements were added to the Liquibase configuration to be executing during startup for the new code. From inspecting the logs at the time, it appears that an instance of the system was still running at the time of the migration. And this happened: The new instance of the system deleted all data from the TRANSACTION_ITEM table, the still-running instance then added more data to that table, and then the new instance restarted the SEQUENCE use for inserting those records. So yesterday, when I received the duplicate key violations, it was because the SEQUENCE finally reached the ID values corresponding to the TRANSACTION_ITEM records that were added by still-running instance back when DB purge and migration occurred.
Long story, but it all makes sense now. Thanks to those who commented on this issue.
I have a table 'client', which has 3 columns - id, siebel_id, phone_number.
PhoneNumber has a unique constraint. If I save a new client with an existing number, I'll get an error ERROR: duplicate key value violates unique constraint "phone_number_unique".
Is it possible to make PSQL or MyBatis showing 'siebel_id' of a record where the phone number already saved?
I mean to get a message like
'ERROR: duplicate key value violates unique constraint "phone_number_unique"
Detail: Key (phone_number)=(+79991234567) already exists on siebel_id...'
No, it's not possible to tweak the internal message that the PostgreSQL database engine returns accompannying an error. Well... unless you recompiled the whole PostgreSQL database from scratch, and I would assume this is off the table.
However, you can easily search for the offending row using SQL, as in:
select siebel_id from client where phone_number = '+79991234567';
I have migrated an AnyDAC app to FireDAC and I can't get to work its Autoinc fields.
The ID field (primary key) has been defined on Postgre SQL as default to nextval('llistapanelspuzzle_id_seq'::regclass), BIGSERIAL, so the server automatically sets its values.
The column was recognized by AnyDAC as an TAutoincField and worked correctly, but when I now open that table on FireDAC it fails saying that the field found is a TLargeIntField. I change the persistent field to a TLargeIntField, but now when inserting records on Delphi, I don't get the new values from the server, it leaves the dataset with a 0 value, and when I add a second record it raises a Key Violation (two records with a 0 value on its primary key).
Do you know how to define AutoInc fields on FireDAC - PostgreSQL, when they are being recognized as LargeInt fields ?.
Update: I have added ID to the UpdateOptions.AutoIncFields, but it doesn't seem to have changed anything.
Thank you.
Looks like you have to activate the ExtendedMetada flag on the FDConnection in order for FireDAC to recognize automatically the PostgreSQL Autoinc columns.
Now it works correctly.
I am using postgresql and sqlalchemy for my flask project.
I recently partitioned one of my big tables based on created_on using postgresql triggers.
But now if a try to insert a record into master table with db.session.add(obj) in sqlalchemy, i am getting error saying
Instance has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such as within a load() event.
Here I am using a sequence to increment my primary key. Please help me with this.
use autoincrement=True while defining your column example in my code sno is an autoincrement field :
class Contact(db.Model):
sno = db.Column(db.Integer, primary_key=True,autoincrement=True)
I am getting a duplicate key error, DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, when I try to INSERT records. The primary key is one column, INTEGER 4, Generated, and it is the first column.
the insert looks like this: INSERT INTO SCHEMA.TABLE1 values (DEFAULT, ?, ?, ...)
It's my understanding that using the value DEFAULT will just let DB2 auto-generate the key at the time of insert, which is what I want. This works most of the time, but sometimes/randomly I get the duplicate key error. Thoughts?
More specifically, I'm running against DB2 9.7.0.3, using Scriptella to copy a bunch of records from one database to another. Sometimes I can process a bunch with no problems, other times I'll get the error right away, other times after 2 records, or 20 records, or 30 records, etc. Does not seem to be a pattern, nor is it the same record every time. If I change the data to copy 1 record instead of a bunch, sometimes I'll get the error one time, then it's fine the next time.
I thought maybe some other process was inserting records during my batch program, and creating keys at the same time. However, the tables I'm copying TO should not have any other users/processes trying to INSERT records during this same time frame, although there could be READS happening.
Edit: adding create info:
Create table SCHEMA.TABLE1 (
SYSTEM_USER_KEY INTEGER NOT NULL
generated by default as identity (start with 1 increment by 1 cache 20),
COL2...,
)
alter table SCHEMA.TABLE1
add constraint SYSTEM_USER_SYSTEM_USER_KEY_IDX
Primary Key (SYSTEM_USER_KEY);
You most likely have records in your table with IDs that are bigger then the next value in your identity sequence. To find out what the current value your sequence is about at, run the following query.
select s.nextcachefirstvalue-s.cache, s.nextcachefirstvalue-s.increment
from syscat.COLIDENTATTRIBUTES as a inner join syscat.sequences as s on a.seqid=s.seqid
where a.tabschema='SCHEMA'
and a.TABNAME='TABLE1'
and a.COLNAME='SYSTEM_USER_KEY'
So basically what happened is that somehow you got records in your table with ids that are bigger then the current last value of your identity sequence. So sooner or later these ids will collide with identity generated ids.
There are different reasons on how this could have happened. One possibility is that data was loaded which already contained values for the id column or that records were inserted with an actual value for the ID. Another option is that the identity sequence was reset to start at a lower value than the max id in the table.
Whatever the cause, you may also want the fix:
SELECT MAX(<primary_key_column>) FROM onsite.forms;
ALTER TABLE <table> ALTER COLUMN <primary_key_column> RESTART WITH <number from previous query + 1>;