I am migrating DB from Oracle to PostgreSQL14 ,
Oracle partitioned table was being used which i am trying to migrate to PG.
Below script i am using to create partition with index and tablespace
and getting below error
ERROR: unique constraint on partitioned table must include all partitioning columns
In oracle below index was being created on the table which i am now using at partitioned level as i come to know there is limitation , but is there any alternative as this error will cause change in the logic .
Oracle index
CREATE UNIQUE INDEX COL_IDX_ID
ON TABLE1(COL_ID,ERROR_TIME_DT,ERROR_TIME_TM);
Postgres script
create table TABLE1
(
COL_DT date constraint N_COL_DT not null
,COL_TM numeric(9) constraint N_COL_TM not null
,COL_INPROC_ID numeric(1) constraint N_COL_INPROC_ID not null
,COL_SUB_ID numeric(7) constraint N_COL_SUB_ID not null
,COL_RECORD_DT date constraint N_COL_RECORD_DT not null
,COL_RECORD_TM numeric(9) constraint N_COL_RECORD_TM not null
,COL_TIMEOFFSET varchar(6) constraint N_COL_TIMEOFFSET not null
,COL_TYPE numeric(1) constraint N_COL_TYPE not null
,COL4 numeric(1) DEFAULT 0 not null
,ERROR_TIME_DT DATE
,ERROR_TIME_TM numeric(9)
,ERROR__ID numeric(6)
,COL_ID VARCHAR(250)
,CONSTRAINT PK_COL primary key (COL_DT,COL_TM,COL_INPROC_ID,COL_SUB_ID) using index tablespace ${TABLESPACE_INDEX}
) partition by range (COL_DT);
CREATE TABLE P_TABLE1_1 PARTITION OF TABLE1 FOR VALUES FROM (MINVALUE) TO (to_date('${PARTITION_DATE_LIMIT}', 'YYYYMM')) partition by list (COL_INPROC_ID);
CREATE UNIQUE INDEX COL_COLID
ON P_TABLE1_1 ( ERROR_TIME_DT ,ERROR_TIME_TM,COL_ID);
CREATE TABLE P_TABLE1_1_P0 PARTITION OF P_TABLE1_1 FOR VALUES IN (0) ;
CREATE TABLE P_TABLE1_1_P1 PARTITION OF P_TABLE1_1 FOR VALUES IN (1) ;
CREATE TABLE PMAXVALUE PARTITION OF TABLE1 FOR VALUES FROM (to_date('${PARTITION_DATE_LIMIT}', 'YYYYMM')) TO (MAXVALUE) partition by list (COL_INPROC_ID);
CREATE UNIQUE INDEX COL_ID_PMAX
ON PMAXVALUE ( ERROR_TIME_DT ,ERROR_TIME_TM,COL_ID);
CREATE TABLE PMAXVALUE_P0 PARTITION OF PMAXVALUE FOR VALUES IN (0) ;
CREATE TABLE PMAXVALUE_P1 PARTITION OF PMAXVALUE FOR VALUES IN (1) ;
is there any workaround of this problem where i can keep the index same as in oracle "ERROR: unique constraint on partitioned table must include all partitioning columns"
You can only have a unique index on a partitioned table if the partitioning key is part of the index keys. There is no other solution for this, and there is no good workaround either (see below for a small remedy).
Since an index on a partitioned table is always a partitioned index, it can only be UNIQUE if it is by definition, that is, if the partitioning key is an index key. The index partitions can only guarantee uniqueness within the partition itself.
You will have to live without the uniqueness of that index. You can instead define a unique index on each partition, which is almost as good, but does not enforce uniqueness across partitions.
Related
I am migrating DB from Oracle to PostgreSQL14 ,
Oracle partitioned table was being used which i am trying to migrate to PG.
Below script i am using to create partition with index and tablespace
and getting
ERROR: cannot specify default tablespace for partitioned relations
create table TABLE1
(
COL_DT date constraint N_COL_DT not null
,COL_TM numeric(9) constraint N_COL_TM not null
,COL_INPROC_ID numeric(1) constraint N_COL_INPROC_ID not null
,COL_SUB_ID numeric(7) constraint N_COL_SUB_ID not null
,COL_RECORD_DT date constraint N_COL_RECORD_DT not null
,COL_RECORD_TM numeric(9) constraint N_COL_RECORD_TM not null
,COL_TIMEOFFSET varchar(6) constraint N_COL_TIMEOFFSET not null
,COL_TYPE numeric(1) constraint N_COL_TYPE not null
,COL4 numeric(1) DEFAULT 0 not null
,CONSTRAINT PK_COL primary key (COL_DT,COL_TM,COL_INPROC_ID,COL_SUB_ID) using index tablespace ${TABLESPACE_INDEX}
) partition by range (COL_DT);
CREATE TABLE P_TABLE1_1 PARTITION OF TABLE1 FOR VALUES FROM (MINVALUE) TO (to_date('${PARTITION_DATE_LIMIT}', 'YYYYMM')) partition by list (COL_INPROC_ID);
CREATE TABLE P_TABLE1_1_P0 PARTITION OF P_TABLE1_1 FOR VALUES IN (0) ;
CREATE TABLE P_TABLE1_1_P1 PARTITION OF P_TABLE1_1 FOR VALUES IN (1) ;
CREATE TABLE PMAXVALUE PARTITION OF TABLE1 FOR VALUES FROM (to_date('${PARTITION_DATE_LIMIT}', 'YYYYMM')) TO (MAXVALUE) partition by list (COL_INPROC_ID);
CREATE TABLE PMAXVALUE_P0 PARTITION OF PMAXVALUE FOR VALUES IN (0) ;
CREATE TABLE PMAXVALUE_P1 PARTITION OF PMAXVALUE FOR VALUES IN (1) ;
i am unable to get why this error is coming whereas default tablespace is not used
I ran your statements, and they work like they should, once I substituted constants for the placeholders. However, before v12 it would not have worked the way you want, see the release note entry:
Allow CREATE TABLE's tablespace specification for a partitioned table to affect the tablespace of its children (David Rowley, Álvaro Herrera)
So your problem is not reproducible, but since you are migrating from Oracle, let me give you a piece of advice: don't use tablespaces. They are the bread and butter of Oracle administration, but in PostgreSQL you usually use the default tablespace. There are very few valid reasons to use tablespaces, and particularly in virtualized environments they usually don't make any sense at all. And they make administration (backup etc.) more complicated.
I am recreating an existing table as a partitioned table in PostgreSQL 11.
After some research, I am approaching it using the following procedure so this can be done online while writes are still happening on the table:
add a check constraint on the existing table, first as not valid and then validating
drop the existing primary key
rename the existing table
create the partitioned table under the prior table name
attach the existing table as a partition to the new partitioned table
My expectation was that the last step would be relatively fast, but I don't really have a number for this. In my testing, it's taking about 30s. I wonder if my expectations are incorrect or if I'm doing something wrong with the constraint or anything else.
Here's a simplified version of the DDL.
First, the inserted_at column is declared like this:
inserted_at timestamp without time zone not null
I want to have an index on the ID even after I drop the PK for existing queries and writes, so I create an index:
create unique index concurrently my_events_temp_id_index on my_events (id);
The check constraint is created in one transaction:
alter table my_events add constraint my_events_2022_07_events_check
check (inserted_at >= '2018-01-01' and inserted_at < '2022-08-01')
not valid;
In the next transaction, it's validated (and the validation is successful):
alter table my_events validate constraint my_events_2022_07_events_check;
Then before creating the partitioned table, I drop the primary key of the existing table:
alter table my_events drop constraint my_events_pkey cascade;
Finally, in its own transaction, the partitioned table is created:
alter table my_events rename to my_events_2022_07;
create table my_events (
id uuid not null,
... other columns,
inserted_at timestamp without time zone not null,
primary key (id, inserted_at)
) partition by range (inserted_at);
alter table my_events attach partition my_events_2022_07
for values from ('2018-01-01') to ('2022-08-01');
That last transaction blocks inserts and takes about 30s for the 12M rows in my test database.
Edit
I wanted to add that in response to the attach I see this:
INFO: partition constraint for table "my_events_2022_07" is implied by existing constraints
That makes me think I'm doing this right.
The problem is not the check constraint, it is the primary key.
If you make the original unique index include both columns:
create unique index concurrently my_events_temp_id_index on my_events (id,inserted_at);
And if you make the new table have a unique index rather than a primary key on those two columns, then the attach is nearly instantaneous.
These seem to me like unneeded restrictions in PostgreSQL, both that the unique index on one column can't be used to imply uniqueness on the both columns, and that the unique index on both columns cannot be used to imply the primary key (nor even a unique constraint--but only a unique index).
postgres 14
I have some table:
CREATE TABLE sometable (
id integer NOT NULL PRIMARY KEY UNIQUE ,
a integer NOT NULL DEFAULT 1,
b varchar(32) UNIQUE)
PARTITION BY RANGE (id);
But when i try to execute it, i get
ERROR: unique constraint on partitioned table must include all partitioning columns
If i execute same table definition without PARTITION BY RANGE (id) and check indexes, i get:
tablename indexname indexdef
sometable, sometable_b_key, CREATE UNIQUE INDEX sometable_b_key ON public.sometable USING btree (b)
sometable, sometable_pkey, CREATE UNIQUE INDEX sometable_pkey ON public.sometable USING btree (id)
So... unique constraints exist
whats the problem? how can i fix it?
On partitioned tables, all primary keys, unique constraints and unique indexes must contain the partition expression. That is because indexes on partitioned tables are implemented by individual indexes on each partition, and there is no way to enforce uniqueness across different indexes.
If you want to use partitioning, you have to sacrifice some consistency guarantees. There is no way around that. What you can do is create unique constraints on the partitions. That will guarantee uniqueness within each partition, but not global uniqueness.
This limitation is also mentioned in the docs
5.11.2.3. Limitations The following limitations apply to partitioned tables:
Unique constraints (and hence primary keys) on partitioned tables must
include all the partition key columns. This limitation exists because
the individual indexes making up the constraint can only directly
enforce uniqueness within their own partitions; therefore, the
partition structure itself must guarantee that there are not
duplicates in different partitions.
There is no way to create an exclusion constraint spanning the whole
partitioned table. It is only possible to put such a constraint on
each leaf partition individually. Again, this limitation stems from
not being able to enforce cross-partition restrictions.
https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE
I'm trying to partitioning a table with existing rows in Postgresql 10.8.
The structure is like this :
I'm trying to create partitions of table Item, it has around 5mill of rows.
I create the partitions with those commands:
CREATE TABLE item_1 (CHECK (id >0 AND id <1000001)) INHERITS (item);
CREATE TABLE item_2 (CHECK (id >1000000 AND id <2000001)) INHERITS (item);
...
Then the rules:
CREATE RULE item_1_rule AS ON INSERT TO item WHERE (id >0 AND id <1000001) DO INSTEAD INSERT INTO item_1 VALUES (NEW.*);
CREATE RULE item_2_rule AS ON INSERT TO item WHERE (id >1000000 AND id <2000001) DO INSTEAD INSERT INTO item_2 VALUES (NEW.*);
...
Then the migration to the partitioned tables:
INSERT INTO item_1 SELECT * FROM item WHERE (id >0 AND id <1000001);
INSERT INTO item_2 SELECT * FROM item WHERE (id >1000000 AND id <2000001);
...
And finally I try to Delete all the rows from the Item table:
DELETE FROM ONLY item;
But I get this error:
ERROR: update or delete on table "item" violates foreign key constraint >"item_audit_item_id_fkey" on table "item_audit"
SQL state: 23503
Detail: Key (id)=(1) is still referenced from table "item_audit".
So is it possible to drop the data from the main table Item to have only the rows on the partitions tables?
There are other alternatives to make a partition in a different way?
If you use inheritance partitioning, you won't be able to have foreign keys pointing to the partitioned table. You have to drop the constraint before you can delete the rows.
I recommend to use declarative partitioning. If you upgrade to v11 or later, you can have a foreign key pointing to a partitioned table, but since the primary key on a partitioned table has to contain the partitioning key, you'd have to add that column to the foreign key as well.
Since partitioning is mostly useful for mass data deletion, it might make sense to partition item_audit the same way.
I'm trying to find an example to create a partition table.
I have some tables with many tuples and I can classify them according to a value of one column, but, I just find examples using range and date (my column is a varchar and, in other table, is a int/foreign key).
I'm trying to speed my SELECT with this technique.
Here one of my CREATE tables (column Source will be used to partition this table):
CREATE TABLE tb_hit_source (
Hit_SourceId bigserial NOT NULL,
Source varchar(50) NOT NULL,
UniqueId varchar(50) NOT NULL,
tb_hit_HitId int8 NOT NULL,
CONSTRAINT tb_hit_source_ak_1 UNIQUE (Source, tb_hit_HitId, UniqueId) NOT DEFERRABLE INITIALLY IMMEDIATE,
CONSTRAINT tb_hit_source_pk PRIMARY KEY (Hit_SourceId)
);
CREATE INDEX tb_hit_source_idx_1 on tb_hit_source (Source ASC);
CREATE INDEX tb_hit_source_idx_2 on tb_hit_source (tb_hit_HitId ASC);
ALTER TABLE tb_hit_source ALTER COLUMN Hit_SourceId SET DEFAULT nextval('"HitSourceId_seq_tb_hit_source"');;
to create the table do.
CREATE TABLE tb_hit_source (
Hit_SourceId bigserial NOT NULL,
Source varchar(50) NOT NULL,
UniqueId varchar(50) NOT NULL,
tb_hit_HitId int8 NOT NULL,
CONSTRAINT tb_hit_source_ak_1
UNIQUE (Source, tb_hit_HitId, UniqueId) NOT DEFERRABLE,
CONSTRAINT tb_hit_source_pk PRIMARY KEY (Hit_SourceId)
PARTITION BY RANGE (Source);
then to create the partitions use the same value at each end of the range to force a single value partition.
CREATE TABLE tb_hit_source_a PARTITION OF tb_hit_source
FOR VALUES FROM ('a') TO ('a');
etc.
podtgresql 11 offers PARTITION BY LIST (source) allowing the partitions to be declared more simply.
CREATE TABLE tb_hit_source_a PARTITION OF tb_hit_source
FOR VALUES IN ('a');
to create the partitions do
create table part_a (check source='part_a' )inherits (tb_hit_source);
create table part_a (check source='part_b' )inherits (tb_hit_source);
etc.
but if there are going to be many partitions it will probably be more convenient to put them in a separate schema.
create schema hit_source_parts;
create table hit_source_parts.a (check(source='a'))inherits (tb_hit_source);
create table hit_source_parts.b (check(source='b'))inherits (tb_hit_source);
etc.
Any partitions you make will also need the aproptiate indexes.
unique constraints won't work across partitions this is one reaon why most uses of partitioning partition on one of the unique colums, by fragmenting this way uniqueness in each partition also enforces global uniqueness.