I am working on a requirement to create partition for customer name table in Postgresql DB. Primary Key of this table is an auto generated sequence number.
Since regular table cannot be turned into a partitioned table, I am planning to create a new partitioned table with primary key as composite key (sequence number + first letter of the customer's name).
I am not sure if it is required to define different sequence number one for each partition of customer name or defining the sequence number for partitioned table once will work.
Related
I am recreating an existing table as a partitioned table in PostgreSQL 11.
After some research, I am approaching it using the following procedure so this can be done online while writes are still happening on the table:
add a check constraint on the existing table, first as not valid and then validating
drop the existing primary key
rename the existing table
create the partitioned table under the prior table name
attach the existing table as a partition to the new partitioned table
My expectation was that the last step would be relatively fast, but I don't really have a number for this. In my testing, it's taking about 30s. I wonder if my expectations are incorrect or if I'm doing something wrong with the constraint or anything else.
Here's a simplified version of the DDL.
First, the inserted_at column is declared like this:
inserted_at timestamp without time zone not null
I want to have an index on the ID even after I drop the PK for existing queries and writes, so I create an index:
create unique index concurrently my_events_temp_id_index on my_events (id);
The check constraint is created in one transaction:
alter table my_events add constraint my_events_2022_07_events_check
check (inserted_at >= '2018-01-01' and inserted_at < '2022-08-01')
not valid;
In the next transaction, it's validated (and the validation is successful):
alter table my_events validate constraint my_events_2022_07_events_check;
Then before creating the partitioned table, I drop the primary key of the existing table:
alter table my_events drop constraint my_events_pkey cascade;
Finally, in its own transaction, the partitioned table is created:
alter table my_events rename to my_events_2022_07;
create table my_events (
id uuid not null,
... other columns,
inserted_at timestamp without time zone not null,
primary key (id, inserted_at)
) partition by range (inserted_at);
alter table my_events attach partition my_events_2022_07
for values from ('2018-01-01') to ('2022-08-01');
That last transaction blocks inserts and takes about 30s for the 12M rows in my test database.
Edit
I wanted to add that in response to the attach I see this:
INFO: partition constraint for table "my_events_2022_07" is implied by existing constraints
That makes me think I'm doing this right.
The problem is not the check constraint, it is the primary key.
If you make the original unique index include both columns:
create unique index concurrently my_events_temp_id_index on my_events (id,inserted_at);
And if you make the new table have a unique index rather than a primary key on those two columns, then the attach is nearly instantaneous.
These seem to me like unneeded restrictions in PostgreSQL, both that the unique index on one column can't be used to imply uniqueness on the both columns, and that the unique index on both columns cannot be used to imply the primary key (nor even a unique constraint--but only a unique index).
Short description of a problem
I need to build a partitioned table "users" with 2 partitions located on separate servers (Moscow and Hamburg), each partition is table with columns:
id - integer primary key with auto increment
region - smallint partition key, which equals either 100 for Hamburg or 200 for Moscow
login - unique character varying with length of 100.
I intended to make sequences for id as n*1000 + 100 for Hamburg, and n*1000 + 200 for Moscow, so just looking on primary key I will know which partition it belongs to.
region is intended to be read only and never change after creation, so no records will move between partitions.
SELECT queries must be able to return records from all partitions and UPDATE queries must be able to modify records on all partitions, INSERT/DELETE queries must be able to add/delete records only to local partition, so data stored in them is not completely isolated.
What was done
Using pgAdmin4
I created a "test" table on Hamburg server, added all column info, marked it as partitioned table with partition key region and partition type List.
I created a "hamburg" partition in this table, adding primary key constraint as id,region and unique key constraint as login,region.
I created a "moscow" table on Moscow server with the same column info as "test"
I added postgres_fdw extension to Hamburg server, created Foreign server pointing to DB on Moscow server and User mapping.
I added "moscow" foreign table to Hamburg server pointing to "moscow" table on Moscow server.
What is my problem
I couldn't figure out how to attach this foreign table as second partition to "test" table.
When I tried to attach partition through pgAdmin dialog in "test" table partitions properties it shows me an error: cannot unpack non-iterable Response object
When I tried to add partition with query as follows:
ALTER TABLE public.test ATTACH PARTITION public.moscow FOR VALUES IN (200);
It shows me an error:
ERROR: cannot attach foreign table "moscow" as partition of partitioned table "test"
DETAIL: Table "test" contains unique indexes.
SQL state: 42809
I removed unique constraint from login column but it shows the same error.
When I make partitioned table with the same properties and both partitions initially located on the same server all works well, except for postgres watch for login uniqueness per-partition rather than in whole table, but I suggest this is its limitation.
So, how can I attach a table located on the second server as partition to partitioned table located on the first one?
The error message is pretty clear: Since you cannot create an index on a partitioned table, PostgreSQL cannot create a partition of the unique index. But the unique index is required to implement the constraint.
See this source comment:
/*
* If we're attaching a foreign table, we must fail if any of the indexes
* is a constraint index; otherwise, there's nothing to do here. Do this
* before starting work, to avoid wasting the effort of building a few
* non-unique indexes before coming across a unique one.
*/
Either drop the unique constraint or don't use foreign tables as partitions.
Ok, I was finally able to add a foreign table as partition to partitioned table.
It was necessary to drop primary key property on id and unique property on login columns for partitioned table
After that I was able to attach foreign table as partition to partitioned table
Later I have added primary key property on id and unique property on login columns for each local partition.
So in the end I have unique global id as it is generated by sequences for each DB with never intersected values. For login uniqueness I have to manually check global table if there is any record with it before inserting.
P.S. Hopefully, this partitioning mechanism in postgres is suitable for geographically distant regions.
I have a table that I would like to partition based on the "deleted" date.
create table resources(
id int,
type varchar(30),
deleted date
)
I want to have a foreign key from another table pointing to this table's id column. However since I have the partition based on the deleted date, I must include it in the primary key. Adding the deleted column to the primary key does not make sense and will also prevent the other table's FK from pointing to this table. Is there a workaround?
Thanks
No, there is no workaround for foreign keys to a partitioned table. You have to add deleted to the primary key or unique constraint and also to the table that references the partitioned table.
If you don't want that, you'll have to go without a foreign key constraint.
Perhaps it would be useful to add deleted to the referencing table and partition that along the same boundaries as resources. Then you could get rid of old data in a coordinated fashion, and if you add deleted to the join condition, you can profit from enable_partitionwise_join = on.
The improvements to declarative range-based partitioning in version 11 seem like they can really work for my use case, but I'm not completely sure how foreign keys work with partitions.
I have tables Files -< Segments -< Entries in which there are hundreds of Segments per File and hundreds of Entries per segment, so Entries is on the order of 10,000 times the size of Files. Files have a CreationDate field and customers will define a retention period so they'll drop old Entries. All of this clearly points to date-based partitions so it's quicker to query the most recent entries first, and easy to drop old ones.
The first issue I encounter is that when I try to create the Files table, it sounds like I have to include createdDate as part of the primary key in order to have it in the RANGE partition:
CREATE TABLE Files2
(
FileId BIGSERIAL PRIMARY KEY,
createdDate timestamp with time zone,
filepath character varying COLLATE pg_catalog."default" NOT NULL
) PARTITION BY RANGE (createdDate)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
ERROR: insufficient columns in PRIMARY KEY constraint definition
DETAIL: PRIMARY KEY constraint on table "files2" lacks column "createddate" which is part of the partition key.
If I drop the "PRIMARY KEY" from the definition of FileId, I don't get the error, but does that affect the efficiency of child lookups?
I also don't know how to declare the partitions for the child table. PARTITION BY RANGE (Files.createdDate) doesn't work.
Since it's only in version 11 that this use case would even be possible, I haven't found much information about it and would appreciate any pointers! Thanks!
I believe it is a new feature of the version 11. There is a document from which you can get the following information:
"A primary key constraint must include a partition key column. Attempting to create a primary key constraint that does not contain partitioned columns results in an error"
https://h50146.www5.hpe.com/products/software/oe/linux/mainstream/support/lcc/pdf/PostgreSQL11NewFeaturesGAen20181022-1.pdf
As the title suggests I want to have a unique ID as a primary key but over multiple schemas. I know about UUID but it's just too costly.
Is there any way to work this around a serial?
You can create a global sequence and use that in your table instead of the automatic sequence that a serial column creates.
create schema global;
create schema s1;
create schema s2;
create sequence global.unique_id;
create table s1.t1
(
id integer default nextval('global.unique_id') primary key
);
create table s2.t1
(
id integer default nextval('global.unique_id') primary key
);
The difference to a serial column is, that the sequence unique_id doesn't "know" it's used by the id columns. A "serial sequence" is automatically dropped if the corresponding column (or table) is dropped which is not what you want with a global sequence.
There is one drawback however: you can't make sure that duplicate values across those two table are inserted manually. If you want to make sure the sequence is always used to insert values, you can create a trigger that always fetches a value from the sequence.