I apologize for the lengthy post. I'm trying to get in all the details.
We recently upgraded our Postgres AWS RDS from 9.5 to 11.1.
We have several large partitioned tables implemented using inheritance that we are considering converting to declarative partitioning.
(I'm talking about 5TB of partitioned data). I want to be sure of my methodology before I push forward.
For example here is how we would have created a partitioned table with inheritance. The table has a primary key and an index. The inherited partition has a check constraint and an index. (Not shown is the trigger on the primary table that would put the new rows in the correct partition.)
CREATE TABLE test
(
date_key numeric(15,0) NOT NULL,
metric numeric(15,0) NOT NULL,
value numeric(28,5) NOT NULL,
CONSTRAINT test_pkey PRIMARY KEY (date_key,metric)
)
TABLESPACE pg_default;
CREATE INDEX test_idx1
ON test USING btree
(metric)
TABLESPACE pg_default;
CREATE TABLE test_201908
(
CONSTRAINT const_test_chk CHECK (date_key >= 20190801::numeric AND date_key <= 20190831::numeric)
)
INHERITS (test)
TABLESPACE pg_default;
CREATE INDEX test_idx1_201908
ON test_201908 USING btree
(metric)
TABLESPACE pg_default;
AMZGQ3DW=> \d+ edibben.test
Table "edibben.test"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------+---------------+-----------+----------+---------+---------+--------------+-------------
date_key | numeric(15,0) | | not null | | main | |
metric | numeric(15,0) | | not null | | main | |
value | numeric(28,5) | | not null | | main | |
Indexes:
"test_pkey" PRIMARY KEY, btree (date_key, metric)
"test_idx1" btree (metric)
Child tables: edibben.test_201908
AMZGQ3DW=> \d+ edibben.test_201908
Table "edibben.test_201908"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------+---------------+-----------+----------+---------+---------+--------------+-------------
date_key | numeric(15,0) | | not null | | main | |
metric | numeric(15,0) | | not null | | main | |
value | numeric(28,5) | | not null | | main | |
Indexes:
"test_idx1_201908" btree (metric)
Check constraints:
"const_test_chk" CHECK (date_key >= 20190801::numeric AND date_key <= 20190831::numeric)
Inherits: edibben.test
I know that I can convert this table into a declarative partitioned table by doing the following:
Create a new partitioned table:
CREATE TABLE test_part
(
date_key numeric(15,0) NOT NULL,
metric numeric(15,0) NOT NULL,
value numeric(28,5) NOT NULL,
CONSTRAINT test_part_pkey PRIMARY KEY (date_key,metric)
) PARTITION BY RANGE (date_key)
TABLESPACE pg_default;
CREATE INDEX test_part_idx1
ON test_part USING btree
(metric)
TABLESPACE pg_default;
Drop the inheritance on the test_201908 table:
alter table test_201908 no inherit test;
And then add this table to the partitioned table. The doco says to keep the check constraint in place until after the data is loaded.
alter table test_part
attach partition test_201908
for VALUES FROM (20190801) TO (20190831);
The partition shows up attached to the table:
\d+ edibben.test_part
Table "edibben.test_part"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------+---------------+-----------+----------+---------+---------+--------------+-------------
date_key | numeric(15,0) | | not null | | main | |
metric | numeric(15,0) | | not null | | main | |
value | numeric(28,5) | | not null | | main | |
Partition key: RANGE (date_key)
Indexes:
"test_part_pkey" PRIMARY KEY, btree (date_key, metric)
"test_part_idx1" btree (metric)
Partitions: edibben.test_201908 FOR VALUES FROM ('20190801') TO ('20190831')
My question is about what happens to the indexes. When you examine the partition you see the primary key inherited from the partition table
and the original index (test_idx1_201908).
AMZGQ3DW-> \d+ edibben.test_201908
Table "edibben.test_201908"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------+---------------+-----------+----------+---------+---------+--------------+-------------
date_key | numeric(15,0) | | not null | | main | |
metric | numeric(15,0) | | not null | | main | |
value | numeric(28,5) | | not null | | main | |
Partition of: edibben.test_part FOR VALUES FROM ('20190801') TO ('20190831')
Partition constraint: ((date_key IS NOT NULL) AND (date_key >= '20190801'::numeric(15,0)) AND (date_key < '20190831'::numeric(15,0)))
Indexes:
"test_201908_pkey" PRIMARY KEY, btree (date_key, metric)
"test_idx1_201908" btree (metric)
Check constraints:
"const_test_chk" CHECK (date_key >= 20190801::numeric AND date_key <= 20190831::numeric)
If I add a new partition to the test_part table
CREATE TABLE test_201909 PARTITION OF test_part
FOR VALUES FROM ('20190901') TO ('20190930');
The new table has the primary key and the index but the index has a system generated name.
$\d+ edibben.test_201909
Table "edibben.test_201909"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------+---------------+-----------+----------+---------+---------+--------------+-------------
date_key | numeric(15,0) | | not null | | main | |
metric | numeric(15,0) | | not null | | main | |
value | numeric(28,5) | | not null | | main | |
Partition of: edibben.test_part FOR VALUES FROM ('20190901') TO ('20190930')
Partition constraint: ((date_key IS NOT NULL) AND (date_key >= '20190901'::numeric(15,0)) AND (date_key < '20190930'::numeric(15,0)))
Indexes:
"test_201909_pkey" PRIMARY KEY, btree (date_key, metric)
"test_201909_metric_idx" btree (metric)
Looking at pg_class for the objects I just created:
AMZGQ3DW=> select relname, reltype, relkind,relowner from pg_class where relname like 'test%';
relname | reltype | relkind | relowner
------------------------+---------+---------+----------
test_201908 | 365444 | r | 98603
test_201908_pkey | 0 | i | 98603
test_idx1_201908 | 0 | i | 98603
test_201909 | 366498 | r | 98603
test_201909_metric_idx | 0 | i | 98603
test_201909_pkey | 0 | i | 98603
test_part | 365449 | p | 98603
test_part_idx1 | 0 | I | 98603
test_part_pkey | 0 | I | 98603
The indexes on the partitioned table have a relkind of I and the indexes on the partitions have a rekind of i. Looking at pg_indexes
there are no entries for the indexes on the primary table:
AMZGQ3DW=> select schemaname, tablename, indexname from pg_indexes where schemaname = 'edibben' and tablename = 'test_part';
schemaname | tablename | indexname
------------+-----------+-----------
(0 rows)
The indexes on the partitions do show up:
AMZGQ3DW=> select schemaname, tablename, indexname from pg_indexes where schemaname = 'edibben' and tablename like 'test%' order by tablename;
schemaname | tablename | indexname
------------+-------------+------------------------
edibben | test_201908 | test_201908_pkey
edibben | test_201908 | test_idx1_201908
edibben | test_201909 | test_201909_pkey
edibben | test_201909 | test_201909_metric_idx
So, is this partitioned table properly indexed? (yes there was a question buried in this mess). I can't find any documentation of how the
partitioned indexes work but it appears that the partitioned 'Index' is just a definition and that the real indexes are on the partitions themselves. Is there a way to list all of the indexes associated with a partitioned index? Is there a way to see if the partitioned index is valid?
Also, the doco talks about creating the index on the partitioned table with the CREATE INDEX ON ONLY option. I don't think this
applies to what I need to do. Am I right?
"As explained above, it is possible to create indexes on partitioned tables and they are applied automatically
to the entire hierarchy. This is very convenient, as not only the existing partitions will become indexed,
but also any partitions that are created in the future will. One limitation is that it's not possible to use
the CONCURRENTLY qualifier when creating such a partitioned index. To overcome long lock times,
it is possible to use CREATE INDEX ON ONLY the partitioned table; such an index is marked invalid,
and the partitions do not get the index applied automatically. The indexes on partitions can be created
separately using CONCURRENTLY, and later attached to the index on the parent using ALTER INDEX .. ATTACH PARTITION.
Once indexes for all partitions are attached to the parent index, the parent index is marked valid automatically."
The index test_idx1_201908 is automatically converted to be a partition of the partitioned index test_201909_metric_idx. It does not matter that its name is different from other index partitions.
You can verify that with the following query:
SELECT relispartition FROM pg_class WHERE relname = 'test_idx1_201908';
The result should be TRUE, signifying that the index is a partition of a partitioned index.
I have two remarks unrelated to your question:
I notice that you didn't define the upper bound of the range for the partitioning key wrong.
The upper bound you specify is excluded, so you should write
CREATE TABLE test_201909 PARTITION OF test_part
FOR VALUES FROM ('20190901') TO ('20191001');
It is probably too late for that, but you should have chosen date rather than numeric for the partitioning column.
That would make everything simpler and more readable, and it would be impossible to enter incorrect dates like 20190335.
I have a partitioned table called datapoint which have child tables like datapoint_s1, datapoint_s2, ...
I have another table called device which has foreign key to datapoint table.
Here is the story;
Datapoint table is about 100GB with all partitioned tables. I truncated datapoint_s2 table. After that, I want to delete devices belongs to datapoints_s2 table. Also, there is no data in other partitioned tables related to devices I want to delete. Although the datapoint_s2 table is empty, COMMIT; is waiting more than one hour to complete process.
Edit: When I cancel the query, this error occured.
ERROR: canceling statement due to user request
CONTEXT: SQL statement "SELECT 1 FROM "public"."datapoint" x WHERE $1 OPERATOR(pg_catalog.=) "device_id" FOR KEY SHARE OF x"
How I can make device deletion process faster in partitioned tables?
Edit:
Parent table description:
Table "public.datapoint"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
------------+--------------------------+-----------+----------+---------------------------------------+----------+--------------+-------------
id | bigint | | not null | nextval('datapoint_id_seq'::regclass) | plain | |
station_id | integer | | not null | | plain | |
device_id | integer | | not null | | plain | |
data | jsonb | | | | extended | |
created_at | timestamp with time zone | | not null | | plain | |
Partition key: LIST (station_id)
Indexes:
"datapoint_uniq" UNIQUE CONSTRAINT, btree (station_id, device_id, created_at)
Foreign-key constraints:
"datapoint_device_id_fk" FOREIGN KEY (device_id) REFERENCES device(id) DEFERRABLE INITIALLY DEFERRED
"datapoint_station_id_fk" FOREIGN KEY (station_id) REFERENCES station(id) DEFERRABLE INITIALLY DEFERRED
Partitions: datapoint_s2 FOR VALUES IN (10),
datapoint_s3 FOR VALUES IN (11),
....
Child table description:
Table "public.datapoint_s2"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
------------+--------------------------+-----------+----------+---------------------------------------+----------+--------------+-------------
id | bigint | | not null | nextval('datapoint_id_seq'::regclass) | plain | |
station_id | integer | | not null | | plain | |
device_id | integer | | not null | | plain | |
data | jsonb | | | | extended | |
created_at | timestamp with time zone | | not null | | plain | |
Partition of: solarify_datapoint FOR VALUES IN (2)
Partition constraint: ((station_id IS NOT NULL) AND (station_id = 2))
Indexes:
"datapoint_s2_station_id_device_id_created_at_key" UNIQUE CONSTRAINT, btree (station_id, device_id, created_at)
Foreign-key constraints:
"datapoint_device_id_fk" FOREIGN KEY (device_id) REFERENCES device(id) DEFERRABLE INITIALLY DEFERRED
"datapoint_station_id_fk" FOREIGN KEY (station_id) REFERENCES station(id) DEFERRABLE INITIALLY DEFERRED
I have two tables:
x_development=# \d+ xref__email_addresses__organizations
Table "public.xref__email_addresses__organizations"
Column | Type | Modifiers | Storage | Stats target | Description
-------------------------+-----------------------------+------------------------+---------+--------------+-------------
email_address_id | integer | not null | plain | |
rorganization_id | integer | not null | plain | |
last_update | timestamp without time zone | not null default now() | plain | |
Indexes:
"email_address_id_idx" UNIQUE, btree (email_address_id)
"xref_idx" UNIQUE, btree (email_address_id, organization_id)
Foreign-key constraints:
"email_address_id_fkey" FOREIGN KEY (email_address_id) REFERENCES email_addresses(email_address_id) ON UPDATE CASCADE ON DELETE RESTRICT
"organization_id_fkey" FOREIGN KEY (realtor_organization_id) REFERENCES realtor_organizations(realtor_organization_id) ON UPDATE CASCADE ON DELETE RESTRICT
x_development=# \d+ email_addresses
Table "public.email_addresses"
Column | Type | Modifiers | Storage | Stats target | Description
--------------------+---------+----------------------------------------------------------------------------+----------+--------------+-------------
email_address_id | integer | not null default nextval('email_addresses_email_address_id_seq'::regclass) | plain | |
email_address | citext | not null | extended | |
unsubscribe_reason | text | not null default ''::text | extended | |
Indexes:
"email_addresses_pkey" PRIMARY KEY, btree (email_address_id)
"email_address_idx" UNIQUE, btree (email_address)
Referenced by:
TABLE "xref__email_addresses__organizations" CONSTRAINT "email_address_id_fkey" FOREIGN KEY (email_address_id) REFERENCES email_addresses(email_address_id) ON UPDATE CASCADE ON DELETE RESTRICT
Note they both have an email_address_id column and
"email_address_id_idx" UNIQUE, btree (email_address_id) index.
I need to drop the email_address_id_idx on the xref__email_addresses__organizations table but it seems if mis-created the table by having two identical index names.
How do I drop the email_address_id_idx from the xref__email_addresses__organizations table?
From what i have understood, the indexes are different. One of them is email_address_id_idx and the other is email_address_idx. So I Think you can just use drop index to remove the desired one.
I have a table with a character varying(12) field in it which is its PRIMARY KEY. I ran this query
SELECT * FROM bg WHERE bg_id ='470370111002'
It selects a row from the table. All looks good. Then I try.
INSERT INTO csapp_center_bgs(bg_id,center_id) VALUES('470370111002',2)
There is a foreign key on bg_id that looks like...
ALTER TABLE csapp_center_bgs
ADD CONSTRAINT csapp_center_bgs_bg_id_65c818f360c84dc5_fk_bg_bg_id
FOREIGN KEY (bg_id)
REFERENCES tiger.bg (bg_id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION DEFERRABLE INITIALLY DEFERRED;
Here is the precise error...
ERROR: insert or update on table "csapp_center_bgs" violates foreign key constraint "csapp_center_bgs_bg_id_65c818f360c84dc5_fk_bg_bg_id"
DETAIL: Key (bg_id)=(470370111002) is not present in table "bg".
********** Error **********
ERROR: insert or update on table "csapp_center_bgs" violates foreign key constraint "csapp_center_bgs_bg_id_65c818f360c84dc5_fk_bg_bg_id"
SQL state: 23503
Detail: Key (bg_id)=(470370111002) is not present in table "bg".
Why did this not work?! Any ideas? Here is \d+ bg...
Column | Type | Modifiers | Storage | Stats target | Description
----------+-----------------------+--------------------------------------------------+----------+--------------+-------------
gid | integer | not null default nextval('bg_gid_seq'::regclass) | plain | |
statefp | character varying(2) | | extended | |
countyfp | character varying(3) | | extended | |
tractce | character varying(6) | | extended | |
blkgrpce | character varying(1) | | extended | |
bg_id | character varying(12) | not null | extended | |
namelsad | character varying(13) | | extended | |
mtfcc | character varying(5) | | extended | |
funcstat | character varying(1) | | extended | |
aland | double precision | | plain | |
awater | double precision | | plain | |
intptlat | character varying(11) | | extended | |
intptlon | character varying(12) | | extended | |
the_geom | geometry | | main | |
Indexes:
"bg_pkey" PRIMARY KEY, btree (bg_id)
"idx_bg_geom" gist (the_geom) CLUSTER
Check constraints:
"enforce_dims_geom" CHECK (st_ndims(the_geom) = 2)
"enforce_geotype_geom" CHECK (geometrytype(the_geom) = 'MULTIPOLYGON'::text OR the_geom IS NULL)
"enforce_srid_geom" CHECK (st_srid(the_geom) = 4269)
Referenced by:
TABLE "csapp_center_bgs" CONSTRAINT "csapp_center_bgs_bg_id_65c818f360c84dc5_fk_bg_bg_id" FOREIGN KEY (bg_id) REFERENCES bg(bg_id) DEFERRABLE INITIALLY DEFERRED
Child tables: tiger_data.tn_bg
Has OIDs: no
And here is \d+ on csapp_...
Column | Type | Modifiers | Storage | Stats target | Description
-----------+-----------------------+---------------------------------------------------------------+----------+--------------+-------------
id | integer | not null default nextval('csapp_center_bgs_id_seq'::regclass) | plain | |
bg_id | character varying(12) | not null | extended | |
center_id | integer | not null | plain | |
Indexes:
"csapp_center_bgs_pkey" PRIMARY KEY, btree (id)
"csapp_center_bgs_5e94e25f" btree (bg_id)
"csapp_center_bgs_c63f1184" btree (center_id)
Foreign-key constraints:
"csapp_center_bgs_bg_id_65c818f360c84dc5_fk_bg_bg_id" FOREIGN KEY (bg_id) REFERENCES bg(bg_id) DEFERRABLE INITIALLY DEFERRED
"csapp_center_bgs_center_id_360e6806f7d3fee_fk_csapp_centers_id" FOREIGN KEY (center_id) REFERENCES csapp_centers(id) DEFERRABLE INITIALLY DEFERRED
Has OIDs: no
Here is the version:
version
------------------------------------------------------------------------------------------------------
PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
And here is my search path....
search_path
---------------
public, tiger
(1 row)
bg is in schema tiger and csapp_center_bgs is in schema public...
My first guess would be that you are dealing with two different tables named bg. One in the schema tiger, and another one in an undisclosed schema that comes before tiger in your search_path - or tiger is not in the search_path at all.
Find all tables named bg (case sensitive) in all schemas in the current db:
SELECT * FROM pg_tables WHERE tablename = 'bg';
To understand the search_path setting:
How does the search_path influence identifier resolution and the "current schema"
To understand the structure of a Postgres DB cluster:
What's the difference between a catalog and a schema in a relational database?
If that's not it, your index may be corrupted. I would first try a REINDEX:
REINDEX bg_pkey;
Inheritance!
I see in your added table definition:
Child tables: tiger_data.tn_bg
Suspecting that the row with bg_id ='470370111002' actually lives in the child table tiger_data.tn_bg. But your FK constraint references the parent table. FK constraints are not inherited.
What do you get if you query:
SELECT * FROM ONLY bg WHERE bg_id ='470370111002'
If my hypothesis holds, you get no row. Read the chapter Caveats on the Inheritance page of the manual.
Related:
Use triggers on inherited tables to replace foreign keys
Here' my current state.
Eonil=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+-------+------------+-------------
public | TestTable1 | table | Eonil | 8192 bytes |
(1 row)
Eonil=# \d+ TestTable1
Did not find any relation named "TestTable1".
Eonil=#
What is the problem and how can I see the table definition?
Postgres psql needs escaping for capital letters.
Eonil=# \d+ "TestTable1"
So this works well.
Eonil=# \d+ "TestTable1"
Table "public.TestTable1"
Column | Type | Modifiers | Storage | Description
--------+------------------+-----------+----------+-------------
ID | bigint | not null | plain |
name | text | | extended |
price | double precision | | plain |
Indexes:
"TestTable1_pkey" PRIMARY KEY, btree ("ID")
"TestTable1_name_key" UNIQUE CONSTRAINT, btree (name)
Has OIDs: no
Eonil=#