My query
delete from test.t1 where t2_id = 1;
My main table is t1 (includes around 1M rows and need to delete about 100k rows)
CREATE TABLE test.t1
(
id bigserial NOT NULL,
t2_id bigint,
... other fields
CONSTRAINT pk_t1 PRIMARY KEY (id),
CONSTRAINT fk_t1_t2 FOREIGN KEY (t2_id)
REFERENCES test.t2 (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
I have index on t2_id and 3 other indexes on plain string fields.
CREATE INDEX t1_t2_idx ON test.t1 USING btree (t2_id);
There are multiple (around 50) tables that reference test.t1. I have an index on t1_id for every table that references it.
CREATE TABLE test.t7
(
id bigserial NOT NULL,
t1_id bigint,
... other fields
CONSTRAINT pk_objekt PRIMARY KEY (id),
CONSTRAINT fk_t7_t1 FOREIGN KEY (t1_id)
REFERENCES test.t1 (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
CREATE INDEX t7_t1_idx ON test.t7 USING btree (t1_id);
//No other indexes here
Contents of t7 are deleted before t1 and it's super fast compared to removing from t1. Ratio of rows to delete is the same (~10%), but the total number of rows is considerably smaller (around 100K).
I have not been able to reduce the time to reasonable length:
I've tried removing all index-is (cancelled after 24h-s)
Kept only t1_t2_idx and t7_t1_idx ... t50_t1_idx indexes (cancelled after 24h-s)
Kept all indexes (cancelled after 24h-s)
Also vacuum analyze is performed before deletion and there should not be any locks (only active query in db).
I have not tried copying to temp table and truncating t1, but this does not seem reasonable since t1 can grow up to 10M rows from which 1M need to be deleted at some point.
Any ideas how to improve removing?
EDIT
Quite sure there are no locks because pg_stat_activity shows only 2 active queries (delete and pg_stat_activity)
"Delete on test.t1 (cost=0.43..6713.66 rows=107552 width=6)"
" -> Index Scan using t1_t2_idx on test.t1 (cost=0.43..6713.66 rows=107552 width=6)"
" Output: ctid"
" Index Cond: (t1.t1_id = 1)"
Related
Given these tables
Foo
id (PK)
name
updated
Bar
foo_id (FK)
name
updated
And this query:
SELECT *
FROM Foo as f
JOIN Bar as b
ON f.id=b.foo_id
WHERE b.name = 'Baz' AND f.name = 'Baz'
ORDER BY f.updated ASC, f.id ASC
LIMIT 10
OFFSET 10
Are these appropriate indexes to add - in MySql InnoDB the primary key column is automatically added to the end of a secondary index. What is the case with Postgres?
CREATE INDEX foo_name_id_idx ON foo(name, id)
CREATE INDEX bar_name_id_idx ON bar(name, id)
PostgreSQL does not make the distinction between primary and secondary indexes, and the primary key index is no different from other indexes. So the primary key is not added to other indexes, and there is no point in doing that unless you have a special reason for it.
Depending on which of the conditions are selective, there are three possible strategies:
If the condition on bar.name is selective, use bar as the driving site:
CREATE INDEX ON bar (name);
-- foo.id is already indexed
If the condition on foo.name is selective:
CREATE INDEX ON foo (name);
CREATE INDEX ON bar(foo_id); -- for a nested loop join
If none of the conditions are selective:
/* here the "id" is actually at the end of the index,
but that is just because it appears in ORDER BY */
CREATE INDEX ON foo (name, updated, id); -- for the ORDER BY
CREATE INDEX ON bar (foo_id); -- for a nested loop join
I have different queries for fetching data from a large table (about 100-200M rows). I've created partial indexes for my table with different predicates to fit the query because I know each query.
For example, the table similar to this:
CREATE TABLE public.contacts (
id int8 NOT NULL DEFAULT ssng_generate_id(8::bigint),
created timestamp NOT NULL DEFAULT timezone('UTC'::text, now()),
contact_pool_id int8 NOT NULL,
project_id int8 NOT NULL,
state_id int4 NOT NULL DEFAULT 10,
order_x int4 NOT NULL,
next_attempt_date timestamp NULL,
CONSTRAINT contacts_pkey PRIMARY KEY (id)
);
And there are two types of query:
SELECT * FROM contacts WHERE contact_pool_id = X AND state_id = 10 ORDER BY order_x LIMIT 1;
and
SELECT * FROM contacts WHERE contact_pool_id = X AND state_id = 20 AND next_attemp_date <= NOW ORDER BY next_attemp_date LIMIT 1;
For those queries I've created partial indexes:
For state_id = 10 (new contacts)
CREATE INDEX ix_contacts_cpid_orderx_id_for_new ON contacts USING btree (contact_pool_id, order_x, id) WHERE state_id = 10;
For state_id = 20 (available contacts)
CREATE INDEX ix_contacts_cpid_nextattepmdate_id_for_available ON contacts USING btree (contact_pool_id, next_attempt_date, id) WHERE state_id = 20;
For me, those partial indexes are faster than a single index.
And what about an update and insert performance? If I change a row with state_id = 20, will it affect only index 2 (for available contacts) or both of them will be affected?
Partial indexes which are not relevant to the tuple will not get updated.
If PostgreSQL can do a HOT update (if the column being changed is not part of an index, and there is room on the same page for the new tuple), then even the relevant index doesn't need to get updated.
Yes, with a partial index you only pay the overhead of modifying the index for rows that meet the WHERE condition, so you will always only need to modify at most one of the indexes at the same time (unless you change state_id from 10 to 20 or vice versa).
I could use some help optimizing a query that compares rows in a single table with millions of entries. Here's the table's definition:
CREATE TABLE IF NOT EXISTS data.row_check (
id uuid NOT NULL DEFAULT NULL,
version int8 NOT NULL DEFAULT NULL,
row_hash int8 NOT NULL DEFAULT NULL,
table_name text NOT NULL DEFAULT NULL,
CONSTRAINT row_check_pkey
PRIMARY KEY (id, version)
);
I'm reworking our push code and have a test bed with millions of records across about 20 tables. I run my tests, get the row counts, and can spot when some of my insert code has changed. The next step is to checksum each row, and then compare the rows for differences between versions of my code. Something like this:
-- Run my test of "version 0" of the push code, the base code I'm refactoring.
-- Insert the ID and checksum for each pushed row.
INSERT INTO row_check (id,version,row_hash,table_name)
SELECT id, 0, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_check_pkey DO UPDATE SET
row_hash = EXCLUDED.row_hash,
table_name = EXCLUDED.table_name;
truncate table record_changes_log;
-- Run my test of "version 1" of the push code, the new code I'm validating.
-- Insert the ID and checksum for each pushed row.
INSERT INTO row_check (id,version,row_hash,table_name)
SELECT id, 1, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_check_pkey DO UPDATE SET
row_hash = EXCLUDED.row_hash,
table_name = EXCLUDED.table_name;
That gets two rows in row_check for every row in record_changes_log, or any other table I'm checking. For the two runs of record_changes_log, I end up with more than 8.6M rows in row_check. They look like this:
id version row_hash table_name
e6218751-ab78-4942-9734-f017839703f6 0 -142492569 record_changes_log
6c0a4111-2f52-4b8b-bfb6-e608087ea9c1 0 -1917959999 record_changes_log
7fac6424-9469-4d98-b887-cd04fee5377d 0 -323725113 record_changes_log
1935590c-8d22-4baf-85ba-00b563022983 0 -1428730186 record_changes_log
2e5488b6-5b97-4755-8a46-6a46317c1ae2 0 -1631086027 record_changes_log
7a645ffd-31c5-4000-ab66-a565e6dad7e0 0 1857654119 record_changes_log
I asked yesterday for some help on the comparison query, and it lead to this:
select v0.table_name,
v0.id,
v0.row_hash as v0,
v1.row_hash as v1
from row_check v0
join row_check v1 on v0.id = v1.id and
v0.version = 0 and
v1.version = 1 and
v0.row_hash <> v1.row_hash
That works, but now I'm hoping to optimize it a bit. As an experiment, I clustered the data on version and then built a BRIN index, like this:
drop index if exists row_check_version_btree;
create index row_check_version_btree
on row_check
using btree(version);
cluster row_check using row_check_version_btree;
drop index row_check_version_btree; -- Eh? I want to see how the BRIN performs.
drop index if exists row_check_version_brin;
create index row_check_version_brin
on row_check
using brin(row_hash);
vacuum analyze row_check;
I ran the query through explain analyze and get this:
Merge Join (cost=1.12..559750.04 rows=4437567 width=51) (actual time=1511.987..14884.045 rows=10 loops=1)
Output: v0.table_name, v0.id, v0.row_hash, v1.row_hash
Inner Unique: true
Merge Cond: (v0.id = v1.id)
Join Filter: (v0.row_hash <> v1.row_hash)
Rows Removed by Join Filter: 4318290
Buffers: shared hit=8679005 read=42511
-> Index Scan using row_check_pkey on ascendco.row_check v0 (cost=0.56..239156.79 rows=4252416 width=43) (actual time=0.032..5548.180 rows=4318300 loops=1)
Output: v0.id, v0.version, v0.row_hash, v0.table_name
Index Cond: (v0.version = 0)
Buffers: shared hit=4360752
-> Index Scan using row_check_pkey on ascendco.row_check v1 (cost=0.56..240475.33 rows=4384270 width=24) (actual time=0.031..6070.790 rows=4318300 loops=1)
Output: v1.id, v1.version, v1.row_hash, v1.table_name
Index Cond: (v1.version = 1)
Buffers: shared hit=4318253 read=42511
Planning Time: 1.073 ms
Execution Time: 14884.121 ms
...which I did not really get the right idea from...so I ran it again to JSON and fed the results into this wonderful plan visualizer:
http://tatiyants.com/pev/#/plans
The tips there are right, the top node estimate is bad. The result is 10 rows, the estimate is for about 443,757 rows.
I'm hoping to learn more about optimizing this kind of thing, and this query seems like a good opportunity. I have a lot of notions about what might help:
-- CREATE STATISTICS?
-- Rework the query to move the where comparison?
-- Use a better index? I did try a GIN index and a straight B-tree on version, but neither was superior.
-- Rework the row_check format to move the two hashes into the same row instead of splitting them over two rows, compare on insert/update, flag non-matches, and add a partial index for the non-matching values.
Granted, it's funny to even try to index something where there are only two values (0 and 1 in the case above), so there's that. In fact, is there any sort of clever trick for Booleans? I'll always be comparing two versions, so "old" and "new", which I can express however makes life best. I understand that Postgres only has bitmap indexes internally at search/merge (?) time and that it does not have a bitmap type index. Would there be some kind of INTERSECT that might help? I don't know how Postgres implements set math operators internally.
Thanks for any suggestions on how to rethink this data or the query to make it faster for comparisons involving millions, or tens of millions, of rows.
I'm going to add an answer to my own question, but am still interested in what anyone else has to say. In the process of writing out my original question, I realized that maybe a redesign is in order. This hinges on my plan to only ever compare two versions at a time. That's a good solution here, but there are other cases where it wouldn't work. Anyway, here's a slightly different table design that folds the two results into a single row:
DROP TABLE IF EXISTS data.row_compare;
CREATE TABLE IF NOT EXISTS data.row_compare (
id uuid NOT NULL DEFAULT NULL,
hash_1 int8, -- Want NULL to defer calculating hash comparison until after both hashes are entered.
hash_2 int8, -- Ditto
hashes_match boolean, -- Likewise
table_name text NOT NULL DEFAULT NULL,
CONSTRAINT row_compare_pkey
PRIMARY KEY (id)
);
The following expression index should, hopefully, be very small as I shouldn't have any non-matching entries:
CREATE INDEX row_compare_fail ON row_compare (hashes_match)
WHERE hashes_match = false;
The trigger below does the column calculation, once hash_1 and hash_2 are both provided:
-- Run this as a BEFORE INSERT or UPDATE ROW trigger.
CREATE OR REPLACE FUNCTION data.on_upsert_row_compare()
RETURNS trigger AS
$BODY$
BEGIN
IF NEW.hash_1 = NULL OR
NEW.hash_2 = NULL THEN
RETURN NEW; -- Don't do the comparison, hash_1 hasn't been populated yet.
ELSE-- Do the comparison. The point of this is to avoid constantly thrashing the expression index.
NEW.hashes_match := NEW.hash_1 = NEW.hash_2;
RETURN NEW; -- important!
END IF;
END;
$BODY$
LANGUAGE plpgsql;
This now adds 4.3M rows instead of 8.6M rows:
-- Add the first set of results and build out the row_compare records.
INSERT INTO row_compare (id,hash_1,table_name)
SELECT id, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_compare_pkey DO UPDATE SET
hash_1 = EXCLUDED.hash_1,
table_name = EXCLUDED.table_name;
-- I'll truncate the record_changes_log and push my sample data again here.
-- Add the second set of results and update the row compare records.
-- This time, the hash is going into the hash_2 field for comparison
INSERT INTO row_compare (id,hash_2,table_name)
SELECT id, hashtext(record_changes_log::text),'record_changes_log'
FROM record_changes_log
ON CONFLICT ON CONSTRAINT row_compare_pkey DO UPDATE SET
hash_2 = EXCLUDED.hash_2,
table_name = EXCLUDED.table_name;
And now the results are simple to find:
select * from row_compare where hashes_match = false;
This changes the query time from around 17 seconds to around 24 milliseconds.
Here is what I have so far:
INSERT INTO Tenants (LeaseStartDate, LeaseExpirationDate, Rent, LeaseTenantSSN, RentOverdue)
SELECT CURRENT_DATE, NULL, NewRentPayments.Rent, NewRentPayments.LeaseTenantSSN, FALSE from NewRentPayments
WHERE NOT EXISTS (SELECT * FROM Tenants, NewRentPayments WHERE NewRentPayments.HouseID = Tenants.HouseID AND
NewRentPayments.ApartmentNumber = Tenants.ApartmentNumber)
So, HouseID and ApartmentNumber together make up the primary key. If there is a tuple in table B (NewRentPayments) that doesn't exist in table A (Tenants) based on the primary key, then it needs to be inserted into Tenants.
The problem is, when I run my query, it doesn't insert anything (I know for a fact there should be 1 tuple inserted). I'm at a loss, because it looks like it should work.
Thanks.
Your subquery was not correlated - It was just a non-correlated join query.
As per description of your problem, you don't need this join.
Try this:
insert into Tenants (LeaseStartDate, LeaseExpirationDate, Rent, LeaseTenantSSN, RentOverdue)
select current_date, null, p.Rent, p.LeaseTenantSSN, FALSE
from NewRentPayments p
where not exists (
select *
from Tenants t
where p.HouseID = t.HouseID
and p.ApartmentNumber = t.ApartmentNumber
)
When I run the following script with Postgres 9.3 (with enable_seqscan set to off), I expect the final query to make use of the "forms_string" partial index, but instead uses the "forms_int" index, which doesn't make sense.
When I've been testing this with actual code with JSON functions and indexes for more types, it consistently seems to use whatever the last index created was, for every query.
Adding more unrelated rows so that the rows relevant to the partial index are only a small percentage of total rows in the table results in a "bitmap heap scan", but still mentions the same incorrect index after that.
Any idea how I can get it to use the correct index?
CREATE EXTENSION IF NOT EXISTS plv8;
CREATE OR REPLACE FUNCTION
json_string(data json, key text) RETURNS TEXT AS $$
var ret = data,
keys = key.split('.'),
len = keys.length;
for (var i = 0; i < len; ++i) {
if (ret) {
ret = ret[keys[i]]
};
}
if (typeof ret === "undefined") {
ret = null;
} else if (ret) {
ret = ret.toString();
}
return ret;
$$ LANGUAGE plv8 IMMUTABLE STRICT;
CREATE OR REPLACE FUNCTION
json_int(data json, key text) RETURNS INT AS $$
var ret = data,
keys = key.split('.'),
len = keys.length;
for (var i = 0; i < len; ++i) {
if (ret) {
ret = ret[keys[i]]
}
}
if (typeof ret === "undefined") {
ret = null;
} else {
ret = parseInt(ret, 10);
if (isNaN(ret)) {
ret = null;
}
}
return ret;
$$ LANGUAGE plv8 IMMUTABLE STRICT;
CREATE TABLE form_types (
id SERIAL NOT NULL,
name VARCHAR(200),
PRIMARY KEY (id)
);
CREATE TABLE tenants (
id SERIAL NOT NULL,
name VARCHAR(200),
PRIMARY KEY (id)
);
CREATE TABLE forms (
id SERIAL NOT NULL,
tenant_id INTEGER,
type_id INTEGER,
data JSON,
PRIMARY KEY (id),
FOREIGN KEY(tenant_id) REFERENCES tenants (id),
FOREIGN KEY(type_id) REFERENCES form_types (id)
);
CREATE INDEX ix_forms_type_id ON forms (type_id);
CREATE INDEX ix_forms_tenant_id ON forms (tenant_id);
INSERT INTO tenants (name) VALUES ('mike'), ('bob');
INSERT INTO form_types (name) VALUES ('type 1'), ('type 2');
INSERT INTO forms (tenant_id, type_id, data) VALUES
(1, 1, '{"string": "unicorns", "int": 1}'),
(1, 1, '{"string": "pythons", "int": 2}'),
(1, 1, '{"string": "pythons", "int": 8}'),
(1, 1, '{"string": "penguins"}');
CREATE OR REPLACE VIEW foo AS
SELECT forms.id AS forms_id,
json_string(forms.data, 'string') AS "data.string",
json_int(forms.data, 'int') AS "data.int"
FROM forms
WHERE forms.tenant_id = 1 AND forms.type_id = 1;
CREATE INDEX "forms_string" ON forms (json_string(data, 'string'))
WHERE tenant_id = 1 AND type_id = 1;
CREATE INDEX "forms_int" ON forms (json_int(data, 'int'))
WHERE tenant_id = 1 AND type_id = 1;
EXPLAIN ANALYZE VERBOSE SELECT "data.string" from foo;
Outputs:
Index Scan using forms_int on public.forms
(cost=0.13..8.40 rows=1 width=32) (actual time=0.085..0.239 rows=20 loops=1)
Output: json_string(forms.data, 'string'::text)
Total runtime: 0.282 ms
Without enable_seqscan=off:
Seq Scan on public.forms (cost=0.00..1.31 rows=1 width=32) (actual time=0.080..0.277 rows=28 loops=1)
Output: json_string(forms.data, 'string'::text)
Filter: ((forms.tenant_id = 1) AND (forms.type_id = 1))
Total runtime: 0.327 ms
\d forms prints
Table "public.forms"
Column | Type | Modifiers
-----------+---------+----------------------------------------------------
id | integer | not null default nextval('forms_id_seq'::regclass)
tenant_id | integer |
type_id | integer |
data | json |
Indexes:
"forms_pkey" PRIMARY KEY, btree (id)
"forms_int" btree (json_int(data, 'int'::text)) WHERE tenant_id = 1 AND type_id = 1
"forms_string" btree (json_string(data, 'string'::text)) WHERE tenant_id = 1 AND type_id = 1
"ix_forms_tenant_id" btree (tenant_id)
"ix_forms_type_id" btree (type_id)
Foreign-key constraints:
"forms_tenant_id_fkey" FOREIGN KEY (tenant_id) REFERENCES tenants(id)
"forms_type_id_fkey" FOREIGN KEY (type_id) REFERENCES form_types(id)
Index vs seqscan, costs
Looks like your random_page_cost is too high compared to the real performance of your machine. Random I/O is faster (costs less) than Pg thinks it does, so it's choosing a slightly less ideal plan.
That's why the cost estimate for the indexscan is (cost=0.13..8.40 rows=1 width=32) and for the seqscan it's slightly lower at (cost=0.00..1.31 rows=1 width=32).
Lower random_page_cost - try SET random_page_cost = 2 then re-running.
To learn more, read the documentation on PostgreSQL query planning, parameters, and tuning, and the relevant wiki pages.
Index selection
PostgreSQL appears to be picking an index scan on forms_int instead of forms_string because it'll be a more compact, smaller index, and both indexes exactly match the search criteria for the view: tenant_id = 1 AND type_id = 1.
If you disable or drop forms_int it'll probably use forms_string and go slightly slower.
The key thing to understand is that while the index does contain the value of interest, PostgreSQL isn't actually using it. It's scanning the index without an index condition, since every tuple in the index matches, to get tuples from the heap. It's then extracting the value from those heap tuples and outputting them.
This can be demonstrated with an expression-index on a constant:
CREATE INDEX "forms_novalue" ON forms((true)) WHERE tenant_id = 1 AND type_id = 1;
PostgreSQL is quite likely to select this index for the query:
regress=# EXPLAIN ANALYZE VERBOSE SELECT "data.string" from foo;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Index Scan using forms_novalue on public.forms (cost=0.13..13.21 rows=4 width=32) (actual time=0.190..0.310 rows=4 loops=1)
Output: json_string(forms.data, 'string'::text)
Total runtime: 0.346 ms
(3 rows)
All the indexes are the same size because they're all so tiny they fit in the minimum allocation:
regress=# SELECT x.idxname, pg_relation_size(x.idxname) FROM (VALUES ('forms_novalue'),('forms_int'),('forms_string')) x(idxname);
idxname | pg_relation_size
---------------+------------------
forms_novalue | 16384
forms_int | 16384
forms_string | 16384
(3 rows)
but the stats for novalue will be somewhat more attractive due to a narrower row width.
Index scan vs index-only scan
It sounds like what you really expect is an index-only scan, where Pg never touches the table's heap and only uses the tuples in the index its self.
I would expect that this query's requirements could be satisfied with forms_string, but can't get Pg to pick an index-only scan plan for it.
It's not immediately clear to me why Pg is not using an index-only scan here, as it should be a candidate, but it doesn't seem to be able to plan one. If I force enable_indexscan = off, it'll pick an inferior bitmap index scan plan instead, and if force disable enable_bitmapscan it'll fall back to a max-cost-estimate seqscan. This is true even after a VACUUM of the table(s) of interest.
That means it must not be being generated as a candidate path in the query planner - Pg doesn't know how to use an index-only scan for this query, or thinks it cannot do so for some reason.
It isn't an issue with view introspection, as an expanded view query is the same.
Your table has insufficient data in it. In short, Postgres won't use an index when the table fits in a single disk page. Ever. When your table contains a few hundred or thousand rows, it'll become too big to fit, and then you'll see Postgres begin to use index scans when relevant.
Another point to consider is you need to analyze your tables after a large import. Without accurate stats on your actual data, Postgres may end up dismissing some index scans as too expensive, when in fact they'd be cheap.
Lastly, there are cases when it is cheaper to not use an index. In essence, whenever Postgres is about to visit most disk pages repeatedly and in a random order to retrieve a large number of rows, it'll seriously consider the cost of visiting most (bitmap index) or all (seq scan) disk pages once sequentially and filtering invalid rows out. The latter wins if you're selecting enough rows.