POSTGRESQL VERSION: 10
HARDWARE: 4 workers / 16GBRAM / 50% used
I'm not a Postgresql expert. I have just read a lot of documentation and did a lot of tests.
I have some postgresql queries whick take a lot of times > 30s because of 10 millions of rows on a table.
Column | Type | Collation | Nullable | Default
------------------------------+--------------------------+-----------+----------+----------------------------------------------------------
id | integer | | not null |
cveid | character varying(50) | | |
summary | text | | not null |
published | timestamp with time zone | | |
modified | timestamp with time zone | | |
assigner | character varying(128) | | |
vulnerable_products | character varying(250)[] | | |
cvss | double precision | | |
cvss_time | timestamp with time zone | | |
cvss_vector | character varying(250) | | |
access | jsonb | | not null |
impact | jsonb | | not null |
score | integer | | not null |
is_exploitable | boolean | | not null |
is_confirmed | boolean | | not null |
is_in_the_news | boolean | | not null |
is_in_the_wild | boolean | | not null |
reflinks | jsonb | | not null |
reflinkids | jsonb | | not null |
created_at | timestamp with time zone | | |
history_id | integer | | not null | nextval('vulns_historicalvuln_history_id_seq'::regclass)
history_date | timestamp with time zone | | not null |
history_change_reason | character varying(100) | | |
history_type | character varying(1) | | not null |
Indexes:
"vulns_historicalvuln_pkey" PRIMARY KEY, btree (history_id)
"btree_varchar" btree (history_type varchar_pattern_ops)
"vulns_historicalvuln_cve_id_850876bb" btree (cve_id)
"vulns_historicalvuln_cwe_id_2013d697" btree (cwe_id)
"vulns_historicalvuln_history_user_id_9e25ebf5" btree (history_user_id)
"vulns_historicalvuln_id_773f2af7" btree (id)
--- TRUNCATE
Foreign-key constraints:
"vulns_historicalvuln_history_user_id_9e25ebf5_fk_custusers" FOREIGN KEY (history_user_id) REFERENCES custusers_user(id) DEFERRABLE INITIALLY DEFERRED
Example of queries:
SELECT * FROM vulns_historicalvuln WHERE history_type <> '+' order by id desc fetch first 10000 rows only; -> 30s without cache
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.43..31878.33 rows=10000 width=1736) (actual time=0.173..32839.474 rows=10000 loops=1)
-> Index Scan Backward using vulns_historicalvuln_id_773f2af7 on vulns_historicalvuln (cost=0.43..26346955.92 rows=8264960 width=1736) (actual time=0.172..32830.958 rows=10000 loops=1)
Filter: ((history_type)::text <> '+'::text)
Rows Removed by Filter: 296
Planning time: 19.514 ms
Execution time: 32845.015 ms
SELECT DISTINCT "vulns"."id", "vulns"."uuid", "vulns"."feedid", "vulns"."cve_id", "vulns"."cveid", "vulns"."summary", "vulns"."published", "vulns"."modified", "vulns"."assigner", "vulns"."cwe_id", "vulns"."vulnerable_packages_versions", "vulns"."vulnerable_products", "vulns"."vulnerable_product_versions", "vulns"."cvss", "vulns"."cvss_time", "vulns"."cvss_version", "vulns"."cvss_vector", "vulns"."cvss_metrics", "vulns"."access", "vulns"."impact", "vulns"."cvss3", "vulns"."cvss3_vector", "vulns"."cvss3_version", "vulns"."cvss3_metrics", "vulns"."score", "vulns"."is_exploitable", "vulns"."is_confirmed", "vulns"."is_in_the_news", "vulns"."is_in_the_wild", "vulns"."reflinks", "vulns"."reflinkids", "vulns"."created_at", "vulns"."updated_at", "vulns"."id" AS "exploit_count", false AS "monitored", '42' AS "org" FROM "vulns" WHERE ("vulns"."score" >= 0 AND "vulns"."score" <= 100) ORDER BY "vulns"."updated_at" DESC LIMIT 10
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=315191.32..315192.17 rows=10 width=1691) (actual time=3013.964..3013.990 rows=10 loops=1)
-> Unique (cost=315191.32..329642.42 rows=170013 width=1691) (actual time=3013.962..3013.986 rows=10 loops=1)
-> Sort (cost=315191.32..315616.35 rows=170013 width=1691) (actual time=3013.961..3013.970 rows=10 loops=1)
Sort Key: updated_at DESC, id, uuid, feedid, cve_id, cveid, summary, published, modified, assigner, cwe_id, vulnerable_packages_versions, vulnerable_products, vulnerable_product_versions, cvss, cvss_time, cvss_version, cvss_vector, cvss_metrics, access, impact, cvss3, cvss3_vector, cvss3_version, cvss3_metrics, score, is_exploitable, is_confirmed, is_in_the_news, is_in_the_wild, reflinks, reflinkids, created_at
Sort Method: external merge Disk: 277648kB
-> Seq Scan on vulns (cost=0.00..50542.19 rows=170013 width=1691) (actual time=0.044..836.597 rows=169846 loops=1)
Filter: ((score >= 0) AND (score <= 100))
Planning time: 3.183 ms
Execution time: 3070.346 ms
I have created a btree varchar index btree_varchar" btree (history_type varchar_pattern_ops) like this:
CREATE INDEX CONCURRENTLY btree_varchar ON vulns_historicalvuln (history_type varchar_pattern_ops);
I have also created a index for vulns score for my second queries:
CREATE INDEX CONCURRENTLY ON vulns (score);
I read a lot of post and documentation about slow queries and index. I'am sure it's the solution about slow queries but the query plan of Postgresql doesn't use the index I have created. It estimates that it processes faster with seq scan than using the index...
SELECT relname, indexrelname, idx_scan FROM pg_catalog.pg_stat_user_indexes;
relname | indexrelname | idx_scan
-------------------------------------+-----------------------------------------------------------------+------------
vulns_historicalvuln | btree_varchar | 0
Could you tell me if my index is well designed ? How I can debug this, feel free to ask more information if needed.
Thanks
After some research, I understand that index is not the solution of my problem here.
Low_cardinality (repeated value) of this field make the index useless.
The time of the query postgresql here is normal because of 30M rows matched.
I close this issue because there is no problem with index here.
Database: Postgres
I have a product(id, title, source, ...) table which contains almost 500K records.
An example of data is:
| Id | title | source |
|:---|---------:|:--------:|
| 1 | product1 | source1 |
| 2 | product2 | source1 |
| 3 | product3 | source1 |
| 4 | product4 | source1 |
| . | ........ | source1 |
| . | ........ | source2 |
| x | productx | source2 |
|x+n |productX+n| sourceN |
There are are 5 distinct source values. And all records have source values random.
I need to get paginated results in such a way that:
If I need to select 20 products then the results set should contain results equally distributed based on source and should be in a repeating sequence. 2 products from each source till the last source and again next 2 products from each source.
For example:
| # | title | source |
|:---|---------:|:--------:|
| 1 | product1 | source1 |
| 2 | product2 | source1 |
| 3 | product3 | source2 |
| 4 | product4 | source2 |
| 5 | product5 | source3 |
| 6 | product6 | source3 |
| 7 | product7 | source4 |
| 8 | product8 | source4 |
| 9 | product9 | source5 |
| 10 |product10 | source5 |
| 11 | ........ | source1 |
| 12 | ........ | source1 |
| 13 | ........ | source2 |
| 14 | ........ | source2 |
| .. | ........ | ....... |
| 20 | ........ | source5 |
What is the optimized PgSql query to achieve the above scenario considering the LIMIT, OFFSET, sources can increase or decrease?
EDIT
As Suggested by George S, the below solution works, however, it is less performant. it takes almost 6 seconds to select only 20 records.
select id, title, source
, (row_number() over(partition by source order by last_modified DESC) - 1) / 2 as ordinal
-- order here can be by created time, id, title, etc
from product p
order by ordinal, source
limit 20
offset 2;
Explain ANALYZE of above query on real data
Limit (cost=147621.60..147621.65 rows=20 width=92) (actual time=5956.126..5956.138 rows=20 loops=1)
-> Sort (cost=147621.60..148813.72 rows=476848 width=92) (actual time=5956.123..5956.128 rows=22 loops=1)
Sort Key: (((row_number() OVER (?) - 1) / 2)), provider
Sort Method: top-N heapsort Memory: 28kB
-> WindowAgg (cost=122683.80..134605.00 rows=476848 width=92) (actual time=5099.059..5772.821 rows=477731 loops=1)
-> Sort (cost=122683.80..123875.92 rows=476848 width=84) (actual time=5098.873..5347.858 rows=477731 loops=1)
Sort Key: provider, last_modified DESC
Sort Method: external merge Disk: 46328kB
-> Seq Scan on product p (cost=0.00..54889.48 rows=476848 width=84) (actual time=0.012..4360.000 rows=477731 loops=1)
Planning Time: 0.354 ms
Execution Time: 5961.670 ms
This can be accomplished easily with a window function:
select id, title, source
, (row_number() over(partition by source order by id) - 1) / 2 as ordinal
--ordering here can be by created time, id, title, etc
from product p
order by ordinal, source
limit 10
offset 2;
As you noted, depending on your table size and other filters being used this may or may not be performant. The best way to tell is to run an EXPLAIN ANALYZE with the query on your actual data. If this isn't performant, you can also add the ordinal field to the table itself if it will always be the same value / ordering. Sadly, you can't create an index using a window function (at least not in PG12).
If you don't want to change the table itself, you can create a materialized view and then query that view so that the calculation only has to be done once:
CREATE MATERIALIZED VIEW ordered_product AS
select id, title, source
, (row_number() over(partition by source order by id) - 1) / 2 as ordinal
from product;
Afterwards, you can query the view like a normal table:
select * from ordered_product order by ordinal, source limit 10 offset 20;
You can also create indexes for it if necessary. Note that to refresh the view you'd run a command like:
REFRESH MATERIALIZED VIEW ordered_product;
After updating postgres, I noticed that one of the queries I was using became much slower. After running EXPLAIN ANALYZE I see that it is now using a different index on the same query.
Among other columns, my table has an applicationid column which is a foreign key BIGINT, and I have an attributes columns which is a jsonb key/value map.
The description on coupons table are (some irrelevant parts were omitted):
+------------------------+--------------------------+-------------------------------------------------------+
| Column | Type | Modifiers |
|------------------------+--------------------------+-------------------------------------------------------|
| id | integer | not null default nextval('coupons_id_seq'::regclass) |
| created | timestamp with time zone | not null default now() |
| campaignid | bigint | |
| value | text | |
| expirydate | timestamp with time zone | |
| startdate | timestamp with time zone | |
| attributes | jsonb | not null default '{}'::jsonb |
| applicationid | bigint | |
| deleted | timestamp with time zone | |
| deleted_changelogid | bigint | not null default 0 |
| accountid | bigint | not null |
| recipientintegrationid | text | |
+------------------------+--------------------------+-------------------------------------------------------+
Indexes:
"coupons_applicationid_value_idx" UNIQUE, btree (applicationid, value) WHERE deleted IS NULL
"coupons_attrs_index" gin (attributes)
"coupons_recipientintegrationid_idx" btree (recipientintegrationid)
"coupons_value_trgm_idx" gin (value gin_trgm_ops)
The query I'm running is (some irrelevant parts were omitted):
EXPLAIN ANALYZE SELECT
*,
COUNT(*) OVER () AS total_rows
FROM
coupons
WHERE
deleted IS NULL
AND coupons.applicationid = 2
AND coupons.attributes #> '{"SessionId":"1070695459"}'
ORDER BY
id ASC
LIMIT 1000;
applicationid doesn't help us much. The index that was previously used was coupons_attrs_index (over attributes column) which produced very good results.
After the update however, the query planner started preferring the index coupons_applicationid_value_idx for some reason!
Here is output from EXPLAIN ANALYZE (some irrelevant parts were omitted):
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| QUERY PLAN |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| -> Sort (cost=64.09..64.10 rows=1 width=237) (actual time=3068.996..3068.996 rows=0 loops=1) |
| Sort Key: coupons.id |
| Sort Method: quicksort Memory: 25kB |
| -> WindowAgg (cost=0.86..64.08 rows=1 width=237) (actual time=3068.988..3068.988 rows=0 loops=1) |
| -> Nested Loop (cost=0.86..64.07 rows=1 width=229) (actual time=3068.985..3068.985 rows=0 loops=1) |
| -> Index Scan using coupons_applicationid_value_idx on coupons (cost=0.43..61.61 rows=1 width=213) (actual time=3068.984..3068.984 rows=0 loops=1) |
| Index Cond: (applicationid = 2) |
| Filter: (attributes #> '{"SessionId": "1070695459"}'::jsonb) |
| Rows Removed by Filter: 2344013 |
| Planning Time: 0.531 ms |
| Execution Time: 3069.076 ms |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
EXPLAIN
Time: 3.159s (3 seconds), executed in: 3.102s (3 seconds)
Can anyone help me understand why the query planner uses a less efficient index (coupons_applicationid_value_idx instead of coupons_attrs_index) after the update?
After adding a mixed (BTREE + GIN) index on (applicationid, attributes) that index was selected effectively solving the issue. I would still like to understand what happened to predict issues like this one in the future.
[EDIT 31-01-20 11:02]: The issue returned after 24 hours. Again the wrong index was chosen by the planner and the query became slow. Running a simple analyze solved it.
It is still very strange that it only started happening after the update to PG 11.
I have a query which is performing very poorly.
It's too big to post it here but basically it selects several columns and checks to which range they belong e.g.
CASE WHEN col < 3 THEN 'A' WHEN col BETWEEN 3 AND 5 THEN 'B' etc.
And then it counts 'A', 'B' etc
The execution time for 380M rows is 15 min.
It has no DISTINCT and no joins except of join to a small dimension table.
When I run explain it shows the following query plan
+-----------------------------------------------------------------------------------------------------------------+
| QUERY PLAN |
|-----------------------------------------------------------------------------------------------------------------|
| XN Merge (cost=1000011702317.03..1000011702317.16 rows=53 width=48) |
| Merge Key: programme_session.start_date |
| -> XN Network (cost=1000011702317.03..1000011702317.16 rows=53 width=48) |
| Send to leader |
| -> XN Sort (cost=1000011702317.03..1000011702317.16 rows=53 width=48) |
| Sort Key: programme_session.start_date |
| -> XN HashAggregate (cost=11702160.75..11702315.51 rows=53 width=48) |
| -> XN Hash Join DS_DIST_ALL_NONE (cost=105.08..11499929.95 rows=2186279 width=48) |
| Hash Cond: ("outer".start_date = "inner".tk) |
| -> XN Seq Scan on programme_session (cost=0.00..5101316.48 rows=510131648 width=48) |
| -> XN Hash (cost=105.00..105.00 rows=30 width=4) |
| -> XN Seq Scan on dim_date dd (cost=0.00..105.00 rows=30 width=4) |
| Filter: (("year" = 2019) AND ("month" = 6)) |
+-----------------------------------------------------------------------------------------------------------------+
Here I see the explosion of cost from XN HashAggregate to XN Sort.
What could be the reason?
The table is sorted by start_date, I run VACUUM SORT ONLY and skew values are not very big:
+-------------------+-----------+----------------+------------+-----------------+-------------+
| table | encoded | diststyle | sortkey1 | skew_sortkey1 | skew_rows |
|-------------------+-----------+----------------+------------+-----------------+-------------|
| programme_session | Y | KEY(device_id) | start_date | 2.26 | 1.00 |
+-------------------+-----------+----------------+------------+-----------------+-------------+
How can I improve the performance of my query?
Check the data types for the columns in your join clause. I've seen redshift broadcast the transaction table when they are not identical.
In a PostgreSQL 9.4.0 database I have a busy table with 22 indexes which are larger than the actual data in the table.
Since most of these indexes are for columns which are almost entirely NULL, I've been trying to replace some of them with partial indexes.
One of the columns is: auto_decline_at timestamp without time zone. This has 5453085 NULLS out of a total 5457088 rows.
The partial index replacement is being used, but according to the stats, the old index is also still in use, so I am afraid to drop it.
Selecting from pg_tables I see:
tablename | indexname | num_rows | table_size | index_size | unique | number_of_scans | tuples_read | tuples_fetched
-----------+---------------------------------------+-------------+------------+------------+--------+-----------------+-------------+----------------
jobs | index_jobs_on_auto_decline_at | 5.45496e+06 | 1916 MB | 3123 MB | N | 17056009 | 26506058607 | 26232155810
jobs | index_jobs_on_auto_decline_at_partial | 5.45496e+06 | 1916 MB | 120 kB | N | 6677 | 26850779 | 26679802
And a few minutes later:
tablename | indexname | num_rows | table_size | index_size | unique | number_of_scans | tuples_read | tuples_fetched
-----------+---------------------------------------+-------------+------------+------------+--------+-----------------+-------------+----------------
jobs | index_jobs_on_auto_decline_at | 5.45496e+06 | 1916 MB | 3124 MB | N | 17056099 | 26506058697 | 26232155900
jobs | index_jobs_on_auto_decline_at_partial | 5.45496e+06 | 1916 MB | 120 kB | N | 6767 | 27210639 | 27039623
So number_of_scans is increasing for both of them.
The index definitions:
"index_jobs_on_auto_decline_at" btree (auto_decline_at)
"index_jobs_on_auto_decline_at_partial" btree (auto_decline_at) WHERE auto_decline_at IS NOT NULL
The only relevant query I can see in my logs follows this pattern:
SELECT "jobs".* FROM "jobs" WHERE (jobs.pending_destroy IS NULL OR jobs.pending_destroy = FALSE) AND "jobs"."deleted_at" IS NULL AND (state = 'assigned' AND auto_decline_at IS NOT NULL AND auto_decline_at < '2015-08-17 06:57:22.325324')
EXPLAIN ANALYSE gives me the following plan, which uses the partial index as expected:
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using index_jobs_on_auto_decline_at_partial on jobs (cost=0.28..12.27 rows=1 width=648) (actual time=22.143..22.143 rows=0 loops=1)
Index Cond: ((auto_decline_at IS NOT NULL) AND (auto_decline_at < '2015-08-17 06:57:22.325324'::timestamp without time zone))
Filter: (((pending_destroy IS NULL) OR (NOT pending_destroy)) AND (deleted_at IS NULL) AND ((state)::text = 'assigned'::text))
Rows Removed by Filter: 3982
Planning time: 2.731 ms
Execution time: 22.179 ms
(6 rows)
My questions:
Why is index_jobs_on_auto_decline_at still being used?
Could this same query sometimes use index_jobs_on_auto_decline_at, or is there likely to be another query I am missing?
Is there a way to log the queries which are using index_jobs_on_auto_decline_at?