PostgreSQL - how to improve this query/index - postgresql

Question

PostgresSQL 11.2
Settings:
shared_buffers = 1024MB
effective_cache_size = 2048MB
maintenance_work_mem = 320MB
checkpoint_completion_target = 0.5
wal_buffers = 3932kB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 64MB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
I've got a table with about 40M rows.
The query I'm doing on it(more fields exist in the query, it's the where clauses that count):
select id,name from my_table where
action_performed = true AND
should_still_perform_action = false AND
action_performed_at <= '2021-09-05 00:00:00.000'
LIMIT 100;
The date is just something I picked for this example.
The point is to get items that need to be processed. The client retrieving this data would then use the metadata to find a file and upload it to a cloud provider. This can take some time.
The timestamp condition is really only there to say "only process entries older than today" or the given timestamp, in general. The order in which they are returned is of no practical importance, since the goal is to perform processing on any entry that has not yet been processed. The LIMIT was introduced to stop the application doing so from hanging, because of the network activity.
Table definition(redacted):
Table "public.my_table"
action_performed_at | timestamp without time zone | | | now()
should_still_perform_action | boolean | | not null | true
action_performed | boolean | | not null | false
Indexes:
"index001" btree (action_performed_at, should_still_perform_action, action_performed) WHERE should_still_perform_action = false AND action_performed = true
"index002" btree (action_performed, should_still_perform_action, action_performed_at DESC) WHERE should_still_perform_action = false AND action_performed = true
These are all indexes added over time, all worked at the start, but are no longer being used now.
Re-indexing also does not seem to work, only dropping and re-creating them works for a while.
While the table hold 40M rows, the amount of rows matching these conditions is roughly around 100K.
The query plan looks like this:
QUERY PLAN
------------------------------------------------------------------------------------
Limit (cost=0.00..707.80 rows=100 width=3595) (actual time=18520.627..100644.933 rows=100 loops=1)
Buffers: shared hit=0 read=1392361 dirtied=26 written=26
-> Seq Scan on my_table (cost=0.00..4164264.45 rows=5883377 width=3595) (actual time=18520.624..100644.073 rows=100 loops=1)
Filter: (action_performed AND (NOT should_still_perform_action) AND (action_performed_at <= '2021-09-05 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 19846606
Buffers: shared hit=0 read=1392361 dirtied=26 written=26
Planning Time: 63.667ms
Execution Time: 100645.548 ms
(10 rows)
Using the query found here: https://github.com/ioguix/pgsql-bloat-estimation/blob/master/btree/btree_bloat-superuser.sql
This is the result:
current_database | schemaname | tblname | idxname | real_size | extra_size | extra_pct | fillfactor | bloat_size | bloat_pct | is_na
------------------+------------+------------------+---------------------------------------+-------------+-------------+------------------+------------+-------------+------------------+-------
mine | public | my_table | index001 | 343244800 | 341598208 | 99.5202863961814 | 90 | 341426176 | 99.4701670644391 | f
mine | public | my_table | index002 | 3290316800 | 2338521088 | 71.0728245985311 | 90 | 2231902208 | 67.832441180132 | f
And I'm looking for a way to do this better. Sure, I could drop and recreate the index after time I see it slow down, but that's not exactly a good way of doing things.
Changing LIMIT to FETCH changes nothing.
I'm wondering if I can improve this without changing SELECT to FETCH, which I've never used before and I'm not even sure the client can handle.
What should I do here?
EDIT:
After an analyze:
QUERY PLAN
------------------------------------------------------------------------------------
Limit (cost=0.00..690.14 rows=1000 width=3591) (actual time=0.044..5840.228 rows=1000 loops=1)
Buffers: shared hit=3 read=81426 dirtied=18 written=18
-> Seq Scan on my_table (cost=0.00..4163978.60 rows=6033500 width=3591) (actual time=0.034..5839.599 rows=100 loops=1)
Filter: (action_performed AND (NOT should_still_perform_action) AND (action_performed_at <= '2021-09-05 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 953640
Buffers: shared hit=3 read=81426 dirtied=18 written=18
Planning Time: 63.667ms
Execution Time: 100645.548 ms
(10 rows)

Answer 1

The estimate (cost=0.00..4164264.45 rows=5883377 width=3595) shows the planner expects over 5M records to match the criteria. It is significantly different from the expected 100K you mention.
In cases like this `ANALYZE public.my_table;' usually helps. It refreshes statistics for the table data.

Answer 2

Your main problem seems to be index bloat, which is caused by more DELETEs or UPDATEs than autovacuum can clean up. Solve that problem, and you should be fine.
Tune autovacuum to be as fast as possible:
ALTER TABLE my_table SET (autovacuum_vacuum_cost_delay = 0);
Also, give autovacuum enough memory by setting maintenance_work_mem to a high value up to 1GB.
Then rebuild the indexes so that they are no longer bloated. If pgstattuple tells you that the table is bloated too, VACUUM (FULL) that table instead.
Make sure that you don't have long running database transactions most of the time.
By the way, this is the perfect index for this query:
CREATE INDEX ON my_table (action_performed_at)
WHERE action_performed AND NOT should_still_perform_action;

Answer 3

Entire time is taking to scan the table as per the plan. Seq scan is happening, even index is available on the required columns. It seems rows return by this query is quite high, not only 100k rows.
If you check the below condition in the plan, around 20M rows are removed by all 3 filter used in where clause in the query.
Rows Removed by Filter: 19846606
Please check
Why index isn't picking by reviewing the cardinality of the columns,
how many exact rows return by the query and When this table last
analyze.
is Autovaccuum enable in this database? when the last autovaccuum run for this table ?

Answer 4

Because of the statistics that are collected behind the index, the distribution histogram present detailed values only for the first column of the index key.
If this first column has only 2 values the accuracy is inconsistent and the optimizer will create a bad execution plan.
To bypass this trouble, you must place the column action_performed_at as the first column of the index key.
Another point is that you do not need to have column stored in an index with a single value. When you create an index with a WHERE clause that's rely on MyColum = A_Single_Value, you can ignore this column into the index key.
Finally you can use the INCLUDE clause, that MS SQL Server invented 16 years ago and arrived in PostGreSQL, to add some more columns that do not participate in any seek, but is necessary for the SELECT. This will use only the index and do not use a two phase access SEEK for index and SCAN for the table.
So I will try an index like this one :
CREATE INDEX SQLpro__B6B13FC3_6F90_4EEC_BA61_CA6C96C7958A__20210914
ON my_table (action_performed_at)
INCLUDE (id, name)
WHERE action_performed = true AND
should_still_perform_action = true;