We have a large table (265M rows), which has about 50 columns in total, mostly either integers, dates, or varchars. The table has a primary key defined on an autoincremented column.
A query loads temp table with Pk values (say, 10000-20000 rows) and then the large table is queried by being joined to the temp table.
Average size of the row in the large table is fairly consistent and is around 280 bytes.
Here is the query plan it produces, when running:
Nested Loop (cost=0.57..115712.21 rows=13515 width=372) (actual time=0.016..54797.581 rows=11838 loops=1)
Buffers: shared hit=49960 read=9261, local hit=53
-> Seq Scan on t_ids ti (cost=0.00..188.15 rows=13515 width=4) (actual time=0.006..6.993 rows=11838 loops=1)
Buffers: local hit=53
-> Index Scan using test_pk on test t (cost=0.57..8.55 rows=1 width=368) (actual time=4.624..4.624 rows=1 loops=11838)
Index Cond: (pk = ti.pk)
Buffers: shared hit=49960 read=9261
Planning Time: 0.128 ms
Execution Time: 54801.600 ms
... where test is the large table (actually, clustered by pk) and t_ids is the temporary table.
It seems to be doing the right thing - scanning temp table and hitting the large table on the pk index, 11k times... But it is sooo slow....
Any suggestions on what can be tried to make it run faster are gretly appreciated!
Related
I have the following table for attributes of different objects
create table attributes(id serial primary key,
object_id int,
attribute_id text,
text_data text,
int_data int,
timestamp_data timestamp,
state text default 'active');
an object will have different type of attributes and attribute value will be in one column among text_data or int_data or timestamp_data , depending on attribute data type.
sample data is here
I want to retrieve the records, my query is
select * from attributes
where attribute_id = 55 and state='active'
order by text_data
which is very slow.
increased the work_mem to 1 GB for current session. using set command
SET work_mem TO '1 GB'; to improve the sort method from external merge Disk to quicksort
But no improvement in query execution. Query executed plan is
Gather Merge (cost=750930.58..1047136.19 rows=2538728 width=128) (actual time=18272.405..27347.556 rows=3462116 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=235635 read=201793
-> Sort (cost=749930.56..753103.97 rows=1269364 width=128) (actual time=14299.222..15494.774 rows=1154039 loops=3)
Sort Key: text_data
Sort Method: quicksort Memory: 184527kB
Worker 0: Sort Method: quicksort Memory: 266849kB
Worker 1: Sort Method: quicksort Memory: 217050kB
Buffers: shared hit=235635 read=201793
-> Parallel Seq Scan on attributes (cost=0.00..621244.50 rows=1269364 width=128) (actual time=0.083..3410.570 rows=1154039 loops=3)
Filter: ((attribute_id = 185) AND (state = 'active'))
Rows Removed by Filter: 8652494
Buffers: shared hit=235579 read=201793
Planning Time: 0.453 ms
Execution Time: 29135.285 ms
the query total runtime in 45 sec
Successfully run. Total query runtime: 45 secs 237 msec.
3462116 rows affected.
To improve filtering and query execution time, created index on attribute_id and state
create index attribute_id_state on attributes(attribute_id,state);
Sort (cost=875797.49..883413.68 rows=3046474 width=128) (actual time=47189.534..49035.361 rows=3462116 loops=1)
Sort Key: text_data
Sort Method: quicksort Memory: 643849kB
Buffers: shared read=406048
-> Bitmap Heap Scan on attributes (cost=64642.80..547711.91 rows=3046474 width=128) (actual time=981.857..10348.441 rows=3462116 loops=1)
Recheck Cond: ((attribute_id = 185) AND (state = 'active'))
Heap Blocks: exact=396586
Buffers: shared read=406048
-> Bitmap Index Scan on attribute_id_state (cost=0.00..63881.18 rows=3046474 width=0) (actual time=751.909..751.909 rows=3462116 loops=1)
Index Cond: ((attribute_id = 185) AND (state = 'active'))
Buffers: shared read=9462
Planning Time: 0.358 ms
Execution Time: 50388.619 ms
but query become very slow after creating index.
Table has 29.5 Million rows. text_data is null in 9 Million rows.
Query is returning almost 3 million records, which is 10% of table.
Is there any other index or the other way like changing parameter etc to improve the query ?
Some suggestions:
ORDER BY clauses can be accelerated by indexes. So if you put your ordering column in your compound index you may get things to go much faster.
CREATE INDEX attribute_id_state_data
ON attributes(attribute_id, state, text_data);
This index is redundant with the one in your question, so drop that one when you create this one.
You use SELECT *, a notorious performance and maintainability antipattern. You're much better off naming the columns you want. This is especially important when your result sets are large: why waste CPU and network resources on data you don't need in your application? So, let's assume you want to do this. If you don't need all those columns, remove some of them from this SELECT.
SELECT id, object_id, attribute_id, text_data, int_data,
timestamp_data, state ...
You can use the INCLUDE clause on your index so it covers your query, that is so the query can be satisfied entirely from the index.
CREATE INDEX attribute_id_state_data
ON attributes(attribute_id, state, text_data)
INCLUDE (id, object_id, int_data, timestamp_data, state)
When you use this BTREE index, your query is satisfied by random-accessing the index to the first eligible row and then scanning the index sequentially. There's no need for PostgreSQL to bounce back to the table's data. It doesn't get much faster than that for a big result set.
If you remove some columns from your SELECT clause, you can also remove them from the index's INCLUDE clause.
You ORDER BY a large-object TEXT column. That's a lot of data to sort in each record, whether during index creation or a query. It's stored out-of-line, so it's not as fast. Can you rework your application to use a limited-length VARCHAR column for this instead? It will be more efficient.
I have a table named snapshots with a column named data in jsonb format
An Index is created on snapshots table
create index on snapshots using(( data->>'creator' ));
The following query was using index initially but not after couple of days
SELECT id, data - 'sections' - 'sharing' AS data FROM snapshots WHERE data->>'creator' = 'abc#email.com' ORDER BY xmin::text::bigint DESC
below is the output by running explain analyze
Sort (cost=19.10..19.19 rows=35 width=77) (actual time=292.159..292.163 rows=35 loops=1)
Sort Key: (((xmin)::text)::bigint) DESC
Sort Method: quicksort Memory: 30kB
-> Seq Scan on snapshots (cost=0.00..18.20 rows=35 width=77) (actual time=3.500..292.104 rows=35 loops=1)
Filter: ((data ->> 'creator'::text) = 'abc#email.com'::text)
Rows Removed by Filter: 152
Planning Time: 0.151 ms
Execution Time: 292.198 ms
A table with 187 rows is very small. For very small tables, a sequential scan is the most efficient strategy.
What is surprising here is the long duration of the query execution (292 milliseconds!). Unless you have incredibly lame or overloaded storage, this must mean that the table is extremely bloated – it is comparatively large, but almost all pages are empty, with only 187 live rows. You should rewrite the table to compact it:
VACUUM (FULL) snapshots;
Then the query will become must faster.
I'm running Postgres 11.
I have a table with 1.000.000 (1 million) rows and each row has a size of 40 bytes (it contains 5 columns). That is equal to 40MB.
When I execute (directly executed on the DB via DBeaver, DataGrid ect.- not called via Node, Python ect.):
SELECT * FROM TABLE
it takes 40 secs first time (is this not very slow even for the first time).
The CREATE statement of my tables:
CREATE TABLE public.my_table_1 (
c1 int8 NOT NULL GENERATED ALWAYS AS IDENTITY,
c2 int8 NOT NULL,
c3 timestamptz NULL,
c4 float8 NOT NULL,
c5 float8 NOT NULL,
CONSTRAINT my_table_1_pkey PRIMARY KEY (id)
);
CREATE INDEX my_table_1_c3_idx ON public.my_table_1 USING btree (c3);
CREATE UNIQUE INDEX my_table_1_c2_idx ON public.my_table_1 USING btree (c2);
On 5 random tables: EXPLAIN (ANALYZE, BUFFERS) select * from [table_1...2,3,4,5]
Seq Scan on table_1 (cost=0.00..666.06 rows=34406 width=41) (actual time=0.125..7.698 rows=34406 loops=1)
Buffers: shared read=322
Planning Time: 15.521 ms
Execution Time: 10.139 ms
Seq Scan on table_2 (cost=0.00..9734.87 rows=503187 width=41) (actual time=0.103..57.698 rows=503187 loops=1)
Buffers: shared read=4703
Planning Time: 14.265 ms
Execution Time: 74.240 ms
Seq Scan on table_3 (cost=0.00..3486217.40 rows=180205440 width=41) (actual time=0.022..14988.078 rows=180205379 loops=1)
Buffers: shared hit=7899 read=1676264
Planning Time: 0.413 ms
Execution Time: 20781.303 ms
Seq Scan on table_4 (cost=0.00..140219.73 rows=7248073 width=41) (actual time=13.638..978.125 rows=7247991 loops=1)
Buffers: shared hit=7394 read=60345
Planning Time: 0.246 ms
Execution Time: 1264.766 ms
Seq Scan on table_5 (cost=0.00..348132.60 rows=17995260 width=41) (actual time=13.648..2138.741 rows=17995174 loops=1)
Buffers: shared hit=82 read=168098
Planning Time: 0.339 ms
Execution Time: 2730.355 ms
When I add a LIMIT 1.000.000 to table_5 (it contains 1.7 million rows)
Limit (cost=0.00..19345.79 rows=1000000 width=41) (actual time=0.007..131.939 rows=1000000 loops=1)
Buffers: shared hit=9346
-> Seq Scan on table_5(cost=0.00..348132.60 rows=17995260 width=41) (actual time=0.006..68.635 rows=1000000 loops=1)
Buffers: shared hit=9346
Planning Time: 0.048 ms
Execution Time: 164.133 ms
When I add a WHERE clause between 2 dates (I'm monitored the query below with DataDog software and the results are here (max.~ 31K rows/sec when fetching): https://www.screencast.com/t/yV0k4ShrUwSd):
Seq Scan on table_5 (cost=0.00..438108.90 rows=17862027 width=41) (actual time=0.026..2070.047 rows=17866766 loops=1)
Filter: (('2018-01-01 00:00:00+04'::timestamp with time zone < matchdate) AND (matchdate < '2020-01-01 00:00:00+04'::timestamp with time zone))
Rows Removed by Filter: 128408
Buffers: shared hit=168180
Planning Time: 14.820 ms
Execution Time: 2673.171 ms
All tables has an unique index on the c3 column.
The size of the database is like 500GB in total.
The server has 16 cores and 112GB M2 memory.
I have tried to optimize Postgres system variables - Like: WorkMem(1GB), shared_buffer(50GB), effective_cache_size (20GB) - But it doesn't seems to change anything (I know the settings has been applied - because I can see a big difference in the amount of idle memory the server has allocated).
I know the database is too big for all data to be in memory. But is there anything I can do to boost the performance / speed of my query?
Make sure CreatedDate is indexed.
Make sure CreatedDate is using the date column type. This will be more efficient on storage (just 4 bytes), performance, and you can use all the built in date formatting and functions.
Avoid select * and only select the columns you need.
Use YYYY-MM-DD ISO 8601 format. This has nothing to do with performance, but it will avoid a lot of ambiguity.
The real problem is likely that you have thousands of tables with which you regularly make unions of hundreds of tables. This indicates a need to redesign your schema to simplify your queries and get better performance.
Unions and date change checks suggest a lot of redundancy. Perhaps you've partitioned your tables by date. Postgres has its own built in table partitioning which might help.
Without more detail that's all I can say. Perhaps ask another question about your schema.
Without seeing EXPLAIN (ANALYZE, BUFFERS), all we can do is speculate.
But we can do some pretty good speculation.
Cluster the tables on the index on CreatedDate. This will allow the data to be accessed more sequentially, allowing more read-ahead (but this might not help much for some kinds of storage). If the tables have high write load, they may not stay clustered and so you would have recluster them occasionally. If they are static, this could be a one-time event.
Get more RAM. If you want to perform as if all the data was in memory, then get all the data into memory.
Get faster storage, like top-notch SSD. It isn't as fast as RAM, but much faster than HDD.
EDIT In my original question I noticed a difference between searching for neighbors using a JOIN and using a WHERE .. IN clause, which #LukaszSzozda rightfully pointed out is a semi-join. It turns out my node list had duplicates, which explains why the JOIN took longer to run. Thanks, #LukaszSzozda. The more important aspect of my question remains, though, which is what is brought below. UPDATE I added the relevant configuration options to the bottom, and updated statistics using ANALYZE (thanks to #joop). Also, I tested with three different indices (B-Tree, hash, BRIN). Finally, I noticed that using different queries returned different number of rows into tmp_nodes, possibly because of different ordering, so I fixed it to a constant set of rather-random 8,000 nodes .
In PostgreSQL, my query to search for neighbors of 8,000 nodes among ~200*106 nodes (within ~1.3*109 edges) is slow (~30 seconds using hash index; see index benchmarking below).
Given the setup I describe below, are there further optimizations for my server software, database, table or query to make the neighbor search faster? I am particularly surprised at this speed considering how well PostgreSQL did on the ArangoDB NoSQL benchmark.
More specifically:
I am aware of AgnesGraph, but do not wish yet to move to a graph-database solution, specifically since I cannot tell from AgnesGraph how well it keeps up-to-date with PostgreSQL. Can someone explain the performance benefits with regard to how the query actually happens in AgnesGraph vs PostgreSQL, so that I can decide whether to migrate?
Are there any configuration tweaks, whether in the server or the OS, that affect my query according to the plan, which cause it to run for longer than needed?
Set up
I have a large graph database ( ~109 edges, ~200*106 nodes) in PostgreSQL (PostgreSQL 10.1, which I had to pull from the zesty PPA) stored on the cloud (DigitalOcean, 6-core, 16GB RAM machine, Ubuntu 17.10, Intel(R) Xeon(R) CPU E5-2650 v4 # 2.20GHz), and set up with parameters suggested by PGTune (see bottom). I am querying on-server.
I have created forward- and backward-edges tables (see this question)
CREATE TABLE edges_fwd (src BIGINT, dest BIGINT, PRIMARY KEY (src, dest));
CREATE TABLE edges_back (src BIGINT, dest BIGINT, PRIMARY KEY (dest, src));
and clustered both by the respective keys (just in case):
CLUSTER edges_fwd USING edges_fwd_pkey;
CLUSTER edges_back USING edges_back_pkey;
I turned off enabled_seqscan for the purpose of testing my queries (see side note below).
I would like to load all the out-edges for 8,000 nodes (which 8,000 nodes these are can change depending on user query), whose identifiers are list in a table tmp_nodes (with a single column, nid). I initially wrote this version on the query (patting myself in the back on already following the lines of the graph talk from PGCon11):
SELECT e.*
FROM tmp_nodes
JOIN edges AS e
ON e.src = tmp_nodes.nid;
I also tried:
SELECT * FROM edges_fwd AS e
WHERE e.src IN (SELECT nid FROM tmp_nodes);
They both are slow, and take about 30 seconds to run at best (using hash indicies). EXPLAIN ANALYZE outputs are brought below.
I expected things to generally run much faster. For looking for 8,000 keys in a clustered table (yes, I know it's not really a clustered index), since the server knows that the rows are ordered, I should expect less page reads than the number of total rows returned. So while 243,708 rows are fetched, which isn't a little, they are associated with 8,000 distinct keys, and the number of reads should not be much larger than that: it's an average of 30 rows per key, which is about 1,400 bytes per read (the table size is 56GB and has 1.3B rows, so it's about 46 bytes per row; which by the way is quite a bloat for 16 bytes of data). This is far below the page size (4K) for the system. I didn't think reading 8,000 pages, even random-access, should take this long.
This brings me back to my questions (above).
Forcing index usage
I took advice from answers to another question and at least for testing (though, since my database is read-only, I might be tempted to use it in production), set enable_seqscan to off, in order to force index usage.
I ran each 5 times - the times varied by a few seconds here and there.
EXPLAIN ANALYZE outputs
Taking care to flush OS disk cached and restart the server to reflect correct random-seek timings, I used EXPLAIN ANALYZE on both queries. I used two types of indexes - B-Tree and hash. I also tried BRIN with different values for the pages_per_range option (2, 8, 32 and 128), but they are all slower (in orders or magnitude) than those mentioned above. I am giving the results below for reference.
B-Tree index, JOIN query:
Nested Loop (cost=10000000000.58..10025160709.50 rows=15783833 width=16) (actual time=4.546..39152.408 rows=243708 loops=1)
-> Seq Scan on tmp_nodes (cost=10000000000.00..10000000116.00 rows=8000 width=8) (actual time=0.712..15.721 rows=8000 loops=1)
-> Index Only Scan using edges_fwd_pkey on edges_fwd e (cost=0.58..3125.34 rows=1973 width=16) (actual time=4.565..4.879 rows=30 loops=8000)
Index Cond: (src = tmp_nodes.nid)
Heap Fetches: 243708
Planning time: 20.962 ms
Execution time: 39175.454 ms
B-Tree index, WHERE .. IN query (semi-join):
Nested Loop (cost=10000000136.58..10025160809.50 rows=15783833 width=16) (actual time=9.578..42605.783 rows=243708 loops=1)
-> HashAggregate (cost=10000000136.00..10000000216.00 rows=8000 width=8) (actual time=5.903..35.750 rows=8000 loops=1)
Group Key: tmp_nodes.nid
-> Seq Scan on tmp_nodes (cost=10000000000.00..10000000116.00 rows=8000 width=8) (actual time=0.722..2.695 rows=8000 loops=1
)
-> Index Only Scan using edges_fwd_pkey on edged_fwd e (cost=0.58..3125.34 rows=1973 width=16) (actual time=4.924..5.309 rows=30 loops=8000)
Index Cond: (src = tmp_nodes.nid)
Heap Fetches: 243708
Planning time: 19.126 ms
Execution time: 42629.084 ms
Hash index, JOIN query:
Nested Loop (cost=10000000051.08..10056052287.01 rows=15783833 width=16) (actual time=3.710..34131.371 rows=243708 loops=1)
-> Seq Scan on tmp_nodes (cost=10000000000.00..10000000116.00 rows=8000 width=8) (actual time=0.960..13.338 rows=8000 loops=1)
-> Bitmap Heap Scan on edges_fwd e (cost=51.08..6986.79 rows=1973 width=16) (actual time=4.086..4.250 rows=30 loops=8000)
Heap Blocks: exact=8094
-> Bitmap Index Scan on ix_edges_fwd_src_hash (cost=0.00..50.58 rows=1973 width=0) (actual time=2.563..2.563 rows=31
loops=8000)
Execution time: 34155.511 ms
Hash index, WHERE .. IN query (semi-join):
Nested Loop (cost=10000000187.08..10056052387.01 rows=15783833 width=16) (actual time=12.766..31834.767 rows=243708 loops=1)
-> HashAggregate (cost=10000000136.00..10000000216.00 rows=8000 width=8) (actual time=6.297..30.760 rows=8000 loops=1)
-> Seq Scan on tmp_nodes (cost=10000000000.00..10000000116.00 rows=8000 width=8) (actual time=0.883..3.108 rows=8000 loops=$
)
-> Bitmap Heap Scan on edges_fwd e (cost=51.08..6986.79 rows=1973 width=16) (actual time=3.768..3.958 rows=30 loops=8000)
Heap Blocks: exact=8094
-> Bitmap Index Scan on ix_edges_fwd_src_hash (cost=0.00..50.58 rows=1973 width=0) (actual time=2.340..2.340 rows=31
loops=8000)
Execution time: 31857.692 ms
postgresql.conf settings
I set the following configuration options as suggested by PGTune:
max_connections = 10
shared_buffers = 4GB
effective_cache_size = 12GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
random_page_cost = 4
effective_io_concurrency = 2
work_mem = 69905kB
min_wal_size = 4GB
max_wal_size = 8GB
max_worker_processes = 6
max_parallel_workers_per_gather = 3
max_parallel_workers = 6
You also need indexes to search in the reversed direction:
CREATE TABLE edges_fwd
(src BIGINT
, dest BIGINT
, PRIMARY KEY (src, dest)
);
CREATE UNIQUE INDEX ON edges_fwd(dest, src);
CREATE TABLE edges_back
(src BIGINT
, dest BIGINT
, PRIMARY KEY (dest, src)
);
CREATE UNIQUE INDEX ON edges_back(src, dest);
SELECT fwd.*
FROM edges_back AS bck
JOIN edges_fwd AS fwd
ON fwd.src = bck.src -- bck.src does not have a usable index
WHERE bck.dest = root_id;
The absence of this index causes the hashjoin (or: tablescan)
Also, you could maybe combine the two tables.
Also, you can force the src and dest columns to be NOT NULL
(a null would make no sense in a edges table)
, and make them FOREIGN KEYs to your nodes table:
CREATE TABLE nodes
(nid BIGINT NOT NULL PRIMARY KEY
-- ... more stuff...
);
CREATE TABLE edges_fwd
(src BIGINT NOT NULL REFERENCES nodes(nid)
, dest BIGINT NOT NULL REFERENCES nodes(nid)
, PRIMARY KEY (src, dest)
);
CREATE TABLE edges_back
(src BIGINT NOT NULL REFERENCES nodes(nid)
, dest BIGINT NOT NULL REFERENCES nodes(nid)
, PRIMARY KEY (dest, src)
);
INSERT INTO nodes(nid)
SELECT a
FROM generate_series(1,1000) a -- 1000 rows
;
INSERT INTO edges_fwd(src, dest)
SELECT a.nid, b.nid
FROM nodes a
JOIN nodes b ON random()< 0.1 --100K rows
;
INSERT INTO edges_back(src, dest)
SELECT a.nid, b.nid
FROM nodes a
JOIN nodes b ON random()< 0.1 --100K rows
;
This will result in this plan:
DROP SCHEMA
CREATE SCHEMA
SET
CREATE TABLE
CREATE TABLE
CREATE TABLE
INSERT 0 1000
INSERT 0 99298
INSERT 0 99671
ANALYZE
ANALYZE
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.50..677.62 rows=9620 width=16) (actual time=0.086..5.299 rows=9630 loops=1)
-> Index Only Scan using edges_back_pkey on edges_back bck (cost=0.25..100.07 rows=97 width=8) (actual time=0.053..0.194 rows=96 loops=1)
Index Cond: (dest = 11)
Heap Fetches: 96
-> Index Only Scan using edges_fwd_pkey on edges_fwd fwd (cost=0.25..5.46 rows=99 width=16) (actual time=0.008..0.037 rows=100 loops=96)
Index Cond: (src = bck.src)
Heap Fetches: 9630
Planning time: 0.480 ms
Execution time: 5.836 ms
(9 rows)
It seems that random-access for this kind of setup is just this slow. Running a script to check random-access of 8,000 different, random 4K blocks within a large file takes nearly 30 seconds. Using Linux time and the linked script, I get in average something like 24 seconds:
File size: 8586524825 Read size: 4096
32768000 bytes read
real 0m24.076s
So it seems the assumption that random access should be quicker is wrong. Together with time taken to read the actual index, it means performance is at its peak without a hardware change. To improve performance, I will likely need to use a RAID set-up or a cluster. If a RAID set-up will improve performance in a close-to-linear fashion, I will accept my own answer.
I have one de-normalized table with 40+ columns (~ 1.5 million rows, 1 Gb).
CREATE TABLE tbl1 (
...
division_id integer,
division_name varchar(10),
...
);
I need to speed up query
SELECT DISTINCT division_name, division_id
FROM table
ORDER BY division_name;
Query return only ~250 rows, but very slow cause size of table.
I have tried to create index:
create index idx1 on tbl1 (division_name, division_id)
But current execution plan:
explain analyze SELECT Distinct division_name, division_id FROM tbl1 ORDER BY 1;
QUERY PLAN
-----------------------------------------------------------------
Sort (cost=143135.77..143197.64 rows=24748 width=74) (actual time=1925.697..1925.723 rows=294 loops=1)
Sort Key: division_name
Sort Method: quicksort Memory: 74kB
-> HashAggregate (cost=141082.30..141329.78 rows=24748 width=74) (actual time=1923.853..1923.974 rows=294 loops=1)
Group Key: division_name, division_id
-> Seq Scan on tbl1 (cost=0.00..132866.20 rows=1643220 width=74) (actual time=0.069..703.008 rows=1643220 loops=1)
Planning time: 0.311 ms
Execution time: 1925.883 ms
Any suggestion why index does not work or how I can speed up query in other way?
Server Postgresql 9.6.
p.s. Yes, table has 40+ columns and de-normalized, but I know all pros and cons for with decision.
Update1
#a_horse_with_no_name suggest to use vacuum analyze instead of analyze to update table statistic. Now query plain is:
QUERY PLAN
------------------------
Unique (cost=0.55..115753.43 rows=25208 width=74) (actual time=0.165..921.426 rows=294 loops=1)
-> Index Only Scan using idx1 on tbl1 (cost=0.55..107538.21 rows=1643044 width=74) (actual time=0.162..593.322 rows=1643220 loops=1)
Heap Fetches: 0
Much better!
The index will probably only help if PostgreSQL chooses an “index only scan”, that means that it does not have to look at the table data at all.
Normally PostgreSQL has to check the table data (“heap”) to see if a row is visible for the current transaction, because visibility information is not stored in the index.
If, however, the table does not change much and has recently been VACUUMed, PostgreSQL knows that most of the pages consist only of items visible for everyone (there is a “visibility map” to keep track of that information), and then it might be cheaper to scan the index.
Try running VACUUM on the table and see if that causes an index only scan to be used.
Other than that, there is no way to speed up such a query.