PostgreSQL Performance check - postgresql

Question

I have heard PostgreSQL being used in cases where the tables run into billion of rows and having satisfactory response times too. But here's my simple experiment to check this out. I have a table with 6 columns and has 4255700 entries exactly. I have used pgtune to tune the configuration according to my setup. Now, when I run a simple "*select * from tab1*", it is taking 173.425 seconds to fetch all the rows. Is this the normal behavior? I have this single table in the DB.
The table definition is as follows -
CREATE TABLE file_group_permissions
(
fgp_id serial NOT NULL,
file_id integer NOT NULL,
pg_id integer NOT NULL,
policy_id integer,
tag_id integer,
inst_id integer,
CONSTRAINT file_group_permissions_pkey PRIMARY KEY (fgp_id)
)
WITH (
OIDS=FALSE
);
ALTER TABLE file_group_permissions
OWNER TO sa;
-- Index: fgp_file_idx
-- DROP INDEX fgp_file_idx;
CREATE INDEX fgp_file_idx
ON file_group_permissions
USING btree
(file_id);
ALTER TABLE file_group_permissions CLUSTER ON fgp_file_idx;
-- Index: fgp_inst_idx
-- DROP INDEX fgp_inst_idx;
CREATE INDEX fgp_inst_idx
ON file_group_permissions
USING btree
(inst_id);
-- Index: fgp_tag_idx
-- DROP INDEX fgp_tag_idx;
CREATE INDEX fgp_tag_idx
ON file_group_permissions
USING btree
(tag_id);
-- Index: pgfgp_idx
-- DROP INDEX pgfgp_idx;
CREATE INDEX pgfgp_idx
ON file_group_permissions
USING btree
(pg_id);
Output of EXPLAIN (ANALYZE, BUFFERS) select * from file_group_permissions -
"Seq Scan on file_group_permissions (cost=0.00..69662.00 rows=4255700
width=24) (actual time=0.019..580.273 rows=4255700 loops=1)"
" Buffers: shared hit=2432 read=24673"
"Planning time: 0.070 ms"
"Execution time: 903.325 ms"
I have a MacBook Pro with 16 Gigs of RAM and 512 Gigs of SSD. I have configured the PostgreSQL to use 2Gigs of RAM.
Edit
EXPLAIN (ANALYZE, BUFFERS) select pg_id, count(distinct file_id) from file_group_permissions where pg_id in (6117,6115,6116,6113,6114) group by 1;
"GroupAggregate (cost=0.44..102028.21 rows=208 width=8) (actual time=4970.884..5013.423 rows=3 loops=1)"
" Group Key: pg_id"
" Buffers: shared hit=50891, temp read=4824 written=4824"
" -> Index Scan using pgfgp_idx on file_group_permissions (cost=0.44..85511.31 rows=3302964 width=8) (actual time=0.062..1080.926 rows=3323389 loops=1)"
" Index Cond: (pg_id = ANY('{6117,6115,6116,6113,6114}'::integer[]))"
" Buffers: shared hit=50891"
"Planning time: 0.219 ms"
"Execution time: 5013.495 ms"
EDIT1
I separated this table into a new DB and followed the suggestions (composite indices and postgresql conf), here's the new plan -
"GroupAggregate (cost=478307.10..502996.67 rows=209 width=8) (actual
time=7500.426..7528.021 rows=3 loops=1)"
" Group Key: pg_id"
" Buffers: shared read=27105, temp read=12137 written=12137"
" -> Sort (cost=478307.10..486536.26 rows=3291664 width=8) (actual
time=2944.597..3647.248 rows=3323389 loops=1)"
" Sort Key: pg_id"
" Sort Method: external sort Disk: 58488kB"
" Buffers: shared read=27105, temp read=7311 written=7311"
" -> Seq Scan on file_group_permissions (cost=0.00..96260.12
rows=3291664 width=8) (actual time=0.016..1516.743 rows=3323389 loops=1)"
" Filter: (pg_id = ANY
('{6117,6115,6116,6113,6114}'::integer[]))"
" Rows Removed by Filter: 932311"
" Buffers: shared read=27105"
"Planning time: 0.514 ms"
"Execution time: 7540.243 ms"
This table is simply jinxed and it is hurting the performance everywhere it is being joined.

Answer 1

I have created a test table, just like yours, and I have populated it with exactly same amount of records:
insert into file_group_permissions (file_id,pg_id,policy_id,tag_id,inst_id)
select
trunc(random()*10000) as file_id,
trunc(random()*10000) as pg_id,
trunc(random()*10000) as policy_id,
trunc(random()*10000) as tag_id,
trunc(random()*10000) as inst_id
from generate_series(1,4255700) g
When I run your query, it executes pretty fast:
EXPLAIN (ANALYZE, BUFFERS)
select pg_id, count(distinct file_id)
from file_group_permissions
where pg_id in (6117,6115,6116,6113,6114) group by 1;
GroupAggregate (cost=0.43..8.32 rows=5 width=8) (actual time=0.339..1.608 rows=5 loops=1)
Buffers: shared hit=2158
-> Index Scan using pgfgp_idx on file_group_permissions (cost=0.43..8.24 rows=5 width=8) (actual time=0.018..1.170 rows=2147 loops=1)
Index Cond: (pg_id = ANY ('{6117,6115,6116,6113,6114}'::integer[]))
Buffers: shared hit=2158
Total runtime: 1.633 ms
I have notice this line in your execution plan:
Buffers: shared hit=50891, temp read=4824 written=4824
temp read=4824 written=4824 tells us that the database is "using" disk somehow to perform the scan operation. Perhaps you have to tweak some other postgresql.conf parameters, like these of mine:
shared_buffers = 1GB
temp_buffers = 32MB
work_mem = 32MB
effective_cache_size = 1GB