Table orders contains total 1,500,000 toples. After a fresh restart of the system, I ran the following query:
SELECT pg_prewarm('orders');
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE o_totalprice < 100
Which gave a buffer output as following:
Buffers: shared hit=15768 read=10327
The select statement returns no records.
Now my question is, how did PostgreSQL calculate that it will take 15768 blocks in buffer?
Your shared_buffers is set to 128MB, right?
128 MB of shared buffers translates to 16384 blocks of size 8KB in the cache.
So when you run pg_prewarm('orders'), PostgreSQL will read the complete table into shared buffers. Now the table is bigger than your shared_buffers, so the first blocks “drop out” of the cache again when the last blocks are read, because shared_buffers cannot fit them all.
Increase shared_buffers if you want to have the whole table in the cache.
Related
I wanted to migrate from BigQuery to CloudSQL to save cost.
My problem is that CloudSQL with PostgreSQL is very very slow compare to BigQuery.
A query that takes 1.5 seconds in BigQuery takes almost 4.5 minutes(!) on CloudSQL with PostgreSQL.
I have CloudSQL with PostgreSQL server with the following configs:
My database have a main table with 16M rows (around 14GB in RAM).
A example query:
EXPLAIN ANALYZE
SELECT
"title"
FROM
public.videos
WHERE
EXISTS (SELECT
*
FROM (
SELECT
COUNT(DISTINCT CASE WHEN LOWER(param) LIKE '%thriller%' THEN '0'
WHEN LOWER(param) LIKE '%crime%' THEN '1' END) AS count
FROM
UNNEST(categories) AS param
) alias
WHERE count = 2)
ORDER BY views DESC
LIMIT 12 OFFSET 0
The table is a videos tables with categories column as text[].
The search condition here looks where there is a categories which is like '%thriller%' and like '%crime%' exactly two times
The EXPLAIN ANALYZE of this query gives this output (CSV): link.
The EXPLAIN (BUFFERS) of this query gives this output (CSV): link.
Query Insights graph:
Memory profile:
BigQuery reference for the same query on the same table size:
Server config: link.
Table describe: link.
My goal is to have Cloud SQL with the same query speed as Big Query
For anyone coming here wondering how to tune their postgres machine on cloud sql they call it flags and you can do it from the UI although not all the config options are edit able.
https://cloud.google.com/sql/docs/postgres/flags#console
The initial query looks overcomplicated. It could be rewritten as:
SELECT v."title"
FROM public.videos v
WHERE array_to_string(v.categories, '^') ILIKE ALL (ARRAY['%thriller%', '%crime%'])
ORDER BY views DESC
LIMIT 12 OFFSET 0;
db<>fiddle demo
PostGreSQL is very slow by design on every queries involving COUNT aggregate function and there is absolutly nothing to do except materialized view to enforces the performances.
The tests I have made on my machine with 48 cores about COUNT performances compare from PostGreSQL to MS SQL Server is clear : SQL Server is between 61 and 561 times faster in all situations, and with columnstore index SQL Server can be 1,533 time faster…
The same speed is reached when using any other RDBMS. The explanation is clearly the PG MVCC that maintain ghost rows inside table and index pages, that needs to browse every rows to know if it is an active or ghost row... In all the other RDBMS, the count is done by reading only one information at the top of the page (number of rows in the page) and also by using parallelized access or in SQL Server a batch access and not a row access...
There is nothing to do to speed up the count in PG until the storage engine will not been enterely rewrite to avoid ghost slots inside pages...
I believe you need to use a full-text search and the special GIN index. The steps:
Create the helper function for index: CREATE OR REPLACE FUNCTION immutable_array_to_string(text[]) RETURNS text as $$ SELECT array_to_string($1, ','); $$ LANGUAGE sql IMMUTABLE;
Create index itself:
CREATE INDEX videos_cats_fts_idx ON videos USING gin(to_tsvector('english', LOWER(immutable_array_to_string(categories))));
Use the following query:
SELECT title FROM videos WHERE (to_tsvector('english', immutable_array_to_string(categories)) ## (to_tsquery('english', 'thriller & crime'))) limit 12 offset 0;
Be aware that this query has a different meaning for 'crime' and 'thriller'. They are not just substrings. They are tokens in English phrases. But it looks that actually it is better for your task. Also, this index is not good for frequently changed data. It should work fine when you have mostly read-only data.
PS
This answer is inspired by answer & comments: https://stackoverflow.com/a/29640418/159923
Apart from the sql syntax optimization, have you tried Postgresql tune?
I check the explaination has found only two workers in parallel and 25KMemory used in sorting.
Workers Planned: 2"
Sort Method: quicksort Memory: 25kB"
For your query, it is typical OLAP query. it performance usually related the memory(memory and cpu cores used(workers). The default postgres use KB level memory and few workers. You can tune your postgresql.conf to optimized it work as OLAP type database.
===================================================
Here is my suggestion: use more memory(9MB as work mem ) and more cpu(max 16)
# DB Version: 13
# OS Type: linux
# DB Type: dw
# Total Memory (RAM): 24 GB
# CPUs num: 16
# Data Storage: ssd
max_connections = 40
shared_buffers = 6GB
effective_cache_size = 18GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 9830kB
min_wal_size = 4GB
max_wal_size = 16GB
max_worker_processes = 16
max_parallel_workers_per_gather = 8
max_parallel_workers = 16
max_parallel_maintenance_workers = 4
You can add it to you postgresql.conf last line. And restart your postgresql server to make it effect.
To further optimization,
reduce the connection and increase the work_mem.
200* 9830 is about 2GB memory for all connections. If you has less( for example, 100) connections, you can get more memory for query working.
====================================
Regarding using text array type and unnest. you can try to add proper index.
That's all, good luck.
WangYong
I have a table with around 270,000,000 rows and this is how I created it.
CREATE TABLE init_package_details AS
SELECT pcont.package_content_id as package_content_id,
pcont.activity_id as activity_id,
pc.org_id as org_id,
pc.bed_type as bed_type,
pc.is_override as is_override,
pmmap.package_id as package_id,
pcont.activity_qty as activity_qty,
pcont.charge_head as charge_head,
pcont.activity_charge as charge,
COALESCE(pc.charge,0) - COALESCE(pc.discount,0) as package_charge
FROM a pc
JOIN b od ON
(od.org_id = pc.org_id AND od.status='A')
JOIN c pm ON
(pc.package_id=pm.package_id)
JOIN d pmmap ON
(pmmap.pack_master_id=pm.package_id)
JOIN e pcont ON
(pcont.package_id=pmmap.package_id);
I need to build index on the init_package_details table.
This table is getting created at around 5-6 mins.
I have created btree index like,
CREATE INDEX init_package_details_package_content_id_idx
ON init_package_details(package_content_id);`
which is taking 10 mins (More than the time to create and populate the table itself)
And, when I create another index like,
CREATE INDEX init_package_details_package_act_org_bt_id_idx
ON init_package_details(activity_id,org_id,bed_type);
It just freezes and taking forever to complete. I waited for around 30 mins before I manually cancelled it.
Below are stats from iotop -o if it helps,
When I created table Averaging around 110-120 MB/s (This is how 270 million rows got inserted in 5-6 mins)
When I created First Index, It was averaging at around 70 MB/s
On second index, it is snailing at 5-7 MB/s
Could someone explain Why is this happening? Is there anyway I can speedup the index creations here?
EDIT 1: There are no other connections accessing the table. And, pg_stat_activity shows active as status throughout the running time. This happens inside a transaction (this is happening between BEGIN and COMMIT, it contains many other scripts in same .sql file).
EDIT 2:
postgres=# show work_mem ;
work_mem
----------
5MB
(1 row)
postgres=# show maintenance_work_mem;
maintenance_work_mem
----------------------
16MB
Building indexes takes a long time, that's normal.
If you are not bottlenecked on I/O, you are probably on CPU.
There are a few things to improve the performance:
Set maintenance_work_mem very high.
Use PostgreSQL v11 or better, where several parallel workers can be used.
My db size is growing up to 41GB but when I create a backup it becomes 2.4 / 2.5GB, Why this much difference?
I have performed some vaccum, Reindexing etc. but still the size is not shrinking and it is showing 19 GB temp file but when I check in C:\Program Files\PostgreSQL\10\data\base\pgsql_tmp the folder is empty..
How can I delete those temp files? Is there a way to reduce my DB size?
EDIT
Query to get temp files
SELECT temp_files AS "Temporary files"
, temp_bytes AS "Size of temporary files"
FROM pg_stat_database db;
The values in pg_stat_database are cummulative since the server was started, so it's not the number of bytes currently used. It's the number of bytes used since the statistics were reset.
As this is a cumulative number, it will increase constantly and will never shrink unless you reset the statistics.
You can reset those statistics (as the superuser) using
select pg_stat_reset();
It seems that your queries are indeed using many temp files and temp space. That could indicate that you have not given Postgres enough memory (e.g. work_mem or temp_buffers) But that is a different question.
Those temp file are also not part of a database dump as they are only used during query execution.
I am using postgres-9.3 (in CenOS 6.9) and trying to understand the pg_buffercache table output.
I ran this:
SELECT c.relname,count(*) AS buffers FROM pg_class c INNER JOIN
pg_buffercache b ON b.relfilenode=c.relfilenode INNER JOIN
pg_database d ON (b.reldatabase=d.oid AND
d.datname=current_database()) GROUP BY c.relname
ORDER BY 2 DESC LIMIT 5;
and the output below showed one of the tables using 6594 buffers. This was during when I had tons of INSERT followed by SELECT and UPDATE in the data_main table).
relname | buffers
------------------+---------
data_main | 6594
objt_main | 1897
objt_access | 788
idx_data_mai | 736
I also ran "select * from pg_buffercache where is dirty" which showed around 50 entries.
How should I interpret these numbers? Does the buffer count correspond to all the transactions since I created the extension or the recent ones. How can I find out if my specific operation using the proper amount of buffers?
Here's my setting:
# show shared_buffers;
shared_buffers
----------------
1GB
# show work_mem;
work_mem
----------
128kB
# show maintenance_work_mem;
maintenance_work_mem
----------------------
64GB
And the current free mem (I have 64GM memory in this machine). And I have a mixed workload machine with period bursts of INSERTS and lots of SELECTS. Currently the database and tables are small but will grow to at least 2 million rows.
$ free -m
total used free shared buffers cached
Mem: 64375 33483 30891 954 15 15731
/+ buffers/cache: 18097 46278
Swap: 32767 38 32729
Basically, I am trying to understand how to properly use this pg_buffercache table. Should I ran this query periodically? And do I need to change my shared_buffers accordingly.
I did some reading and testing and this is what I have found. Found a userful query here: How large is a "buffer" in PostgreSQL
Here are a few notes for others that have similar questions.
You will need to create the extension for each database. So "\c db_name" then "create extension pg_buffercache".
Same for running the queries.
Restarting the database clears the queries.
This is a follow-up to the question at Slow Postgres 9.3 queries.
The new indexes definitely help. But what we're seeing is sometimes queries are much slower in practice than when we run EXPLAIN ANALYZE. An example is the following, run on the production database:
explain analyze SELECT * FROM messages WHERE groupid=957 ORDER BY id DESC LIMIT 20 OFFSET 31980;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=127361.90..127441.55 rows=20 width=747) (actual time=152.036..152.143 rows=20 loops=1)
-> Index Scan Backward using idx_groupid_id on messages (cost=0.43..158780.12 rows=39869 width=747) (actual time=0.080..150.484 rows=32000 loops=1)
Index Cond: (groupid = 957)
Total runtime: 152.186 ms
(4 rows)
With slow query logging turned on, we see instances of this query taking over 2 seconds. We also have log_lock_waits=true, and no slow locks are reported around the same time. What could explain the vast difference in execution times?
LIMIT x OFFSET y generally performs not much faster than LIMIT x + y. A large OFFSET is always comparatively expensive. The suggested index in the linked question helps, but while you cannot get index-only scans out of it, Postgres still has to check visibility in the heap (the main relation) for at least x + y rows to determine the correct result.
SELECT *
FROM messages
WHERE groupid = 957
ORDER BY id DESC
LIMIT 20
OFFSET 31980;
CLUSTER on your index (groupid,id) would help to increase locality of data in the heap and reduce the number of data pages to be read per query. Definitely a win. But if all groupid are equally likely to be queried, that's not going to remove the bottleneck of too little RAM for cache. If you have concurrent access, consider pg_repack instead of CLUSTER:
Optimize Postgres timestamp query range
Do you actually need all columns returned? (SELECT *) A covering index enabling index-only scans might help if you only need a few small columns returned. (autovacuum must be strong enough to cope with writes to the table, though. Read-only table would be ideal.)
Also, according to your linked question, your table is 32 GB on disk. (Typically a bit more in RAM). The index on (groupid,id) adds another 308 MB at least (without any bloat):
SELECT pg_size_pretty(7337880.0 * 44); -- row count * tuple size
Making sense of Postgres row sizes
You have 8 GB RAM, of which you expect around 4.5 GB to be used for cache (effective_cache_size = 4608MB). That's enough to cache the index for repeated use, but not nearly enough to also cache the whole table.
If your query happens to find data pages in cache, it's fast. Else, not so much. Big difference, even with SSD storage (much more with HDD).
Not directly related to this query, but 8 MB of work_mem (work_mem = 7864kB) seems way to small for your setup. Depending on various other factors I would set this to at least 64MB (unless you have many concurrent queries with sort / hash operations). Like #Craig commented, EXPLAIN (BUFFERS, ANALYZE) might tell us more.
The best query plan also depends on value frequencies. If only few rows pass the filter, the result might be empty for certain groupid and the query is comparatively fast. If a large portion of the table has to be fetched, a plain sequential scan wins. You need valid table statistics (autovacuum again). And possibly a larger statistics target for groupid:
Keep PostgreSQL from sometimes choosing a bad query plan
Since OFFSET is slow, an alternative is to simulate OFFSET using another column and some index preparation. We require a UNIQUE column (like a PRIMARY KEY) on the table. If there is none, one can be added with:
CREATE SEQUENCE messages_pkey_seq ;
ALTER TABLE messages
ADD COLUMN message_id integer DEFAULT nextval('messages_pkey_seq');
Next we create the position column for the OFFSET simulation:
ALTER TABLE messages ADD COLUMN position INTEGER;
UPDATE messages SET position = q.position FROM (SELECT message_id,
row_number() OVER (PARTITION BY group_id ORDER BY id DESC) AS position
FROM messages ) AS q WHERE q.message_id=messages.message_id ;
CREATE INDEX ON messages ( group_id, position ) ;
Now we are ready for the new version of the query in the OP:
SELECT * FROM messages WHERE group_id = 957 AND
position BETWEEN 31980 AND (31980+20-1) ;