We have a query we have been using for several months. It recently started hanging. The query is a join from 4 tables. One of those tables is only a few thousand records, one a hundred thousand, and 2 are about 2 million. It has been running in about 20 seconds for several months.
After several attempts to identify the issue by adding indexes to unindexed fields with no avail we modified one of the large tables to be a sub query with the result of 100000 records versus 2 million. The query now runs in 20 about seconds.
Explain of the query that hangs produces:
Limit (cost=1714850.81..1714850.81 rows=1 width=79)
While explain of the query that runs in 20 seconds produces:
Limit (cost=1389451.40..1389451.40 rows=1 width=79)
The query that hangs is larger, but does not indicate a significant difference.
Questions:
Are there thresh holds in postgres that cause it to use system
resources differently, i.e. disk buffering versus memory buffering?
The query that hangs shows one cpu with 100% usage. The system is
Linux. Iotop does not show extraordinary io usage. The system has 32
GB RAM and 8 processors. Postgres does not appear to by loading the
system heavily.
Are there other tools that can be applied? The query sub select
worked in this case but we may not be able to reduce a join
dimensions in this way in the future.
As a note the full explain does not show a markedly different execution plan.
Thanks, Dan
Related
So I am writing some performance tests for our application and I am having the following behavior.
Let's image I have N unrelated documents to query(in the example they will be in the same collection, but in general they can be in different collections).
Instead of querying them 1 by 1 i would like to query them all in parallel. Since each firestore query depends on the size of the result set they all should take approximately the same time. So running them in parallel should be approximately the same as running a single one(well a small overhead at best).
My results are the following:
1 document -> around 200ms
10 documents -> around 400 ms
50 documents -> around 1 second
100 documents -> around 2 seconds
I am on 100/50 Mbps network so it's not a bandwidth problem. (documents are about 100kb each)
Does anybody have an idea why does this happen?
Thank you
The problem was in my testing I used names for documents which were lexicographically close so there was a read hot spot.
After generating random names, the results are as expected (around 600-800ms) for 100 documents(so a small overhead, and a there is a bit more variability in the results).
More info is available here: https://firebase.google.com/docs/firestore/best-practices#high_read_write_and_delete_rates_to_a_narrow_document_range
We need to store 1 billion documents of 1KB each. Each shard is planned to have 8GB of RAM. The platform is Open Shift Red Hat Linux.
Initially we had 10 shards for 300 million. We started inserting documents with 2000 inserts/second. Everything went well till 250 million. After that the insert slowed down drastically to 300/400 insert per second.
The queries are also taking long time (more than 1 minute) even all the queries are covered queries.(Queries which need to scan all the indexes).
Hence we assumed, that 20 million per shard is the optimal value and hence we require 50 shards for the current hardware to achieve 1 billion.
Is this reasonable estimate or we can improve it (less shards) by tweaking mongo db parameters for better performance with the current hardware?
There are two compound indexes and one unique index(long).insertion is done using bulk write( with unordered option) with 10 threads and 200 records per (thread) bulk write using java script directly on the mongos.Shardkey is nodeId(prefix of compound index) which has cardinality upto 10k. For 300 million, the total index size comes to 45 GB.40 GB for the 2 compound indexes.Almost 9500 chunks are distributed across 10 nodes.One interesting fact is that if I increase RAM to 12 GB, the speed increases to 1500 inserts/sec.Is RAM limiting factor?
Update:
Using mongostat tool, we found that the flush(fysnc) takes more than 55 seconds to complete.MongoDB cluster runs on kubernetes based on RedHat OpenShift platform. It runs on Dell EMC server with NFS (EXT4 disk format).Is it a problem in the I/O that it supports only 2MB/second. It takes 60 seconds to write 2000 records per second and another 55 seconds to flush completely to disk.(during which all the operations of DB are blocked)
The disk utilization does not even reach 4 %.
Have you tried not sharding at all?
There's a common tendency to shard prematurely. I've seen a MongoDB consultant who suggested a rule of thumb, to not shard until your total data size is at least 2 TB. Your 1B documents of 1KB each should be around 1 TB. While it's only a rule of thumb, maybe it's worth trying.
If nothing else, it'll be much simpler to design the db without sharding and performance will be much more predictable.
I know that count(*) in Postgres is generally slow, however I have a database where it's super slow. I'm talking about minutes even hours.
There is approximately 40M rows in a table and the table consists of 29 columns (most of the are text, 4 are double precision). There is an index on one column which should be unique and I've already run vacuum full. It took around one hour to complete but without no observable results.
Database uses dedicated server with 32GB ram. I set shared_buffers to 8GB and work_mem to 80MB but no speed improvement. I'm aware there are some techniques to get approximated count or to use external table to keep the count but I'm not interested in the count specifically, I'm more concerned about performance in general, since now it's awful. When I run the count there are no CPU peeks or something. Could someone point where to look? Can it be that data are structured so badly that 40M rows are too much for postgres to handle?
Postgres 9.6; Centos 6.7 ; 24 cores
BigTable1 contains 1,500,000,000 rows; weight 180GB.
max_worker_processes = 20
max_parallel_workers_per_gather = 12
1)
When running
EXPLAIN
SELECT
date_id, id1, id2, id3, id4, topdomain, ftype, SUM(imps), SUM(cls)
FROM BigTable1
WHERE
date_id BETWEEN 2017021200 AND 2017022400
AND date_id BETWEEN 2017020000 AND 2017029999
GROUP BY
date_id, id1, id2, id3, id4, topdomain, ftype;
No “Workers Planned:” used at all. Why?
2)
When running the same query when in the session defined
set max_parallel_workers_per_gather = 5;
“Workers Planned: 5” appear. The execution time was improved only by 25%.
2.1) Why “Workers Planned:” appears only after this setting?
2.2) Why we could not see a much better improvement when running with max_parallel_workers_per_gather = 5 ?
Thank you!.
When PostgreSQL considers a parallel sequential scan, it decides how many workers should be used based on the relation size (or the parallel_workers storage parameter for the driving table) and computes the cost of a parallel plan with that number of workers. This is compared to the cost of a serial plan, and the cheaper plan wins. Plans with other numbers of workers are not considered, so it can happen that the cost of the serial plan is less than the cost of the plan considered but more than the cost of some plan with a different number of workers. This probably happened in your case.
Since you didn't post the EXPLAIN ANALYZE output, we can't see how many groups your query is producing, but my guess is that it's a fairly large number. In PostgreSQL 9.6, a parallel aggregate must be performed by aggregating a portion of the data in each worker (a PartialAggregate) and then merging groups with the same keys in the leader (a FinalizeAggregate). Between these two steps, a Gather node is required to transmit the partially grouped data from the workers to the leader. This Gather node is somewhat expensive, so the most likely reason why you saw only limited speedup is that the number of groups being transferred was large. The cost of sending all of those groups, and of merging groups that occurred in more than one worker, may have looked too high to justify parallelism with a higher number of workers, but may have looked like a win with a lesser number of workers. These same costs probably account for the fact that even when parallel query was used, you saw only a 25% speedup.
If you post the EXPLAIN ANALYZE output with and without parallel query (that is, with "Workers Planned: 5" and with no parallelism), it might be possible to understand more clearly what is happening in your case.
(Source: I am one of the principal authors of PostgreSQL's parallel query support.)
If you are just looking to test the parallel query bits then you can have a look at force_parallel_mode and setting it to on.
force_parallel_mode (enum)
Allows the use of parallel queries for testing purposes even in
cases where no performance benefit is expected. The allowed values of
force_parallel_mode are off (use parallel mode only when it is
expected to improve performance), on (force parallel query for all
queries for which it is thought to be safe), and regress (like on, but
with additional behavior changes as explained below).
And, just as robert-haas mentioned below, Without force_parallel_modethe optimizer will potentially decide parallel query isn't the quickest... see the params below:
select
name,
setting,
unit,
short_desc
from pg_settings
where name in (
'force_parallel_mode',
'min_parallel_relation_size',
'parallel_setup_cost',
'parallel_tuble_cost',
'max_parallel_workers_per_gather' )
limit 10 ;
postgres=>
---------------------------------+---------+------+---------------------------------------------------------------------------------------------
force_parallel_mode | off | | Forces use of parallel query facilities.
max_parallel_workers_per_gather | 0 | | Sets the maximum number of parallel processes per executor node.
min_parallel_relation_size | 1024 | 8kB | Sets the minimum size of relations to be considered for parallel scan.
parallel_setup_cost | 1000 | | Sets the planner's estimate of the cost of starting up worker processes for parallel query.
(4 rows)
I am building a large postgres 9.1 database on ubuntu 12.04, with one table that holds about 80 million rows or so. Whenever I run a SELECT statement:
SELECT * FROM db WHERE ID=1;
It takes almost 2.5 minutes to execute the query which returns only a few thousand rows. After running a few diagnostics on the disk I/O, I think that is not the problem, but just in case below is the output from a diagnostic. (I have 2GB of RAM) I am not exactly sure what a good output is here, but it seems ballpark given stats found for other servers on the internet.
time sh -c "dd if=/dev/zero of=bigfile bs=8k count=500000 && sync"
500000+0 records in
500000+0 records out
4096000000 bytes (4.1 GB) copied, 106.969 s, 38.3 MB/s
real 1m49.091s
user 0m0.248s
sys 0m9.369s
I have modified postgresql.conf considerably, boosting the effective_cache to 75% of ram, shared_buffers to 25%, checkpoint_segments to 15, work_mem to 256MB, autovacuum, SHMMAX on the kernel, etc. I have had some performance increases, but not more than 5% better. Networking shouldnt be an issue since it still takes a long time even running on localhost. I am planning to add even more data, and the query time seems to be growing quickly with the number of rows.
It seems like I should be able to run these SELECT statements in a few seconds, not a few minutes. Any suggestions on where this bottleneck could be?
sorry if this is inexcusably obvious, but do you have an index on the ID column?
also, though I'm not blaming the disk, you merely tested sequential bandwidth, which tells you very little about latency. though I have to say that 38 MB/s is underwhelming even for that measure...