I am currently analyzing why the application installed on top of PostgreSQL we are using is sometimes soo slow. The logfiles are showing that queries to a specific table have extremely long execution times.
I further found out, that it is one column on the table, which contains XML documents (ranging from a few bytes to one entry with ~7MB XML data), which is the cause of the slow query.
There are 1100 Rows in the table and a
SELECT * FROM mytable
has the same query execution time of 5 Seconds as
SELECT [XML-column-only] FROM mytable
But in contrast, a
SELECT [id-only] FROM mytable
has a query execution time of only 0.2s!
I couldn't produce any noticeable differences depending on the settings (the usual ones, work_mem, shared_buffers,...), there is even almost no difference in comparison between our production server (PostgreSQL 9.3) and running it in a VM on PostgreSQL 9.4 on my workstation PC.
Disk monitoring shows almost no I/O activity for the query.
So the last thing I went to analyze was the Network I/O.
Of course, as mentioned before, it's a lot of data in these XML Column. Total size for the 1100 rows (XML column only) is 36 MB. Divided for the 5 seconds running time, this are a mere 7.2MB/s Network Transfer, which equal around 60MBit/s. Which is a little bit slow, as we all are on Gbit Ethernet, aren't we? :D Windows Taskmanger also show a utilization of 6% for the Networking during the runtime of the query, which is in concordance with the manual calculation from before.
Furthermore, the query execution time is almost linear to the amount of XML data in the table. For testing I deleted the top 10% rows with the largest amount of data on the XML column, and the execution time (now ~18 instead of 36MB to transfer) dropped to 2.5s instead of 5s.
So, to get to the point: What are my options on the database administration side (we cannot touch or change the application itself), to make the simple SELECT for this XML-Data noticeable faster? Is there any bottleneck I didn't take into account yet? Or is this the normal PostgreSQL behaviour?
EDIT: I use pgAdmin III. The Execution plan (explain (analyze, verbose) select * from gdb_items) shows a much shorter total runtime, than the actual query and the statement duration entry in the log:
Seq Scan on sde.gdb_items (cost=0.00..181.51 rows=1151 width=1399) (actual time=0.005..0.193 rows=1151 loops=1)
Output: objectid, uuid, type, name, physicalname, path, url, properties, defaults, datasetsubtype1, datasetsubtype2, datasetinfo1, datasetinfo2, definition, documentation, iteminfo, shape
Total runtime: 0.243 ms
Related
I have a very simple query:
SELECT count(*) FROM link_list ll
WHERE ll.tsvector ## to_tsquery('english', 'advertising|slogan|publicity|ads|advertise|brand|promo|ad|propaganda|sales|promote|logo|hype|commercial|consumerism|pamphlet|viral|salesperson|selling|blurb|advert|merchandise|marking|products|campaign|packaging|billboard|advertisement|promotion|questionnaire|marketing')
It runs fast (Here's the output from EXPLAIN ANALYZE). It uses the GIN index, and works exactly as expected.
Life is good.
But now, let me tweak the query slightly, and now it takes 2x as long!
WITH query AS
(
SELECT to_tsquery('english',('advertising|slogan|publicity|ads|advertise|brand|promo|ad|propaganda|sales|promote|logo|hype|commercial|consumerism|pamphlet|viral|salesperson|selling|blurb|advert|merchandise|marking|products|campaign|packaging|billboard|advertisement|promotion|questionnaire|marketing')) query
)
SELECT count(*) FROM link_list ll
WHERE ll.tsvector ## (SELECT query FROM query);
(Output from EXPLAIN ANALYZE)
I would just use the first query... but unfortunately, the list of synonyms has to be dynamically generated, which I pull from another table.
For some strange reason, putting the tsquery inside WITH makes Postgresql not use the index as efficiently (It thinks it'll be a quick and dirty job that doesn't need an index, and it ends up being dead wrong).
Why in the world is this happening??
Neither one of those executions seems all that fast.
Your use of the WITH inhibits the parallel plan. If your bottleneck is IO (which seems likely) you can get parallel IO without parallel query by setting effective_io_concurrency to a value > 1.
The time spent setting up JIT is over 10% of the faster plan, and is probably a complete waste. You can set jit = off (or turn it off globally in the .conf file) to spare that time.
Your recheck of lossy blocks also wastes time. You should increase work_mem to get rid of those. But the waste is of mostly CPU time, so the effect will be small if the bottleneck is IO, not CPU. It still has to visit the same set of blocks. (An exception to this is if TOAST is heavily used, then rows not rechecked don't need to be assembled from TOAST, so those TOAST block reads are avoided.)
Usual when improving my queries I see a coinciding improvement with both cost and actual time when running an explain analyze on both before and after queries.
However, in one case, the before query reports
"Hash Join (cost=134.06..1333.57 rows=231 width=70)
(actual time=115.349..115.650 rows=231 loops=1)"
<cut...>
"Planning time: 4.060 ms"
"Execution time: 115.787 ms"
and the after reports
"Hash Join (cost=4.63..1202.61 rows=77 width=70)
(actual time=0.249..0.481 rows=231 loops=1)"
<cut...>
"Planning time: 2.079 ms"
"Execution time: 0.556 ms"
So as you can see, the costs are similar but actual and real execution times are vastly different, regardless of the order in which I run the tests.
Using Postgres 8.4.
Can anyone clear up my understanding as to why the cost does not show an improvement?
There isn't much information available in the details given in the question but a few pointers can may be help others who come here searching on the topic.
The cost is a numerical estimate based on table statistics that are calculated when analyze is run on the tables that are involved in the query. If the table has never been analyzed then the plan and the cost may be way sub optimal. The query plan is affected by the table statistics.
The actual time is the actual time taken to run the query. Again this may not correlate properly to the cost depending on how fresh the table statistics are. The plan may be arrived upon depending on the current table statistics, but the actual execution may find real data conditions different from what the table statistics tell, resulting in a skewed execution time.
Point to note here is that, table statistics affect the plan and the cost estimate, where as the plan and actual data conditions affect the actual time. So, as a best practice, before working on query optimization, always run analyze on the tables.
A few notes:
analyze <table> - updates the statistics of the table.
vacuum analyze <table> - removes stale versions of the updated records from the table and then updates the statistics of the table.
explain <query> - only generates a plan for the query using statistics of the tables involved in the query.
explain (analyze) <query> - generates a plan for the query using existing statistics of the tables involved in the query, and also runs the query collecting actual run time data. Since the query is actually run, if the query is a DML query, then care should be taken to enclose it in begin and rollback if the changes are not intended to be persisted.
Cost meaning
The costs are in an arbitrary unit. A common misunderstanding is that they are in milliseconds or some other unit of time, but that’s not the case.
The cost units are anchored (by default) to a single sequential page read costing 1.0 units (seq_page_cost).
Each row processed adds 0.01 (cpu_tuple_cost)
Each non-sequential page read adds 4.0 (random_page_cost).
There are many more constants like this, all of which are configurable.
Startup cost
The first numbers you see after cost= is known as the “startup cost”. This is an estimate of how long it will take to fetch the first row.
The startup cost of an operation includes the cost of its children.
Total cost
After the startup cost and the two dots, is known as the “total cost”. This estimates how long it will take to return all the rows.
example
QUERY PLAN |
--------------------------------------------------------------+
Sort (cost=66.83..69.33 rows=1000 width=17) |
Sort Key: username |
-> Seq Scan on users (cost=0.00..17.00 rows=1000 width=17)|
We can see that the total cost of the Seq Scan operation is 17.00, and the startup cost of the Seq Scan is 0.00. For the Sort operation, the total cost is 69.33, which is not much more than its startup cost (66.83).
Actual time meaning
The “actual time” values are in milliseconds of real time, it is the result of EXPLAIN's ANALYZE. Note: the EXPLAIN ANALYZE option performs the query (be careful with UPDATE and DELETE)
EXPLAIN ANALYZE could be used to compare the estimated number of rows with the actual rows returned by each operation.
Helping the planner estimate more accurately
Gather better statistics
tables also change over time, so tuning the autovacuum settings to make sure it runs frequently enough for your workload can be very helpful.
If you’re having trouble with bad estimates for a column with a skewed distribution, you may benefit from increasing the amount of information Postgres gathers by using the ALTER TABLE SET STATISTICS command, or even the default_statistics_target for the whole database.
Another common cause of bad estimates is that, by default, Postgres will assume that two columns are independent. You can fix this by asking it to gather correlation data on two columns from the same table via extended statistics.
Tune the constants it uses for the calculations
Assuming you’re running on SSDs, you’ll likely at minimum want to tune your setting of random_page_cost. This defaults to 4, which is 4x more expensive than the seq_page_cost we looked at earlier. This ratio made sense on spinning disks, but on SSDs it tends to penalize random I/O too much.
Source:
PG doc - using explain
Postgres explain cost
Is there a reason why the same query executed a number of times have huge variance in response times? from 50% - 200% what the projected response time is? They range from 6 seconds to 20 seconds even though it is the only active query in the database.
Context:
Database on Postgres 9.6 on AWS RDS (with Provisioned IOPS)
Contains one table comprising five numeric columns, indexed on id, holding 200 million rows
The query:
SELECT col1, col2
FROM calculations
WHERE id > 0
AND id < 100000;
The query's explain plan:
Bitmap Heap Scan on calculation (cost=2419.37..310549.65 rows=99005 width=43)
Recheck Cond: ((id > 0) AND (id <= 100000))
-> Bitmap Index Scan on calculation_pkey (cost=0.00..2394.62 rows=99005 width=0)
Index Cond: ((id > 0) AND (id <= 100000))
Is there any reasons why a simple query like this isn't more predictable in response time?
Thanks.
When you see something like this in PostgreSQL EXPLAIN ANALYZE:
(cost=2419.37..310549.65)
...it doesn't mean the cost is between 2419.37 and 310549.65. These are in fact two different measures. The first value is the startup cost, and the second value is the total cost. Most of the time you'll care only about the total cost. The times that you should be concerned with startup cost is when that component of the execution plan is in relation to (for example) an EXISTS clause, where only the first row needs to be returned (so you only care about startup cost, not total, as it exits almost immediately after startup).
The PostgreSQL documentation on EXPLAIN goes into greater detail about this.
A query may be (and should be, excluding special cases) more predictable in response time when you are a sole user of the server. In the case of a cloud server, you do not know anything about the actual server load, even if your query is the only one performed on your database, because the server most likely supports multiple databases at the same time. As you asked about response time, there may be also various circumstances involved in accessing a remote server over the network.
After investigation of the historical load, we have found out that the provisioned IOPS we originally configured had been exhausted during the last set of load tests performed on the environment.
According to Amazon's documentation #http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html, after this point, Amazon does not guarantee consistency in execution times and the SLAs are no longer applicable.
We have confirmed that replicating the database onto a new instance of AWS RDS with same configuration yields consistent response times when executing the query multiple times.
I am running a vacuum on a very large table.
When I run it, it says:
bacula=# VACUUM FULL VERBOSE file_partition_19
bacula-# ;
INFO: vacuuming "public.file_partition_19"
INFO: "file_partition_19": found 16242451 removable, 21024161 nonremovable row versions in 900380 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU 5.14s/14.42u sec elapsed 19.61 sec.
VACUUM
Time: 163784.767 ms
bacula=#
When it does this, it shows up to the CPU line fairly quickly, then waits a long time before it shows the final two lines (+ the prompt). This is reflected in the difference in time - "elapsed 19.61 sec", compared to to the "Time:" of 163 seconds (shown because I set \timing on).
While I haven't timed them, both times are about right - start the command, wait 20 seconds, it then shows up to the "CPU" line, then waits about 3 minutes, then prints the rest.
Is this normal? Why is it happening?
It's mostly rebuilding all indizes on the table, which it has to do since basically "VACUUM FULL" does a full rewrite of the table. If you remove all indizes from your table, there should be almost no delay after the "CPU" line.
AFAICT, the CPU usage line is printed by a generic routine that does most of the work for other (non-FULL) vacuum modes. It is meaningless in the "VACUUM FULL" case.
If you are concerned that it takes too long, I recommend that you have a look at the "When to use VACUUM FULL and when not to" from the PostgreSQL wiki. 9 times out of 10 when people are using VACUUM FULL they actually shouldn't.
Based on the tag "postgres-9.3" you used for your question I am assuming that you have Postgres 9.3 version.
you can refer this link just for your own knowledge about "VACUUM" and "VACUUM FULL" for the pre-9.0 versions of Postgres.
VACUUM VS VACUUM FULL For Pre-9.0 versions of Postgres
So as you have Postgres-9.3, the documentation says following:
For clarity, 9.0 changes VACUUM FULL. As covered in the documentation, the VACUUM FULL implementation has been changed to one that's similar to using CLUSTER in older versions. This gives a slightly different set of trade-offs from the older VACUUM FULL described here. While the potential to make the database slower via index bloating had been removed by this change, it's still something you may want to avoid doing, due to the locking and general performance overhead of a VACUUM FULL.
As per the current documentation, VACUUM FULL operation not only retrieves the space from the table where records are marked deleted but it also touches every valid record in the table and tries to reorganize them in DB pages so that's how it frees up more space then just VACUUM operation. So in the VERBOS result when we see the line
CPU 5.14s/14.42u sec elapsed 19.61 sec
it is the time taken by system process to go through the table and analyze the table and retrieve the space that is already marked. Then it starts the organizing the records into page file and hence depending on how much table pages are fragmented the process will take time.
For example, if you have a new table and keep adding new records incrementally/sequentially so that new records gets added at the bottom of the page (based on the primary key defined). Now you perform delete operation in a reverse order so that records only gets deleted from the bottom of the page. Let's say you delete half of the records from the table. In this situation, there is no much page fragmentation(virtually 0) and hence when VACUMME FULL runs the second phase, it will still try to organize the valid records but because there is no fragmentation and hence it will not have to actually move any records and will finish faster.
But, above explain situation is not the way update/delete happens in real world. Real word Update/Delete on table create lots of page fragmentation and hence during the second phase VACUUM FULL process has to actually move valid records into free space at the beginning of each page and hence takes more time.
check the following sample output,
I ran for very small dummy table. even though It has only 7 rows. VACUME PROCESS (First Phase) finishes in 0.03sec(30ms) but total query reported to finish in 61ms. So that tells me even though there is nothing to reorganize the process still checks how much if it can be reorganized and hence takes time. But if I have actually lots of fragmentation and reorganize happens then it would be much more completion time depending on page fragmentation.
I just want to know what is the reason for having different time while executing the same query in PostgreSQL.
For Eg: select * from datas;
For the first time it takes 45ms
For the second time the same query takes 55ms and the next time it takes some other time.Can any one say What is the reason for having non static time.
Simple, everytime the database has to read the whole table and retrieve the rows. There might be 100 different things happening in database which might cause a difference of few millis. There is no need to panic. This is bound to happen. You can expect the operation to take same time with some millis accuracy. If there is a huge difference then it is something which has to be looked.
Have u applied indexing in your table . it also increases speed to a great deal!
Compiling the explanation from
Reference by matt b
EXPLAIN statement? helps us to display the execution plan that the PostgreSQL planner generates for the supplied statement.
The execution plan shows how the
table(s) referenced by the statement will be scanned — by plain
sequential scan, index scan, etc. — and if multiple tables are
referenced, what join algorithms will be used to bring together the
required rows from each input table
And Reference by Pablo Santa Cruz
You need to change your PostgreSQL configuration file.
Do enable this property:
log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements
# and their durations, > 0 logs only
# statements running at least this number
# of milliseconds
After that, execution time will be logged and you will be able to figure out exactly how bad (or good) are performing your queries.
Well that's about the case with every app on every computer. Sometimes the operating system is busier than other times, so it takes more time to get the memory you ask it for or your app gets fewer CPU time slices or whatever.