All,
I'm trying to understand the impact of PostgreSQL server setting log_duration. Would it cause any performance issues setting it to ON. I was trying to setup a confluence server with Postgres backend. When this value is set as ON, the response of the service is slow, whereas when I set it to OFF, it is fast.
log_min_duration_statement is set to -1 => No query texts should be logged.
Queries:
Why it is slow when it is set to ON
Logging => does it log in any log file? If yes, where to find the same in server
There is another feature pg_stat_statements.track, which can be set to TOP, ALL. Why setting this is not giving performance issue. It is also tracking queries. But, it gives total_time and calls, but not individual.
Could some one help me understand the impact.
refer to this, let me know if you need more info:
log min duration
This allows logging a sample of statements, without incurring excessive
log traffic (which may impact performance). This can be useful when
analyzing workloads with lots of short queries.
The sampling is configured using two new GUC parameters:
log_min_duration_sample - minimum required statement duration
log_statement_sample_rate - sample rate (0.0 - 1.0)
Only statements with duration exceeding log_min_duration_sample are
considered for sampling. To enable sampling, both those GUCs have to
be set correctly.
The existing log_min_duration_statement GUC has a higher priority, i.e.
statements with duration exceeding log_min_duration_statement will be
always logged, irrespectedly of how the sampling is configured. This
means only configurations
log_min_duration_sample < log_min_duration_statement
do actually sample the statements, instead of logging everything.
The best way to know is to measure it.
For simple select-only queries, such as pgbench -S, the impact is quite high. I get 22,000 selects per second with it off, and 13,000 with it on. For more complex select queries or for DML, the percentage impact will be much less.
Formatting and printing log messages, and the IPC to the log collector, take time. pg_stat_statements doesn't do that for every query, it just increments some counters.
Of course log_duration logs to a log file (or pipe), otherwise there wouldn't be any point in turning it on. The location is highly configurable, look in postgresql.conf for things like log_destination, log_directory, and log_filename. (But if you don't know where to find the logs, why would you even bother to turn this logging on in the first place?)
Related
I have a Mongo db version 4.4.1 and I am profiling for slow queries. With profiling level 1 and verbosity level 0 nothing gets logged, but with profiling level 1 and verbosity level 1 all queries get logged as slow even though durationMillis is lower than slowms. All I found was the official information in the documentation:
https://www.mongodb.com/docs/manual/tutorial/manage-the-database-profiler/#profiling-levels
https://www.mongodb.com/docs/manual/reference/method/db.setLogLevel/#db.setloglevel--
https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-operationProfiling.slowOpThresholdMs
At higher logLevel settings, all operations appear in the diagnostic log regardless of their latency with the following exception: the logging of slow oplog entry messages by the secondaries. The secondaries log only the slow oplog entries; increasing the logLevel does not log all oplog entries.
It is unclear if verbosity level 1 and higher automatically logs all queries as slow.
What is the behavour of the different verbosity levels and why are queries with duratonMillis lower than slowms logged as slow
I think this question is conflating a few related topics, for understandable reasons as we'll see below.
The first pair of concepts are the profiler and slow query logging. The database profiler is a capped collection that collects runtime information about CRUD operations (among other things). Details about CRUD operations can separately be captured in a log file directly. You adjust both of these using the a single db.setProfilingLevel() method.
Now the aforementioned log file is the same diagnostic log file that is used by the mongod process for its other operational needs. So that single file is serving double duty as it will also capture details about many other components such as networking, replication, and access control. To compound the problem further one of the components that can be configured and logged is indeed the QUERY one. Adjusting the verbosity for components is done via the db.setLogLevel() method. The system defaults to a verbosity level of 0 for all components, which includes "Informational" (I) details.
Now "slow query" log lines are one of the types of messages generated by and are classified as QUERY components at the "Informational" level. The result of this fact is that you can increase the number of slow query entries that get logged by either:
Decreasing the slowms threshold via the db.setProfilingLevel() method
Increasing the verbosity for the QUERY component via the db.setLogLevel() method.
Finally to return back to the beginning, slow query logging and profiling also overlap with their usage of the slowms (and other related settings). Specifically if you set the (profiling) level to 1, then slowms will now be used to control both what entries go into the log file as well as what entries get captured by the profiler. A setting of 0 or 2 makes profiling ignore slowms (either not capturing anything in the profiler or capturing everything, respectively).
So the section of the documentation that you quoted, including the sentence before it, is what attempts to tie all of this together (when it comes to behavior on the PRIMARY):
When logLevel is set to 0, MongoDB records slow operations to the diagnostic log at a rate determined by slowOpSampleRate. At higher logLevel settings, all operations appear in the diagnostic log regardless of their latency
Noting that:
"slow operations" here are defined as those that exceed the slowms setting
logLevel here refers to that verbosity setting of db.setLogLevel() as opposed to the level parameter of db.setProfilingLevel().
I can find no details on why the component strings for the db.setLogLevel() method are shown in lowercase yet they appear to be uppercase when logged.
Returning back to the original questions with this additional context, the described behavior is aligned with the documented behavior if we tighten up the wording a bit. I assume that the statement about "profiling for slow queries" is only about capturing slow queries in the log file. If so, then:
With profiling level 1 and verbosity level 0 nothing gets logged
That is correct if all of the queries that are executed are also below the slowms threshold (which defaults to 100ms).
with profiling level 1 and verbosity level 1 all queries get logged as slow even though durationMillis is lower than slowms
It is unclear if verbosity level 1 and higher automatically logs all queries as slow.
That is also correct per the documentation quoted earlier, or the Logging Slow Operations section here which succinctly states that "Client operations (such as queries) appear in the log if their duration exceeds the slow operation threshold or when the log verbosity level is 1 or higher."
On RDS PostgreSQL instance, below are the vacuum parameters set.
autovacuum_vacuum_cost_delay 5
autovacuum_vacuum_cost_limit 3000
vacuum_cost_page_dirty 20
vacuum_cost_page_hit 1
vacuum_cost_page_miss 0
From what I understand from this blog - https://www.2ndquadrant.com/en/blog/autovacuum-tuning-basics/
There is a cost associated with if the page being vacuumed is in shared buffer or not. If the vacuum_cost_page_miss is set to zero, I am thinking its going to assume the cost of reading from disk is free and since the cost limit is set to 3000, the autovacuum will be performing lots of IO until it reaches the cost limit. Is my understanding correct? Would it mean something else by setting this parameter to 0?
0 is not a special value here, it means the same thing it does in ordinary arithmetic. So, no throttling will get applied on the basis of page misses.
This is my preferred setting for vacuum_cost_page_miss. Page misses are inherently self-limiting. Once a page is needed and not found, then the process stalls until the page is read. No more read requests will get issued by that process while it is waiting. This is in contrast to page dirtying. There is nothing other than vacuum_cost_page_dirty-driven throttling to prevent the vacuum process from dirtying pages far faster than they can be written to disk, leading to IO constipation which will then disturb everyone else on the system.
If you are going to reduce vacuum_cost_page_miss to zero, you should also set vacuum_cost_page_hit to zero. Having the latter high than the former is weird. Maybe whoever came up with those setting just figured that 1 was already low enough, so there was no point in changing yet another setting.
vacuum_cost_page_miss throttling could be particularly bad before v9.6 (when freeze map was introduced) when freezing large tables which have hit autovacuum_freeze_max_age, but have seen few changes since the last freeze. PostgreSQL will charge vacuum_cost_page_miss for every page, even though most of them will be found already in the kernel page cache (but not in shared_buffers) through the magic of readahead. So it will slow-walk the table as if it were doing random reads, while doing no useful work and holding the table lock hostage. This might be the exact thing that lead your predecessor to make the changes he made.
the autovacuum will be performing lots of IO until it reaches the cost limit.
Autovacuum once begun has a mostly fixed task to do, and will do the amount of IO it needs to do to get it done. At stake is not how much IO it will do, but over how much time it does it.
These settings are silly in my opinion.
Setting the cost for a page found in shared buffers higher than the cost for a page read from disk does not make any sense. Also, if you want to discount I/O costs, why leave vacuum_cost_page_dirty at 20?
Finally, increasing the cost limit and leaving the cost delay at a high value like 5 (the default from v12 on is 2) can only be explained if RDS is based on a PostgreSQL version older than v12.
It feels like whoever dabbled with the autovacuum settings had a vague idea to make autovacuum more aggressive, but didn't understand it well enough to do it right. I think the default settings of v12 are better.
We're experiencing some constant outages in our back-end that seem to correlate with peaks of high CPU usage for our Cloud SQL Postgres instance (v9.6)
Taking a look to the cloudsql.googleapis.com/postgres.log, those high CPU peaks seems to also correlate to when the database is running an automatic vacuum of table cloudsqladmin.public.heartbeat
We haven't found any documentation on what this table is and why is running autovacuum so often (our own tables doesn't seem to be affected by it).
Is this normal? Should we tune the values for the autovacuum? Thanks in advance.
By looking at your graphs there is no correlation between the CPU and the cloudsqladmin.public.heartbeat autovacuum.
Lets start by what the cloudsqladmin.public.heartbeat table is, this is a table used by the Cloud SQL High Availability process, this is better explained here:
Each second, the primary instance writes to a system database as a
heartbeat signal.
So the table is used internally to keep track of your instance's health. The autovacuum is triggered based on the doc David shared.
Now, if the Vacuum process generated the CPU spike, you would see the spike every minute/second.
So, straight answers to your questions:
Is this normal? : Yes, the autovacuum and the cloudsqladmin.public.heartbeat table are completely normal from a Cloud SQL internal perspective, they should not impact in any way the Instance.
Should we tune the values for the autovacuum? : No need for that, as mentioned, this process is not the one impacting the CPU Instance, you can hide the similar logs including "cloudsqladmin.public.heartbeat" and analyze the ones left on the time the Spike was presented.
It is worth looking at the backup processes triggered too (there could be one on the same time) Cloud SQL > Instance Details > Backups, but of course, that's a different topic than the one described here :) .
Here's a recommendation that seems very relevant to your situation: https://www.netiq.com/documentation/cloud-manager-2-5/ncm-install/data/vacuum.html
We have some queries that are running extremely slowly intermittently in our production environment. These are JSONB intersection queries which normally return in milliseconds, but are taking 30-90 seconds.
We have tried to look at co-occurring server conditions such as RAM, CPU and query load, but there is nothing obvious. This affects a very small minority of queries - probably less than 1%. This does not appear to be a query optimization issue as the affected queries themselves are varied and in some cases very simple.
We've reproduced the same environment as far as possible on a staging server and loaded it heavily and the issue does not occur.
Can anyone suggest possible steps to investigate what is occurring in Postgres when this happens, or anything else we should consider? We have been working on this for over a week and are running out of ideas.
It is difficult to guess the cause of that problem; one explanation would be locks.
You should use auto_explain to investigate the problem.
In postgresql.conf, use the following settings:
# log if somebody has to wait for a lock for more than one second
log_lock_waits = on
# log slow statements with their parameters
log_min_duration_statement = 1000
# log the plans of slow statements
shared_preload_libraries = 'auto_explain'
# configuration for auto_explain
auto_explain.log_nested_statements = on
auto_explain.log_min_duration = 1000
Then restart PostgreSQL.
Now all statements that exceed one second will have their plan dumped in the PostgreSQL log, so all you have to do is to wait for the problem to happen again, so that you can analyze it.
You can also get EXPLAIN (ANALYZE, BUFFERS) output if you set
auto_explain.log_buffers = on
auto_explain.log_analyze = on
That would make the log much more valuable, but it will slow down processing considerably, so I'd be reluctant to do it on a production system.
I am writing logging information asynchronously to mongodb. Since this is an non-essential function, I am looking for a way to throttle these writes so it does not impact read/writes from other part of the application. Essentially, only write when certain stat is below acceptable level.
One stats I thought of using is "globalLock.ratio" from serverStatus. However, this does not seem to be a moving average and not a good way to measure current usage on the database.
What would be a good stats to use for what I am looking to do? Write lock % would be ideal, but how would I get moving average from serverStatus?
There are a number of things to note about your question:
1) If you want moving averages, then you'll need to keep track of them yourself in your client program. If you're running a multi-threaded program, you could dedicate one thread to polling MongoDB at regular (1 second? 5 second?) intervals, and calculating the moving average yourself. This is the way that MMS does it.
2) When you calculate this average, you need to figure out what a 'loaded database' means to you. There could be many things to check: do you care about write lock percentage? read percentage? I/O usage? Replication delay? Unfortunately, there is no single metric that will work for all use cases at all times: you'll have to figure out what you care about and measure that.
3) Another strategy that you could take to achieve this goal is to do the writes to the logging collection using write concern, a 'w' value of 'majority', and a reasonable timeout (say 10 seconds). Using this, you won't be able to write to your database faster than your replication. If you start getting timeouts, you know that you need to scale back. If you can't write fast enough to drain the queue, then you start dropping log entries at that time.