We're upgrading our Postgresql from 9.6 to 14, using pglogical (latest installed via yum). The replication is working fine without errors. What we are not seeing, however, is any autovacuum activity on the v14 database, even though we continue to see normal autovacuum activity on the v9.6 database. Also, strangely, the dead tuple counts do not change on the v14 database and are mostly 0. I did run VACUUM ANALYZE on the v14 database.
The command we are using to see autovacuum activity is
SELECT relname, last_vacuum, last_autovacuum, last_autoanalyze FROM pg_stat_user_tables;
The command we are using to see dead tuple counts is
SELECT relname, n_dead_tup FROM pg_stat_user_tables;
There's nothing in the logs except checkpoint notifications. Here is one line picked at random:
2022-09-22 11:59:46 PDT [2877]: [15846-1] user=,db=,app=,client= LOG: checkpoint complete: wrote 38220 buffers (0.9%); 0 WAL file(s) added, 0 removed, 17 recycled; write=269.923 s, sync=0.025 s, total=269.962 s; sync files=264, longest=0.007 s, average=0.001 s; distance=313936 kB, estimate=329901 kB
The v14 database is streaming to another v14 database acting as a replica.
Is this expected behavior?
After experimenting, it's clear that the n_dead_tup counts are not updated while pglogical replication is running. This also means that autovacuum never runs while pglogical replication is running. Restarting the v14 node causes the n_dead_tup counts to be updated and does trigger autovacuum, but that is a one time event (the tuples are not updated again until another restart).
Once you disable pglogical, the n_dead_tup counts are immediately updated, and autovacuum starts to work again as expected (even without a restart).
Related
On my production server sometimes backup failed with below messages. This backup is scheduled on standby node. (In my environment streaming replication is configured.) There is no process running during this backup time. This is a nightly cron job on an Ubuntu machine.
2020-07-01 05:21:07.567 CEST [27925] postgres#DBname LOG: process 27925 still waiting for AccessShareLock on relation 2610 of database 17948 after 1000.096 ms
2020-07-01 05:21:07.567 CEST [27925] postgres#DBname DETAIL: Process holding the lock: 25802. Wait queue: 1559, 27925.
2020-07-01 05:21:17.120 CEST [25802] postgres#DBname ERROR: canceling statement due to conflict with recovery
2020-07-01 05:21:17.120 CEST [25802] postgres#DBname DETAIL: User was holding a relation lock for too long.
2020-07-01 05:21:17.120 CEST [25802] postgres#DBname STATEMENT: COPY public.tablename(id, col1,col2,col3, geom) TO stdout;
2020-07-01 05:21:17.127 CEST [27925] postgres#DBname LOG: process 27925 acquired AccessShareLock on relation 2610 of database 17948 after 10560.447 ms
relation 2610 is pg_index. I tried to lock this table and reproduce the table but didn't get any error.
Has anyone faced this issue? Any hint/fix is appreciated.
Somebody is locking the table on the primary server in ACCESS EXCLUSIVE mode.
Such locks are taken by VACUUM (FULL), TRUNCATE, ALTER TABLE and similar commands, but can also happen during a regular VACUUM, if it decides to truncate empty pages off the end of the table when it is done.
While it is easy to avoid the former statements, autovacuum cannot be avoided, but from v12 on you can set the vacuum_truncate storage option to off on a table to avoid it.
Anyway, that ACCESS EXCLUSIVE lock gets replicated to the standby server, where it conflicts with the ACCESS SHARE lock that pg_dump has on the table (to read it).
That causes a conflict. I guess that max_standby_streaming_delay is set to 10 seconds, so after delaying replication of the lock for 10 seconds, the startup process terminates the query and hence pg_dump.
Either take no ACCESS EXCLUSIVE locks on the primary server, or set max_standby_streaming_delay to a higher value.
Since upgrading to Postgres 11 I cannot get my production standby server to catch up. In the logs things look fine eventually:
2019-02-06 19:23:53.659 UTC [14021] LOG: consistent recovery state reached at 3C772/8912C508
2019-02-06 19:23:53.660 UTC [13820] LOG: database system is ready to accept read only connections
2019-02-06 19:23:53.680 UTC [24261] LOG: started streaming WAL from primary at 3C772/8A000000 on timeline 1
But the following queries show everything is not fine:
warehouse=# SELECT coalesce(abs(pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn())), -1) / 1024 / 1024 / 1024 AS replication_delay_gbytes;
replication_delay_gbytes
-------------------------
208.2317776754498486
(1 row)
warehouse=# select now() - pg_last_xact_replay_timestamp() AS replication_delay;
replication_delay
-------------------
01:54:19.150381
(1 row)
After a while (a couple hours) replication_delay stays about the same but replication_delay_gbytes grows, although note replication_delay is behind from the beginning and replication_delay_gbytes starts near 0. During startup there were a number of these messages:
2019-02-06 18:24:36.867 UTC [14036] WARNING: xlog min recovery request 3C734/FA802AA8 is past current point 3C700/371ED080
2019-02-06 18:24:36.867 UTC [14036] CONTEXT: writing block 0 of relation base/16436/2106308310_vm
but Googling suggests these are fine.
Replica was created using repmgr by running pg_basebackup to perform the clone and then starting up the replica and seeing it catch up. This previously was working with Postgres 10.
Any thoughts on why this replica comes up but is perpetually lagging?
I'm still not sure what the issue is/was, but I was able to get the standby caught up with these two changes:
set use_replication_slots=true in the repmgr config
set wal_compression=on in the postgres config
Use replication slots didn't seem to change anything other than to cause replication_delay_gbytes to stay roughly flat. Turing on WAL compression did help, somehow, although I'm not entirely sure how. Yes, in theory it made it possible to ship WAL files to the standby faster, but reviewing network logs I see a drop in sent/received bytes that matches the effects of compression, so it seems to be shipping WAL files at the same speed just using less network.
It still seems like there is some underlying issue at play here, though, because for example when I do pg_basebackup to create the standby it generates roughly 500 MB/s of network traffic, but then when it is streaming WALs after the standby finishes recovery it drops to ~250 MB/s without WAL compression and ~100 MB/s with WAL compression, but there is no decrease in network traffic after it caught up with WAL compression, so I'm not sure what's going on there that allowed it to catch up.
I have a process running that can not be killed safely:
autovacuum: VACUUM public.mytable (to prevent wraparound)
This table has been cleared (aside from some entries that can not be deleted due to the table's corruption during a hardware issue) and can not be dropped, because the vacuum is blocking this. I had to run a kill -9 to stop this process and restarted the database, but you can't disable this autovacuum (to prevent [transaction] wraparound), so the autovacuum is coming back up and immediately getting stuck by this corrupt table.
Any insight into this?
First of all, shutdown database server and make a physical copy of data directory to a safe place.
Then you could truncate the datafile of corrupted table. E.g.:
--Get datafile path
db=# SELECT pg_relation_filepath('corrupted_table');
pg_relation_filepath
----------------------
base/1234/56789
(1 row)
Enter database directory (e.g: data/base/1234)
Rename the file to 56789_bkp
Create an empty file called 56789: touch 56789
Start database server
Issue a truncate table to force PostgreSQL overwrite datafile: TRUNCATE TABLE corrupted_table;
You may want to VACUUM and make a backup afterwards
Hope this helps.
I run an update on a large table (e.g. 8 GB). It is a simple update of 3 fields in the table. I had no problems running it under postgresql 9.1, it would take 40-60 minutes but it worked. I run the same query in 9.4 database (freshly created, not upgraded) and it starts the update fine but then slows down. It uses only ~2% CPU, the level if IO is 4-5MB/s and it is sitting there. No locks, no other queries or connections, just this single update SQL on the server.
The SQL is below. "lookup" table has 12 records. The lookup can return only one row, it breaks a discrete scale (SMALLINT, -32768 .. +32767) into non-overlapping regions. "src" and "dest" tables are ~60 million records.
UPDATE dest SET
field1 = src.field1,
field2 = src.field2,
field3_id = (SELECT lookup.id FROM lookup WHERE src.value BETWEEN lookup.min AND lookup.max)
FROM src
WHERE dest.id = src.id;
I thought my disk slowed down but I can copy 1 GB files in parallel to query execution and it runs fast at >40MB/s and I have only one disk (it is a VM with ISCSI media). All other disk operations are not impacted, there is plenty of IO bandwidth. At the same time PostgreSQL is just sitting there doing very little, running very slowly.
I have 2 virtualized linux servers, one runs postgresql 9.1 and another runs 9.4. Both servers have close to identical postgresql configuration.
Has anyone else had similar experience? I am running out of ideas. Help.
Edit
The query "ran" for 20 hours I had to kill the connections and restart the server. Surprisingly it didn't kill the connection via query:
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid() AND datname = current_database();
and sever produced the following log:
2015-05-21 12:41:53.412 EDT FATAL: terminating connection due to administrator command
2015-05-21 12:41:53.438 EDT FATAL: terminating connection due to administrator command
2015-05-21 12:41:53.438 EDT STATEMENT: UPDATE <... this is 60,000,000 record table update statement>
Also server restart took long time, producing the following log:
2015-05-21 12:43:36.730 EDT LOG: received fast shutdown request
2015-05-21 12:43:36.730 EDT LOG: aborting any active transactions
2015-05-21 12:43:36.730 EDT FATAL: terminating connection due to administrator command
2015-05-21 12:43:36.734 EDT FATAL: terminating connection due to administrator command
2015-05-21 12:43:36.747 EDT LOG: autovacuum launcher shutting down
2015-05-21 12:44:36.801 EDT LOG: received immediate shutdown request
2015-05-21 12:44:36.815 EDT WARNING: terminating connection because of crash of another server process
2015-05-21 12:44:36.815 EDT DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
"The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory" - is this an indication of a bug in PostgreSQL?
Edit
I tested 9.1, 9.3 and 9.4. Both 9.1 and 9.3 don't experience the slow down. 9.4 consistently slows down on large transactions. I noticed that when a transaction starts htop monitor indicates high CPU and the process status is "R" (running). Then it gradually changes to low CPU usage and status "D" - disk (see screenshot ). My biggest question is why 9.4 is different from 9.1 and 9.3? I have a dozen of servers and this effect is observed across the board.
Thanks everyone for the help. No matter how much I tried to emphasize on the difference of performance between identical configuration of 9.4 and previous versions no one seemed to pay attention to that.
The problem was solved by disabling transparent huge pages:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
Here are some resources I found helpful in reserching the issue:
* https://dba.stackexchange.com/questions/32890/postgresql-pg-stat-activity-shows-commit/34169#34169
* https://lwn.net/Articles/591723/
* https://blogs.oracle.com/linux/entry/performance_issues_with_transparent_huge
I'd suspect a lot of disk seeking - 5MB/s is just about right for a very random IO on ordinary (spinning) hard drive.
As you constantly replace basically all your rows I'd try to set dest table fillfactor to about 45% (alter table dest set (fillfactor=45);) and then cluster test using test_pkey;. This would allow updated row versions to be placed in the same disk sector.
Additionally using cluster src using src_pkey; so both tables would have data in the same physical order on disk also can help.
Also remember to vacuum table dest; after every update that large, so old row versions could be used again in subsequent updates.
Your old server probably evolved it's fillfactor naturally during multiple updates. On new server it is packed 100%, so updated rows have to be placed at the end.
If only few of the target rows are actually updated, you can avoid new row versions to be generated by using DISTICNT FROM. This can prevent a lot of useless disk traffic.
UPDATE dest SET
field1 = src.field1,
field2 = src.field2,
field3_id = lu.id
FROM src
JOIN lookup lu ON src.value BETWEEN lu.min AND lu.max
WHERE dest.id = src.id
-- avoid unnecessary row versions to be generated
AND (dest.field1 IS DISTINCT FROM src.field1
OR dest.field1 IS DISTINCT FROM src.field1
OR dest.field3_id IS DISTINCT FROM lu.id
)
;
How can one tell if the autovacuum daemon in Postgres 9.x is running and maintaining the database cluster?
PostgreSQL 9.3
Determine if Autovacuum is Running
This is specific to Postgres 9.3 on UNIX.
For Windows, see this question.
Query Postgres System Table
SELECT
schemaname, relname,
last_vacuum, last_autovacuum,
vacuum_count, autovacuum_count -- not available on 9.0 and earlier
FROM pg_stat_user_tables;
Grep System Process Status
$ ps -axww | grep autovacuum
24352 ?? Ss 1:05.33 postgres: autovacuum launcher process (postgres)
Grep Postgres Log
# grep autovacuum /var/log/postgresql
LOG: autovacuum launcher started
LOG: autovacuum launcher shutting down
If you want to know more about the autovacuum activity, set log_min_messages to DEBUG1..DEBUG5. The SQL command VACUUM VERBOSE will output information at log level INFO.
Regarding the Autovacuum Daemon, the Posgres docs state:
In the default configuration, autovacuuming is enabled and the related configuration parameters are appropriately set.
See Also:
http://www.postgresql.org/docs/current/static/routine-vacuuming.html
http://www.postgresql.org/docs/current/static/runtime-config-autovacuum.html
I'm using:
select count(*) from pg_stat_activity where query like 'autovacuum:%';
in collectd to know how many autovacuum are running concurrently.
You may need to create a security function like this:
CREATE OR REPLACE FUNCTION public.pg_autovacuum_count() RETURNS bigint
AS 'select count(*) from pg_stat_activity where query like ''autovacuum:%'';'
LANGUAGE SQL
STABLE
SECURITY DEFINER;
and call that from collectd.
In earlier Postgres, "query" was "current_query" so change it according to what works.
You can also run pg_activity to see the currently running queries on your database. I generally leave a terminal open with this running most of the time anyway as it's very useful.
set log_autovacuum_min_duration to the time length that you desire and the autovacuum execution exceeds the time length will be logged.