Multiple autovacuum: ANALYZE/VACUUM on the same table - postgresql

I am currently using Postgres 11. For some reason whenever I check running autovacuums, I would see several multiple autovacuum: ANALYZE or autovacuum: VACUUM on the same table.
postgres=# SELECT backend_type,
pid,
now() - xact_start AS duration,
query,
state
FROM pg_stat_activity
WHERE query LIKE 'autovacuum%'
ORDER BY duration DESC LIMIT 64;
backend_type | pid | duration | query | state
-------------------+------+-----------------+-------------------------------------------------------------------+--------
autovacuum worker | 5900 | 00:48:29.175656 | autovacuum: ANALYZE someschema.foo | active
autovacuum worker | 3552 | 00:48:18.72438 | autovacuum: ANALYZE someschema.foo | active
autovacuum worker | 6122 | 00:47:29.907655 | autovacuum: VACUUM someschema.foo | active
autovacuum worker | 5897 | 00:42:45.380395 | autovacuum: VACUUM someschema.bar | active
autovacuum worker | 5583 | 00:38:45.406516 | autovacuum: VACUUM someschema.bar | active
The DDL looks like the following:
Table "someschema.foo"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-------------------+------+-----------+----------+---------+----------+--------------+-------------
foo_id | text | | not null | | extended | |
baz_id | text | | | | extended | |
Indexes:
"foo_pkey" PRIMARY KEY, btree (foo_id)
"baz_reference_id_idx" UNIQUE, btree (foo_id, baz_id)
Options: autovacuum_vacuum_cost_limit=2000, autovacuum_vacuum_cost_delay=10
I am unsure what is going on, and I cannot find my answers from the official documentations:
https://www.postgresql.org/docs/11/routine-vacuuming.html#AUTOVACUUM
https://www.postgresql.org/docs/11/sql-analyze.html
max_worker_processes = 15
autovacuum_max_workers = 64

Related

PostgreSQL insert performance - why would it be so slow?

I've got a PostgreSQL database running inside a docker container on an AWS Linux instance. I've got some telemetry running, uploading records in batches of ten. A Python server inserts these records into the database. The table looks like this:
postgres=# \d raw_journey_data ;
Table "public.raw_journey_data"
Column | Type | Collation | Nullable | Default
--------+-----------------------------+-----------+----------+---------
email | character varying | | |
t | timestamp without time zone | | |
lat | numeric(20,18) | | |
lng | numeric(21,18) | | |
speed | numeric(21,18) | | |
There aren't that many rows in the table; about 36,000 presently. But committing the transactions that insert the data is taking about a minute each time:
postgres=# SELECT pid, age(clock_timestamp(), query_start), usename, query
FROM pg_stat_activity
WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'
ORDER BY query_start desc;
pid | age | usename | query
-----+-----------------+----------+--------
30 | | |
32 | | postgres |
28 | | |
27 | | |
29 | | |
37 | 00:00:11.439313 | postgres | COMMIT
36 | 00:00:11.439565 | postgres | COMMIT
39 | 00:00:36.454011 | postgres | COMMIT
56 | 00:00:36.457828 | postgres | COMMIT
61 | 00:00:56.474446 | postgres | COMMIT
35 | 00:00:56.474647 | postgres | COMMIT
(11 rows)
The load average on the system's CPUs is zero and about half of the 4GB system RAM is available (as shown by free). So what causes the super-slow commits here?
The insertion is being done with SqlAlchemy:
db.session.execute(import_table.insert([
{
"email": current_user.email,
"t": row.t.ToDatetime(),
"lat": row.lat,
"lng": row.lng,
"speed": row.speed
}
for row in data.data
]))
Edit Update with the state column:
postgres=# SELECT pid, age(clock_timestamp(), query_start), usename, state, query
FROM pg_stat_activity
WHERE query NOT ILIKE '%pg_stat_activity%'
ORDER BY query_start desc;
pid | age | usename | state | query
-----+-----------------+----------+-------+--------
32 | | postgres | |
30 | | | |
28 | | | |
27 | | | |
29 | | | |
46 | 00:00:08.390177 | postgres | idle | COMMIT
49 | 00:00:08.390348 | postgres | idle | COMMIT
45 | 00:00:23.35249 | postgres | idle | COMMIT
(8 rows)

Timescaledb: retention policy isn't removing data from hypertable

(note: I've also posted this as a github issue https://github.com/timescale/timescaledb/issues/3653)
I have a hypertable request_logs configured with a 24 hour retention policy. The retention policy is being marked as running successfully, however no old data from the table is being removed. The table continues to grow day by day.
I checked and don't see any errors in the postgresql log files.
Could use additional guidance on where to look for information to troubleshoot this issue.
request_logs table structure
\d+ request_logs;
Table "public.request_logs"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
-----------+--------------------------+-----------+----------+---------+---------+--------------+-------------
time | timestamp with time zone | | not null | | plain | |
referer | bigint | | | | plain | |
useragent | bigint | | | | plain | |
Indexes:
"request_logs_time_idx" btree ("time" DESC)
Triggers:
ts_insert_blocker BEFORE INSERT ON request_logs FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()
Child tables: _timescaledb_internal._hyper_1_37_chunk,
_timescaledb_internal._hyper_1_38_chunk,
_timescaledb_internal._hyper_1_40_chunk
Access method: heap
This is the hypertable description retrieved by running select * from _timescaledb_catalog.hypertable;
id | schema_name | table_name | associated_schema_name | associated_table_prefix | num_dimensions | chunk_sizing_func_schema | chunk_sizing_func_name | chunk_target_size | compression_state | compressed_hypertable_id | replication_factor
----+-------------+--------------+------------------------+-------------------------+----------------+--------------------------+--------------------------+-------------------+-------------------+--------------------------+--------------------
1 | public | request_logs | _timescaledb_internal | _hyper_1 | 1 | _timescaledb_internal | calculate_chunk_interval | 0 | 0 | |
(1 row)
This is the retention_policy retrieved by running SELECT * FROM timescaledb_information.job_stats;.
hypertable_schema | hypertable_name | job_id | last_run_started_at | last_successful_finish | last_run_status | job_status | last_run_duration | next_start | total_runs | total_successes | total_failures
-------------------+-----------------+--------+-------------------------------+-------------------------------+-----------------+------------+-------------------+-------------------------------+------------+-----------------+----------------
public | request_logs | 1002 | 2021-10-05 23:59:01.601404+00 | 2021-10-05 23:59:01.638441+00 | Success | Scheduled | 00:00:00.037037 | 2021-10-06 23:59:01.638441+00 | 8 | 8 | 0
| | 1 | 2021-10-05 08:38:20.473945+00 | 2021-10-05 08:38:21.153468+00 | Success | Scheduled | 00:00:00.679523 | 2021-10-06 08:38:21.153468+00 | 45 | 45 | 0
(2 rows)
Relevant system information:
OS: Ubuntu 20.04.3 LTS
PostgreSQL version (output of postgres --version): 12
TimescaleDB version (output of \dx in psql): 2.4.1
Installation method: apt install process described https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/#installation-apt-ubuntu
It looks as though this might be related to a bug that has been fixed in version 2.4.2 of TimescaleDB. The GitHub report has been updated, if you find that the issue remains after upgrade, please update the issue on GitHub with your example. Thanks for reporting!
Transparency: I work for Timescale

DROP TABLE on a Child table getting blocked by a SELECT on the parent table in Postgres 9.5

I have a parent/child relationship between table T and T_CHILD. When I have an Active transaction SELECTING from the parent - T, and I try to drop the child table (T_CHILD), the DROP statement just hangs, as this session tries to get an Access Exclusivelock on both the parent and child tables.
1. Is this expected behavior?
2. Why should Postgres prevent a child table from being dropped, eventhough dropping the child table has no impact on the SELECT on the parent table? Can you please explain this?
dev_sporuran=# \d+ t
Table "public.t"
Column | Type | Modifiers | Storage | Stats target | Description
--------+---------+-----------+---------+--------------+-------------
id | integer | not null | plain | |
Indexes:
"t_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "t_child" CONSTRAINT "t_child_fk" FOREIGN KEY (t_id) REFERENCES t(id)
dev_sporuran=# \d+ t_child
Table "public.t_child"
Column | Type | Modifiers | Storage | Stats target | Description
--------+-----------------------+-----------+----------+--------------+-------------
id | integer | | plain | |
t_id | integer | | plain | |
name | character varying(10) | | extended | |
Foreign-key constraints:
"t_child_fk" FOREIGN KEY (t_id) REFERENCES t(id)
***********26727 session starts a SELECT ********
dev_sporuran=# begin;
BEGIN
dev_sporuran=# select * from t;
id
----
(0 rows)
*****26727 gets a shared lock for the SELECT on the table and the PK******
dev_sporuran=# select locktype,database,relation::regclass,transactionid,pid,mode,granted
from pg_locks where pid in (26727,26780) order by pid;
locktype | database | relation | transactionid | pid | mode | granted
------------+----------+----------+---------------+-------+-----------------+---------
relation | 19752133 | pg_locks | | 26727 | AccessShareLock | t
relation | 19752133 | t_pkey | | 26727 | AccessShareLock | t
relation | 19752133 | t | | 26727 | AccessShareLock | t
virtualxid | | | | 26727 | ExclusiveLock | t
(4 rows)
**************26827 tries to DROP the t_child table ************
dev_sporuran=# select pg_backend_pid();
pg_backend_pid
----------------
26867
(1 row)
dev_sporuran=# drop table t_child;
********* This just hangs;
Looking at pg_locks - 26867 requires not only an Access Exclusive lock on t_child, it also needs an Access Exclusive Lock on the parent table - T.
But it cannot get it, because 26727 has a Share lock on it thru the SELECT statement. So it has to wait*****
dev_sporuran=# select locktype,virtualxid,relation::regclass,transactionid,pid,mode,granted
from pg_locks where pid in (26727,26867) and relation::regclass in ('t','t_pkey','t_child') order by pid;
locktype | virtualxid | relation | transactionid | pid | mode | granted
----------+------------+----------+---------------+-------+---------------------+---------
relation | | t_pkey | | 26727 | AccessShareLock | t
relation | | t | | 26727 | AccessShareLock | t
relation | | t_child | | 26867 | AccessExclusiveLock | t
relation | | t | | 26867 | AccessExclusiveLock | f
(4 rows)
Thanks
Murali
Seems like something you could raise for discussion on pgsql-hackers. At a glance I don't see a good reason it shouldn't be ExclusiveLock not AccessExclusiveLock, so it'd still permit SELECT. But most table structure alterations require A.E.L.
Take a look at the ALTER TABLE lock strengths reduction patch for how complex this sort of thing can be.

postgresql remove stale lock

After a system crash my Postgresql database does have a lock on a row.
The pg_locks table contains a lot of rows without a pid. i.e.
select locktype,database,relation,virtualtransaction, pid,mode,granted from pg_locks p1;
locktype | database | relation | virtualtransaction | pid | mode | granted
---------------+----------+----------+--------------------+-------+------------------+---------
relation | 16408 | 31459 | -1/40059 | | AccessShareLock | t
relation | 16408 | 31459 | -1/40059 | | RowExclusiveLock | t
relation | 16408 | 31022 | -1/40060 | | AccessShareLock | t
transactionid | | | -1/40060 | | ExclusiveLock | t
relation | 16408 | 31485 | -1/40060 | | AccessShareLock | t
How do I get the transaction 40060 killed and the locks removed?
Ok, solution found by myself:
Find the gid to the transaction (i.e. 40060 in the case above) by select * from pg_prepared_xacts where transaction = 40060;
Find an awful long gid.
ROLLBACK PREPARED gid;
This will clear the locks.

PostgreSQL CREATE INDEX CONCURRENTLY waiting column

Im trying to create index on large table:
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | waiting | state | backend_xid | backend_xmin | query
-------+----------------------+-------+----------+-------------------------+--- --------------------------+--------------+-----------------+-------------+------ -------------------------+-------------------------------+---------------------- ---------+-------------------------------+---------+--------+-------------+----- ---------+---------------------------------------------------------------------- ---------------------
25439 | messengerdb | 30692 | 25438 | messengerdb_rw | pgAdmin III - Przegl??darka | 10.167.12.52 | | 50593 | 2016-08-11 05:27:12.101452+02 | 2016-08-11 05:28:01.535943+02 | 2016-08-11 05:28:01.535943+02 | 2016-08-11 05:28:01.535958+02 | t | active | | 1173740991 | CREATE INDEX CONCURRENTLY user_time_idx +
| | | | | | | | | | | | | | | | | ON core.conversations (user_id ASC NULLS LAST, last_message_timestamp ASC NULLS LAST);+
Is this query working? I'm worried about 'waiting' column === 't' does it mean that it is waiting for lock or sth?
Creating an index concurrently may take a long time since it does not lock the table from writes and it waits until other transactions are finished. However, it may wait forever if you have connections that stay idle in transactions (for example when a client or application keeps an open connection without rollback/commit).
Check if there are some idle connections in transactions (you should be able to see them in the processes list). You can also check PostgreSQL logs.
Section about creating index concurrently in PostgreSQL documentation can be helpful. There is also a nice article about concurrent indexes under this link.