In my postgresql 9.6 instance I have 1 production database. When I query the size of all databases :
combit=> Select pg_database.datname,pg_size_pretty(pg_database_size(pg_database.datname)) as size from pg_database;
datname | size
-----------+---------
template0 | 7265 kB
combit | 285 GB
postgres | 7959 kB
template1 | 7983 kB
repmgr | 8135 kB
(5 rows)
When I check what are the big tables in my database (includes indexes) :
combit=> SELECT nspname || '.' || relname AS "relation",
combit-> pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
combit-> FROM pg_class C
combit-> LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
combit-> WHERE nspname NOT IN ('pg_catalog', 'information_schema')
combit-> AND C.relkind <> 'i'
combit-> AND nspname !~ '^pg_toast'
combit-> ORDER BY pg_total_relation_size(C.oid) DESC
combit-> LIMIT 20;
relation | total_size
-----------------------------+------------
rep.ps_rf_inst_prod | 48 GB
rep.nap_inter_x5 | 46 GB
rep.man_x5 | 16 GB
rep.tc_fint_x5 | 9695 MB
rep.nap_ip_debit_x5 | 7645 MB
rep.ip__billing | 5458 MB
rep.ps_rd | 3417 MB
rep.nap_ip_discount | 3147 MB
rep.custo_x5 | 2154 MB
rep.ip_service_discou_x5 | 1836 MB
rep.tc_sub_rate__x5 | 294 MB
The total sum is not more than 120G.
When I check the fs directly :
[/data/base] : du -sk * | sort -n
7284 13322
7868 13323
7892 1
8156 166694
298713364 16400
[/data/base] :
16400 is the oid of the combit database. As you can see the size of combit on the fs is about 298G.
I checked for dead tuples in the biggest tables :
combit=>select relname,n_dead_tup,last_autoanalyze,last_analyze,last_autovacuum,last_vacuum from pg_stat_user_tables order by n_live_tup desc limit4;
-[ RECORD 1 ]----+------------------------------
relname | ps_rf_inst_prod
n_dead_tup | 0
last_autoanalyze | 2017-12-04 09:00:16.585295+02
last_analyze | 2017-12-05 16:08:31.218621+02
last_autovacuum |
last_vacuum |
-[ RECORD 2 ]----+------------------------------
relname | man_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-05 06:02:07.189184+02
last_analyze | 2017-12-05 16:12:58.130519+02
last_autovacuum |
last_vacuum |
-[ RECORD 3 ]----+------------------------------
relname | tc_fint_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-05 06:04:06.698422+02
last_analyze |
last_autovacuum |
last_vacuum |
-[ RECORD 4 ]----+------------------------------
relname | nap_inter_x5
n_dead_tup | 0
last_autoanalyze | 2017-12-04 08:54:16.764392+02
last_analyze | 2017-12-05 16:10:23.411266+02
last_autovacuum |
last_vacuum |
I run vacuum full on all 5 top tables 2 hours ago and it didnt free alot of space...
On this database the only operations that happen are truncate , insert and select. So how can it be that I had dead tuples on some of my tables ? If I only run truncate,select,insert query tuples shouldnt be created..
And the bigger question, Where are the missing 180G ?
Just wanted to mention that the solution was dumping the database with pg_dump into a file, dropping the database and then restoring it. I had in the database`s directory files that represented objects that were no longer exist.
Related
Table queried size is not same as on physical layout
and table fsm file is of same size as table file. Ideally It should have been very small comparatively.
Postgresql Engine= 13
OS = centos7
Table file size:
# du -sh 16385_vm
8.0K 16385_vm
# du -sh 16385
24K 16385
# du -sh 16385_fsm
24K 16385_fsm
but when I query table sizes are below:
testing=# select pg_size_pretty(pg_relation_size('test1'));
pg_size_pretty
----------------
24 kB
(1 row)
testing=# select pg_size_pretty(pg_total_relation_size('test1'));
pg_size_pretty
----------------
64 kB
(1 row)
testing=# select pg_size_pretty(pg_table_size('test1'));
pg_size_pretty
----------------
64 kB
(1 row)
testing=# \d+ test1
Table "public.test1"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+-------------------+-----------+----------+---------+----------+--------------+-------------
id | integer | | | | plain | |
name | character varying | | | | extended | |
Access method: heap
testing=#
**Free space Map:**
testing=# SELECT * FROM pg_freespace('test1');
blkno | avail
-------+-------
0 | 1088
1 | 1120
2 | 3456
(3 rows)
toast table size is also zero:
testing=# select * from pg_class where oid=16385;
-[ RECORD 1 ]-------+------
oid | 16385
relname | test1
relnamespace | 2200
reltype | 16387
reloftype | 0
relowner | 10
relam | 2
relfilenode | 16385
reltablespace | 0
relpages | 3
reltuples | 17
relallvisible | 3
reltoastrelid | 16388
relhasindex | f
relisshared | f
relpersistence | p
relkind | r
relnatts | 2
relchecks | 0
relhasrules | f
relhastriggers | f
relhassubclass | f
relrowsecurity | f
relforcerowsecurity | f
relispopulated | t
relreplident | d
relispartition | f
relrewrite | 0
relfrozenxid | 487
relminmxid | 1
relacl |
reloptions |
relpartbound |
testing=#
[root#ip-10-15-11-219 16384]# du -sh 16388
0 16388
[root#ip-10-15-11-219 16384]#
Then how come sql query is returning 64 KB table and total relation size instead of 24 KB.
Why table fsm file size is 24 kB which is equal to actual table size of 24 KB?
I'm struggling with a bloated table that I'm unable to shrink. It has just 6 rows but its size is 140MB+ and it's continously updated\deleted by quick transactions. I tried using VACUUM and VACUUM FULL but there's no result.
These are the table structure and the related statistics:
\d bloated_table
COLUMN | TYPE | Collation | Nullable | DEFAULT
-------------------------+-----------------------------+-----------+----------+---------
col1 | BIGINT | | NOT NULL |
<omissis> | CHARACTER varying(100) | | |
<omissis> | CHARACTER varying(50) | | |
<omissis> | TIMESTAMP WITHOUT TIME ZONE | | |
<omissis> | BIGINT | | |
<omissis> | BIGINT | | |
<omissis> | BIGINT | | |
<omissis> | TEXT | | |
INDEXES:
"<omissis>" PRIMARY KEY, btree (col1)
Referenced BY:
TABLE "<omissis>" CONSTRAINT "<omissis>" FOREIGN KEY (col1) REFERENCES <omissis>(col1)
SELECT ROUND(n_dead_tup::NUMERIC/NULLIF(n_live_tup::NUMERIC,0),2), *
FROM pg_catalog.pg_stat_user_tables
WHERE n_dead_tup>0
ORDER BY 1 DESC NULLS LAST
FETCH FIRST ROW ONLY;
round | relid | schemaname | relname | seq_scan | seq_tup_read | idx_scan | idx_tup_fetch | n_tup_ins | n_tup_upd | n_tup_del | n_tup_hot_upd | n_live_tup | n_dead_tup | n_mod_since_analyze | n_ins_since_vacuum | last_vacuum | last_autovacuum | last_analyze | last_autoanalyze | vacuum_count | autovacuum_count | analyze_count | autoanalyze_count
-----------+----------+----------------------+----------------------------+----------+--------------+----------+---------------+-----------+-----------+-----------+---------------+------------+------------+---------------------+--------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------+------------------+---------------+-------------------
215689.17 | 59328017 | db_bloated_table | bloated_table | 27462255 | 279950918 | 679 | 691 | 20 | 25895488 | 14 | 25476514 | 6 | 1294135 | 7 | 0 | 2022-07-06 07:32:24.031073+00 | 2022-07-06 07:39:54.601903+00 | 2022-07-05 22:06:37.492046+00 | 2022-07-06 07:39:54.657717+00 | 30 | 39195 | 26 | 38875
The table size is 143MB:
SELECT pg_size_pretty(pg_total_relation_size('bloated_table'::regclass));
pg_size_pretty
----------------
143 MB
UPDATE HERE sorry, I badly pasted the VACUUM output:
Following the VACUUM output:
> vacuum (verbose) bloated_table;
INFO: vacuuming "argodb.bloated_table"
INFO: "bloated_table": found 0 removable, 25570 nonremovable row versions in 343 out of 343 pages
DETAIL: 25564 dead row versions cannot be removed yet, oldest xmin: 87657915
There were 16 unused item identifiers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
INFO: vacuuming "pg_toast.pg_toast_59328017"
INFO: "pg_toast_59328017": found 0 removable, 0 nonremovable row versions in 0 out of 0 pages
DETAIL: 0 dead row versions cannot be removed yet, oldest xmin: 87657915
There were 0 unused item identifiers.
Skipped 0 pages due to buffer pins, 0 frozen pages.
0 pages are entirely empty.
CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.
VACUUM
> vacuum (full,verbose) bloated_table;
INFO: vacuuming "argodb.bloated_table"
INFO: "bloated_table": found 0 removable, 29068 nonremovable row versions in 389 pages
DETAIL: 29062 dead row versions cannot be removed yet.
CPU: user: 0.03 s, system: 0.01 s, elapsed: 0.06 s.
VACUUM
And finally, there aren't any long-last opened transaction, abandoned replication slots nor orphaned prepared transactions:
--Abandoned replication slots
>SELECT slot_name, slot_type, DATABASE, xmin
FROM pg_replication_slots
ORDER BY AGE(xmin) DESC;
slot_name | slot_type | DATABASE | xmin
-----------+-----------+----------+------
(0 ROWS)
--Orphaned prepared transactions
> SELECT gid, PREPARED, OWNER, DATABASE, TRANSACTION AS xmin
FROM pg_prepared_xacts
ORDER BY AGE(TRANSACTION) DESC;
gid | PREPARED | OWNER | DATABASE | xmin
-----+----------+-------+----------+------
(0 ROWS)
My environment is the following: PG 13.6 on MS Azure Flexible Server
Thank you in advance for your help.
We did a vacuum full on our table and toast. The dead tuples dropped drastically, however the max transaction id stays pretty much the same. My question is, why did it the max transaction id not go down as dead tuples go down drastically?
Before
select relname,last_autovacuum ,n_tup_upd,n_tup_del,n_tup_hot_upd,n_live_tup,n_dead_tup,n_mod_since_analyze,vacuum_count,autovacuum_count from pg_stat_all_tables where relname in ('examples','pg_toast_16450');
relname | last_autovacuum. | n_tup_upd | n_tup_del | n_tup_hot_upd | n_live_tup | n_dead_tup | n_mod_since_analyze | vacuum_count | autovacuum_count
----------------+-------------------------------+-----------+------------+---------------+------------+------------+---------------------+--------------+------------------
examples | 2022-01-18 23:26:52.432808+00 | 57712813 | 9818 | 48386674 | 3601588 | 306558 | 42208 | 0 | 44
pg_toast_16450 | 2022-01-17 23:14:42.516933+00 | 0 | 5735566377 | 0 | 3763818 | 805501171 | 11472355929 | 0 | 51
SELECT max(age(datfrozenxid)) FROM pg_database;
max
-----------
199857797
After
select relname,last_autovacuum ,n_tup_upd,n_tup_del,n_tup_hot_upd,n_live_tup,n_dead_tup,n_mod_since_analyze,vacuum_count,autovacuum_count from pg_stat_all_tables where relname in ('examples','pg_toast_16450');
relname | last_autovacuum | n_tup_upd | n_tup_del | n_tup_hot_upd | n_live_tup | n_dead_tup | n_mod_since_analyze | vacuum_count | autovacuum_count
----------------+-------------------------------+-----------+-------------+--------------+------------+------------+---------------------+--------------+------------------
examples | 2022-02-01 15:41:17.722575+00 | 120692014 | 9818 | 98148003 | 4172134 | 17666 | 150566 | 1 | 4064
pg_toast_16450 | 2022-02-01 20:49:30.552251+00 | 0 | 16169731895 | 0 | 5557218 | 33365 | 32342853690 | 0 | 15281
SELECT max(age(datfrozenxid)) FROM pg_database;
max
-----------
183888023
Yes, that is as expected. You need VACUUM to freeze tuples. VACUUM (FULL) doesn't.
Users tend to be confused, because both are triggered by the VACUUM statement, but VACUUM (FULL) is actually something entirely different from VACUUM. It is not just “a more thorough VACUUM”. The only thing they have in common is that they get rid of dead tuples. VACUUM (FULL) does not modify tuples, as freezing has to do, it just copies them around (or doesn't, if they are dead).
I created a temp table using below query
Drop table if exists tmp_a;
Create temp table tmp_a
(
id int
);
Insert into tmp_a select generate_series(1,10000);
When I queried pg_stat_activity, it is showing as "IDLE" in current_query column for the above session.
I will get the size of all temp table from pg_class table using this query.
But I want the list of temp tables created for a particular session and the size of those temp tables i.e if I created two temp tables from two different sessions then the result should be like below
procpid | temp table name | size | username
12345 | tmp_a | 20 | gpadmin
12346 | tmp_b | 30 | gpadmin
Please share the query if anyone has it
It's actually simpler than you think --
The temporary schema namesapce is the same as the session id --
So...
SELECT
a.procpid as ProcessID,
a.sess_id as SessionID,
n.nspname as SchemaName,
c.relname as RelationName,
CASE c.relkind
WHEN 'r' THEN 'table'
WHEN 'v' THEN 'view'
WHEN 'i' THEN 'index'
WHEN 'S' THEN 'sequence'
WHEN 's' THEN 'special'
END as RelationType,
pg_catalog.pg_get_userbyid(c.relowner) as RelationOwner,
pg_size_pretty(pg_relation_size(n.nspname ||'.'|| c.relname)) as RelationSize
FROM
pg_catalog.pg_class c
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_catalog.pg_stat_activity a ON 'pg_temp_' || a.sess_id::varchar = n.nspname
WHERE c.relkind IN ('r','s')
AND (n.nspname !~ '^pg_toast' and nspname like 'pg_temp%')
ORDER BY pg_relation_size(n.nspname ||'.'|| c.relname) DESC;
And you get --
processid | sessionid | schemaname | relationname | relationtype | relationowner | relationsize
-----------+-----------+------------+--------------+--------------+---------------+--------------
5006 | 9 | pg_temp_9 | tmp_a | table | gpadmin | 384 kB
5006 | 9 | pg_temp_9 | tmp_b | table | gpadmin | 384 kB
(2 rows)
Let's put that process to sleep -- and startup another....
gpadmin=#
[1]+ Stopped psql
[gpadmin#gpdb-sandbox ~]$ psql
psql (8.2.15)
Type "help" for help.
gpadmin=# SELECT nspname
FROM pg_namespace
WHERE oid = pg_my_temp_schema();
nspname
---------
(0 rows)
gpadmin=# Create temp table tmp_a( id int );
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
gpadmin=# SELECT nspname
FROM pg_namespace
WHERE oid = pg_my_temp_schema();
nspname
---------
pg_temp_10
(1 row)
... run the same query ...
processid | sessionid | schemaname | relationname | relationtype | relationowner | relationsize
-----------+-----------+------------+--------------+--------------+---------------+--------------
5006 | 9 | pg_temp_9 | tmp_a | table | gpadmin | 384 kB
5006 | 9 | pg_temp_9 | tmp_b | table | gpadmin | 384 kB
27365 | 10 | pg_temp_10 | tmp_a | table | gpadmin | 384 kB
(3 rows)
I imported data into 30 tables (every table also has indexes and constraints) in my PostgreSQL database. After some check I truncated all the tables and performed vacuum full analyze on all the tables. Some space indeed was freed but I`m sure that more data need to be freed. Before the import my PostgreSQL directory was about 20G. After the import it grew to be 270G. Currently, the size of the data directory is 215G.
I run this select :
SELECT
relname as "Table",
pg_size_pretty(pg_total_relation_size(relid)) As "Size",
pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) as
"External Size"
FROM pg_catalog.pg_statio_user_tables ORDER BY
pg_total_relation_size(relid)
DESC;
and the result was that the biggest tables are 660M (and right now there are only 4 tables that their size bigger than 100M).
Table | Size | External Size
-------------------------------+------------+---------------
my_table_1 | 660 MB | 263 MB
my_Table_2 | 609 MB | 277 MB
my_table_3 | 370 MB | 134 MB
my_table_4 | 137 MB | 37 MB
my_table_5 | 83 MB | 31 MB
my_table_6 | 5056 kB | 24 kB
mariel_test_table | 4912 kB | 8192 bytes
..........
The data/base directory size is 213G.
I run also this select :
SELECT nspname || '.' || relname AS "relation",
pg_size_pretty(pg_relation_size(C.oid)) AS "size"
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_relation_size(C.oid) DESC
LIMIT 20;
output :
relation | size
-----------------------------------+--------
my_table_1 | 397 MB
my_Table_2 | 332 MB
my_Table_3 | 235 MB
my_table_7 | 178 MB
my_table_4 | 100 MB
my_table_8 | 99 MB
The outputs of the selects aren't identical.
tempfile sizes :
SELECT temp_files AS "Temporary files"
, temp_bytes AS "Size of temporary files"
FROM pg_stat_database db;;
Temporary files | Size of temporary files
-----------------+-------------------------
0 | 0
0 | 0
0 | 0
100 | 47929425920
0 | 0
I also tried to restart the PostgreSQL instance and the Linux server. What can I try next?