It is very Noob question about compressing Archive Logs in PostgreSQL 10.
My situation.
Ubuntu 18
postgresql 10
pg_wal - 10Gb
path - /var/lib/postgresql/10/main/pg_wal/
I have a big pg_wal folder right now it is 10 Gg and growing. In pg_wal I saw a very large number journal of files capacity 16mb.
Right now I don't have a capacity.
I read the postgresql manual 25.3.6.2 (gzip)
and I saw two command it is
archive_command = 'gzip < %p > /var/lib/pgsql/archive/%f'
restore_command = 'gunzip < /mnt/server/archivedir/%f > %p'
Can I use the path /var/lib/postgresql/10/main/pg_wal/
in archive_command?
archive_command = 'gzip <%p> /var/lib/postgresql/10/main/pg_wal/%f'.
restore_command = ''gunzip < /var/lib/postgresql/10/main/pg_wal/%f > %p'
Is it a possible or it is stupid idea?
No, you can only archive the WAL segments to a different destination.
First, remove whatever obstacle kept the WAL segments from being deleted:
failing archive_command (check pg_stat_archiver)
abandoned replication slot (check pg_replication_slots)
high wal_keep_segments
Do not manually remove or modify files in pg_wal.
Related
Trying to achieve sync streaming to barman server and i need to add an entry to postgresql.conf for this parameter, which already has an entry and tried a few variations but does not work. Any ideas? Also tried '&&' but in vain
synchronous_standby_names='ANY 1 (*)',barman-wal-archive
2022-06-10 16:50:54.272 BST [11241-43] # app= LOG: syntax error in
file "/var/lib/pgsql/13/data/postgresql.conf" line 22, near token ","
2022-06-10 16:50:54.272 BST [11241-44] # app= LOG: configuration file
"/var/lib/pgsql/13/data/postgresql.conf" contains errors; no changes
were applied
The syntax you are using is not valid, and you won't be able to specify that Barman should be kept synchronous and any one of the others. The best you can do is
synchronous_standby_names = 'FIRST 2 ("barman-wal-archive", standby1, standby2, standby3)'
(You have to double quote all names that are not standard SQL identifiers, for example if they contain -.)
Then PostgreSQL will always keep Barman synchronized, as well as the first available standby server. But that won't have transactions fail if Barman is not available, which seems to be what you want.
Keep just
synchronous_standby_names='ANY 1 (*)'
and set
synchronous_commit = on
or
synchronous_commit = remote_write
I have two Postgres databases set up in a Primary/Secondary configuration. I tried to setup replication between them, but have hit a road block. Where am I going wrong?
I have checked various configuration files: recovery.conf, postgresql.conf, pg_hba.conf, and all seem to be set up correctly.
This is the error I have found in the pg_log folder:
cp: cannot stat ‘/var/lib/pgsql/walfiles/00000002000001CA0000003E’: No such file or directory
cp: cannot stat ‘/var/lib/pgsql/walfiles/00000003.history’: No such file or directory
2019-04-16 16:17:19 AEST FATAL: database system identifier differs between the primary and standby
2019-04-16 16:17:19 AEST DETAIL: The primary's identifier is 6647133350114885049, the standby's identifier is 6456613398298492847.
I am using PostgreSQL 9.2.23.
This is my recovery.conf:
standby_mode = 'on'
primary_conninfo = 'host=10.201.108.25 port=5432 user=repl-master password=111222333'
restore_command = 'cp -p /var/lib/pgsql/walfiles/%f %p'
trigger_file = '/var/lib/pgsql/i_am_master.pg.trigger'
recovery_target_timeline = 'latest'
archive_cleanup_command = 'pg_archivecleanup /var/lib/pgsql/walfiles %r'
I'd expect replication from Primary to Secondary. So far, nothing.
Appreciate any input/ideas.
You didn't set up replication correctly. You cannot use pg_dump to create the replica, you have to use a physical backup technique like pg_basebackup.
See the documentation for details.
Do not use PostgreSQL 9.2, it is out of support.
My Server has following resources :
[postgres#srv2813 ~]$ free -m
total used free shared buffers cached
Mem: 15929 15118 810 142 12 219
-/+ buffers/cache: 14885 1043
Swap: 8031 2007 6024
[postgres#srv2813 ~]$ cat /proc/cpuinfo | grep processor | wc -l
8
[root#srv2813 postgres]# sysctl kernel.shmall
kernel.shmall = 4194304
[root#srv2813 postgres]# sysctl kernel.shmmax
kernel.shmmax = 17179869184
and My PostgreSQL conf :
default_statistics_target = 100
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = 12GB
work_mem = 32MB
wal_buffers = 16MB
shared_buffers = 3840MB
max_connections = 500
fsync = off
temp_buffers=32MB
But its getting "too many connection" error. The nginx_status page of the webserver shows around 500 active connections when this happens. The server hosts an api severver, so every "http request" invariably initiate a database "read". Its not a "write" heavy thing, but very "read" heavy.
Its possible that i maxed out our sever, but still i expected a little more from a 16G/8 core box considering the "read only" nature of the application. Can i push the PostgreSQL in any other possible direction?
PostgreSQL is process based vs thread based so it does not generally work well with a lot of connections.
I would look at using something like PgBouncer. PgBouncer is a lightweight connection pooler for PostgreSQL.
I have the following problem: i am trying to set up a streaming replication scenario with load balancing. I read various tutorials but i cannot find the mistake. The replication does not work. I do not have a "wal sender/receiver process". The archiving works and everytime the master restarts, the archived wal files are copied to the slave. I even do not get any error. And in configuration file(s) everything looks like fine for me, e.g. master:
wal_level = hot_standby
wal_keep_segments = 32
max_wal_senders = 5
max_replication_slots = 5
wal_sender_timeout = 60s
What irritates me the most is that there is no "wal sender process" and there is no error thrown.
Thank you for any idea,
Sven
UPDATE 1: my recovery.conf:
standby_mode = 'on'
primary_conninfo = 'host=arcserver1 port=5432 user=postgres pass=postgres'
restore_command = 'pg_standby /db/pg_archived %f %p >> /var/log/standby.log'
primary_slot_name='standby1'
and my client postgresql.conf contains:
hot_standby = on
I found the solution:i replaced pg_standby with cp, because pg_standby seems to be only for warm standby, not hot standby.
I'm running Percona 5.5 on a Centos 6.3. I'm using the prepackaged "huge" my.cnf which ships with percona, it matches my server specs pretty well. My database uses innodb tables. Reading the my.cnf file, there is a section pertaining to innodb:
# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 100M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
I uncommented the above lines (obviously leaving the actual comments commented), restarted percona and I get the following message:
Starting MySQL (Percona Server). ERROR! The server quit without updating PID file (/var/lib/mysql/rmdb.pid).
I'm new to managing a database server. What benefit is there to uncommenting the above lines, and why does it crash mySQL when I restart when I do?
Thanks