we have a postgres 9.6 replica configured among 2 servers. We used the following configuration to create the replica:
postgresql.conf
wal_level = hot_standby
max_wal_senders = 5
wal_keep_segments = 32
archive_mode = on
archive_command = 'cp %p /archive/%f'
The problem is that the servers has been restarted due to some maintenance tasks and now they are out of sync.
Since the DB is very large, how can we restore the replica and then synchronize the data without having the application down more then 5/10 minutes? Can it be done in background while the application on the master site is being used?
Add a restore_command to recovery.conf that can restore archived WALs, then the standby can use them to catch up.
You need to restart the standby server to activate changes in recovery.conf.
Related
Environment: Postgresql 13.x (dockerized)
I was trying to test the DR setup for PostgreSQL nodes.
pg_basebackup and wal_files archive was taken from the standby mode.
Done restoration on a new node by copying pg_basebackup and configured postgresql.conf to use restore_command pointing to walfiles archive.
#----------------------- RECOVERY CONFIGS -----------------------
restore_command = 'cp /db-restore/mydb/walfiles/%f "%p"'
recovery_target_timeline = 'latest'
recovery_target_action = promote
recovery seems to be fine. Some random select queries returning correct results.
But logfile is throwing below error frequently.
2022-04-19 10:19:53 UTC [291] rep_usr#[unknown] ERROR: replication slot "slot_name" does not exist
2022-04-19 10:19:58 UTC [296] rep_usr#[unknown] ERROR: replication slot "slot_name" does not exist
As I have taken backup from standby, is this restoration making new node as a standby and looking for the replication_slot it used in the previous generation?
How can I make new node as a Master (remove replication_slot info)
What are the proper steps to recover if the backup was taken from standby.
I have 1 master and 2 standby nodes. And planning to take a backup from a standby. So is there any specific changes required for archive_mode and archive_command when using this on a standby node? Current commands:
archive_mode = always
archive_level = logical
archive_command = 'test ! -f /db-archives/walfiles/%f && cp %p /db-archives/walfiles/%f'"
Could someone help with this? Any pointers?
I am sure, db-backup will have info about replication_slot and connection_info as the pg_basebackup itself is a clone of entire DB. To revert configs, I am manually removing postgresql.auto.conf in main directory which contains above parameters.
So how can I remove any other references of replication_slot if there are any in the DB backup?
These error messages don't seem to be thrown by recovery, but by some other tool that connects as database user rep_usr.
Create the replication slot if your application needs it!
I removed all configs and started with fresh.
removed main/postgresql.auto.conf which was present in the backup.
main/postgresql.auto.conf is present in standby nodes when we take pg_basebackup. contains the configs used for pg_basebackup in standby nodes. (slot_name, and connect_info).
As I was restoring backup from standby to a Master, I don't need that postgresql.auto.conf.
I am running postgresql 9.6 and trying to set up the cascading physical replications.
However, when I notice that the replication slots that set up on the master is not shown on the cascading standby units so that the downstream standby fails the basebackup when a replication slot is specified.
on my master:
wal_level = replica
wal_log_hints = on
max_wal_senders = 10
wal_keep_segments = 1024
archive_mode = on
archive_command = 'test ! -f /backup/pg_archive_5432/%f && cp %p /backup/pg_archive_5432/%f'
on my standby:
hot_standby = on
Is this normal behavior on 9.6? If anyone is doing active-standby setup, can you check on your standby unit?
Thank you very much
Replication slots are not replicated. So if you want to use cascading replication with replication slots, you have to create another replication slot on the first standby server. That replication slot can be used by the second standby server.
If you think about it, that makes sense: the cascading standby is not at the same place in the WAL stream as the first one, so they need different replication slots.
We have a PostgreSQL setup with 3 node in each availability of zone and each availability zone is replicated across Availability zone. All replication are hot standby with async replication. We leverage replication slots for each slave AZ cluster.
We often do automated rolling upgrades of DB containers. After sometime, slaves stops replicating from master cluster with messages like following:
2019-08-06 16:57:11.061 UTC [2987482] LOG: replication terminated by primary
server
2019-08-06 16:57:11.061 UTC [2987482] DETAIL: End of WAL reached on timeline 3
at 0/293D7AE0.
2019-08-06 16:57:11.136 UTC [2987477] LOG: new timeline 4 forked off current
database system timeline 3 before current recovery point 0/293D7D28
I am looking for some pointers to fix this problem.
We tried to scale down each cluster to single node in each AZ and we dont see this problem there. My understanding is that since replication across AZ happens with a proxy in front, during rolling upgrade once the master changes, the slaves are not sure how to continue replication.
I wanted to understand if there is something in protcol during replication which can handle something like this.
recovery.conf on slaves looks like:
standby_mode = 'on'
primary_conninfo = 'host=foo-bar.com port=5432 user=repluser password=
sslmode=verify-ca sslcert=/etc/db_repl_secrets/tls.crt
sslkey=/etc/db_repl_secrets/tls.key sslrootcert=/etc/db_repl_secrets/ca.crt'
primary_slot_name = 'stolon_us_east_1'
recovery_target_timeline = 'latest'
That message probably indicates that a standby server was promoted before it could catch up with the primary, and now the primary cannot take the role of a standby server.
Did you run pg_rewind on the server before using it as standby?
If my explanation is off, please describe your procedure in more detail.
im trying to set up streaming replication, but for some reason when i update the database on the master, the changes are not reflected on the standby server UNTIL i restart the postgresql service on the master. (i see new xlog files in master server but these do not get synced to the standby server). when i restart the service on master, i finally see new files added to my shared wal_archive folder
the only way I can make it sync automatically is if i set the archive_timeout.
Master:
wal_level = 'hot_standby' # minimal, archive, hot_standby, or logical
archive_mode = on # allows archiving to be done
# (change requires restart)
archive_command = 'copy "%p" "\\\\VBOXSVR\\wal_archive\\%f"'
max_wal_senders = 3 # max number of walsender processes
# (change requires restart)
wal_keep_segments = 10 # in logfile segments, 16MB each; 0 disables
pb_hba.conf
host replication postgres slaveip/32 trust
It sounds like you're using archive-based replication without streaming. So it's only replicating when a WAL archive is finished and a new one is opened, which happens:
When the server does a checkpoint before a clean shutdown
When a WAL archive is filled by write activity and a new one is needed
at archive_timeout time
If you want continuous replication you need to use streaming replication. See the manual for details. This involves setting a connection string in your downstream server's recovery.conf so it can connect directly to the upstream master to receive new writes in near-real-time.
You should still leave archive based replication enabled, because this allows the replica to recover if it's disconnected for a while. It's also useful for point-in-time recovery.
We have a master-slave replication configuration as follows.
On the master:
postgresql.conf has replication configured as follows (commented line taken out for brevity):
max_wal_senders = 1
wal_keep_segments = 8
On the slave:
Same postgresql.conf as on the master. recovery.conf looks like this:
standby_mode = 'on'
primary_conninfo = 'host=master1 port=5432 user=replication password=replication'
trigger_file = '/tmp/postgresql.trigger.5432'
When this was initially setup, we performed some simple tests and confirmed the replication was working. However, when we did the initial data load, only some of the data made it to the slave.
Slave's log is now filled with messages that look like this:
< 2015-01-23 23:59:47.241 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:47.241 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
< 2015-01-23 23:59:52.259 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:52.260 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
< 2015-01-23 23:59:57.270 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:57.270 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
After some analysis and help on the #postgresql IRC channel, I've come to the conclusion that the slave cannot keep up with the master. My proposed solution is as follows.
On the master:
Set max_wal_senders=5
Set wal_keep_segments=4000 . Yes I know it is very high, but I'd like to monitor the situation and see what happens. I have room on the master.
On the slave:
Save configuration files in the data directory (i.e. pg_hba.conf pg_ident.conf postgresql.conf recovery.conf)
Clear out the data directory (rm -rf /var/lib/pgsql/9.3/data/*) . This seems to be required by pg_basebackup.
Run the following command:
pg_basebackup -h master -D /var/lib/pgsql/9.3/data --username=replication --password
Am I missing anything ? Is there a better way to bring the slave up-to-date w/o having to reload all the data ?
Any help is greatly appreciated.
The two important options for dealing with the WAL for streaming replication:
wal_keep_segments should be set high enough to allow a slave to catch up after a reasonable lag (e.g. high update volume, slave being offline, etc...).
archive_mode enables WAL archiving which can be used to recover files older than wal_keep_segments provides. The slave servers simply need a method to retrieve the WAL segments. NFS is the simplest method, but anything from scp to http to tapes will work so long as it can be scripted.
# on master
archive_mode = on
archive_command = 'cp %p /path_to/archive/%f'
# on slave
restore_command = 'cp /path_to/archive/%f "%p"'
When the slave can't pull the WAL segment directly from the master, it will attempt to use the restore_command to load it. You can configure the slave to automatically remove segments using the archive_cleanup_commandsetting.
If the slave comes to a situation where the next WAL segment it needs is missing from both the master and the archive, there will be no way to consistently recover the database. The only reasonable option then is to scrub the server and start again from a fresh pg_basebackup.
You can configure replication slots for postgress to keep WAL segments for replica mentioned in such slot.
Read more at https://www.percona.com/blog/2018/11/30/postgresql-streaming-physical-replication-with-slots/
On master server run
SELECT pg_create_physical_replication_slot('standby_slot');
On slave server add next line to recovery.conf
primary_slot_name = 'standby_slot'
actually to recover, you don't have to drop the whole DB and start from scratch. since master has up-to-date binary, you can do following to recover the slave and bring them back to in-sync:
psql -c "select pg_start_backup('initial_backup');"
rsync -cva --inplace --exclude=*pg_xlog* <data_dir> slave_IP_address:<data_dir>
psql -c "select pg_stop_backup();"
Note:
1. slave has to be turned down by service stop
2. master will turn to read-only due to query pg_start_backup
3. master can continue serving read only queries
4. bring back slave at the end of the steps
I did this in prod, it works perfect for me.
slave and master are in sync and there is no data loss.
You will get that error if keep_wal_segments setting is too low.
When you set the value for keep_wal_segments consider that "How long is the pg_basebackup taking?"
Remember that segments are generated about every 5 minutes, so if the backup takes an hour, you need at least 12 segments saved. At 2 hours, you need 24, etc. I would set the value to about 12.2 segments/hour of backup.
As Ben Grimm suggested in the comments, this is a question of making sure to set segments to the maximum possible value to allow the slave to catch up.