Stop PostgreSQL streaming replication - postgresql

I want to make my PostgreSQL master / slave streaming replication setup into a single master slave setup without replication and without HA.
How is it possible to tell master that it no longer has slave and it should not replicate its data to the slave ?
Of course it should also not keep WALs to be sent to the slave as is done when the slave is temporarily down.

Here is what I did:
on both master / slave, edit pg_hba.conf, remove:
host replication replicator x.x.x.x/32 md5
in master, reload config, via:
select pg_reload_conf()
in slave's data dir, remove file: standby.signal
restart slave,
Then slave will keep the data, but don't replica from master any more, and slave is writable.

Depending on the version of Postgres you're using, it's possible your replication is done using "replication slots". If you don't have a client streaming from a replication slot anymore, you can drop the slot. Here are some useful queries:
Get disk usage per replication slot (for Postgres 9.6):
SELECT
redo_location,
slot_name,restart_lsn,
round((redo_location-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM
pg_control_checkpoint(),
pg_replication_slots;
Get disk usage per replication slot (for Postgres 10+):
SELECT redo_lsn,
slot_name,
restart_lsn,
round((redo_lsn-restart_lsn) / 1024 / 1024 / 1024, 2) AS GB_behind
FROM pg_control_checkpoint(),
pg_replication_slots;
Drop replication slot:
select pg_drop_replication_slot('slot_name');
References:
https://severalnines.com/database-blog/using-postgresql-replication-slots :

it is very important to also stop walsender to recover server memory with the following command:
select pg_terminate_backend(pid) from pg_stat_replication;
this will free up your server storage memory.

Related

Can we do streaming and logical replication in postgresql Slave at same time?

We have PostgresSQL cluster with 1 master and 2 slave configuration , we want to enable logical replication from Slave , as the master a pretty work heavy database , we don’t want to put more load on DB , Is it possible if we can start both type of replication from any slave currently replicating through streaming replication
we have setup with master it worked , not sure with slave , we did few test run in older vesrion which failed
No, the logical replication primary cannot be a streaming replication standby server.

Postgresql 9.6 cascading stream replication issue: physical replication slots not synced from master to slave

I am running postgresql 9.6 and trying to set up the cascading physical replications.
However, when I notice that the replication slots that set up on the master is not shown on the cascading standby units so that the downstream standby fails the basebackup when a replication slot is specified.
on my master:
wal_level = replica
wal_log_hints = on
max_wal_senders = 10
wal_keep_segments = 1024
archive_mode = on
archive_command = 'test ! -f /backup/pg_archive_5432/%f && cp %p /backup/pg_archive_5432/%f'
on my standby:
hot_standby = on
Is this normal behavior on 9.6? If anyone is doing active-standby setup, can you check on your standby unit?
Thank you very much
Replication slots are not replicated. So if you want to use cascading replication with replication slots, you have to create another replication slot on the first standby server. That replication slot can be used by the second standby server.
If you think about it, that makes sense: the cascading standby is not at the same place in the WAL stream as the first one, so they need different replication slots.

Primary and standby server at different timelines in postgres

I am very new to postgres and being new I got stuck at a point and need some help, please pardon if you find it silly.
I am doing a pgpool HA and at postgres level i have streaming replication between 3 nodes of postgresql-9.5 - 1 master and 2 slaves
I was trying to configure auto failover but when i switched back to my original master, and restarted the postgres service, I am getting the following error:
slave 1-highest timeline 1 of the primary is behind recovery timeline 11
slave 2-highest timeline 1 of the primary is behind recovery timeline 10
slave 3-highest timeline 1 of the primary is behind recovery timeline 3
I tried deleting pg_xlog files in slaves and copying all the files from master pg_xlog into the slaves and then did a rsync.
i also did a pg_rewind but it says:
target server needs to use either data checksums or wal_log_hints = on
(I have wal_log_hints = on set in postgresql.conf already)
I've tried doing a pg_basebackup but since the data base server in slaves are still starting up its not able to connect to the server
Is there any way to bring the master and the slave at a same timeline?
In my case, it happened because ( experimentally ), I updated the standby database tables and again when I simulate the master-standby streaming replication I got the same errors.
So once again I cleaned the whole standby database directory and migrate the master database using cmd like
"pg_basebackup -P -R -X stream -c fast -h 10.10.40.105 -U postgres -D standby/"
I think something is wrong in your pgpool configuration. What tool you have been using for manement of replication and master-slave control? Is it post master or repmgr?
I was trying to configure pgpool with 3 data nodes using a tutorial from http://jensd.be/591/linux/setup-a-redundant-postgresql-database-with-repmgr-and-pgpool and have done it correctly.
Also you can lean auto failover here.
(These question is obviously duplicate of this one, so I'll repeat the answer also.)
I'm not sure what you exactly mean by "when i switched back to my original master", but it looks that you are doing the wrongest possible thing in PostgreSQL streaming replication - introducing the second master.
The most important thing you should know about PostgreSQL replication is that once the failover is performed, you cannot simply "switch back to original master" - there's now a new master in cluster, and existence of two masters will make damage.
After a slave is promoted to master, the only way for you to re-join the old master is to:
Destroy it (delete the data directory);
Join it as a slave.
If you want it to be master again you'll continue with the following:
Let it run for awhile as a slave so that it can sync the data;
Kill temporary master and failover to old master;
Rejoin temporary master again as a slave.
You cannot simply switch master servers! Master can be created ONLY by failover (promoting a slave)
You should also know that whenever you are performing failover (whenever the master is changed), all slaves (except for the one that is promoted) need to be reconfigured to target the new master.
I suggest you reading this tutorial - it'll help.

postgresql streaming replication -- continuous archiving?

im trying to set up streaming replication, but for some reason when i update the database on the master, the changes are not reflected on the standby server UNTIL i restart the postgresql service on the master. (i see new xlog files in master server but these do not get synced to the standby server). when i restart the service on master, i finally see new files added to my shared wal_archive folder
the only way I can make it sync automatically is if i set the archive_timeout.
Master:
wal_level = 'hot_standby' # minimal, archive, hot_standby, or logical
archive_mode = on # allows archiving to be done
# (change requires restart)
archive_command = 'copy "%p" "\\\\VBOXSVR\\wal_archive\\%f"'
max_wal_senders = 3 # max number of walsender processes
# (change requires restart)
wal_keep_segments = 10 # in logfile segments, 16MB each; 0 disables
pb_hba.conf
host replication postgres slaveip/32 trust
It sounds like you're using archive-based replication without streaming. So it's only replicating when a WAL archive is finished and a new one is opened, which happens:
When the server does a checkpoint before a clean shutdown
When a WAL archive is filled by write activity and a new one is needed
at archive_timeout time
If you want continuous replication you need to use streaming replication. See the manual for details. This involves setting a connection string in your downstream server's recovery.conf so it can connect directly to the upstream master to receive new writes in near-real-time.
You should still leave archive based replication enabled, because this allows the replica to recover if it's disconnected for a while. It's also useful for point-in-time recovery.

How do I fix a PostgreSQL 9.3 Slave that Cannot Keep Up with the Master?

We have a master-slave replication configuration as follows.
On the master:
postgresql.conf has replication configured as follows (commented line taken out for brevity):
max_wal_senders = 1
wal_keep_segments = 8
On the slave:
Same postgresql.conf as on the master. recovery.conf looks like this:
standby_mode = 'on'
primary_conninfo = 'host=master1 port=5432 user=replication password=replication'
trigger_file = '/tmp/postgresql.trigger.5432'
When this was initially setup, we performed some simple tests and confirmed the replication was working. However, when we did the initial data load, only some of the data made it to the slave.
Slave's log is now filled with messages that look like this:
< 2015-01-23 23:59:47.241 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:47.241 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
< 2015-01-23 23:59:52.259 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:52.260 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
< 2015-01-23 23:59:57.270 EST >LOG: started streaming WAL from primary at F/52000000 on timeline 1
< 2015-01-23 23:59:57.270 EST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000F00000052 has already been removed
After some analysis and help on the #postgresql IRC channel, I've come to the conclusion that the slave cannot keep up with the master. My proposed solution is as follows.
On the master:
Set max_wal_senders=5
Set wal_keep_segments=4000 . Yes I know it is very high, but I'd like to monitor the situation and see what happens. I have room on the master.
On the slave:
Save configuration files in the data directory (i.e. pg_hba.conf pg_ident.conf postgresql.conf recovery.conf)
Clear out the data directory (rm -rf /var/lib/pgsql/9.3/data/*) . This seems to be required by pg_basebackup.
Run the following command:
pg_basebackup -h master -D /var/lib/pgsql/9.3/data --username=replication --password
Am I missing anything ? Is there a better way to bring the slave up-to-date w/o having to reload all the data ?
Any help is greatly appreciated.
The two important options for dealing with the WAL for streaming replication:
wal_keep_segments should be set high enough to allow a slave to catch up after a reasonable lag (e.g. high update volume, slave being offline, etc...).
archive_mode enables WAL archiving which can be used to recover files older than wal_keep_segments provides. The slave servers simply need a method to retrieve the WAL segments. NFS is the simplest method, but anything from scp to http to tapes will work so long as it can be scripted.
# on master
archive_mode = on
archive_command = 'cp %p /path_to/archive/%f'
# on slave
restore_command = 'cp /path_to/archive/%f "%p"'
When the slave can't pull the WAL segment directly from the master, it will attempt to use the restore_command to load it. You can configure the slave to automatically remove segments using the archive_cleanup_commandsetting.
If the slave comes to a situation where the next WAL segment it needs is missing from both the master and the archive, there will be no way to consistently recover the database. The only reasonable option then is to scrub the server and start again from a fresh pg_basebackup.
You can configure replication slots for postgress to keep WAL segments for replica mentioned in such slot.
Read more at https://www.percona.com/blog/2018/11/30/postgresql-streaming-physical-replication-with-slots/
On master server run
SELECT pg_create_physical_replication_slot('standby_slot');
On slave server add next line to recovery.conf
primary_slot_name = 'standby_slot'
actually to recover, you don't have to drop the whole DB and start from scratch. since master has up-to-date binary, you can do following to recover the slave and bring them back to in-sync:
psql -c "select pg_start_backup('initial_backup');"
rsync -cva --inplace --exclude=*pg_xlog* <data_dir> slave_IP_address:<data_dir>
psql -c "select pg_stop_backup();"
Note:
1. slave has to be turned down by service stop
2. master will turn to read-only due to query pg_start_backup
3. master can continue serving read only queries
4. bring back slave at the end of the steps
I did this in prod, it works perfect for me.
slave and master are in sync and there is no data loss.
You will get that error if keep_wal_segments setting is too low.
When you set the value for keep_wal_segments consider that "How long is the pg_basebackup taking?"
Remember that segments are generated about every 5 minutes, so if the backup takes an hour, you need at least 12 segments saved. At 2 hours, you need 24, etc. I would set the value to about 12.2 segments/hour of backup.
As Ben Grimm suggested in the comments, this is a question of making sure to set segments to the maximum possible value to allow the slave to catch up.