We are using streaming replication in a Postgres v10 system. We have managed to get the failover from Primary to Standby to work correctly, but we cannot work out how to failback to the original setup.
Our setup may be a bit unusual, as each server is in a different data-center. Failover consists of not only moving the primary database to a different data-center, but the associated client applications. So due to speed/bandwidth reasons it is important that after a failover we can revert (failback) to our normal configuration.
Our setup is as follows:
Primary1 ----replicate---> Standby1 ---replicate---> HotStandby2
As stated we can kill the master and failover to the following situation
Standby1 ---replicate---> HotStandby2
We now would like to failback, so we build a new database called Primary2 and set it as a replication of Standby1
Standby1 ---replicate---> HotStandby2
---replicate---> Primary2
After a while Primary2 is in sync with Standby1, so we would like to promote Primary2 to be a active server, and then reconfigure Standby2 to be a replication of Primary2 to get back to the original state
Primary2 ----replicate---> Standby1 ---replicate---> HotStandby2
It is easy to promote Primary2 to be active, but we cannot find any way of moving Standby2 from an active to standby state.
We have tried
insuring that Primary2 is in sync with Standby1
Changing the recover.done to recover.conf
adding recovery_target_timeline = 'latest' to the recover.conf
Setting up new replication slots
Using pg_rewind
Using pg_rewind along with archive WAR files
Most of our attempts result in Standby1 still running as a master and some strange messages in the logs files about diverging timelines....
Has anyone have any ideas or can point us to some articles - tutorials, etc.
Is this even possible in Postgres? Or should we then rebuild Standby1 and if so what about HotStandby2?
Related
I have set up Postgres 11 streaming replication cluster. Standby is a "hot standby". Is it possible to attach the second standby as a warm standby?
I assume that you are talking about WAL file shipping when you are speaking of a “warm standby”.
Sure, there is nothing that keeps you from adding a second standby that ships WAL files rather than directly attaching to the primary, but I don't see the reason for that.
According to this decent documentation of Postgres 11 streaming replication architecture, you can set the sync_state of a 2nd slave instance to be potential. This means that if/when the 1st sync slave fails, the detected failure (through ACK communication) will result in the 2nd slave will move from potential to sync becoming the active replication server. --see Section 11.3 - Managing Multiple Stand-by Servers in that link for more details.
I'm reading the article below how to achieve streaming replication in Postgres DB.
https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql
Some things are not quite clear
1) Are both DB instances active OR the slave instance is just a clone of master (o it communicates with master, but not the backend?
2) If DB master node failed, what will happen until second node will get back online? Is this covered by default by just having wal sender and wal receiver processes or something else needs to be added?
3) Which DB_HOST:PORT should be configured in the backend app if for example I have two backend nodes (both of them are active)?
If hot_standby = on in postgresql.conf, clients can connect to the standby, but only read data and not modify them. The standby is an identical physical cooy of the primary, just as if you had copied it file by file.
If the primary fails, the standby will remain up and running, but you still can only read data until somebody promotes the standby. You have to understand that PostgreSQL does not ship with cluster software that allows this to happen automatically. You have to usr some other software like Patroni for that.
That depends on the API your software is using. With libpq (the C API) or JDBC you can have a connection string that contains both servers and will select the primary automatically, but with other clients you may have to use external load balancing software.
I am running two databases (PostgreSQL 9.5.7) in a master/slave setup. My application is connecting to a pgpool instance which routes to the master database (and slave for read only queries).
Now I am trying to scale out some data to another read-only database instance containing only a few tables.
This works perfectly using pglogical directly on the master database.
However if the master transitions to slave for some reason, pglogical can't replicate any longer because the node is in standby.
Tried following things:
subscribed on the slave since it's less likely to go down, or overheated: Can't replicate on standby node.
subscribed via pgpool server: pgpool doesn't accept replication connections.
subscribed to both servers: pglogical config gets replicated along, so can't give them different node names.
The only thing I can think of now is to write my own tcp proxy which regularly checks for the state of the server to which I can subscribe to.
Is there any other/easier way I can solve this ?
Am I using the wrong tools perhaps ?
Ok so it seems that there are no solutions for this problem just yet.
Since the data in my logically replicated database is not changing fast, there is no harm if the replication stops for a moment.
Actions on failover could be:
Re-subscribe to the promoted master.
or promote standby node back to master after failover.
We are putting together an architecture to support High Availability for our Postgres 9.5 Database. We have 1 master and 3 slaves Replicating the data of the master. When The master goes down Slave 1 is promoted to new master but Slave 2 and Slave 3 are still pointing to the previous master and not the updated master node.
Is there a way to make the slaves to read from the new master dynamically . Or does it require changing the configurations manually and restarting the slaves?
There's no short answer, but I'll try:
When primary server fails you'll promote one slave, and reconfigure all other slaves to target the new master. However there's one scenario in which reconfiguring other slaves might not be needed: if you are using "WAL archiving", and your archive is stored on a shared drive which survived the failure of the old primary. If the new primary continues to use the same shared store you might not need to reconfigure other slaves. Again, I've never tried this - you can try.
If your replication mechanism is based on "replication slots" (introduced in PostgreSQL 9.4) - then you have to reconfigure all the slaves. In this case actually you'll have to rebuild replication on all other slaves from scratch (as if they've never been slaves at all). Nevertheless, in my opinion "replication slots" are better choice.
Regarding automation: You've asked if it is possible to automatically reconfigure other slaves, but thing you've missed to mention is if you have any failover automation implemented. What I'm trying to say is that PostgreSQL itself will not automatically perform failover (promote one of slaves when the master fails). At least you have to create "trigger file" on the slave to be promoted, and you have to do this manually or by using another product (for example pgpool2).
If you use pgpool2 - you can setup automatic slave reconfiguration by setting follow_master_command pgpool.conf value.
Finally I'll strongly recommend reading this tutorial - it'll make your life easier.
Edit:
I've forgot to say two things:
Automatically reconfiguring all other slaves as soon as the new master is promoted might not be a good idea, especially if you have many slaves. It'll put additional pressure on your new primary, and on your network, so in some cases it is better to postpone this for night hours for example. More on this in the mentioned tutorial.
I've wrote the tutorial.
As e4c5 commented, you can use repmgr for managing this type of tasks. I have tried repmgr and I was done without a problem.
I have followed a tutorial for doing that and here is the link:
http://jensd.be/591/linux/setup-a-redundant-postgresql-database-with-repmgr-and-pgpool
I hope following this tutorial you can do what you want without any problem.
I have one master and two standby servers setup via Streaming Replication(postgresql 9.0.5). Now if master fails i bring one of the standby out of recovery mode by creating trigger file. Now will the secondary standby follow the new master assuming i fix recovery_target_timeline = latest in recovery.conf file?
No, because the new master doesn't know it has a slave. And the new slave doesn't know it has a new master. You have to change the corresponding configuration files to get the required situation.
Check repmgr as well, maybe this is what you're looking for.