I did the stream replication (Slave-Master) using postghreSQL 9.6 on Centos 7.
During the fail Over we promoted the Slave to become the new master.
Now we have solved the problem concerning the old master ,and we want to return to our old architecture where the old Master has to be the master and the new Master has to become the Slave.
When we try to turn new Master to Slave the replication is not working neither the Read-Only.
In fact we are facing challenges when making the new Master to become the Slave or to get to the initial state before the Fail Over.
Note:When we do replication,it just copy(pg_basebackup) data from one server to another ,when we make changes in the master it does not effect the Slave and the Slave still read and write what is not normal.
Is it possible that because of the Fail Over that the new Master Can't become Slave again ?
That is possible if the "old master" was ahead of the promoted slave.
Either start from scratch with a new pg_basebackup or try to reset the old master with pg_rewind.
Of course you have to add an appropriately configured recovery.conf.
Related
I've built a Postgresql 12 HA system, one server is Master (s1) and the other is Slave (s2).
Assume the Master server (s1) failed due to power outage, and the Slave server (s2) got promoted to Master, what will happen when I turn on the Master server (s1) again? will it become Master and the Slave Server (s2) which currently Master be promoted again to Slave ? and will the Master Server (s1) resync what the Slave server (s2) handled while it was down ?
PostgreSQL doesn't do automated failover, you need to be using some other software to do that, and you haven't described what that is.
If you manually promote the replica to be the new master, then if you just turn on the old master it will still think it is the master. Unless the networking level prevents anyone from connecting to it, you will soon have corrupted data as some transactions are sent to one master and some to the other one.
I have 2 PostgreSQL servers and I have set up streaming replication between them.
I have built a shell script which will ping to master server every minute and will promote the slave as master when master won't respond. I am using rh-postgresql95 and 3rd party tools are not working with this version.
My JDBC connection string has comma separated nodes with targetServerType=master like below
jdbc:postgresql://node1,node2/accounting?targetServerType=master.
I just want to know how can I avoid split brain scenario if the slave is promoted to master and old master also comes up somehow?
or
Is there anyway so that old master never comes up automatically?
EDIT
node1 is master and node2 is the slave in my JDBC connection string.
I stopped postgres service on master and promoted the slave to New Master. In this case, service was pointing to the new master.
Then I restarted postgres service on old master and service started pointing to the old master(node1 is old master's ip and it comes first in JDBC connection string.).
So, I didn't get split brain issue, but this scenario will lead to a data inconsistency.
As an idea, your ping script could check if both servers think they are masters:
select pg_is_in_recovery();
A server that is not in recovery is a master. Then you could check the last received WAL's number:
select pg_last_wal_receive_lsn();
The server with the highest LSN is the server that was promoted last. You could then shut down the other server.
If you change your mind about third party options, have a look at the PostgresSQL wiki.
We are putting together an architecture to support High Availability for our Postgres 9.5 Database. We have 1 master and 3 slaves Replicating the data of the master. When The master goes down Slave 1 is promoted to new master but Slave 2 and Slave 3 are still pointing to the previous master and not the updated master node.
Is there a way to make the slaves to read from the new master dynamically . Or does it require changing the configurations manually and restarting the slaves?
There's no short answer, but I'll try:
When primary server fails you'll promote one slave, and reconfigure all other slaves to target the new master. However there's one scenario in which reconfiguring other slaves might not be needed: if you are using "WAL archiving", and your archive is stored on a shared drive which survived the failure of the old primary. If the new primary continues to use the same shared store you might not need to reconfigure other slaves. Again, I've never tried this - you can try.
If your replication mechanism is based on "replication slots" (introduced in PostgreSQL 9.4) - then you have to reconfigure all the slaves. In this case actually you'll have to rebuild replication on all other slaves from scratch (as if they've never been slaves at all). Nevertheless, in my opinion "replication slots" are better choice.
Regarding automation: You've asked if it is possible to automatically reconfigure other slaves, but thing you've missed to mention is if you have any failover automation implemented. What I'm trying to say is that PostgreSQL itself will not automatically perform failover (promote one of slaves when the master fails). At least you have to create "trigger file" on the slave to be promoted, and you have to do this manually or by using another product (for example pgpool2).
If you use pgpool2 - you can setup automatic slave reconfiguration by setting follow_master_command pgpool.conf value.
Finally I'll strongly recommend reading this tutorial - it'll make your life easier.
Edit:
I've forgot to say two things:
Automatically reconfiguring all other slaves as soon as the new master is promoted might not be a good idea, especially if you have many slaves. It'll put additional pressure on your new primary, and on your network, so in some cases it is better to postpone this for night hours for example. More on this in the mentioned tutorial.
I've wrote the tutorial.
As e4c5 commented, you can use repmgr for managing this type of tasks. I have tried repmgr and I was done without a problem.
I have followed a tutorial for doing that and here is the link:
http://jensd.be/591/linux/setup-a-redundant-postgresql-database-with-repmgr-and-pgpool
I hope following this tutorial you can do what you want without any problem.
I have a master and slave configured on different servers. When master is down, my slave becomes master and everything seems work as it is. BUT when master is recovered, I cant get any keys from the current master(which was slave at first).
Any helps?
Thanks
What probably happens is that the master recovers without reloading the data properly, and the slave syncs with its master, resetting all its data.
A better practice would be to either:
if the master is down, treat the slave as a read only node, not adding any data to it. and make sure the master recovers all the data properly. This will mean no inconsistencies caused by the down time. This is of course only if you can afford read only operation.
Or - when you fail over to the slave, treat it as the new master, and when the old master goes back up, it MUST become a slave and not assume its former role. Redis sentinel does that automatically for you.
I have one master and two standby servers setup via Streaming Replication(postgresql 9.0.5). Now if master fails i bring one of the standby out of recovery mode by creating trigger file. Now will the secondary standby follow the new master assuming i fix recovery_target_timeline = latest in recovery.conf file?
No, because the new master doesn't know it has a slave. And the new slave doesn't know it has a new master. You have to change the corresponding configuration files to get the required situation.
Check repmgr as well, maybe this is what you're looking for.