PostgreSql 9.1 has master-slave synchronous replication. Suppose the master is machine A and the slave is machine B.
If the master fails, how does PostgreSQL know when to make the slave the master? What if the slave incorrectly thought the master was down because of a temporary network glitch on the master where the client program could still contact the master though.
And moreover, how would my client program know the slave in the new master and more importantly is ready to accept writes. Does the slave send a message to the client?
Check repmgr, it's one of its jobs is to deal with this issue.
Typically you want to use a promotion-management system like repmgr or patroni. Then you want to use some sort of a high availability proxy (could be pgbouncer or haproxy) to handle the actual abstraction so your applications do not need to know what system is master.
In answer to your question, most of these systems use a heartbeat to determine if there is a problem. Patroni goes out over the etcd heartbeat. Repmgr has its own heartbeat check. With Repmgr you need to write hook scripts to take care of stonith, and so forth.
Related
I'm searching for HA solutions without load balancing in the master-slave model, using postgresql. My favorite solution so far is log shipping synchronous replication. But I have one main concern, and that is, if my slave server becomes unavailable, will my master server continue it's operation? Or will it wait for the acknowledgment of my slave server until it's up again?
If you have only one standby, the master will halt ( by design ).
The master will still serve read-only statements, but all writes will be blocked until the standby comes back.
You can avoid this scenario by providing multiple candidates in synchronous_standby_names.
See SYNCHRONOUS-REPLICATION in the PostgreSQL Docs.
I found another way to prevent the master halt at slave crash. We can use wal_sender_timeout in masters postgresql.conf file to disconnect from the slave if it's been crashed.
I have 2 PostgreSQL servers and I have set up streaming replication between them.
I have built a shell script which will ping to master server every minute and will promote the slave as master when master won't respond. I am using rh-postgresql95 and 3rd party tools are not working with this version.
My JDBC connection string has comma separated nodes with targetServerType=master like below
jdbc:postgresql://node1,node2/accounting?targetServerType=master.
I just want to know how can I avoid split brain scenario if the slave is promoted to master and old master also comes up somehow?
or
Is there anyway so that old master never comes up automatically?
EDIT
node1 is master and node2 is the slave in my JDBC connection string.
I stopped postgres service on master and promoted the slave to New Master. In this case, service was pointing to the new master.
Then I restarted postgres service on old master and service started pointing to the old master(node1 is old master's ip and it comes first in JDBC connection string.).
So, I didn't get split brain issue, but this scenario will lead to a data inconsistency.
As an idea, your ping script could check if both servers think they are masters:
select pg_is_in_recovery();
A server that is not in recovery is a master. Then you could check the last received WAL's number:
select pg_last_wal_receive_lsn();
The server with the highest LSN is the server that was promoted last. You could then shut down the other server.
If you change your mind about third party options, have a look at the PostgresSQL wiki.
I am running two databases (PostgreSQL 9.5.7) in a master/slave setup. My application is connecting to a pgpool instance which routes to the master database (and slave for read only queries).
Now I am trying to scale out some data to another read-only database instance containing only a few tables.
This works perfectly using pglogical directly on the master database.
However if the master transitions to slave for some reason, pglogical can't replicate any longer because the node is in standby.
Tried following things:
subscribed on the slave since it's less likely to go down, or overheated: Can't replicate on standby node.
subscribed via pgpool server: pgpool doesn't accept replication connections.
subscribed to both servers: pglogical config gets replicated along, so can't give them different node names.
The only thing I can think of now is to write my own tcp proxy which regularly checks for the state of the server to which I can subscribe to.
Is there any other/easier way I can solve this ?
Am I using the wrong tools perhaps ?
Ok so it seems that there are no solutions for this problem just yet.
Since the data in my logically replicated database is not changing fast, there is no harm if the replication stops for a moment.
Actions on failover could be:
Re-subscribe to the promoted master.
or promote standby node back to master after failover.
I am new at postgresql replication and reading documentations. I looked some replication tutorials. There is one primary machine and multiple stand by machines.
what will be if primary machine fails? Can I convert stand by machines to primary machine?
if stand by machines fails, will system gives error? (for example database errors because of connection error.)
I need some fail scenarios answer.
if primary machine fails slaves won't be able to get replication from master. To promote slave a new master, run pg_ctl promote ont it or just create a trigger file. If other slaves were configured to stream from this one (cascaded replication) you will have to define latest timeline in recovery.conf.
if postgres dies on slave, usually you have reason for it in postgres.log. If you ask if master will notice death of any of slaves - no.
If you want some tools, check https://github.com/2ndQuadrant/repmgr
We are putting together an architecture to support High Availability for our Postgres 9.5 Database. We have 1 master and 3 slaves Replicating the data of the master. When The master goes down Slave 1 is promoted to new master but Slave 2 and Slave 3 are still pointing to the previous master and not the updated master node.
Is there a way to make the slaves to read from the new master dynamically . Or does it require changing the configurations manually and restarting the slaves?
There's no short answer, but I'll try:
When primary server fails you'll promote one slave, and reconfigure all other slaves to target the new master. However there's one scenario in which reconfiguring other slaves might not be needed: if you are using "WAL archiving", and your archive is stored on a shared drive which survived the failure of the old primary. If the new primary continues to use the same shared store you might not need to reconfigure other slaves. Again, I've never tried this - you can try.
If your replication mechanism is based on "replication slots" (introduced in PostgreSQL 9.4) - then you have to reconfigure all the slaves. In this case actually you'll have to rebuild replication on all other slaves from scratch (as if they've never been slaves at all). Nevertheless, in my opinion "replication slots" are better choice.
Regarding automation: You've asked if it is possible to automatically reconfigure other slaves, but thing you've missed to mention is if you have any failover automation implemented. What I'm trying to say is that PostgreSQL itself will not automatically perform failover (promote one of slaves when the master fails). At least you have to create "trigger file" on the slave to be promoted, and you have to do this manually or by using another product (for example pgpool2).
If you use pgpool2 - you can setup automatic slave reconfiguration by setting follow_master_command pgpool.conf value.
Finally I'll strongly recommend reading this tutorial - it'll make your life easier.
Edit:
I've forgot to say two things:
Automatically reconfiguring all other slaves as soon as the new master is promoted might not be a good idea, especially if you have many slaves. It'll put additional pressure on your new primary, and on your network, so in some cases it is better to postpone this for night hours for example. More on this in the mentioned tutorial.
I've wrote the tutorial.
As e4c5 commented, you can use repmgr for managing this type of tasks. I have tried repmgr and I was done without a problem.
I have followed a tutorial for doing that and here is the link:
http://jensd.be/591/linux/setup-a-redundant-postgresql-database-with-repmgr-and-pgpool
I hope following this tutorial you can do what you want without any problem.