I have a postgresql 10 master db with 2 hot standby servers with streaming replication, and the replication is working correctly. The synchronous_commit is setted to remote_write
Also I have a pgpool 3.7.5 configured with the params:
delay_threshold = 1
sr_check_period = 1
And the the following weights:
master: 1
node1: 3
node2: 3
In the log I can see the node1 and node2 are lagging:
Replication of node:1 is behind 75016 bytes from the primary server (node:0)
The pgpool docs says:
delay_threshold (integer)
Specifies the maximum tolerance level of replication delay in WAL bytes on the standby server against the primary server. If the delay exceeds this configured level, Pgpool-II stops sending the SELECT queries to the standby server and starts routing everything to the primary server even if load_balance_mode is enabled, until the standby catches-up with the primary. Setting this parameter to 0 disables the delay checking. This delay threshold check is performed every sr_check_period. Default is 0.
The problem it's that pgpool sends queries to the hot standbys before they obtained the new data from master through streaming replication.
I enabled the log_per_node_statement = on temporally to be able to see which node the query executes and I can see that queries are sent to the nodes even if there aren't sync when delay_threshold should avoid that.
Am I missing something? When the nodes are behind master the queries are not supposed to go the master?
Thanks in advance.
Other config values of pgpool are:
num_init_children = 120
max_pool = 3
connection_cache = off
load_balance_mode = on
master_slave_sub_mode = 'stream'
replication_mode = off
sr_check_period = 1
first, I think you should check the result of "show pool_nodes" and check if three nodes are properly set with right role (primary, standby, standby).
second, did you set "app_name_redirect_preference_list" or "database_redirect_preference_list" ? If so, That can affect on selecting the node for SELECT query.
And in my opinion, I think delay_threshold = 1 is strict, the unit is bytes and in my case, I use "10000000" on PROD. why don't you just put "/NO LOAD BALANCE/" comment to send specific queries to only master?
And I simply recommend you to upgrade the version of pgpool to 4.0.0 (2018-10-19 released). 3.7.x has mysterious bug on load balancing.
I also faced a similar problem that load balancing is not working properly with the version (3.7.5) even when our configuration has no problem. The pgpool randomly We even contact pgpool developer team to solve this problem but they couldn't find the root cause.
You can check the details in the link below.
https://www.pgpool.net/mantisbt/view.php?id=435.
And this was resolved like charm by upgrading to version 4.0.0.
Related
I have set up Postgres 11 streaming replication cluster. Standby is a "hot standby". Is it possible to attach the second standby as a warm standby?
I assume that you are talking about WAL file shipping when you are speaking of a “warm standby”.
Sure, there is nothing that keeps you from adding a second standby that ships WAL files rather than directly attaching to the primary, but I don't see the reason for that.
According to this decent documentation of Postgres 11 streaming replication architecture, you can set the sync_state of a 2nd slave instance to be potential. This means that if/when the 1st sync slave fails, the detected failure (through ACK communication) will result in the 2nd slave will move from potential to sync becoming the active replication server. --see Section 11.3 - Managing Multiple Stand-by Servers in that link for more details.
My setup:
pgpool-2 V4.0.2
OS Ubuntu
2 aws rds read replicas (master db not included in the setup)
pgpool mode: master_slave mode + sub-mode streaming replication
Purpose of using pgpool (have not achieved)
Evenly split incoming db connections between two replicas, e.g. when there are 20 db connections come to pgpool, pgpool will open 10 connections to replica 1 and open 10 connections to replica 2.
Things that my current setup can do
Load balancing queries, cache connections, watch-dog fail over.
I got reply from official Pgpool-2 developer. Pgpool-2 does not split connections, it handles load balancing for query only, not for connections.
I have pgpool-II for HA and repmgr for automatic failover. Pgpool-II also can also run failover I am just wondering doing automatic failover using pgpool or repmgr? If pgpool can do the failover do I need to use repmgr? and use shell scripts to the promotion of new master?
I want to do the automatic failover via repmgrd and use pgpool only as entry point for the applications. It almost works: I have set the failover_command in pgpool configuration file to an empty string (''). When the primary database fails pgpool executes the failover_command (which does nothing because failover_command is empty), and then it will repeatedly try to find a new master. When repmgrd has done the failover, then pgpool see the new master and it is fine.
If the master is stopped, repmgrd will do the failover after x seconds (depending on reconnect interval and attemps parameters). You have to make sure that the health check interval specified in pgpool is bigger than that otherwise pgpool will detach the stopped database too early : the parameters are health_chek_max_retries and health_check_retry_delay (there is also health_check_period which is not clear). You must also set failover_on_backend_error to no.
I am not very confident in this set-up though, I still need to test it more intensively.
pgpool-II: Connection Pooling, Load Balancing, Automated failover, Limiting Exceeding Connections
In pgpool virtual IP is their application connect to VIP( a delegated_ip parameter on pgpool. conf file) primary goes down then failover happens standby promote to primary ( automatic failover )then there are no-changes on application end.
failover_command to write a script then configure the failover.sh script location on pgpool.conf file failover_command="" parameter
repmgr: replication and failover
In Repmgr If the primary is stopped, it will do the failover after xx seconds (depending on reconnecting interval connect_timeout parameter on /etc/repmgr.conf
I am running two databases (PostgreSQL 9.5.7) in a master/slave setup. My application is connecting to a pgpool instance which routes to the master database (and slave for read only queries).
Now I am trying to scale out some data to another read-only database instance containing only a few tables.
This works perfectly using pglogical directly on the master database.
However if the master transitions to slave for some reason, pglogical can't replicate any longer because the node is in standby.
Tried following things:
subscribed on the slave since it's less likely to go down, or overheated: Can't replicate on standby node.
subscribed via pgpool server: pgpool doesn't accept replication connections.
subscribed to both servers: pglogical config gets replicated along, so can't give them different node names.
The only thing I can think of now is to write my own tcp proxy which regularly checks for the state of the server to which I can subscribe to.
Is there any other/easier way I can solve this ?
Am I using the wrong tools perhaps ?
Ok so it seems that there are no solutions for this problem just yet.
Since the data in my logically replicated database is not changing fast, there is no harm if the replication stops for a moment.
Actions on failover could be:
Re-subscribe to the promoted master.
or promote standby node back to master after failover.
I'd like to use mongodb to distribute a cached database to some distributed worker nodes I'll be firing up in EC2 on demand. When a node goes up, a local copy of mongo should connect to a master copy of the database (say, mongomaster.mycompany.com) and pull down a fresh copy of the database. It should continue to replicate changes from the master until the node is shut down and released from the pool.
The requirements are that the master need not know about each individual slave being fired up, nor should the slave have any knowledge of other nodes outside the master (mongomaster.mycompany.com).
The slave should be read only, the master will be the only node accepting writes (and never from one of these ec2 nodes).
I've looked into replica sets, and this doesn't seem to be possible. I've done something similar to this before with a master/slave setup, but it was unreliable. The master/slave replication was prone to sudden catastrophic failure.
Regarding replicasets: While I don't imagine you could have a set member invisible to the primary (and other nodes), due to the need for replication, you can tailor a particular node to come pretty close to what you want:
Set the newly-launched node to priority 0 (meaning it cannot become primary)
Set the newly-launched node to 'hidden'
Here are links to more info on priority 0 and hidden nodes.