PostgreSQL - Etcd - Patroni Cluster Restoration Problem - postgresql

I am trying to create PostgreSQL - Etcd - Patroni(PEP) cluster. There are lots of examples on the internet and I have created one and it runs perfect. Yet, this architecture should comply with my company' s backup solution which is NetApp. We are putting the database into backup mode with "SELECT pg_start_backup('test_backup', true);" and then copy all the data files to backup directory.
PEP cluster has a small problem with this solution. Taking backup is running fine, but restoration point is not that much good. In order to restore the leader of the PEP cluster I need stop the database and then move the backup files to the data directory and finally start the restoration. At this point Patroni says the restoration node is a new cluster. Here is the error:
raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
2022-04-11 12:49:29,930 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-04-11 12:49:29,942 INFO: Lock owner: None; I am pgsql_node1
2022-04-11 12:49:29,962 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
Also, when I check the patroni cluster status I saw this:
root#4cddca032454:/data/backup# patronictl -c /etc/patroni/config.yml list
+ Cluster: pgsql (7085327534197401486) --------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+------------+---------+---------+----+-----------+
| pgsql_node1 | 172.17.0.6 | Replica | stopped | | unknown |
| pgsql_node2 | 172.17.0.7 | Replica | running | 11 | 0 |
| pgsql_node3 | 172.17.0.8 | Replica | running | 11 | 0 |
+-------------+------------+---------+---------+----+-----------+
At this point I have a PEP cluster without a leader. So, how can I solve this issue?
(Note: The restoration node attempted to join right cluster because, before starting the restoration I check cluster status and got this result:
root#4cddca032454:/data/backup# patronictl -c /etc/patroni/config.yml list
+ Cluster: pgsql (7085327534197401486) --------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+------------+---------+---------+----+-----------+
| pgsql_node2 | 172.17.0.7 | Replica | running | 11 | 0 |
| pgsql_node3 | 172.17.0.8 | Replica | running | 11 | 0 |
+-------------+------------+---------+---------+----+-----------+
pgsql_node1 is not there.
)
As explained here, "https://patroni.readthedocs.io/en/latest/existing_data.html#existing-data" I can create a new cluster after restoration but my priority saving the cluster. Or do I think wrong, all this steps are same with the converting a standalone PostgreSQL database to PEP cluster?
Please let me know if you need any data or something is not clear.
Here is my leader node patroni config file:
scope: "cluster"
namespace: "/cluster/"
name: 8d454a228d251
restapi:
listen: 172.17.0.2:8008
connect_address: 172.17.0.2:8008
etcd:
host: 172.17.0.2:2379
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
check_timeline: true
postgresql:
use_pg_rewind: true
remove_data_directory_on_rewind_failure: true
remove_data_directory_on_diverged_timelines: true
use_slots: true
postgresql:
listen: 0.0.0.0:5433
connect_address: 172.17.0.2:5432
use_unix_socket: true
data_dir: /data/postgresql/
bin_dir: /usr/lib/postgresql/14/bin
config_dir: /etc/postgresql/14/main
authentication:
replication:
username: "patroni_replication"
password: "123123"
superuser:
username: "patroni_superuser"
password: "123123"
parameters:
unix_socket_directories: '/var/run/postgresql/'
logging_collector: 'on'
log_directory: '/var/log/postgresql'
log_filename: 'postgresql-14-8d454a228d25.log'
restore_command: 'cp /data/backup/%f %p'
recovery_target_timeline: 'latest'
promote_trigger_file: '/tmp/promote'
Thanks!

If I understand well you want to restore the primary server (leader) restoring the data directory with a new set of backup files.
After doing the restore in data directory of leader you need to recreate the patroni cluster (remove the keys in DCS) with patronictl remove option.
Example:
stop pgsql_node2
stop pgsql_node3
stop pgsql_node1
on pgsql_node1:
patronictl -c /etc/patroni/config.yml remove <clustername>
start pgsql_node1
start pgsql_node2
start pgsql_node3

Related

Why is id as SERIAL discontinuous values after failover in RDS Aurora PostgreSQL?

I'm testing failover using RDS Aurora PostgreSQL.
First, create RDS Aurora PostgreSQL and access the writer cluster to create users table.
$ CREATE TABLE users (
id SERIAL PRIMARY KEY NOT NULL,
name varchar(10) NOT NULL,
createAt TIMESTAMP DEFAULT Now() );
And I added one row and checked the table.
$ INSERT INTO users(name) VALUES ('test');
$ SELECT * FROM users;
+----+--------+----------------------------+
| id | name | createdAt |
+----+--------+----------------------------+
| 1 | test | 2022-02-02 23:09:57.047981 |
+----+--------+----------------------------+
After failover of RDS Aurora Cluster, I added another row and checked the table.
$ INSERT INTO users(name) VALUES ('temp');
$ SELECT * FROM users;
+-----+--------+----------------------------+
| id | name | createdAt |
+-----+--------+----------------------------+
| 1 | test | 2022-02-01 11:09:57.047981 |
| 32 | temp | 2022-02-01 11:25:57.047981 |
+-----+--------+----------------------------+
After failover, the id value that should be 2 became 32.
Why is this happening?
Is there any way to solve this problem?
That is to be expected. Index modifications are not WAL logged whenever nextval is called, because that could become a performance bottleneck. Rather, a WAL record is written every 32 calls. That means that the sequence can skip some values after a crash or failover to the standby.
You may want to read my ruminations about gaps in sequences.

pgbouncer login rejected

The second day I can not overcome the connection error through pgbouncer if I use auth_type = hba:
postgres=# create user monitoring with password 'monitoring';
postgres=# create database monitoring owner monitoring;
postgres=# \du+ monitoring
List of roles
Role name | Attributes | Member of | Description
------------+------------+-----------+-------------
monitoring | | {} |
postgres=# \l+ monitoring
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
------------+------------+----------+-------------+-------------+-------------------+---------+------------+-------------
monitoring | monitoring | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 7861 kB | pg_default |
/var/lib/pgsql/10/data/pg_hba.conf:
# TYPE DATABASE USER ADDRESS METHOD
host monitoring monitoring 0.0.0.0/0 trust
local monitoring monitoring trust
/etc/pgbouncer/pgbouncer.ini:
pidfile = /var/run/pgbouncer/pgbouncer.pid
reserve_pool_size = 5
reserve_pool_timeout = 2
listen_port = 6432
listen_addr = *
auth_type = hba
auth_hba_file = /etc/pgbouncer/hba_bouncer.conf
auth_file = /etc/pgbouncer/userlist.txt
logfile = /var/log/pgbouncer/pgbouncer.log
log_connections = 0
log_disconnections = 0
log_pooler_errors = 1
max_client_conn = 5000
server_idle_timeout = 30
pool_mode = transaction
server_reset_query =
admin_users = root
stats_users = root,monitoring
[databases]
* = client_encoding=UTF8 host=localhost port=5432 pool_size=1000
In pg_hba.conf of pgbouncer I also tried to specify specific addresses of interfaces of the server with mask /32, also mask /8, /16 (real mask of my network segment).
The result is only one: login rejected!
/etc/pgbouncer/hba_bouncer.conf:
host monitoring monitoring 0.0.0.0/0 trust
host monitoring monitoring 127.0.0.1/32 trust
/etc/pgbouncer/userlist.txt:
"monitoring" "monitoring"
Connection attempt:
# psql -U monitoring -p 5432 -h 127.0.0.1
psql (10.1)
Type "help" for help.
monitoring=>
# psql -U monitoring -p 6432 -h 127.0.0.1
psql: ERROR: login rejected
We have a use case similar to yours. We are running version 1.12.0 and we ran into the same issue where we also got the "ERROR: login rejected" message.
Turned out after investigation that the permissions on our pg_hba.conf for pg_bouncer was incorrect. Once we gave pgbouncer read permissions, it was working as expected. Unfortunately nothing in the higher logging that we turned on revealed this, and we happened to stumbled across this solution through testing on our own.
Ps. the password hash in the pgbouncer config we had left as "" as we're using trust on our connection. I don't think there is anything different in our config to what you posted otherwise.

Postgresql - 100% sure to use SSL connection with server?

is it possible to check (as not rooted user/SQL question) to check, if my connection from client to server uses SSL (my destination server cas uses both - secured and not secured connection)?
as of 9.5:
https://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-SSL-VIEW
The pg_stat_ssl view will contain one row per backend or WAL sender
process, showing statistics about SSL usage on this connection. It can
be joined to pg_stat_activity or pg_stat_replication on the pid column
to get more details about the connection.
t=# set role notsu ;
SET
Time: 9.289 ms
t=> select * from pg_stat_ssl where pid = pg_backend_pid();
pid | ssl | version | cipher | bits | compression | clientdn
-------+-----+---------+--------+------+-------------+----------
43767 | f | | | | |
(1 row)
Time: 10.846 ms
t=> \du+ notsu
List of roles
Role name | Attributes | Member of | Description
-----------+------------+-----------+-------------
notsu | | {} |
the above shows my connection is not SSL

Postgres BDR replication stopped - replication slot inactive

Our Postgres BDR database system stopped replicating data between the nodes.
When I did a check using the pg_xlog_location_diff I noticed that there is a growing buffer in the replication slot.
SELECT slot_name, database, active, pg_xlog_location_diff(pg_current_xlog_insert_location(), restart_lsn) AS retained_bytes
FROM pg_replication_slots
WHERE plugin = 'bdr';
slot_name | database | active | retained_bytes
-----------------------------------------+--------------+--------+----------------
bdr_26702_6275336279642079463_1_20305__ | ourdatabase | f | 32253352
I also noticed that the slot is marked as active=false.
SELECT * FROM pg_replication_slots;
-[ RECORD 1 ]+----------------------------------------
slot_name | bdr_26702_6275336279642079463_1_20305__
plugin | bdr
slot_type | logical
datoid | 26702
database | ourdatabase
active | f
xmin |
catalog_xmin | 8041
restart_lsn | 0/5F0C6C8
I increased the Postgres logging level, but then only messages I see in the log are:
LOCATION: LogicalIncreaseRestartDecodingForSlot, logical.c:886
DEBUG: 00000: updated xmin: 1 restart: 0
LOCATION: LogicalConfirmReceivedLocation, logical.c:958
DEBUG: 00000: failed to increase restart lsn: proposed 0/7DCE6F8, after 0/7DCE6F8, current candidate 0/7DCE6F8, current after 0/7DCE6F8, flushed up to 0/7DCE6F8
Please let me know if you have an idea how I can re-activate the replication slot and allow the replication to resume.
Except if you have really huuuuuge amount of data, I cannot see any reason for not recreating the replication from scratch. Stop the slave, delete the slot on master, delete data directory on slave, create new slot (with the same name to avoid further changes on slave), do pg_basebackup.
You can find a good tutorial here.

MongoDB vs Couchbase performance on single node

I am doing POC on document data store so for that I have selected MongoDB and CouchBase for evaluation.
Environment Detail is as below
Machine : Centos 6.7, 2 core cpu, CPU MHz: 2494.078, RAM : 7 GB (6 GB Free)
MongoDB db version v3.2.0 with default configuration
CouchBase Version: 4.1.0-5005 Enterprise Edition (Cluster RAM: 3GB, Bucket RAM: 1GB)
Document Size : 326 B
Following is the result of POC
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
| Operation | insert (in 10 batch each bacth is of 100K ) | select query | select query | select query | select query | select query | range query | range query |
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
| Record Count | 1000K | 0 | 100 | 33k | 140k | 334k | 114k | 460k |
| Mongo | 99 sec | 568ms | 792ms | 1500ms | 3800ms | 7800ms | - | 15387ms |
| CouchBase | 370 sec | 8ms | 250ms | 6700ms | 28000ms | 69000ms | 28644ms | - |
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
Client: I have used JAVA sdk and spring data.
There is big diffrence in performance of couchbase and mongodb on single node. Is there there any configuration parameter to increase performnace of couchbase?
It appears the current version of Spring Data MongoDB uses WriteConcern.UNACKNOWLEGED - it's fire and forget. You should enable WriteResultChecking.EXCEPTION or use WriteConcern.ACKNOWLEDGED.
http://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongo.mongo-3.write-concern
What durability options are you using? Are you running out of bucket memory? 2 cpus is on the the low side for couchbase minimum requirements, if compaction is running at the same time as your test I would expect that to make a difference. This can be disabled in the settings.
Couchbase on a single node is not something I would ever run in production, minium 3 nodes, so if you have the time increasing your node count might give you some more meaningful figures
If you have 6GB of memory available, you might want to increase the amount of memory allocated to your bucket. In MongoDB 3.2, WiredTiger will use 60% of memory minus 1GB. For 7GB, that's 2.6 to 3.2GB (I'm not sure if it is 60% of available or 60% of total). Perhaps configure your bucket to match that.