I have the following problem: i am trying to set up a streaming replication scenario with load balancing. I read various tutorials but i cannot find the mistake. The replication does not work. I do not have a "wal sender/receiver process". The archiving works and everytime the master restarts, the archived wal files are copied to the slave. I even do not get any error. And in configuration file(s) everything looks like fine for me, e.g. master:
wal_level = hot_standby
wal_keep_segments = 32
max_wal_senders = 5
max_replication_slots = 5
wal_sender_timeout = 60s
What irritates me the most is that there is no "wal sender process" and there is no error thrown.
Thank you for any idea,
Sven
UPDATE 1: my recovery.conf:
standby_mode = 'on'
primary_conninfo = 'host=arcserver1 port=5432 user=postgres pass=postgres'
restore_command = 'pg_standby /db/pg_archived %f %p >> /var/log/standby.log'
primary_slot_name='standby1'
and my client postgresql.conf contains:
hot_standby = on
I found the solution:i replaced pg_standby with cp, because pg_standby seems to be only for warm standby, not hot standby.
Related
I have been working on the timescaleDB multinode clustering concept. When I go to add a data node, I run the add_data_node query in the access node at that time I got an error like SSL Connection has been closed unexpectedly
Config in the access node
postgresql.conf
listen_addresses = '*'
enable_partitionwise_aggregate = on
jit = off
Config in Data Node
Postgresql.conf
listen_addresses = '*'
max_prepared_transactions = 150
wal_level = logical
If you know the root cause of the problem let me know
While running the add_data_node query, the connection will be made. if any unexpected error will be thrown from the data_node service. it throws the error with the SSL connection has been closed unexpectedly.
To check the PostgreSQL database error log we need to enable two of the configurations.
log_connections = on
log_disconnections = on
Through the log what I have found is, in the postgresql.conf file, 'shared_preload_libraries' was with the empty string. I want to add the string 'timescaldb' for that variable (shared_preload_libraries).
I will not recommend this enabling log for production.
https://www.digitalocean.com/community/questions/how-can-i-investigate-postgres-managed-server-error-ssl-connection-has-been-closed-unexpectedly
I am using postgres v14.4. I would use barman to make a backup of this instance. Log archives are not needed and hence archive_mode = off.
The problem is that barman refuses to make a backup with the archive mode disabled. Is there a way to do a consistent backup with barman without keeping the archives?
Below is the barman configuration used:
[xxxxxxx]
description = "Cluster x of type oltp"
active = true
archiver = true
streaming_archiver = false
; The mandatory connection info to PostgreSQL
conninfo = host=xxxxxxx port=xxxxx user=barman dbname=postgres
; Activate pg_basebackup
backup_method = postgres
reuse_backup = off
; The mandatory connection info to login with REPLICATION privileges
streaming_conninfo = host=xxxxxxxxx port=xxxxx user=streaming_barman
slot_name = barman_ge
; recover WAL files with barman-wal-restore command
recovery_options = get-wal
path_prefix=/usr/pgsql-14/bin
backup_directory=/barman/aa/bb
I have a postgres active/standby cluster of 2 nodes, I have used repmgr for creating the cluster. The issue is, automatic failover is not happening when I stop the postgres services on master node. Contents of file repmgr.conf on master are as follows:
node_id=1
data_directory='/data/pgdatabase/masterdb/data'
node_name=node1
conninfo='host=IP-Of-Master user=repmgr dbname=repmgr'
failover=automatic
promote_command='repmgr standby promote -f /etc/repmgr/11/repmgr.conf --log-to-file'
follow_command='repmgr standby follow -f /etc/repmgr/11/repmgr.conf --log-to-file --upstream-node-id=%n'
log_file='/var/log/repmgr/repmgr.log'
log_level=NOTICE
reconnect_attempts=4
reconnect_interval=5
repmgrd_service_start_command='sudo systemctl repmgr11 start'
repmgrd_service_stop_command='sudo systemctl repmgr11 stop'
Contents of postgresql.conf are as follows:
listen_addresses = '*'
shared_preload_libraries = 'repmgr'
max_wal_senders = 15
max_replication_slots = 15
wal_level = 'replica'
hot_standby = on
archive_mode = on
archive_command = 'cp %p /var/lib/pgsql/11/archive/%f'
The contents are same on master and slave expect the name, which is node2 on slave.
Can anyone guide me what can be possible reason for automatic failover not happening
I have resolved it myself, I needed to create file /etc/default/repmgrd and add following lines into it
REPMGRD_ENABLED=yes
REPMGRD_CONF="/etc/repmgr.conf"
I have pgbouncer.ini file with the below configuration
[databases]
test_db = host=localhost port=5432 dbname=test_db
[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 0.0.0.0
listen_port = 5433
unix_socket_dir = /var/run/postgresql
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres
#pool_mode = transaction
pool_mode = session
server_reset_query = RESET ALL;
ignore_startup_parameters = extra_float_digits
max_client_conn = 25000
autodb_idle_timeout = 3600
default_pool_size = 250
max_db_connections = 250
max_user_connections = 250
and I have in my postgresql.conf file
max_connections = 2000
does it effect badly on the performance ? because of max_connections in my postgresql.conf ? or it doesn't mean anything and already the connection handled by the pgbouncer ?
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
Is it better to set default_pool_size on PgBouncer equal to the number of CPU cores available on this server?
Shall all of default_pool_size, max_db_connections and max_user_connections to be set with the same value ?
So the idea of using pgbouncer is to pool connections when you can't afford to have a higher number of max_connections in PG itself.
NOTE: Please DO NOT set max_connections to a number like 2000 just like that.
Let's start with an example, if you have a connection limit of 20 and then your app or organization wants to have a 1000 connections at a given time, that is where pooler comes into picture and in this specific case you want the 20 connections to pool that 1000 coming in from the application.
To understand how it actually works let's take a step back and understand what happens when you do not have a connection pooler and only rely on PG config setting for the max connections which in our case is 20.
So when a connection comes in from a client\application etc. the main process of postgresql(PID, i.e. parent ID) spawns a child for that. So each new connection spawns a child process under the main postgres process, like so:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24379 postgres 20 0 346m 148m 122m R 61.7 7.4 0:46.36 postgres: sysbench sysbench ::1(40120)
24381 postgres 20 0 346m 143m 119m R 62.7 7.1 0:46.14 postgres: sysbench sysbench ::1(40124)
24380 postgres 20 0 338m 137m 121m R 57.7 6.8 0:46.04 postgres: sysbench sysbench ::1(40122)
24382 postgres 20 0 338m 129m 115m R 57.4 6.5 0:46.09 postgres: sysbench sysbench ::1(40126)
So now once a connection request is sent, it is received by the POSTMASTER process and creates a child process at OS level under the main parent process. This connection then has a life span of "unlimited" unless close by the application or you have a time out set for idle connections in postgresql.
Now here comes the situation where it can be a very costly affair to manage the connections with a given compute, if they exceed a certain limit. Meaning n number of connections when served have a given compute cost and after some time the OS won't be able to handle a situation with HUGE connections and will in turn cause contentions at different compute level(i.e. Memory, CPU, I/O).
What if you can use the presently spawned child processes(backends) if they are not doing any work. You will save time on getting the child process(backends) and the additional cost as well(this can be different at times). This is where the pool of connections that are always open help to serve different client requests comes in and is also called pooling.
So basically now you have only n connections available but the pooler can manage n+i number of connections to serve the client requests.
This where pg-bouncer helps to reuse the connections. It can be configured with 3 types of pooling i.e Session pooling, Statement pooling and Transaction pooling. Basically bouncer returns the connection back to the pool once it has done, statement level work or transaction level work etc. Only during session pooling it keeps the connections unless it disconnects.
So basically lower down the number of connections at PG conf file level and tune all settings in the bouncer.ini.
To answer the second part:
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
It depends if you have a standalone deployment, server etc.
basically if its on the server itself and you want it to allow connections from everywhere(incoming) use "*" if you want only the local network to be allowed use "127.0.0.0".
For the rest of your questions check this link: pgbouncer docs
I have tried to share a little of what I know, feel free to ask away if anything was unclear or or correct if it was incorrectly mentioned.
I have two Postgres databases set up in a Primary/Secondary configuration. I tried to setup replication between them, but have hit a road block. Where am I going wrong?
I have checked various configuration files: recovery.conf, postgresql.conf, pg_hba.conf, and all seem to be set up correctly.
This is the error I have found in the pg_log folder:
cp: cannot stat ‘/var/lib/pgsql/walfiles/00000002000001CA0000003E’: No such file or directory
cp: cannot stat ‘/var/lib/pgsql/walfiles/00000003.history’: No such file or directory
2019-04-16 16:17:19 AEST FATAL: database system identifier differs between the primary and standby
2019-04-16 16:17:19 AEST DETAIL: The primary's identifier is 6647133350114885049, the standby's identifier is 6456613398298492847.
I am using PostgreSQL 9.2.23.
This is my recovery.conf:
standby_mode = 'on'
primary_conninfo = 'host=10.201.108.25 port=5432 user=repl-master password=111222333'
restore_command = 'cp -p /var/lib/pgsql/walfiles/%f %p'
trigger_file = '/var/lib/pgsql/i_am_master.pg.trigger'
recovery_target_timeline = 'latest'
archive_cleanup_command = 'pg_archivecleanup /var/lib/pgsql/walfiles %r'
I'd expect replication from Primary to Secondary. So far, nothing.
Appreciate any input/ideas.
You didn't set up replication correctly. You cannot use pg_dump to create the replica, you have to use a physical backup technique like pg_basebackup.
See the documentation for details.
Do not use PostgreSQL 9.2, it is out of support.