Run PostgreSQL streaming replication synchronous and asynchronous simultaneously - postgresql

I am using PostgreSQL 14 and ubuntu as my OS.
I have done the database replication in synchronous mode
Now I want to add another server and the relationship between primary and new_standby node will be asynchronous.
Another problem, how to set application_name for different nodes? I have got same application_name=14/main for my two standby servers
Can anyone help me to solve these issues?

Yes, you can run it simultaneously.
You should have this configuration in your primary node:
listen_addresses = '*'
port = 5432
wal_level = hot_standby
max_wal_senders = 16
wal_keep_segments = 32
synchronous_commit = on
synchronous_standby_names = 'pgsql_0_node_0'
Restart the node to take the changes:
$ systemctl restart postgresql-14
Create the replication role:
$ CREATE ROLE replication_user WITH LOGIN PASSWORD 'PASSWORD' REPLICATION;
And configure this in your standby nodes:
Both:
wal_level = hot_standby
hot_standby = on
Sync:
standby_mode = 'on'
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_0 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Async:
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_1 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Restart the node to take the changes:
$ systemctl restart postgresql-14
Then, you can run this in your Primary node to see the replication nodes:
$ SELECT pid,usename,application_name,state,sync_state FROM pg_stat_replication;
pid | usename | application_name | state | sync_state
-------+------------------+------------------+-----------+------------
10951 | replication_user | pgsql_0_node_1 | streaming | async
10952 | replication_user | pgsql_0_node_0 | streaming | sync
(2 rows)

Related

Can't connect PostGis to database and server

I followed these steps to set up QWC services https://github.com/qwc-services/qwc-services-core#quick-start and I can run the demo. But if load my own QGIS project, I receive the following error message:
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 WARNING ClearCapabilities[99]: Cached cleared : /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING PostGIS[99]: Connection to database failed
qwc-qgis-server_1 | could not connect to server: No such file or directory
qwc-qgis-server_1 | Is the server running locally and accepting
qwc-qgis-server_1 | connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 CRITICAL Server[99]: Error, Layer(s) measurement_b46e976f_2d0f_4bf0_942a_9d9462b40c3e not valid in project /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-config-service_1 | [2022-01-04 07:50:09,360] WARNING in config_generator: Skipping theme item '': Could not get capabilities for /ows/MeasurementDemo
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The generation of the configuration files resulted in a failure
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The configuration files were not updated!
qwc-config-service_1 | [2022-01-04 07:50:20,856] CRITICAL in config_generator: The generation of the permission files resulted in a failure.
qwc-config-service_1 | [2022-01-04 07:50:20,857] CRITICAL in config_generator: The permission files were not updated!
qwc-config-service_1 | [pid: 15|app: 0|req: 18/18] 172.18.0.11 () {30 vars in 408 bytes} [Tue Jan 4 07:50:05 2022] POST /generate_configs?tenant=default => generated 2881 bytes in 15083 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
As the error is quite similar to this question: PostgreSQL: Why psql can't connect to server?, I followed the answers but with no result.
ps -ef | grep postgres gives me the following result:
postgres 203911 1 0 07:35 ? 00:00:00 /usr/lib/postgresql/13/bin/postgres -D /var/lib/postgresql/13/main -c config_file=/etc/postgresql/13/main/postgresql.conf
Also I found the socket in
/var/run/postgresql/.s.PGSQL.5432
And I run the command
psql -h /var/run/postgresql/ GeoDB
But without result. After that I checked the ph_hba.conf File:
# "local" is for Unix domain socket connections only
local all all peer
Running the command pg_lsclusters gives me:
Ver Cluster Port Status Owner Data directory Log file
13 main 5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
Also after restarting the pg_ctlcluster and PostgreSQL the error remained the same.
Edit 1
After the answer from cnaimi I checked the postgresql.confFile:
# - Connection Settings -
#listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 100 # (change requires restart)
#superuser_reserved_connections = 3 # (change requires restart)
unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories
# (change requires restart)
#unix_socket_group = '*' # (change requires restart)
#unix_socket_permissions = 0777 # begin with 0 to use octal notation
# (change requires restart)
#bonjour = off # advertise server via Bonjour
# (change requires restart)
#bonjour_name = '' # defaults to the computer name
# (change requires restart)
But I can't find an error there as the port is 5432 and it listen to all adresses.
Edit 2
During my search I found several pg_service.conf Files:
./qwc-services/qwc-docker/wsgi-service/pg_service.conf
./qwc-services/qwc-docker/qgis-server/pg_service.conf
./qwc-services/qwc-docker/postgis/pg_service.conf
./qwc-services/qwc-docker/pg_service.conf
Each if them contain one or more credentials for databases like the one below:
[qwc_geodb]
host=qwc-postgis
port=5432
dbname=qwc_demo
user=qwc_service
password=qwc_service
sslmode=disable
The port is in all files correct, as far as I saw. But of course the db name and user/password are wrong. Does this could cause the error? Or does QWS get the credentials through the .qgs file?
Edit 3
Thanks to the hints from Devdatta Tengshe I set the host for PostgreSQL to 127.0.0.1. By using sudo docker-compose ps one can see the used container and their ports:
Name Command State Ports
-------------------------------------------------------------------------------------------------------------------------------
qwc-docker_qwc-admin-gui_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5031->9090/tcp
qwc-docker_qwc-api-gateway_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:8088->80/tcp,:::8088->80/tcp
qwc-docker_qwc-auth-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5017->9090/tcp
qwc-docker_qwc-config-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5010->9090/tcp
qwc-docker_qwc-data-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5012->9090/tcp
qwc-docker_qwc-elevation-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5002->9090/tcp
qwc-docker_qwc-fulltext-search-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5011->9090/tcp
qwc-docker_qwc-map-viewer_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5030->9090/tcp
qwc-docker_qwc-mapinfo-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5016->9090/tcp
qwc-docker_qwc-ogc-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5013->9090/tcp
qwc-docker_qwc-permalink-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5001->9090/tcp
qwc-docker_qwc-postgis_1 docker-entrypoint.sh postgres Up (healthy) 127.0.0.1:5439->5432/tcp
qwc-docker_qwc-qgis-server_1 /sbin/my_init Up 127.0.0.1:8001->80/tcp
qwc-docker_qwc-solr_1 docker-entrypoint.sh solr- ... Up 127.0.0.1:8983->8983/tcp
Can you check the postgres.conf file located in
/etc/postgresql/13/main/postgresql.conf
specially the parameter listen_address
Maybe you have to specify from which host you are listening.
But if the demo example is working the database configuration should be ok.
You can also check the port for postgres on postgres.conf and validate it's 5432.
There are a couple of things that need to be fixed to get this working.
I'm assuming that you have the Postgres Server running on the host machine, and not within any Docker container.
When you configured your QGIS Map file, you probably connected to localhost, and this information got saved in the .qgs file.
This is why your first error message says that it trying to connect to localhost, and no server was found. This error was thrown within the qwc docker container.
This error is occuring, because QGIS server (within the docker container) is not able to connect to the postgres server which is running on the host, using 'localhost' as the hostname
To solve this, you need to do the following:
In QGIS, connect to the Postgres Server using 127.0.0.1 and not localhost.
Save your qgs file using this new connection.
When you run the docker container for qwc, use --network="host" as the commandline parameter.
See: From inside of a Docker container, how do I connect to the localhost of the machine?
After this, the qgis server (within docker container) should be able to connect to the Postgres Server running on your host, using 127.0.0.1 as IP address.

pcp_attach_node gives EOFError in pgpool

I have successfully setup replication for my Postgres database using pg_pool.
Then I stopped the master server and checked the pool status. It is as like below
postgres=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role
---------+------------+------+--------+-----------+--------
0 | 10.140.0.9 | 5432 | 3 | 0.500000 | slave
1 | 10.140.0.7 | 5432 | 2 | 0.500000 | master
(2 rows)
Then I started the server, but it still shows the same status for the slave. So I used the following command to start the node:
/usr/sbin/pcp_node_info 10 10.140.0.9 5432 postgres postgres 1
But it is giving "EOFError" error. Please help to solve this issue.
Or please let me know a way to bring back the status 2 from status 3?
I solved the issue myself. In configuration the pcp port is 9898. Also there should be no space before password in pcp.conf file.
The pcp command should be as follows
/usr/sbin/pcp_node_info 10 localhost 9898 postgres postgres 1

About coordinator and datanode in Postgres-XL

thanks for your help.
I failed to build a Postgres-XL cluster.
My gtm.confg :
listen_addresses = '0.0.0.0'
port = 6668
at 10.80.19.47.
My gtm_proxy.confg:
listen_addresses = '0.0.0.0'
port = 6666
gtm_host = '10.80.19.47'
gtm_port = 6668
after start gtm and gtm_proxy ,i show net information use lsof -i :
gtm 11730 postgres 6u IPv4 355914 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44017 (ESTABLISHED
gtm 11730 postgres 7u IPv4 355915 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44018 (ESTABLISHED)
gtm 11730 postgres 8u IPv4 355916 0t0 TCP 10.80.19.47:ircu-4->10.80.19.46:44019 (ESTABLISHED)
Then i configure datanode like:
listen_addresses = '0.0.0.0'
port = 5432
pooler_port = 20002
gtm_host = '127.0.0.1'
gtm_port = 6666
and coordinator like:
listen_addresses = '0.0.0.0'
port = 3456
pooler_port = 20003
gtm_host = '127.0.0.1'
gtm_port = 6666
others are default ,then i start datanode with :
/usr/postgres-xl-9.2/bin/postgres --datanode -D /data1/pgsql/data &
start coordinator with:
/usr/postgres-xl-9.2/bin/postgres --coordinator -D /data1/pgsql/coordinator &
when i connect to Postgres-XL use coordinator with :
psql -h 127.0.0.1 -p 3456 postgres;
when i want to create a database with:
create database test;
but failed,it shows :
No Datanode defined in cluster
You need to define at least 1 Datanode with CREATE NODE.
then i check the cluster with :
psql -h 127.0.0.1 -p 3456 postgres -c "select * from pgxc_node order by 1";
it shows just one node like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
-------------------+-----------+-----------+-----------+----------------+------------------+-----------
coordinator_node2 | C | 5432 | localhost | f | f | 738118815
and i connect to port 5432 it shows the similar information like:
node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred | node_id
------------+-----------+-----------+-----------+----------------+------------------+------------
data_node2 | C | 5432 | localhost | f | f | -923817565
what wrong with my Configuration?
Thanks all.
You need to define datanode something like this.
./psql -c "CREATE NODE datanode1 WITH (TYPE = 'datanode', PORT = 7777)" postgres -p 3456 -h 127.0.0.1
I'm answering this question a year after. However, I'd suggest you to use pgxc_ctl command line tool as suggested by the Postgres XL developers themselves. It will keep you away from the hectic tasks of manually configuring the cluster.
You can refer to the official documentation of Postgres XL for this. After you successfully installed pgxc_ctl command line tool, type in
prepare config minimal
in order to generate a basic configuration file to configure the cluster. Edit your changes in pgxc_ctl/pgxc_ctl.conf accordingly. Type in exit and finally you can initialize all the clusters if your configuration is okay. For this, type in
pgxc_ctl init all

pg_basebackup fails with " too many connections for role "replication""

I am trying to set up a standby server and keep getting this error. My primary server has more than enough connections to handle the load:
listen_addresses = '*'
wal_level = hot_standby
max_wal_senders = 10
max_connections=100
checkpoint_segments = 8
wal_keep_segments = 8
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/archive/%f'
This is the command that fails on the standby server:
pg_basebackup -h ${MASTER_PORT_5432_TCP_ADDR} -D ${PGDATA} -U ${REP_USER} -vPw --xlog-method=stream
I don't understand why this happens.

Postgres replication not starting due to wal error

I am using postgres version 9.3.2 on two servers one master, one primary.
I am setting up replication as follows:-
On master:-
sudo -u postgres psql -c "CREATE USER replicator REPLICATION LOGIN ENCRYPTED PASSWORD 'FOO’;"
Edit postgresql.conf
listen_address = '*'
wal_level = hot_standby
max_wal_senders = 32
checkpoint_segments = 8
wal_keep_segments = 100
Edit pg_hba.conf
hostssl replication replicator <SLAVE IP>/32 md5
On Slave:-
Edit postgresql.conf
wal_level = hot_standby
max_wal_senders = 3
checkpoint_segments = 8
wal_keep_segments = 8
hot_standby = on
Run
sudo service postgresql stop
sudo -u postgres rm -rf /var/lib/postgresql/9.3/main
sudo -u postgres pg_basebackup -h <MASTER IP> -D /var/lib/postgresql/9.3/main -U replicator -v -P
CREATE /var/lib/postgresql/9.3/main/recovery.conf
standby_mode = 'on'
primary_conninfo = 'host=<MASTER IP> port=5432 user=replicator password=FOO sslmode=require'
trigger_file = '/tmp/postgresql.trigger'
Run:-
sudo service postgresql restart
When I restart postgres on the slave I get this error message:-
LOG: database system was shut down at 2015-01-14 09:10:50 GMT 2015-01-14 09:11:01 GMT [16741-2]
LOG: entering standby mode 2015-01-14 09:11:01 GMT [16741-3] WARNING: WAL was generated with wal_level=minimal, data may be missing 2015-01-14 09:11:01 GMT [16741-4] HINT: This happens if you temporarily set wal_level=minimal without taking a new base backup. 2015-01-14 09:11:01 GMT [16741-5]
FATAL: hot standby is not possible because wal_level was not set to "hot_standby" on the master server 2015-01-14 09:11:01 GMT [16741-6] HINT: Either set wal_level to "hot_standby" on the master, or turn off hot_standby here. 2015-01-14 09:11:01 GMT [16740-1]
LOG: startup process (PID 16741) exited with exit code 1 2015-01-14 09:11:01 GMT [16740-2] LOG: aborting startup due to startup process failure ... failed!
Why is this happening? I have checked and rechecked that on the master wal_level is set to hot_standby. On the master running "show all" shows that this is the case? I am at a loss as to what I am doing wrong here.
You have to restart primary database again to let the current WAL file replayed on the standby, since this WAL file was generated when wal_level=archive.