Cannot connect Barman to PostgreSQL 12 - postgresql

I have 2 ubuntu-20.04 VM on VMWARE with Postgres 12 installed on each
pgprimary on ip 192.168.1.131
pgbackup on ip 192.168.1.130
barman CLI tools are installed on pgprimary
barman is installed on pgbackup
I want to backup data from pgprimary on pgbackupsame 2 users as Postgress users
on each machine I created
2 Linux sudoist users
useradd barman
useradd streaming_barman
also created the same two user as Postgress users
createuser --superuser --replication -P barman
createuser --superuser --replication -P streaming_barman
here are relevant parts on the configuration files
On pgprimary
postgressql.conf
listen_addresses = '*' # what IP address(es) to listen on;
port = 5432
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/12/arc/%f'
wal_level = replica
restore_command = 'cp /var/lib/postgresql/12/arc/%f %p'
recovery_target_time = '2021-03-24 16:18:11.319298+05:30'
recovery_target_inclusive = false
pg_hba.conf
local all postgres peer
# TYPE DATABASE USER ADDRESS METHOD
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
#local replication all peer
#host replication all 127.0.0.1/32 md5
#host replication all ::1/128 md5
# FOR TESTING
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
also did
firewall-cmd --permanent --add-port=5432/tcp
firewall-cmd --reload
========================
con
On pgbackup
sudo cat <<'EOF' >> /etc/barman.d/pgprimary.conf
[pgprimary]
description = "Example of PostgreSQL Database (Streaming-Only)"
conninfo = host=192.168.1.131 user=barman dbname=training
streaming_conninfo = host=192.168.1.131 user=streaming_barman dbname=training
backup_method = postgres
streaming_archiver = on
slot_name = barman
create_slot = auto
EOF
pg_hba.conf
cat <<'EOF' >>~/.pgpass
pgprimary:*:*:barman:barman
pgprimary:*:*:streaming_barman:barman
EOF
Then I did
barman cron
Output
Starting WAL archiving for server pgprimary
Starting streaming archiver for server pgprimary
barman check pgprimary
Then I get this error
[13643] barman.utils WARNING: Failed opening the requested log file. Using standard error instead.
Server pgprimary:
2021-10-30 21:39:15,982 [13643] barman.server ERROR: Check 'WAL archive' failed for server 'pgprimary'
WAL archive: FAILED (please make sure WAL shipping is setup)
2021-10-30 21:39:37,006 [13643] barman.postgres WARNING: Error retrieving PostgreSQL status: connection to server at "192.168.131" (192.168.0.131), port 5432 failed: Connection refused
2021-10-30 21:39:58,021 [13643] barman.server ERROR: Check 'check timeout' failed for server 'pgprimary'
check timeout: FAILED (barman check command timed out)
Why cannot connect barman to the server ?
UPDATE:
psql -h 192.168.1.131 -U barman -d training
Password for user barman:
psql (12.8 (Ubuntu 12.8-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
I also can connect to server via netstat

Related

PgBouncer - Server DNS lookup failed

I am trying to test PgBouncer connection with pgbench on a PostgreSQL server. But I'm getting error as closing because: server DNS lookup failed and at pgbench getting this message:
pgbench -c 10 -t 10 -C -f C:\Users\Administrator\Downloads\query.sql -U postgres -p 6432 -n tags
Password:
pgbench: error: connection to server at "localhost" (::1), port 6432 failed: FATAL: client_login_timeout (server down)
This is observed in PgBouncer log:
LOG C-01715ff0: tags/postgres#[::1]:52768 login attempt: db=tags user=postgres tls=no
WARNING DNS lookup failed: localhost: result=11001
LOG S-0174d218: tags/postgres#(bad-af):0 closing because: server DNS lookup failed (age=0s)
I'm running PostgreSQL-14.5 on Windows Server 2019
PgBouncer Config:
[databases]
postgres = host=localhost port=5435
tags = dbname=postgres host=localhost port=5435 user=postgres password=Admin#!23 auth_user=postgres
[pgbouncer]
logfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.log
pidfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.pid
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = C:\Program Files (x86)\PgBouncer\etc\userlist.txt
admin_users = postgres
stats_users = postgres
pool_mode = session
max_client_conn = 100
default_pool_size = 20
I tried solutions suggested in this post, like, setting
listen_addresses = '*' in postgresql.conf
but could not get the issue resolved. I also searched over the internet for the solution & did not find any.
Am I missing anything or please suggest a solution to this issue. Thanks in advance.
I can connect to Postgres DB after setting [databases] configuration in pgbouncer.ini like this : tags = host=127.0.0.1 port=5435 auth_user=postgres dbname=postgres.
Turns out the issue was with localhost not added in pg_hba.conf.

Can't connect PostGis to database and server

I followed these steps to set up QWC services https://github.com/qwc-services/qwc-services-core#quick-start and I can run the demo. But if load my own QGIS project, I receive the following error message:
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 WARNING ClearCapabilities[99]: Cached cleared : /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING PostGIS[99]: Connection to database failed
qwc-qgis-server_1 | could not connect to server: No such file or directory
qwc-qgis-server_1 | Is the server running locally and accepting
qwc-qgis-server_1 | connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
qwc-qgis-server_1 |
qwc-qgis-server_1 | 07:50:07 CRITICAL Server[99]: Error, Layer(s) measurement_b46e976f_2d0f_4bf0_942a_9d9462b40c3e not valid in project /data/MeasurementDemo.qgs
qwc-qgis-server_1 | 07:50:07 WARNING Server[99]: <ServerException>Layer(s) not valid</ServerException>
qwc-qgis-server_1 |
qwc-config-service_1 | [2022-01-04 07:50:09,360] WARNING in config_generator: Skipping theme item '': Could not get capabilities for /ows/MeasurementDemo
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The generation of the configuration files resulted in a failure
qwc-config-service_1 | [2022-01-04 07:50:19,468] CRITICAL in config_generator: The configuration files were not updated!
qwc-config-service_1 | [2022-01-04 07:50:20,856] CRITICAL in config_generator: The generation of the permission files resulted in a failure.
qwc-config-service_1 | [2022-01-04 07:50:20,857] CRITICAL in config_generator: The permission files were not updated!
qwc-config-service_1 | [pid: 15|app: 0|req: 18/18] 172.18.0.11 () {30 vars in 408 bytes} [Tue Jan 4 07:50:05 2022] POST /generate_configs?tenant=default => generated 2881 bytes in 15083 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
As the error is quite similar to this question: PostgreSQL: Why psql can't connect to server?, I followed the answers but with no result.
ps -ef | grep postgres gives me the following result:
postgres 203911 1 0 07:35 ? 00:00:00 /usr/lib/postgresql/13/bin/postgres -D /var/lib/postgresql/13/main -c config_file=/etc/postgresql/13/main/postgresql.conf
Also I found the socket in
/var/run/postgresql/.s.PGSQL.5432
And I run the command
psql -h /var/run/postgresql/ GeoDB
But without result. After that I checked the ph_hba.conf File:
# "local" is for Unix domain socket connections only
local all all peer
Running the command pg_lsclusters gives me:
Ver Cluster Port Status Owner Data directory Log file
13 main 5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
Also after restarting the pg_ctlcluster and PostgreSQL the error remained the same.
Edit 1
After the answer from cnaimi I checked the postgresql.confFile:
# - Connection Settings -
#listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 100 # (change requires restart)
#superuser_reserved_connections = 3 # (change requires restart)
unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories
# (change requires restart)
#unix_socket_group = '*' # (change requires restart)
#unix_socket_permissions = 0777 # begin with 0 to use octal notation
# (change requires restart)
#bonjour = off # advertise server via Bonjour
# (change requires restart)
#bonjour_name = '' # defaults to the computer name
# (change requires restart)
But I can't find an error there as the port is 5432 and it listen to all adresses.
Edit 2
During my search I found several pg_service.conf Files:
./qwc-services/qwc-docker/wsgi-service/pg_service.conf
./qwc-services/qwc-docker/qgis-server/pg_service.conf
./qwc-services/qwc-docker/postgis/pg_service.conf
./qwc-services/qwc-docker/pg_service.conf
Each if them contain one or more credentials for databases like the one below:
[qwc_geodb]
host=qwc-postgis
port=5432
dbname=qwc_demo
user=qwc_service
password=qwc_service
sslmode=disable
The port is in all files correct, as far as I saw. But of course the db name and user/password are wrong. Does this could cause the error? Or does QWS get the credentials through the .qgs file?
Edit 3
Thanks to the hints from Devdatta Tengshe I set the host for PostgreSQL to 127.0.0.1. By using sudo docker-compose ps one can see the used container and their ports:
Name Command State Ports
-------------------------------------------------------------------------------------------------------------------------------
qwc-docker_qwc-admin-gui_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5031->9090/tcp
qwc-docker_qwc-api-gateway_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:8088->80/tcp,:::8088->80/tcp
qwc-docker_qwc-auth-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5017->9090/tcp
qwc-docker_qwc-config-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5010->9090/tcp
qwc-docker_qwc-data-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5012->9090/tcp
qwc-docker_qwc-elevation-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5002->9090/tcp
qwc-docker_qwc-fulltext-search-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5011->9090/tcp
qwc-docker_qwc-map-viewer_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5030->9090/tcp
qwc-docker_qwc-mapinfo-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5016->9090/tcp
qwc-docker_qwc-ogc-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5013->9090/tcp
qwc-docker_qwc-permalink-service_1 /bin/sh -c uwsgi --http-so ... Up 127.0.0.1:5001->9090/tcp
qwc-docker_qwc-postgis_1 docker-entrypoint.sh postgres Up (healthy) 127.0.0.1:5439->5432/tcp
qwc-docker_qwc-qgis-server_1 /sbin/my_init Up 127.0.0.1:8001->80/tcp
qwc-docker_qwc-solr_1 docker-entrypoint.sh solr- ... Up 127.0.0.1:8983->8983/tcp
Can you check the postgres.conf file located in
/etc/postgresql/13/main/postgresql.conf
specially the parameter listen_address
Maybe you have to specify from which host you are listening.
But if the demo example is working the database configuration should be ok.
You can also check the port for postgres on postgres.conf and validate it's 5432.
There are a couple of things that need to be fixed to get this working.
I'm assuming that you have the Postgres Server running on the host machine, and not within any Docker container.
When you configured your QGIS Map file, you probably connected to localhost, and this information got saved in the .qgs file.
This is why your first error message says that it trying to connect to localhost, and no server was found. This error was thrown within the qwc docker container.
This error is occuring, because QGIS server (within the docker container) is not able to connect to the postgres server which is running on the host, using 'localhost' as the hostname
To solve this, you need to do the following:
In QGIS, connect to the Postgres Server using 127.0.0.1 and not localhost.
Save your qgs file using this new connection.
When you run the docker container for qwc, use --network="host" as the commandline parameter.
See: From inside of a Docker container, how do I connect to the localhost of the machine?
After this, the qgis server (within docker container) should be able to connect to the Postgres Server running on your host, using 127.0.0.1 as IP address.

Is it needed to use more than one computer for master slave replication?

I am trying to create master slave replication physically with postgresql11 in debian10. I got same errors from different linux dists so is it something related making the replication in same computer? Or something related with master-slave accounts ?
root#dlp:~# vi /etc/postgresql/11/main/postgresql.conf
listen_addresses = '*'
wal_level = replica
synchronous_standby_names = '*'
wal_keep_segments = 10
I configured the pg_hba.conf as follows:
#host replication all 127.0.0.1/32 md5
#host replication all ::1/128 md5
host replication rep_user 127.0.0.1/32 trust
host replication rep_user 10.0.0.30/32 trust
host replication rep_user 10.0.0.51/32 trust
After this segment:
#i didnt create new user as node1 i just used my root/postgres account for all the processes
root#node01:~# systemctl stop postgresql
root#node01:~# rm -rf /var/lib/postgresql/11/main/*
root#node01:~# su - postgres
postgres#node01:~$ pg_basebackup -R -h dlp.srv.world -U rep_user -D /var/lib/postgresql/11/main -P
I got the following error:
pg_basebackup: could not connect to server: could not connect to server: Connection refused
Is the server running on host "dlp.srv.world" (180.43.145.38) and accepting
TCP/IP connections on port 5432?
also in the root account postgres is working fine but i doest show that port 5432 listening?
netstat -nlt
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
when i check the status of "postgresql" is says active but when i type :
systemctl status postgresql#11-main
Dec 14 06:20:52 debian postgresql#11-main[6260]: Error: /usr/lib/postgresql/11/bin/pg_ctl /
Dec 14 06:20:52 debian postgresql#11-main[6260]: pg_ctl: directory "/var/lib/postgresql/11/
Dec 14 06:20:52 debian systemd[1]: postgresql#11-main.service: Can't open PID file /run/pos
Dec 14 06:20:52 debian systemd[1]: postgresql#11-main.service: Failed with result 'protocol
Dec 14 06:20:52 debian systemd[1]: Failed to start PostgreSQL Cluster 11-main.
i got that error.

PostgreSQL SSL Configuration

I have tried a lot but did't get exact knowledge to copy which files to client side
Getting error " Valid authentication certificate required"
Below is postgres DB server directory
total 80
skipping unwanted directory
-rw-------. 1 postgres postgres 1285 Feb 13 20:16 rootCA.crt
-rw-------. 1 postgres postgres 1168 Feb 13 20:16 server.crt
-rw-------. 1 postgres postgres 1679 Feb 13 20:17 server.key
Below is postgresql.conf file content related to SSL configuration.
[postgres#munmvs2951 raj]$ cat postgresql.conf | grep ssl
ssl = on # (change requires restart)
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
#ssl_prefer_server_ciphers = on # (change requires restart)
#ssl_ecdh_curve = 'prime256v1' # (change requires restart)
#ssl_renegotiation_limit = 512MB # amount of data between renegotiations
ssl_cert_file = 'server.crt' # (change requires restart)
ssl_key_file = 'server.key' # (change requires restart)
ssl_ca_file = 'rootCA.crt' # (change requires restart)
#ssl_crl_file = '' # (change requires restart)
Below is the pg_hba file.
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres trust
#host replication postgres 127.0.0.1/32 trust
#host replication postgres ::1/128 trust
#secure client
hostssl postgres postgres clientip/32 md5 clientcert=1
Below is the client side PostgreSQL user home directory from where psql will connect to server.
[postgres#clinet ~]$ ll
total 32
-rw------- 1 postgres postgres 1168 Feb 15 10:27 client.crt
-rw------- 1 postgres postgres 989 Feb 15 10:27 client.csr
-rw------- 1 postgres postgres 1285 Feb 15 11:47 rootCA.crt
-rw------- 1 postgres postgres 1675 Feb 15 11:47 rootCA.key
-rw------- 1 postgres postgres 17 Feb 15 11:47 rootCA.srl
-rw------- 1 postgres postgres 1168 Feb 15 11:47 server.crt
-rw------- 1 postgres postgres 989 Feb 15 11:47 server.csr
-rw------- 1 postgres postgres 1679 Feb 15 11:47 server.key
[postgres#client ~]$ pwd
/home/postgres
when i going to connect, getting below error.
[postgres#client ~]$ /opt/PostgresPlus/9.4AS/bin/psql -U postgres -p5443 -hserverip
psql.bin: FATAL: connection requires a valid client certificate
FATAL: no pg_hba.conf entry for host "10.80.x.x", user "postgres", database "postgres", SSL off
However, I can access without clientcert=1 in pg_hba file.
Ref:
[postgres#client ~]$ /opt/PostgresPlus/9.4AS/bin/psql -U postgres -p5443 -hserverip
psql.bin (9.4.1.3)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.
postgres=#
In above successful connection, i think ssl checking only at server side not client side.
but i want to know which files should i copy to PostgreSQL home directory #client side. to check certificate as per clientcert=1
Thanks for your help.
See https://www.postgresql.org/docs/current/libpq-ssl.html
Below is the client side PostgreSQL user home directory from where psql will connect to server
They do not belong in the home directory, they belong in a subdirectory of the home directory named .postgresql. Or in Windows, in a directory named "%APPDATA% \ postgresql". And file rootCA.crt has to be named root.crt, unless you have gone out of your way to change the name/location on the client side configuration, which it doesn't seem you have done (you have changed it on the server side, but that doesn't change it on the client side). And client.crt should be named postgresql.crt and you are missing the key file for it. And the fact that you have a file named rootCA.key in the client's directory is frightening, no one but the CA should have access to that.

Postgres replication not starting due to wal error

I am using postgres version 9.3.2 on two servers one master, one primary.
I am setting up replication as follows:-
On master:-
sudo -u postgres psql -c "CREATE USER replicator REPLICATION LOGIN ENCRYPTED PASSWORD 'FOO’;"
Edit postgresql.conf
listen_address = '*'
wal_level = hot_standby
max_wal_senders = 32
checkpoint_segments = 8
wal_keep_segments = 100
Edit pg_hba.conf
hostssl replication replicator <SLAVE IP>/32 md5
On Slave:-
Edit postgresql.conf
wal_level = hot_standby
max_wal_senders = 3
checkpoint_segments = 8
wal_keep_segments = 8
hot_standby = on
Run
sudo service postgresql stop
sudo -u postgres rm -rf /var/lib/postgresql/9.3/main
sudo -u postgres pg_basebackup -h <MASTER IP> -D /var/lib/postgresql/9.3/main -U replicator -v -P
CREATE /var/lib/postgresql/9.3/main/recovery.conf
standby_mode = 'on'
primary_conninfo = 'host=<MASTER IP> port=5432 user=replicator password=FOO sslmode=require'
trigger_file = '/tmp/postgresql.trigger'
Run:-
sudo service postgresql restart
When I restart postgres on the slave I get this error message:-
LOG: database system was shut down at 2015-01-14 09:10:50 GMT 2015-01-14 09:11:01 GMT [16741-2]
LOG: entering standby mode 2015-01-14 09:11:01 GMT [16741-3] WARNING: WAL was generated with wal_level=minimal, data may be missing 2015-01-14 09:11:01 GMT [16741-4] HINT: This happens if you temporarily set wal_level=minimal without taking a new base backup. 2015-01-14 09:11:01 GMT [16741-5]
FATAL: hot standby is not possible because wal_level was not set to "hot_standby" on the master server 2015-01-14 09:11:01 GMT [16741-6] HINT: Either set wal_level to "hot_standby" on the master, or turn off hot_standby here. 2015-01-14 09:11:01 GMT [16740-1]
LOG: startup process (PID 16741) exited with exit code 1 2015-01-14 09:11:01 GMT [16740-2] LOG: aborting startup due to startup process failure ... failed!
Why is this happening? I have checked and rechecked that on the master wal_level is set to hot_standby. On the master running "show all" shows that this is the case? I am at a loss as to what I am doing wrong here.
You have to restart primary database again to let the current WAL file replayed on the standby, since this WAL file was generated when wal_level=archive.