Automatic creation of replication slot not working on Barman 2.11? - postgresql

I am using barman 2.11 and postgres 9.5 in my setup. I specified "create_slot = auto" in the server config for automatic replication slot creation as mentioned in the docs but it unfortunately appears to have no effect & the barman check reports the issue as below,
My server config:
[postgres-source-db]
; Configuration options for the server named 'postgres-source-db'
description = "Config for PostgreSQL Database Backup via rsync/SSH with WAL streaming"
ssh_command = ssh -q postgres#postgres-source-db
conninfo = host=postgres-source-db user=barman dbname=dcmdb
backup_method = rsync
parallel_jobs = 1
reuse_backup = link
archiver = on
backup_options = exclusive_backup
streaming_conninfo = host=postgres-source-db user=barman
streaming_archiver = on
slot_name = barman
create_slot = auto
===
Barman check output:
barman#4f5c93878899:~$ barman check postgres-source-db
Server postgres-source-db:
WAL archive: FAILED (please make sure WAL shipping is setup)
PostgreSQL: OK
superuser or standard user with backup privileges: OK
PostgreSQL streaming: OK
wal_level: OK
replication slot: FAILED (replication slot 'barman' doesn't exist. Please execute 'barman receive-wal --create-slot postgres-source-db')
directories: OK
retention policy settings: OK
backup maximum age: FAILED (interval provided: 1 day, latest backup age: No available backups)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: FAILED (have 0 backups, expected at least 1)
ssh: OK (PostgreSQL server)
not in recovery: OK
systemid coherence: OK (no system Id stored on disk)
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: FAILED (See the Barman log file for more details)
archive_mode: OK
archive_command: OK
archiver errors: OK
barman#4f5c93878899:~$
I should note that the test check succeeds as shown below,
barman#4f5c93878899:~$ psql -U barman -h postgres-source-db -c "IDENTIFY_SYSTEM" replication=1
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
systemid | timeline | xlogpos | dbname
---------------------+----------+-----------+--------
6854705426793291833 | 1 | 0/3000AE0 |
(1 row)
barman#4f5c93878899:~$
Am i missing something?
UPDATE (made partial headway, but still not out of the woods):
One more update & info to add. I am setting this up on docker containers & notice that the cron setup was missing despite my installing this from the PostgreSQL apt-repository. Once i logged into the container & ran
'/usr/bin/barman -q cron'
to start the WAL receiver i see that the status has changed to success. Not sure why it did not run automatically, any clue?
Doesn't look like a permission issue but the syntax of the content in '/etc/cron.d/barman' seems strange to me,
barman#bef22f0beec3:~$ cat /etc/cron.d/barman
# /etc/cron.d/barman: crontab entries for the barman package
MAILTO=root
* * * * * barman [ -x /usr/bin/barman ] && /usr/bin/barman -q cron
barman#bef22f0beec3:~$
Below are the terminal outputs,
barman#4f5c93878899:~$ crontab -l
no crontab for barman
barman#4f5c93878899:~$ su root
Password:
root#4f5c93878899:/var/lib/barman# crontab -l
no crontab for root
root#4f5c93878899:/var/lib/barman# exit
exit
barman#4f5c93878899:~$ id
uid=102(barman) gid=103(barman) groups=103(barman)
barman#4f5c93878899:~$ pwd
/var/lib/barman
barman#4f5c93878899:~$ cat /etc/cron.d/barman
# /etc/cron.d/barman: crontab entries for the barman package
MAILTO=root
* * * * * barman [ -x /usr/bin/barman ] && /usr/bin/barman -q cron
barman#4f5c93878899:~$ ls -ltr /etc/cron.d/barman
-rw-r--r-- 1 root root 140 Jul 9 11:18 /etc/cron.d/barman
barman#4f5c93878899:~$ /usr/bin/barman -q cron
barman#4f5c93878899:~$ ps -aef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 05:00 ? 00:00:00 /bin/bash /docker-entrypoint.sh barman
root 24 1 0 05:00 ? 00:00:00 /usr/sbin/sshd -D
root 27 24 0 05:03 ? 00:00:00 sshd: barman [priv]
barman 33 27 0 05:03 ? 00:00:00 sshd: barman#pts/0
barman 34 33 0 05:03 pts/0 00:00:00 -bash
barman 107 1 4 05:18 ? 00:00:00 /usr/bin/python3 /usr/bin/barman -c /etc/barman.conf -q receive-wal postgres-source-db
barman 111 107 1 05:18 ? 00:00:00 /usr/lib/postgresql/12/bin/pg_receivewal --dbname=dbname=replication host=postgres-source-db options=-cdatestyle=iso replication=true user=barman application_name
barman 114 34 0 05:18 pts/0 00:00:00 ps -aef
barman#4f5c93878899:~$
barman#4f5c93878899:~$ barman check postgres-source-db
Server postgres-source-db:
PostgreSQL: OK
superuser or standard user with backup privileges: OK
PostgreSQL streaming: OK
wal_level: OK
replication slot: OK
directories: OK
retention policy settings: OK
backup maximum age: FAILED (interval provided: 1 day, latest backup age: No available backups)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: FAILED (have 0 backups, expected at least 1)
ssh: OK (PostgreSQL server)
not in recovery: OK
systemid coherence: OK (no system Id stored on disk)
pg_receivexlog: OK
pg_receivexlog compatible: OK
receive-wal running: OK
archive_mode: OK
archive_command: OK
continuous archiving: OK
archiver errors: OK
barman#4f5c93878899:~$
Thanks

Just for others that might be facing the same issue, i resolved it.
The problem appears to be due to two bugs in barman (i see it on Debian, not sure of others),
The cron.d entry (/etc/cron.d/barman) is missing a new line at the end & it turns out that is needed for cron to execute them at-least on Debian.
The cron was not started automatically by default & i had to start it from my entry point script.
With the above two fixed it works like a charm.

Related

How to run a process in daemon mode with systemd service?

I've googled and read quite a bit of blogs, posts, etc. on this. I've also been trying them out manually on my EC2 instance. However, I'm still not able to properly configure the systemd service unit to have it run the process in background as I expect. The process I'm running is nessus service. Here's my service unit definition:
$ cat /etc/systemd/system/nessusagent.service
[Unit]
Description=Nessus
[Service]
ExecStart=/opt/myorg/bin/init_nessus
Type=simple
[Install]
WantedBy=multi-user.target
and here is my script /opt/myorg/bin/init_nessus:
$ cat /opt/apiq/bin/init_nessus
#!/usr/bin/env bash
set -e
NESSUS_MANAGER_HOST=...
NESSUS_MANAGER_PORT=...
NESSUS_CLIENT_GROUP=...
NESSUS_LINKING_KEY=...
#-------------------------------------------------------------------------------
# link nessus agent with manager host
#-------------------------------------------------------------------------------
/opt/nessus_agent/sbin/nessuscli agent link --key=${NESSUS_LINKING_KEY} --host=${NESSUS_MANAGER_HOST} --port=${NESSUS_MANAGER_PORT} --groups=${NESSUS_CLIENT_GROUP}
if [ $? -ne 0 ]; then
echo "Cannot link the agent to the Nessus manager, quitting."
exit 1
fi
/opt/nessus_agent/sbin/nessus-service -q -D
When I run the service, I always get the following:
$ systemctl status nessusagent.service
● nessusagent.service - Nessus
Loaded: loaded (/etc/systemd/system/nessusagent.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2020-08-24 06:40:40 UTC; 9min ago
Process: 27787 ExecStart=/opt/myorg/bin/init_nessus (code=exited, status=0/SUCCESS)
Main PID: 27787 (code=exited, status=0/SUCCESS)
...
Aug 24 06:40:40 ip-10-27-0-104 init_nessus[27787]: + /opt/nessus_agent/sbin/nessuscli agent link --key=... --host=... --port=8834 --groups=...
Aug 24 06:40:40 ip-10-27-0-104 init_nessus[27787]: [info] [agent] HostTag::getUnix: setting TAG value to '8596420322084e3ab97d3c39e5c92e00'
Aug 24 06:40:40 ip-10-27-0-104 init_nessus[27787]: [info] [agent] Successfully linked to <myorg.com>:8834
Aug 24 06:40:40 ip-10-27-0-104 init_nessus[27787]: + '[' 0 -ne 0 ']'
Aug 24 06:40:40 ip-10-27-0-104 init_nessus[28506]: + /opt/nessus_agent/sbin/nessus-service -q -D
However, I can't see the process that I expect to see:
$ ps faux | grep nessus
root 28565 0.0 0.0 12940 936 pts/0 S+ 06:54 0:00 \_ grep --color=auto nessus
If I run the last command manually, I can see it:
$ /opt/nessus_agent/sbin/nessus-service -q -D
$ ps faux | grep nessus
root 28959 0.0 0.0 12940 1016 pts/0 S+ 07:00 0:00 \_ grep --color=auto nessus
root 28952 0.0 0.0 6536 116 ? S 07:00 0:00 /opt/nessus_agent/sbin/nessus-service -q -D
root 28953 0.2 0.0 69440 9996 pts/0 Sl 07:00 0:00 \_ nessusd -q
What is it that I'm missing here?
Eventually figured out that this was because of the extra -D option in the last command. Removing the -D option fixed the issue. Running the process in daemon mode inside a system manager is not the way to go. We need to run it in the foreground and let the system manager handle it.

`pg_ls_dir` can query some directories, but not others

On my system, /home and /etc have exactly the same permissions:
$ ls -ld /home /etc
drwxr-xr-x 67 root root 4096 Nov 13 15:59 /etc
drwxr-xr-x 3 root root 4096 Oct 18 13:45 /home
However, Postgres can read one, but not the other:
test=# select count(*) from (select pg_ls_dir('/etc')) a;
count
-------
149
(1 row)
test=# select count(*) from (select pg_ls_dir('/home')) a;
ERROR: could not open directory "/home": Permission denied
Even though the user the DB is running as can, in fact, run ls /home:
$ sudo -u postgres ls /home > /dev/null && echo "ls succeeded"
ls succeeded
What is going on?
My postgres version is 11.5, running on Arch Linux.
I figured it out, it is because Arch's bundled postgresql.service file set ProtectHome=true, causing systemd to use Linux mount namespaces to block the postgres processes from accessing /home.

Postgres No such interface 'org.freedesktop.DBus.Properties'

Postgres database crashed after restart, tried just about everything including reinstalling postgres. It will not start on ubuntu 14.04,
$ systemctl status postgresql#9.6-main.service
Failed to issue method call: No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/systemd1/unit/postgresql_409_2e6_2dmain_2eservice
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
9.6 main 5432 down postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
$ sudo service postgresql start
* Starting PostgreSQL 9.6 database server
* Failed to issue method call: Unit postgresql#9.6-main.service failed to
load: No such file or directory. See system logs and 'systemctl status
postgresql#9.6-main.service' for details.
$ ps uxa|grep dbus-daemon
message+ 751 0.0 0.0 40812 4064 ? Ss 18:39 0:03 dbus-daemon --system --fork
dominic 3058 0.0 0.0 40840 4252 ? Ss 18:40 0:02 dbus-daemon --fork --session --address=unix:abstract=/tmp/dbus-S1LhlCDwl2
dominic 3145 0.0 0.0 39400 3536 ? S 18:40 0:00 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
dominic 17462 0.0 0.0 15956 2244 pts/4 S+ 21:45 0:00 grep --color=auto dbus-daemon
Postgres log file is empty.
I had the same error after install snap on Ubuntu 14.04. It was install some parts from systemd and broke postgresql init script.
You need to add parameter --skip-systemctl-redirect to pg_ctlcluster in file /usr/share/postgresql-common/init.d-functions
The function you need to change:
do_ctl_all() {
...
# --skip-systemctl-redirect fix postgresql No such interface 'org.freedesktop.DBus.Properties'
if [ "$1" = "stop" ] || [ "$1" = "restart" ]; then
ERRMSG=$(pg_ctlcluster --skip-systemctl-redirect --force "$2" "$name" $1 2>&1)
else
ERRMSG=$(pg_ctlcluster --skip-systemctl-redirect "$2" "$name" $1 2>&1)
fi
...
}
Ubuntu 14.04 did not switch to systemd yet. I highly recommend upgrading to 16.04 or even better, 18.04.

PostgreSQL: Permission denied + has wrong ownership loop?

I'm trying to run postgresql on my local machine like I usually do, however it's putting me in a situation where I can't fix. I installed postgresql91 with macports.
These are the three commands I usually have to run to get it running:
sudo sysctl -w kern.sysv.shmall=4096
sudo sysctl -w kern.sysv.shmmax=16777216
sudo su postgres -c "/opt/local/lib/postgresql91/bin/postgres -D /opt/local/var/db/postgresql91/defaultdb -p 55432"
However, it's giving me this error today:
Nets-Mac-Pro:~ emai$ sudo sysctl -w kern.sysv.shmall=4096
Password:
kern.sysv.shmall: 4096 -> 4096
Nets-Mac-Pro:~ emai$ sudo sysctl -w kern.sysv.shmmax=16777216
kern.sysv.shmmax: 16777216 -> 16777216
Nets-Mac-Pro:~ emai$ sudo su postgres -c "/opt/local/lib/postgresql91/bin/postgres -D /opt/local/var/db/postgresql91/defaultdb -p 55432"
postgres cannot access the server configuration file "/opt/local/var/db/postgresql91/defaultdb/postgresql.conf": Permission denied
When I go to /opt/local/var/db/postgresql91/ and do an ls -l this is what comes up:
drwx------ 18 root wheel 612 Jun 28 12:44 defaultdb
So I decided to add the postgres user to the wheel group, and then chmod defaultdb to 770.
drwxrwx--- 18 root wheel 612 Jun 28 12:44 defaultdb
I still get the error:
FATAL: could not open configuration file "/opt/local/var/db/postgresql91/defaultdb/postgresql.conf": Permission denied
And so I change the file rights from:
-rw------- 1 root wheel 19170 Jan 7 11:52 postgresql.conf
to:
-rw-rw---- 1 root wheel 19170 Jan 7 11:52 postgresql.conf
And now it complains that when I run the command again:
Nets-Mac-Pro:~ emai$ sudo su postgres -c "/opt/local/lib/postgresql91/bin/postgres -D /opt/local/var/db/postgresql91/defaultdb -p 55432"
FATAL: data directory "/opt/local/var/db/postgresql91/defaultdb" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
I have no clue how I used to run the postgres server considering the file permissions of the files. Where do I find the data folder that it is hinting me about? Is there a better way to fix this?
Postgres should be owner, and the only user capable of writing to, data directory.
So, do:
sudo chown -Rf postgres:postgres /opt/local/var/db/postgresql91/defaultdb
sudo chmod 700 /opt/local/var/db/postgresql91/defaultdb
and it should be fine.

Error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory

I am trying to execute pg_dump on PostgreSQL 9.0.4 server running on Debian and I am getting the error below:
./pg_dump: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory
libpq.so.5 is a link to libpq.so.5.3 as shown below
lrwxrwxrwx 1 root root 12 Jun 27 16:24 libpq.so.5 -> libpq.so.5.3
-rwxr-xr-x 1 root root 180749 Jun 21 02:43 libpq.so.5.3
What is it that I am doing wrong?
Try this:
1: Know the path of libpq.so.5
find / -name libpq.so.5
Output example:
/usr/pgsql-9.4/lib/libpq.so.5
If found nothing, check if you have already installed the suitable postgresql-libs for your postgresql version and your OS platform
2: Symbolic link that library in a "well known" library path like /usr/lib:
ln -s /usr/pgsql-9.4/lib/libpq.so.5 /usr/lib/libpq.so.5
Attention:
If your platform is 64 bit, you MUST also symbolic link to 64 bit libraries path:
ln -s /usr/pgsql-9.4/lib/libpq.so.5 /usr/lib64/libpq.so.5
3: Be happy !
In which directory are these libpq files? You can try setting environment variable LD_LIBRARY_PATH to point to this directory or make sure it's in standard place.
Also, why isn't the libpq.so.5 link shown in the "as shown below" section? Maybe you should just run ldconfig?
Ubuntu 21.10+
Since this is the top search result for the error. I'll add an updated answer. I received the error when trying to start a django server.
I hadn't installed the postgres stuff.
Try:
sudo apt install libpq-dev
See:
https://packages.ubuntu.com/impish/libpq-dev
I was getting the same error message on Postgres 9.5 on RHEL 6.5 which lead me to this post. But a find for the file libpq.so.5 returned nothing, which made things more confusing.
In the end the following symbolic links made it run
ln -s /opt/rh/rh-postgresql95/root/usr/lib64/libpq.so.rh-postgresql95-5 /usr/lib64/libpq.so.rh-postgresql95-5
ln -s /opt/rh/rh-postgresql95/root/usr/lib64/libpq.so.rh-postgresql95-5 /usr/lib/libpq.so.rh-postgresql95-5
These paths are for RHEL, use find / -name libpq.so to location your installation and add it to the same destination folders /usr/lib/ and /usr/lib64/ using the orginal file name.
The root cause appears that the installation did not place this file into a shared location.
This error probably occurs because of $LD_LIBRARY_PATH environment variable is not set.
When you install your application from source code using prefix (./configure --prefix=/some/path), you have to inform where your lib/ path is. I just found a solution for this, and I added this variable to postgres user init bash script:
printf 'export PATH=$PATH:/opt/apl/pgsql/bin\nexport LD_LIBRARY_PATH=/opt/apl/pgsql/lib:$LD_LIBRARY_PATH\n' > /etc/profile.d/postgres.sh
redhat 7 is missing few steps after installing yum install pgadmin4:
sudo ln -s /usr/lib64/pgdg-libpq5/lib/libpq.so /usr/lib64/libpq.so.5
sudo ln -s /usr/lib64/pgdg-libpq5/lib/libpq.so /usr/lib/libpq.so.5
sudo chown -R apache:apache /var/lib/pgadmin4/
then you can run
sudo python3 /usr/lib/python3.6/site-packages/pgadmin4-web/setup.py
and if all successful:
systemctl start httpd
systemctl status httpd
apachectl configtest
and make sure the httpd starts ok
I had exactly the same problem with the pg 9.6 install. I fixed it like this. Rather irritating that the installer doesn't factor this in.
***********post yum install & running initdb *********
Success. You can now start the database server using:
/opt/rh/rh-postgresql96/root/usr/bin/pg_ctl -D /var/opt/rh/rh-postgresql96/lib/pgsql/data -l logfile start
-bash-4.2$ /opt/rh/rh-postgresql96/root/usr/bin/pg_ctl -D /var/opt/rh/rh-postgresql96/lib/pgsql/data -l logfile start
/opt/rh/rh-postgresql96/root/usr/bin/pg_ctl: **error while loading shared libraries: libpq.so.rh-postgresql96-5: cannot open shared object file: No such file or directory**
-bash-4.2$ id
uid=26(postgres) gid=26(postgres) groups=26(postgres) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
**************
-bash-4.2$ cat LibFix
ln -s /opt/rh/rh-postgresql96/root/usr/lib64/libpq.so.rh-postgresql96-5 /usr/lib64/libpq.so.rh-postgresql96-5
ln -s /opt/rh/rh-postgresql96/root/usr/lib64/libpq.so.rh-postgresql96-5 /usr/lib/libpq.so.rh-postgresql96-5
**************
[root#****lab ~]# ln -s /opt/rh/rh-postgresql96/root/usr/lib64/libpq.so.rh-postgresql96-5 /usr/lib64/libpq.so.rh-postgresql96-5
[root#****lab ~]# ln -s /opt/rh/rh-postgresql96/root/usr/lib64/libpq.so.rh-postgresql96-5 /usr/lib/libpq.so.rh-postgresql96-5
[root#****lab ~]# su - postgres
Last login: Thu Apr 5 08:57:21 CEST 2018 on pts/0
-bash-4.2$ /opt/rh/rh-postgresql96/root/usr/bin/pg_ctl -D /var/opt/rh/rh-postgresql96/lib/pgsql/data -l logfile start
server starting
-bash-4.2$ ps -ef | grep postgres
root 12778 7883 0 09:07 pts/0 00:00:00 su - postgres
postgres 12779 12778 0 09:07 pts/0 00:00:00 -bash
postgres 12802 1 0 09:08 pts/0 00:00:00 /opt/rh/rh-postgresql96/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql96/lib/pgsql/data
postgres 12803 12802 0 09:08 ? 00:00:00 postgres: logger process
postgres 12805 12802 0 09:08 ? 00:00:00 postgres: checkpointer process
postgres 12806 12802 0 09:08 ? 00:00:00 postgres: writer process
postgres 12807 12802 0 09:08 ? 00:00:00 postgres: wal writer process
postgres 12808 12802 0 09:08 ? 00:00:00 postgres: autovacuum launcher process
postgres 12809 12802 0 09:08 ? 00:00:00 postgres: stats collector process
postgres 12810 12779 0 09:08 pts/0 00:00:00 ps -ef
-bash-4.2$ id
uid=26(postgres) gid=26(postgres) groups=26(postgres) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
-bash-4.2$ psql
psql (9.6.5)
postgres=# \conninfo
You are connected to database "postgres" as user "postgres" via socket in "/var/run/postgresql" at port "5432".