Postgres is restarting continuously on using shared_preload_libraries extension.
https://postgresqlco.nf/doc/en/param/shared_preload_libraries/
I am running postgres-15.1 using a python-based daemon in CentOS7-32bit arch. It is working fine if we do not use "shared_preload_libraries" extension. But after enabling this extension using "ALTER SYSTEM SET shared_preload_libraries" command, the postgres is restarting every few seconds.
Initially it was working fine with postgres-9.6.4.
Postgres logs:
waiting for server to start....2023-02-15 07:13:45.676 GMT [28605] LOG: skipping missing configuration file "/home/runtime/pgsql/data/postgresql.auto.conf"
2023-02-15 07:13:45.825 GMT [28605] LOG: starting PostgreSQL 15.1 on i686-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 32-bit
2023-02-15 07:13:45.825 GMT [28605] LOG: listening on IPv4 address "127.0.0.1", port 5432
2023-02-15 07:13:45.933 GMT [28605] LOG: listening on Unix socket "/home/runtime/pgsql/.s.PGSQL.5432"
2023-02-15 07:13:45.969 GMT [28608] LOG: database system was shut down at 2023-02-15 07:13:35 GMT
2023-02-15 07:13:45.989 GMT [28605] LOG: database system is ready to accept connections
done
server started
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
2023-02-15 07:13:51.480 GMT [28605] LOG: received fast shutdown request
waiting for server to shut down....2023-02-15 07:13:51.512 GMT [28605] LOG: aborting any active transactions
2023-02-15 07:13:51.513 GMT [28605] LOG: background worker "logical replication launcher" (PID 28611) exited with exit code 1
2023-02-15 07:13:51.513 GMT [28606] LOG: shutting down
2023-02-15 07:13:51.536 GMT [28606] LOG: checkpoint starting: shutdown immediate
2023-02-15 07:13:51.908 GMT [28606] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.090 s, sync=0.028 s, total=0.395 s; sync files=2, longest=0.021 s, average=0.014 s; distance=0 kB, estimate=0 kB
2023-02-15 07:13:51.909 GMT [28605] LOG: database system is shut down
done
server stopped
I tried to use postgres-15.0 and postgres-14.4, got the same behavior with both. I am not able to find any open issues w.r.t. shared_preload_libraries extension with new versions of Postgres.
PS: I have built this Postgres from the source code with openssl-1.1.1i.
I am using "citus" library with this.
ALTER SYSTEM SET shared_preload_libraries="citus";
I have generated a new citus.so file from it's source code using postgres-15.1. github.com/citusdata/citus
I've update my Airbyte image from 0.35.2-alpha to 0.35.37-alpha.
[running in kubernetes]
When the system rolled out the db pod wouldn't terminate and I [a terrible mistake] deleted the pod.
When it came back up, I get an error -
PostgreSQL Database directory appears to contain a database; Skipping initialization
2022-02-24 20:19:44.065 UTC [1] LOG: starting PostgreSQL 13.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
2022-02-24 20:19:44.065 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-02-24 20:19:44.065 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-02-24 20:19:44.071 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-24 20:19:44.079 UTC [21] LOG: database system was shut down at 2022-02-24 20:12:55 UTC
2022-02-24 20:19:44.079 UTC [21] LOG: invalid resource manager ID in primary checkpoint record
2022-02-24 20:19:44.079 UTC [21] PANIC: could not locate a valid checkpoint record
2022-02-24 20:19:44.530 UTC [1] LOG: startup process (PID 21) was terminated by signal 6: Aborted
2022-02-24 20:19:44.530 UTC [1] LOG: aborting startup due to startup process failure
2022-02-24 20:19:44.566 UTC [1] LOG: database system is shut down
Pretty sure the WAL file is corrupted, but I'm not sure how to fix this.
Warning - there is a potential for data loss
This is a test system, so I wasn't concerned with keeping the latest transactions, and had no backup.
First I overrode the container command to keep the container running but not try to start postgres.
...
spec:
containers:
- name: airbyte-db-container
image: airbyte/db
command: ["sh"]
args: ["-c", "while true; do echo $(date -u) >> /tmp/run.log; sleep 5; done"]
...
And spawned a shell on the pod -
kubectl exec -it -n airbyte airbyte-db-xxxx -- sh
Run pg_reset_wal
# dry-run first
pg_resetwal --dry-run /var/lib/postgresql/data/pgdata
Success!
pg_resetwal /var/lib/postgresql/data/pgdata
Write-ahead log reset
Then removed the temp command in the container, and postgres started up correctly!
I have edited my pg_hba file and copied it to server and restarted the services by "sudo service postgresql restart" but after that the server is not connecting.
Showing the below error, Your database returned: "Connection to 138.2xx.1xx.xx:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections."
The Jenkins job and data visualization tools are failing which was working fine previously. What could be the reason.
Getting this in PostgreSQL Log
2019-10-23 07:21:25.829 CEST [11761] LOG: received fast shutdown request
2019-10-23 07:21:25.829 CEST [11761] LOG: aborting any active transactions
2019-10-23 07:21:25.829 CEST [11766] LOG: autovacuum launcher shutting down
2019-10-23 07:21:25.832 CEST [11763] LOG: shutting down
2019-10-23 07:21:25.919 CEST [11761] LOG: database system is shut down
2019-10-23 07:21:27.068 CEST [22633] LOG: database system was shut down at 2019-10-23 07:21:25 CEST
2019-10-23 07:21:27.073 CEST [22633] LOG: MultiXact member wraparound protections are now enabled
2019-10-23 07:21:27.075 CEST [22631] LOG: database system is ready to accept connections
2019-10-23 07:21:27.075 CEST [22637] LOG: autovacuum launcher started
2019-10-23 07:21:27.390 CEST [22639] [unknown]#[unknown] LOG: incomplete startup packet
Below shows no response.
root#Ubuntu-1604-xenial-64-minimal ~ # pg_isready -h localhost -p 5432
localhost:5432 - no response
Below was already added to the postgresql.config file.
listen_addresses = '*'
Do i need to restart the entire server?
Can anyone please help me to resolve this.
I'd like to try out the quick start example at https://github.com/wrouesnel/postgres_exporter on a MacOS host device. In one terminal, I run a postgres image in interactive mode on the host network:
> docker run --network=host -it --rm --env POSTGRES_PASSWORD=password postgres
Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
8f91359f1fff: Pull complete
c6115f5efcde: Pull complete
28a9c19d8188: Pull complete
2da4beb7be31: Pull complete
fb9ca792da89: Pull complete
cedc20991511: Pull complete
b866c2f2559e: Pull complete
5d459cf6645c: Pull complete
cf66247ad881: Pull complete
35e29440d9da: Pull complete
362779251360: Pull complete
fb82d778d08d: Pull complete
b1f8d21ff25d: Pull complete
6d49eb0e8dd0: Pull complete
Digest: sha256:be456a40361cd836e0e1b35fc4d872e20e138f214c93138425169c4a2dfe1b0e
Status: Downloaded newer image for postgres:latest
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2019-10-03 19:06:10.222 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-03 19:06:10.235 UTC [43] LOG: database system was shut down at 2019-10-03 19:06:10 UTC
2019-10-03 19:06:10.239 UTC [42] LOG: database system is ready to accept connections
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down....2019-10-03 19:06:10.320 UTC [42] LOG: received fast shutdown request
2019-10-03 19:06:10.322 UTC [42] LOG: aborting any active transactions
2019-10-03 19:06:10.325 UTC [42] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
2019-10-03 19:06:10.325 UTC [44] LOG: shutting down
2019-10-03 19:06:10.338 UTC [42] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2019-10-03 19:06:10.436 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2019-10-03 19:06:10.438 UTC [1] LOG: listening on IPv6 address "::", port 5432
2019-10-03 19:06:10.440 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-03 19:06:10.450 UTC [51] LOG: database system was shut down at 2019-10-03 19:06:10 UTC
2019-10-03 19:06:10.453 UTC [1] LOG: database system is ready to accept connections
while in another terminal, I run postgres_exporter and connect it to that database:
> docker run --net=host --env DATA_SOURCE_NAME="postgresql://postgres:password#localhost:5432/postgres?sslmode=disable" wrouesnel/postgres_exporter
Unable to find image 'wrouesnel/postgres_exporter:latest' locally
latest: Pulling from wrouesnel/postgres_exporter
0d6d2d2516f9: Pull complete
e9d7b571ef5e: Pull complete
Digest: sha256:aeea975f0efeacb49c170f0f7c4a4000d3f0099cc33437aedd3f276e628cde1c
Status: Downloaded newer image for wrouesnel/postgres_exporter:latest
time="2019-10-03T19:09:20Z" level=info msg="Established new database connection to \"localhost:5432\"." source="postgres_exporter.go:778"
time="2019-10-03T19:09:20Z" level=info msg="Semantic Version Changed on \"localhost:5432\": 0.0.0 -> 11.5.0" source="postgres_exporter.go:1238"
time="2019-10-03T19:09:20Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1459"
As I understand it, I should be able to go to localhost:9187 in my browser and see the exported metrics. What I get, however, is that the connection gets refused:
> curl http://localhost:9187
curl: (7) Failed to connect to localhost port 9187: Connection refused
What I suspect is that this is because, as documented at https://docs.docker.com/network/host/,
The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.
Is this what is causing the issue and if so, how can I adapt this quickstart example to work using Docker Desktop for Mac?
A little late, but maybe for someone in the future. I had the same problem on Docker Desktop for Windows. For it to work I used the following command:
docker run -p 9187:9187 -e DATA_SOURCE_NAME="postgresql://postgres:password#localhost:5432/postgres?sslmode=disable" wrouesnel/postgres_exporter
I use postgres:10 (https://hub.docker.com/_/postgres/) image for DB.
It is deployed in docker swarm cluster.
After running DB replica I got database system is shut down in DB's log.
2018-05-11 10:26:53.073 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 10:26:53.073 UTC [1] LOG: listening on IPv6 address "::", port 5432,
2018-05-11 10:26:53.077 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 10:26:53.092 UTC [20] LOG: database system was shut down at 2018-05-11 10:26:20 UTC,
2018-05-11 10:26:53.100 UTC [1] LOG: database system is ready to accept connections,
The files belonging to this database system will be owned by user "postgres".,
This user must also own the server process.,
,
The database cluster will be initialized with locale "en_US.utf8".,
The default database encoding has accordingly been set to "UTF8".,
The default text search configuration will be set to "english".,
,
Data page checksums are disabled.,
,
fixing permissions on existing directory /var/lib/postgresql/data ... ok,
creating subdirectories ... ok,
selecting default max_connections ... 100,
selecting default shared_buffers ... 128MB,
selecting dynamic shared memory implementation ... posix,
creating configuration files ... ok,
running bootstrap script ... ok,
performing post-bootstrap initialization ... ok,
,
WARNING: enabling "trust" authentication for local connections,
You can change this by editing pg_hba.conf or using the option -A, or,
--auth-local and --auth-host, the next time you run initdb.,
syncing data to disk ... ok,
,
Success. You can now start the database server using:,
,
pg_ctl -D /var/lib/postgresql/data -l logfile start,
,
waiting for server to start....2018-05-11 09:39:21.129 UTC [37] LOG: listening on IPv4 address "127.0.0.1", port 5432,
2018-05-11 09:39:21.130 UTC [37] LOG: could not bind IPv6 address "::1": Cannot assign requested address,
2018-05-11 09:39:21.130 UTC [37] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.,
2018-05-11 09:39:21.133 UTC [37] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.147 UTC [38] LOG: database system was shut down at 2018-05-11 09:39:20 UTC,
2018-05-11 09:39:21.152 UTC [37] LOG: database system is ready to accept connections,
done,
server started,
CREATE DATABASE,
,
CREATE ROLE,
,
,
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*,
,
2018-05-11 09:39:21.595 UTC [37] LOG: received fast shutdown request,
waiting for server to shut down....2018-05-11 09:39:21.596 UTC [37] LOG: aborting any active transactions,
2018-05-11 09:39:21.598 UTC [37] LOG: worker process: logical replication launcher (PID 44) exited with exit code 1,
2018-05-11 09:39:21.599 UTC [39] LOG: shutting down,
2018-05-11 09:39:21.613 UTC [37] LOG: database system is shut down,
done,
server stopped,
,
PostgreSQL init process complete; ready for start up.,
,
2018-05-11 09:39:21.706 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 09:39:21.706 UTC [1] LOG: listening on IPv6 address "::", port 5432,
2018-05-11 09:39:21.709 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.724 UTC [64] LOG: database system was shut down at 2018-05-11 09:39:21 UTC,
2018-05-11 09:39:21.729 UTC [1] LOG: database system is ready to accept connections,
2018-05-11 10:26:20.444 UTC [1] LOG: received smart shutdown request,
2018-05-11 10:26:20.449 UTC [1] LOG: worker process: logical replication launcher (PID 70) exited with exit code 1,
2018-05-11 10:26:20.449 UTC [65] LOG: shutting down,
2018-05-11 10:26:20.460 UTC [1] LOG: database system is shut down,
Image :
FROM postgres:10
COPY healthcheck /usr/local/bin/
RUN chmod +x /usr/local/bin/healthcheck
HEALTHCHECK --interval=30s --timeout=30s --retries=3 \
CMD healthcheck
Snippet from docker-compose :
db_jackrabbit:
build: ./images/pgsql_jackrabbit
container_name: db_jackrabbit
environment:
- POSTGRES_DB=${JACK_POSTGRES_DB}
- POSTGRES_USER=${JACK_POSTGRES_USER}
- POSTGRES_PASSWORD=${JACK_POSTGRES_PASSWORD}
volumes:
- pgsql_jackrabbit_local:/var/lib/postgresql/data
ports:
- ${PORT_DB_JACKRABBIT}:5432
healthcheck:
#!/bin/bash
set -eo pipefail
host="$(hostname -i || echo '127.0.0.1')"
user="${POSTGRES_USER:-postgres}"
db="${POSTGRES_DB:-$POSTGRES_USER}"
export PGPASSWORD="${POSTGRES_PASSWORD:-}"
args=(
# force postgres to not use the local unix socket (test "external" connectibility)
--host "$host"
--username "$user"
--dbname "$db"
--quiet --no-align --tuples-only
)
if select="$(echo 'SELECT 1' | psql "${args[#]}")" && [ "$select" = '1' ]; then
exit 0
fi
exit 1
But DB still alive. It is shutdown periodically and accept connections again (
What it the problem?
Thanks in advance!
Ok, so I solved my issue. This issue helped me.
It seems like Postgres initialization process does stop the initialization process once done, and it is another process which follows up and accepts connexion.
Hence I had:
postgres:
deploy:
restart_policy:
condition: on-failure
window: 15m
and apparently docker received a end-of-process status code, so it stopped without going to the next process so never accepting connexions.
My interpretation may be incorrect, but at least if you face the issue, try removing restart_policy key to see if it fixes it.
I haven't tried to restore healthcheck yet, as it may also have undesired side effects.