Postgres is restarting continuously on using shared_preload_libraries extension.
https://postgresqlco.nf/doc/en/param/shared_preload_libraries/
I am running postgres-15.1 using a python-based daemon in CentOS7-32bit arch. It is working fine if we do not use "shared_preload_libraries" extension. But after enabling this extension using "ALTER SYSTEM SET shared_preload_libraries" command, the postgres is restarting every few seconds.
Initially it was working fine with postgres-9.6.4.
Postgres logs:
waiting for server to start....2023-02-15 07:13:45.676 GMT [28605] LOG: skipping missing configuration file "/home/runtime/pgsql/data/postgresql.auto.conf"
2023-02-15 07:13:45.825 GMT [28605] LOG: starting PostgreSQL 15.1 on i686-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 32-bit
2023-02-15 07:13:45.825 GMT [28605] LOG: listening on IPv4 address "127.0.0.1", port 5432
2023-02-15 07:13:45.933 GMT [28605] LOG: listening on Unix socket "/home/runtime/pgsql/.s.PGSQL.5432"
2023-02-15 07:13:45.969 GMT [28608] LOG: database system was shut down at 2023-02-15 07:13:35 GMT
2023-02-15 07:13:45.989 GMT [28605] LOG: database system is ready to accept connections
done
server started
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
ALTER SYSTEM
2023-02-15 07:13:51.480 GMT [28605] LOG: received fast shutdown request
waiting for server to shut down....2023-02-15 07:13:51.512 GMT [28605] LOG: aborting any active transactions
2023-02-15 07:13:51.513 GMT [28605] LOG: background worker "logical replication launcher" (PID 28611) exited with exit code 1
2023-02-15 07:13:51.513 GMT [28606] LOG: shutting down
2023-02-15 07:13:51.536 GMT [28606] LOG: checkpoint starting: shutdown immediate
2023-02-15 07:13:51.908 GMT [28606] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.090 s, sync=0.028 s, total=0.395 s; sync files=2, longest=0.021 s, average=0.014 s; distance=0 kB, estimate=0 kB
2023-02-15 07:13:51.909 GMT [28605] LOG: database system is shut down
done
server stopped
I tried to use postgres-15.0 and postgres-14.4, got the same behavior with both. I am not able to find any open issues w.r.t. shared_preload_libraries extension with new versions of Postgres.
PS: I have built this Postgres from the source code with openssl-1.1.1i.
I am using "citus" library with this.
ALTER SYSTEM SET shared_preload_libraries="citus";
I have generated a new citus.so file from it's source code using postgres-15.1. github.com/citusdata/citus
I have restored a backup from psql server A to psql server B. When the postgresql service starts on server B, the log below is produced. From what I read the invalid record length message is not a problem, just an indicator that the end of the last WAL file was reached.
However, when I attempt to run 'psql' it gives a fatal error that the database is starting up. (I waited over an hour and still in that state). I still can't run psql. Note that standby.signal file is present on B, in case that matters.
What does this mean, and how do I fix it? Or is this normal while the standby.signal file is present?
2022-10-18 16:50:47.953 EDT [126144] LOG: starting PostgreSQL 13.5 (Debian 13.5-0+deb11u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-10-18 16:50:47.954 EDT [126144] LOG: listening on IPv6 address "::1", port 5432
2022-10-18 16:50:47.954 EDT [126144] LOG: listening on IPv4 address "127.0.0.1", port 5432
2022-10-18 16:50:47.955 EDT [126144] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-18 16:50:47.963 EDT [126145] LOG: database system was shut down in recovery at 2022-10-18 16:50:47 EDT
2022-10-18 16:50:48.052 EDT [126145] LOG: entering standby mode
2022-10-18 16:50:48.142 EDT [126145] LOG: redo starts at 0/70000028
2022-10-18 16:50:48.142 EDT [126145] LOG: invalid record length at 0/700000D8: wanted 24, got 0
pg_ctl: server did not start in time
2022-10-18 16:52:16.600 EDT [126876] postgres#postgres FATAL: the database system is starting up
In the Datastore logs, I encountered the following error, Not sure what has gone wrong.
[7804] LOG: starting PostgreSQL 13.1, compiled by Visual C++ build 1914, 64-bit
2021-08-23 22:56:15.980 CEST [7804] LOG: listening on IPv4 address "127.0.0.1", port 9003
2021-08-23 22:56:15.983 CEST [7804] LOG: listening on IPv4 address "10.91.198.36", port 9003
2021-08-23 22:56:16.041 CEST [8812] LOG: database system was shut down at 2021-08-23 22:54:51 CEST
2021-08-23 22:56:16.044 CEST [8812] LOG: invalid primary checkpoint record
2021-08-23 22:56:16.045 CEST [8812] PANIC: could not locate a valid checkpoint record
2021-08-23 22:56:16.076 CEST [7804] LOG: startup process (PID 8812) was terminated by exception 0xC0000409
2021-08-23 22:56:16.076 CEST [7804] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value.
2021-08-23 22:56:16.078 CEST [7804] LOG: aborting startup due to startup process failure
2021-08-23 22:56:16.094 CEST [7804] LOG: database system is shut down
Somebody deleted crucial WAL files (to free space?), and now your cluster is corrupted
Restore from backup. If you have no backup, running pg_resetwal is an option, since it seems there was a clean shutdown.
We built a debian docker image with postgresql to run one of our service. The database is for internal container use and does not need port mapping. I believe it is installed via apt-get in the Dockerbuild file.
We stop and start this service often, and it is a performance issue that the database is slow to startup. Although empty, takes sightly over 20s to accept connection on the first time we start the docker image. The log is as follow :
2019-04-05 13:05:30.924 UTC [19] LOG: could not bind IPv6 socket: Cannot assign requested address
2019-04-05 13:05:30.924 UTC [19] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-04-05 13:05:30.982 UTC [20] LOG: database system was shut down at 2019-04-05 12:57:16 UTC
2019-04-05 13:05:30.992 UTC [20] LOG: MultiXact member wraparound protections are now enabled
2019-04-05 13:05:30.998 UTC [19] LOG: database system is ready to accept connections
2019-04-05 13:05:30.998 UTC [24] LOG: autovacuum launcher started
2019-04-05 13:05:31.394 UTC [26] [unknown]#[unknown] LOG: incomplete startup packet
2019-04-19 13:21:58.974 UTC [37] LOG: could not bind IPv6 socket: Cannot assign requested address
2019-04-19 13:21:58.974 UTC [37] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-04-19 13:21:59.025 UTC [38] LOG: database system was interrupted; last known up at 2019-04-05 13:05:34 UTC
2019-04-19 13:21:59.455 UTC [39] [unknown]#[unknown] LOG: incomplete startup packet
2019-04-19 13:21:59.971 UTC [42] postgres#postgres FATAL: the database system is starting up
[...]
2019-04-19 13:22:15.221 UTC [85] root#postgres FATAL: the database system is starting up
2019-04-19 13:22:15.629 UTC [38] LOG: database system was not properly shut down; automatic recovery in progress
2019-04-19 13:22:15.642 UTC [38] LOG: redo starts at 0/14EEBA8
2019-04-19 13:22:15.822 UTC [38] LOG: invalid record length at 0/24462D0: wanted 24, got 0
2019-04-19 13:22:15.822 UTC [38] LOG: redo done at 0/24462A8
2019-04-19 13:22:15.822 UTC [38] LOG: last completed transaction was at log time 2019-04-05 13:05:36.602318+00
2019-04-19 13:22:16.084 UTC [38] LOG: MultiXact member wraparound protections are now enabled
2019-04-19 13:22:16.094 UTC [37] LOG: database system is ready to accept connections
2019-04-19 13:22:16.094 UTC [89] LOG: autovacuum launcher started
2019-04-19 13:22:21.528 UTC [92] root#test LOG: could not receive data from client: Connection reset by peer
2019-04-19 13:22:21.528 UTC [92] root#test LOG: unexpected EOF on client connection with an open transaction
Any suggetion in fixing this startup issue ?
EDIT : Some requested the dockerfile, here is relevant lines
RUN apt-get update \
&& apt-get install -y --force-yes \
postgresql-9.6-pgrouting \
postgresql-9.6-postgis-2.3 \
postgresql-9.6-postgis-2.3-scripts \
[...]
# Download, compile and install GRASS 7.2
[...]
USER postgres
# Create a database 'grass_backend' owned by the "root" role.
RUN /etc/init.d/postgresql start \
&& psql --command "CREATE USER root WITH SUPERUSER [...];" \
&& psql --command "CREATE EXTENSION postgis; CREATE EXTENSION plpython3u;" --dbname [dbname] \
&& psql --command "CREATE EXTENSION postgis_sfcgal;" --dbname [dbname] \
&& psql --command "CREATE EXTENSION postgis; CREATE EXTENSION plpython3u;" --dbname grass_backend
WORKDIR [...]
End of file after workdir, meaning I guess the database isn't properly shut down
Answer I stopped properly postgresql inside the docker install. It now starts 15s faster. Thanks for replying
Considering the line database system was not properly shut down; automatic recovery in progress that would definitely explain slow startup, please don't kill the service, send the stop command and wait for it to close properly.
Please note that the system might kill the process if it takes to long to stop, this will happen in the case of postgresql if there are connections still held to it (probably from your application). If you disconnect all the connections and than stop, postgresql should be able to stop relatively quickly.
Also make sure you stop the postgresql service inside the container before turning it off.
TCP will linger connections for a while, if you are starting and stopping in quick succession without properly stopping the service inside that would explain your error of why the port is unavailable, normally the service can start/stop in very quick succession on my machine if nothing is connected to it.
3 start-stop cycles of postgresql on my machine (I have 2 decently sized databases)
$ time bash -c 'for i in 1 2 3; do /etc/init.d/postgresql-11 restart; done'
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
* Stopping PostgreSQL 11 (this can take up to 92 seconds) ... [ ok ]
* /run/postgresql: correcting mode
* Starting PostgreSQL 11 ... [ ok ]
real 0m1.188s
user 0m0.260s
sys 0m0.080s
I use postgres:10 (https://hub.docker.com/_/postgres/) image for DB.
It is deployed in docker swarm cluster.
After running DB replica I got database system is shut down in DB's log.
2018-05-11 10:26:53.073 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 10:26:53.073 UTC [1] LOG: listening on IPv6 address "::", port 5432,
2018-05-11 10:26:53.077 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 10:26:53.092 UTC [20] LOG: database system was shut down at 2018-05-11 10:26:20 UTC,
2018-05-11 10:26:53.100 UTC [1] LOG: database system is ready to accept connections,
The files belonging to this database system will be owned by user "postgres".,
This user must also own the server process.,
,
The database cluster will be initialized with locale "en_US.utf8".,
The default database encoding has accordingly been set to "UTF8".,
The default text search configuration will be set to "english".,
,
Data page checksums are disabled.,
,
fixing permissions on existing directory /var/lib/postgresql/data ... ok,
creating subdirectories ... ok,
selecting default max_connections ... 100,
selecting default shared_buffers ... 128MB,
selecting dynamic shared memory implementation ... posix,
creating configuration files ... ok,
running bootstrap script ... ok,
performing post-bootstrap initialization ... ok,
,
WARNING: enabling "trust" authentication for local connections,
You can change this by editing pg_hba.conf or using the option -A, or,
--auth-local and --auth-host, the next time you run initdb.,
syncing data to disk ... ok,
,
Success. You can now start the database server using:,
,
pg_ctl -D /var/lib/postgresql/data -l logfile start,
,
waiting for server to start....2018-05-11 09:39:21.129 UTC [37] LOG: listening on IPv4 address "127.0.0.1", port 5432,
2018-05-11 09:39:21.130 UTC [37] LOG: could not bind IPv6 address "::1": Cannot assign requested address,
2018-05-11 09:39:21.130 UTC [37] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.,
2018-05-11 09:39:21.133 UTC [37] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.147 UTC [38] LOG: database system was shut down at 2018-05-11 09:39:20 UTC,
2018-05-11 09:39:21.152 UTC [37] LOG: database system is ready to accept connections,
done,
server started,
CREATE DATABASE,
,
CREATE ROLE,
,
,
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*,
,
2018-05-11 09:39:21.595 UTC [37] LOG: received fast shutdown request,
waiting for server to shut down....2018-05-11 09:39:21.596 UTC [37] LOG: aborting any active transactions,
2018-05-11 09:39:21.598 UTC [37] LOG: worker process: logical replication launcher (PID 44) exited with exit code 1,
2018-05-11 09:39:21.599 UTC [39] LOG: shutting down,
2018-05-11 09:39:21.613 UTC [37] LOG: database system is shut down,
done,
server stopped,
,
PostgreSQL init process complete; ready for start up.,
,
2018-05-11 09:39:21.706 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 09:39:21.706 UTC [1] LOG: listening on IPv6 address "::", port 5432,
2018-05-11 09:39:21.709 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.724 UTC [64] LOG: database system was shut down at 2018-05-11 09:39:21 UTC,
2018-05-11 09:39:21.729 UTC [1] LOG: database system is ready to accept connections,
2018-05-11 10:26:20.444 UTC [1] LOG: received smart shutdown request,
2018-05-11 10:26:20.449 UTC [1] LOG: worker process: logical replication launcher (PID 70) exited with exit code 1,
2018-05-11 10:26:20.449 UTC [65] LOG: shutting down,
2018-05-11 10:26:20.460 UTC [1] LOG: database system is shut down,
Image :
FROM postgres:10
COPY healthcheck /usr/local/bin/
RUN chmod +x /usr/local/bin/healthcheck
HEALTHCHECK --interval=30s --timeout=30s --retries=3 \
CMD healthcheck
Snippet from docker-compose :
db_jackrabbit:
build: ./images/pgsql_jackrabbit
container_name: db_jackrabbit
environment:
- POSTGRES_DB=${JACK_POSTGRES_DB}
- POSTGRES_USER=${JACK_POSTGRES_USER}
- POSTGRES_PASSWORD=${JACK_POSTGRES_PASSWORD}
volumes:
- pgsql_jackrabbit_local:/var/lib/postgresql/data
ports:
- ${PORT_DB_JACKRABBIT}:5432
healthcheck:
#!/bin/bash
set -eo pipefail
host="$(hostname -i || echo '127.0.0.1')"
user="${POSTGRES_USER:-postgres}"
db="${POSTGRES_DB:-$POSTGRES_USER}"
export PGPASSWORD="${POSTGRES_PASSWORD:-}"
args=(
# force postgres to not use the local unix socket (test "external" connectibility)
--host "$host"
--username "$user"
--dbname "$db"
--quiet --no-align --tuples-only
)
if select="$(echo 'SELECT 1' | psql "${args[#]}")" && [ "$select" = '1' ]; then
exit 0
fi
exit 1
But DB still alive. It is shutdown periodically and accept connections again (
What it the problem?
Thanks in advance!
Ok, so I solved my issue. This issue helped me.
It seems like Postgres initialization process does stop the initialization process once done, and it is another process which follows up and accepts connexion.
Hence I had:
postgres:
deploy:
restart_policy:
condition: on-failure
window: 15m
and apparently docker received a end-of-process status code, so it stopped without going to the next process so never accepting connexions.
My interpretation may be incorrect, but at least if you face the issue, try removing restart_policy key to see if it fixes it.
I haven't tried to restore healthcheck yet, as it may also have undesired side effects.