I don't understand this postgresql log :
2022-03-27 08:00:19.441 UTC [584262] postgres#boutique2 FATAL: database "boutique2" does not exist
2022-03-27 08:00:19.704 UTC [584264] postgres#boutique2 FATAL: database "boutique2" does not exist
2022-03-27 08:01:54.770 UTC [781] LOG: received fast shutdown request
2022-03-27 08:01:54.773 UTC [781] LOG: aborting any active transactions
2022-03-27 08:01:54.779 UTC [781] LOG: background worker "logical replication launcher" (PID 800) exited with exit code 1
2022-03-27 08:01:54.780 UTC [795] LOG: shutting down
2022-03-27 08:01:54.797 UTC [781] LOG: database system is shut down
2022-03-27 08:02:16.254 UTC [770] LOG: starting PostgreSQL 13.5 (Debian 13.5-0+deb11u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-03-27 08:02:16.255 UTC [770] LOG: listening on IPv4 address "127.0.0.1", port 5432
2022-03-27 08:02:16.256 UTC [770] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-03-27 08:02:16.271 UTC [772] LOG: database system was shut down at 2022-03-27 08:01:54 UTC
2022-03-27 08:02:16.285 UTC [770] LOG: database system is ready to accept connections
2022-03-27 08:02:17.243 UTC [891] postgres#boutique2 FATAL: database "boutique2" does not exist
2022-03-27 08:02:17.640 UTC [1044] postgres#boutique2 FATAL: database "boutique2" does not exist
I dropped this database, which is not present in my mojolicious scripts, the only ones on this server.
root#perso:/etc/postgresql/13/main# grep postgresql.conf -e 'boutique2'
root#perso:/etc/postgresql/13/main# grep pg_hba.conf -e 'boutique2'
root#perso:/etc/postgresql/13/main#
Has please someone an idea about ?
After enabling log_connection, the postgres log file starts logging many rows like these:
19:2021-09-17 00:00:00 CEST [26025]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56911
22:2021-09-17 00:00:00 CEST [26026]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56912
25:2021-09-17 00:00:00 CEST [26027]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56913
28:2021-09-17 00:00:00 CEST [26028]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56914
31:2021-09-17 00:00:00 CEST [26029]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56915
34:2021-09-17 00:00:00 CEST [26030]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56916
37:2021-09-17 00:00:00 CEST [26031]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56917
40:2021-09-17 00:00:00 CEST [26032]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56918
43:2021-09-17 00:00:00 CEST [26033]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56919
46:2021-09-17 00:00:00 CEST [26034]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56920
49:2021-09-17 00:00:00 CEST [26035]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=56921
The client/host parameters happen to be the application server IP.
Yesterday I found 452.583 rows like these.
What could be the cause?
Thanks
an piece of log files
2021-09-17 01:55:04 CEST [3093]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=54868
2021-09-17 01:55:04 CEST [3094]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: ho/st=192.168.7.61 port=54869
2021-09-17 01:55:04 CEST [3093]: user=xxxxxxx,db=xxxxx,app=[unknown],client=192.168.7.61 LOG: connection authorized: user=xxxxxxxdatabase=xxxxxxx
2021-09-17 01:55:04 CEST [3094]: user=xxxxxxx,db=xxxxxxx,app=[unknown],client=192.168.7.61 LOG: connection authorized: user=xxxxxxxdatabase=xxxxxxx
2021-09-17 01:55:04 CEST [3093]: user=xxxxxxx,db=xxxxxxx,app=[unknown],client=192.168.7.61 LOG: disconnection: session time: 0:00:00.013 user=supreme_web database=xxxxxxxhost=192.168.7.61 port=54868
2021-09-17 01:55:04 CEST [3094]: user=xxxxxxx,db=xxxxxxx,app=[unknown],client=192.168.7.61 LOG: disconnection: session time: 0:00:00.014 user=xxxxxxxdatabase=xxxxxxxhost=192.168.7.61 port=54869
2021-09-17 01:55:04 CEST [3095]: user=[unknown],db=[unknown],app=[unknown],client=192.168.7.61 LOG: connection received: host=192.168.7.61 port=54870
2021-09-17 01:55:04 CEST [3095]: user=xxxxxxx,db=xxxxxxx,app=[unknown],client=192.168.7.61 LOG: connection authorized: user=xxxxxxxdatabase=xxxxxxx
2021-09-17 01:55:04 CEST [3095]: user=xxxxxxx,db=xxxxxxx,app=[unknown],client=192.168.7.61 LOG: disconnection: session time: 0:00:00.062 user=xxxxxxxdatabase=xxxxxxxhost=192.168.7.61 port=54870
If you have log_connections enabled then for a successful connection attempt you will see two log entries.
The first is when the client attempts to make a tcp/ip connection to the PostgreSQL socket. The second is when it connects to a database and authenticates.
Those are the first log entry and the process is split into two steps so that you can deny connections in pg_hba.conf cheaply before needing to create a new server backend.
I have a PostgreSQL 12 database running on CentOS 8 and until today everything was good.
After restarting my database server, my postgresql-12.service doesn't start anymore, I've tried with systemctl like systemctl start postgresql-12.service and I got this:
[postgres#perseus ~]$ sudo systemctl start postgresql-12.service
[sudo] password for postgres:
Job for postgresql-12.service failed because the control process exited with error code.
See "systemctl status postgresql-12.service" and "journalctl -xe" for details.
[postgres#perseus ~]$ sudo systemctl status postgresql-12.service
● postgresql-12.service - PostgreSQL 12 database server
Loaded: loaded (/usr/lib/systemd/system/postgresql-12.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-08-11 08:57:51 -03; 11s ago
Docs: https://www.postgresql.org/docs/12/static/
Process: 2822 ExecStart=/usr/pgsql-12/bin/postmaster -D ${PGDATA} (code=exited, status=1/FAILURE)
Process: 2816 ExecStartPre=/usr/pgsql-12/bin/postgresql-12-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
Main PID: 2822 (code=exited, status=1/FAILURE)
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.040 -03 [2822] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.040 -03 [2822] LOG: listening on IPv6 address "::", port 5432
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.041 -03 [2822] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.042 -03 [2822] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.068 -03 [2822] LOG: redirecting log output to logging collector process
Aug 11 08:57:49 perseus postmaster[2822]: 2020-08-11 08:57:49.068 -03 [2822] HINT: Future log output will appear in directory "log".
Aug 11 08:57:51 perseus systemd[1]: postgresql-12.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 08:57:51 perseus systemd[1]: postgresql-12.service: Killing process 2824 (postmaster) with signal SIGKILL.
Aug 11 08:57:51 perseus systemd[1]: postgresql-12.service: Failed with result 'exit-code'.
Aug 11 08:57:51 perseus systemd[1]: Failed to start PostgreSQL 12 database server.
And then, googling this error, i've tried some workarounds, like using pg_ctl and initdb comands, but it's also fails, like this:
[postgres#perseus ~]$ /usr/pgsql-12/bin/pg_ctl start -D /var/lib/pgsql/12/data
waiting for server to start....2020-08-11 09:04:44.197 -03 [2897] LOG: starting PostgreSQL 12.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5), 64-bit
2020-08-11 09:04:44.197 -03 [2897] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-08-11 09:04:44.197 -03 [2897] LOG: listening on IPv6 address "::", port 5432
2020-08-11 09:04:44.197 -03 [2897] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-08-11 09:04:44.198 -03 [2897] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-08-11 09:04:44.222 -03 [2897] LOG: redirecting log output to logging collector process
2020-08-11 09:04:44.222 -03 [2897] HINT: Future log output will appear in directory "log".
.. stopped waiting
pg_ctl: could not start server
Examine the log output.
[postgres#perseus ~]$ /usr/pgsql-12/bin/initdb -D '/var/lib/pgsql/12/data'
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
initdb: error: directory "/var/lib/pgsql/12/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/pgsql/12/data" or run initdb
with an argument other than "/var/lib/pgsql/12/data".
journalctl -xe results:
[postgres#perseus ~]$ sudo systemctl start postgresql-12.service
[sudo] password for postgres:
Job for postgresql-12.service failed because the control process exited with error code.
See "systemctl status postgresql-12.service" and "journalctl -xe" for details.
[postgres#perseus ~]$ sudo journalctl -xe
Aug 11 11:16:49 perseus sudo[3022]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Aug 11 11:16:49 perseus sudo[3022]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Aug 11 11:16:49 perseus sudo[3022]: pam_unix(sudo:session): session closed for user root
Aug 11 11:20:01 perseus systemd[1]: Starting system activity accounting tool...
-- Subject: Unit sysstat-collect.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sysstat-collect.service has begun starting up.
Aug 11 11:20:01 perseus systemd[1]: Started system activity accounting tool.
-- Subject: Unit sysstat-collect.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sysstat-collect.service has finished starting up.
--
-- The start-up result is RESULT.
Aug 11 11:26:10 perseus sudo[3095]: postgres : TTY=pts/0 ; PWD=/var/lib/pgsql ; USER=root ; COMMAND=/bin/journalctl -xe
Aug 11 11:26:10 perseus sudo[3095]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Aug 11 11:26:10 perseus sudo[3095]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Aug 11 11:26:49 perseus sudo[3095]: pam_unix(sudo:session): session closed for user root
Aug 11 11:26:52 perseus sudo[3107]: postgres : TTY=pts/0 ; PWD=/var/lib/pgsql ; USER=root ; COMMAND=/bin/journalctl -xe
Aug 11 11:26:52 perseus sudo[3107]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Aug 11 11:26:52 perseus sudo[3107]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Aug 11 11:27:48 perseus sudo[3107]: pam_unix(sudo:session): session closed for user root
Aug 11 11:30:01 perseus systemd[1]: Starting system activity accounting tool...
-- Subject: Unit sysstat-collect.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sysstat-collect.service has begun starting up.
Aug 11 11:30:01 perseus systemd[1]: Started system activity accounting tool.
-- Subject: Unit sysstat-collect.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit sysstat-collect.service has finished starting up.
--
-- The start-up result is RESULT.
Aug 11 11:34:13 perseus sudo[3167]: postgres : TTY=pts/0 ; PWD=/var/lib/pgsql ; USER=root ; COMMAND=/bin/systemctl start postgresql-12.service
Aug 11 11:34:13 perseus sudo[3167]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Aug 11 11:34:13 perseus sudo[3167]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Aug 11 11:34:13 perseus systemd[1]: Starting PostgreSQL 12 database server...
-- Subject: Unit postgresql-12.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit postgresql-12.service has begun starting up.
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.373 -03 [3176] LOG: starting PostgreSQL 12.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5), 64-bit
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.373 -03 [3176] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.373 -03 [3176] LOG: listening on IPv6 address "::", port 5432
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.375 -03 [3176] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.380 -03 [3176] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.399 -03 [3176] LOG: redirecting log output to logging collector process
Aug 11 11:34:13 perseus postmaster[3176]: 2020-08-11 11:34:13.399 -03 [3176] HINT: Future log output will appear in directory "log".
Aug 11 11:34:17 perseus systemd[1]: postgresql-12.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 11:34:17 perseus systemd[1]: postgresql-12.service: Killing process 3177 (postmaster) with signal SIGKILL.
Aug 11 11:34:17 perseus systemd[1]: postgresql-12.service: Failed with result 'exit-code'.
Aug 11 11:34:17 perseus systemd[1]: Failed to start PostgreSQL 12 database server.
-- Subject: Unit postgresql-12.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit postgresql-12.service has failed.
--
-- The result is RESULT.
Aug 11 11:34:17 perseus sudo[3167]: pam_unix(sudo:session): session closed for user root
Aug 11 11:34:38 perseus sudo[3188]: postgres : TTY=pts/0 ; PWD=/var/lib/pgsql ; USER=root ; COMMAND=/bin/journalctl -xe
Aug 11 11:34:38 perseus sudo[3188]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Aug 11 11:34:38 perseus sudo[3188]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Database log with verbose mode:
2020-08-11 15:23:01.400 -03 [7405] LOG: 00000: database system was interrupted; last known up at 2020-08-07 11:37:10 -03
2020-08-11 15:23:01.400 -03 [7405] LOCATION: StartupXLOG, xlog.c:6305
2020-08-11 15:23:05.160 -03 [7405] FATAL: 58P01: could not open directory "": No such file or directory
2020-08-11 15:23:05.160 -03 [7405] LOCATION: ReadDirExtended, fd.c:2563
2020-08-11 15:23:05.161 -03 [7403] LOG: 00000: startup process (PID 7405) exited with exit code 1
2020-08-11 15:23:05.161 -03 [7403] LOCATION: LogChildExit, postmaster.c:3675
2020-08-11 15:23:05.161 -03 [7403] LOG: 00000: aborting startup due to startup process failure
2020-08-11 15:23:05.161 -03 [7403] LOCATION: reaper, postmaster.c:2945
2020-08-11 15:23:05.163 -03 [7403] LOG: 00000: database system is shut down
2020-08-11 15:23:05.163 -03 [7403] LOCATION: UnlinkLockFiles, miscinit.c:860
The PGDATA directory:
[postgres#perseus ~]$ ll -la /var/lib/pgsql/12/data/
total 88
drwx------. 20 postgres postgres 4096 Aug 11 15:49 .
drwx------. 4 postgres postgres 66 Aug 11 14:55 ..
drwx------. 26 postgres postgres 4096 Jul 28 09:29 base
-rw------- 1 postgres postgres 30 Aug 11 15:49 current_logfiles
drwx------. 2 postgres postgres 4096 Aug 11 09:34 global
-rw-r--r-- 1 root root 560 May 29 10:05 global.sql
drwx------. 2 postgres postgres 188 Aug 11 15:50 log
drwx------. 2 postgres postgres 6 May 22 09:16 pg_commit_ts
drwx------. 2 postgres postgres 6 May 22 09:16 pg_dynshmem
-rw------- 1 postgres postgres 4329 May 28 09:07 pg_hba.conf
-rw-------. 1 postgres postgres 1636 May 22 09:16 pg_ident.conf
drwx------. 4 postgres postgres 68 Aug 7 11:37 pg_logical
drwx------. 4 postgres postgres 36 May 22 09:16 pg_multixact
drwx------. 2 postgres postgres 18 Aug 11 15:49 pg_notify
drwx------. 2 postgres postgres 6 May 22 09:16 pg_replslot
drwx------. 2 postgres postgres 6 May 22 09:16 pg_serial
drwx------. 2 postgres postgres 6 May 22 09:16 pg_snapshots
drwx------. 2 postgres postgres 6 Jul 24 14:45 pg_stat
drwx------. 2 postgres postgres 4096 Aug 11 15:49 pg_stat_tmp
drwx------. 2 postgres postgres 18 Jul 28 09:36 pg_subtrans
drwx------. 2 postgres postgres 6 May 22 09:16 pg_tblspc
drwx------. 2 postgres postgres 6 May 22 09:16 pg_twophase
-rw-------. 1 postgres postgres 3 May 22 09:16 PG_VERSION
drwx------. 3 postgres postgres 4096 Jul 31 17:16 pg_wal
drwx------. 2 postgres postgres 4096 Jun 3 11:51 pg_xact
-rw-------. 1 postgres postgres 88 May 22 09:16 postgresql.auto.conf
-rw------- 1 postgres postgres 26808 Aug 11 15:22 postgresql.conf
-rw------- 1 postgres postgres 58 Aug 11 15:49 postmaster.opts
-rw-r--r-- 1 root root 1160 May 29 09:08 teste.backup
Edited with more data.
I'm using PostgreSQL v11.0. I am executing a simple SQL query
delete from base.sys_attribute where id=20;
It fails and returns the error:
server process (PID 29) was terminated by signal 11 (see log below)
Any idea how I can troubleshoot this issue? I'm completely stuck...
listof-db | 2019-05-10 08:54:46.425 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
listof-db | 2019-05-10 08:54:46.425 UTC [1] LOG: listening on IPv6 address "::", port 5432
listof-db | 2019-05-10 08:54:46.425 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
listof-db | 2019-05-10 08:54:46.435 UTC [20] LOG: database system was shut down at 2019-05-10 08:54:26 UTC
listof-db | 2019-05-10 08:54:46.438 UTC [1] LOG: database system is ready to accept connections
listof-db | 2019-05-10 08:56:52.295 UTC [1] LOG: server process (PID 29) was terminated by signal 11
listof-db | 2019-05-10 08:56:52.295 UTC [1] DETAIL: Failed process was running: delete from base.sys_attribute where id=20
listof-db | 2019-05-10 08:56:52.295 UTC [1] LOG: terminating any other active server processes
listof-db | 2019-05-10 08:56:52.295 UTC [24] WARNING: terminating connection because of crash of another server process
listof-db | 2019-05-10 08:56:52.295 UTC [24] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
listof-db | 2019-05-10 08:56:52.295 UTC [24] HINT: In a moment you should be able to reconnect to the database and repeat your command.
listof-db | 2019-05-10 08:56:52.486 UTC [1] LOG: all server processes terminated; reinitializing
listof-db | 2019-05-10 08:56:53.093 UTC [30] LOG: database system was interrupted; last known up at 2019-05-10 08:54:46 UTC
listof-db | 2019-05-10 08:56:53.187 UTC [30] LOG: database system was not properly shut down; automatic recovery in progress
listof-db | 2019-05-10 08:56:53.189 UTC [30] LOG: redo starts at 0/75955A8
listof-db | 2019-05-10 08:56:53.189 UTC [30] LOG: invalid record length at 0/75971B8: wanted 24, got 0
listof-db | 2019-05-10 08:56:53.189 UTC [30] LOG: redo done at 0/7597180
listof-db | 2019-05-10 08:56:53.194 UTC [1] LOG: database system is ready to accept connections
Postgresql wont start in ubuntu 16.04,
Please give support.
These are the error log read:
2018-10-12 13:42:34 UTC [3371-2] LOG: received fast shutdown request
2018-10-12 13:42:34 UTC [3371-3] LOG: aborting any active transactions
2018-10-12 13:42:34 UTC [3376-2] LOG: autovacuum launcher shutting down
2018-10-12 13:42:34 UTC [3373-1] LOG: shutting down
2018-10-12 13:42:34 UTC [3373-2] LOG: database system is shut down
2018-10-12 13:42:52 UTC [3855-1] LOG: database system was shut down at
2018-10-12 13:42:34 UTC
2018-10-12 13:42:52 UTC [3855-2] LOG: MultiXact member wraparound
protections are now enabled
2018-10-12 13:42:52 UTC [3853-1] LOG: database system is ready to accept
connections
2018-10-12 13:42:52 UTC [3859-1] LOG: autovacuum launcher started
2018-10-12 13:42:52 UTC [3861-1] [unknown]#[unknown] LOG: incomplete
startup
packet