PgPool-II backend authentication failed - postgresql

I'm trying to configure pgpool as the load balancer for my Postgres cluster.
I have two postgres nodes, 1 master and 1 slave.
My pg_hba.conf looks like
hostssl user mydb 1.1.1.1/32 md5
hostssl user postgres 1.1.1.1/32 md5
host user mydb 1.1.1.1/32 md5
host user postgres 1.1.1.1/32 md5
where 1.1.1.1/32 is my actual pgpool server IP.
If I try to establish a connection to ether master or slave using psql right from the pgpool container, I can do it without any problems.
But when I start pgpool I got this error message:
2021-10-26 13:50:13: pid 753: ERROR: backend authentication failed
2021-10-26 13:50:13: pid 753: DETAIL: backend response with kind 'E' when expecting 'R'
2021-10-26 13:50:13: pid 753: HINT: This issue can be caused by version mismatch (current version 3)
2021-10-26 13:50:13: pid 736: ERROR: backend authentication failed
2021-10-26 13:50:13: pid 736: DETAIL: backend response with kind 'E' when expecting 'R'
2021-10-26 13:50:13: pid 736: HINT: This issue can be caused by version mismatch (current version 2)
If I edit pool_passwd file and set some invalid password I got a proper error
2021-10-26 13:59:03: pid 736: ERROR: md5 authentication failed
2021-10-26 13:59:03: pid 736: DETAIL: password does not match
So I guess that's not a problem with my postgres credentials.
Any ideas?

Related

Postgresql shutdown by itself

2021-11-03 07:15:23.704 UTC [354507] postgres#postgres FATAL: password authentication failed for user "postgres"
2021-11-03 07:15:23.704 UTC [354507] postgres#postgres DETAIL: Password does not match for user "postgres".
Connection matched pg_hba.conf line 105: "host all all 0.0.0.0/0 md5"
2021-11-03 07:33:29.904 UTC [354788] pgsql#postgres FATAL: password authentication failed for user "pgsql"
2021-11-03 07:33:29.904 UTC [354788] pgsql#postgres DETAIL: Role "pgsql" does not exist.
Connection matched pg_hba.conf line 105: "host all all 0.0.0.0/0 md5"
2021-11-03 07:52:40.628 UTC [355083] pgsql#postgres FATAL: password authentication failed for user "pgsql"
2021-11-03 07:52:40.628 UTC [355083] pgsql#postgres DETAIL: Role "pgsql" does not exist.
Connection matched pg_hba.conf line 105: "host all all 0.0.0.0/0 md5"
2021-11-03 07:53:02.963 UTC [327839] LOG: received smart shutdown request
2021-11-03 07:53:02.976 UTC [327839] LOG: background worker "logical replication launcher" (PID 327846) exited with exit code 1
2021-11-03 07:53:02.980 UTC [327841] LOG: shutting down
2021-11-03 07:53:03.011 UTC [327839] LOG: database system is shut down
I am hosting PostgreSQL on a DigitalOcean droplet and since this server is just for my toy project only so I have all the ports open. I understand this is bad practice but from my understanding, unless the hacker somehow gains access to my username and password the DB will be safe.
But last month and yesterday my Postgre just shut down itself and according to the log, it seems it was shut down after a shutdown request?
I am using "postgres" as my user name and from the log, I can see someone keep trying to log with the username "pgsql"?
So I want to know am I being hacked or I did something stupid and somehow shut down the server by myself?

Docbase could not connect to the database error in documentum content server installation

While I am installing Documentum content server on AWS EKS, I am receiving this Error.
Postgress DB is installed on EC2 VM.
14:20:47,013 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerModifyDocbaseDirectory
- The installer will create the folder structure for repository postgres. 14:20:47,021 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerPasswordFileGenerator
- The installer is generating database password file... 14:20:47,111 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerIniGenerator
- The installer will create server.in file for repository postgres. 14:20:47,152 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateTableSpaceScriptGenerator
- The installer will create scripts to for Postgresql Database. 14:20:47,152 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateTableSpaceScriptGenerator
- The URL is jar:file:/tmp/install.dir.208/InstallerData/installer.zip!/dm_CreateTableSpace.sql 14:20:47,209 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCopyDeleteTableSpaceScript
- The installer will move file /opt/dctm/dba/config/postgres/dm_DeleteTableSpace.sql to a new location /opt/dctm/server_uninstall/delete_db/postgres/dm_DeleteTableSpace.sql. 14:20:47,214 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerCreateTableSpace
- The installer is executing the : Creating the database script. 14:20:47,355 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerWebCacheIniGenerator
- The installer will create webcache.ini file for the repository. 14:20:47,394 INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerTestServerIni
- The installer is testing the database connection information 14:20:47,395 INFO [main] com.documentum.install.server.common.services.db.DiServerPostgresqlServer
- The installer is validating the database connection information in the server.ini file. 14:20:47,563 ERROR [main] com.documentum.install.server.installanywhere.actions.DiWAServerTestServerIni
- Docbase could not connect to the database. Please check output file for more information: /tmp/291406.tmp/DBTestResult18051870723865753931.tmp com.documentum.install.shared.common.error.DiException: Docbase could not connect to the database. Please check output file for more information: /tmp/291406.tmp/DBTestResult18051870723865753931.tmp
This is log of dm_CreateTableSpace.out
psql:/opt/dctm/dba/config/postgres/dm_CreateTableSpace.sql:1: ERROR: role "postgres" already exists
psql:/opt/dctm/dba/config/postgres/dm_CreateTableSpace.sql:3: ERROR: zero-length delimited identifier at or near """"
LINE 1: GRANT "postgres" TO "";
^
psql:/opt/dctm/dba/config/postgres/dm_CreateTableSpace.sql:6: ERROR: database "dm_postgres_docbase" already exists
ALTER DATABASE
GRANT
psql:/opt/dctm/dba/config/postgres/dm_CreateTableSpace.sql:9: ERROR: zero-length delimited identifier at or near """"
LINE 1: REVOKE "postgres" FROM "";
^
You are now connected to database "dm_postgres_docbase" as user "postgres".
CREATE SCHEMA
SET
GRANT
GRANT
GRANT
I don't understand how to make it working.
Even I am facing issues with Postgres RDS Instant i.e.
GRANT "postgres" TO ""
The log is getting generated on PostgreSQL ec2 instance
2021-07-20 11:53:46.434 UTC [7854] dctm#dm_dctm_docbase FATAL:
password authentication fail
ed for user "dctm" 2021-07-20 11:53:46.434 UTC [7854]
dctm#dm_dctm_docbase DETAIL: Role "dctm" does not exist.
Connection matched pg_hba.conf line 99: "host all all 172
.16.0.0/16 md5" 2021-07-20 11:53:46.436 UTC [7855]
dctm#dm_dctm_docbase FATAL: password authentication fail
ed for user "dctm" 2021-07-20 11:53:46.436 UTC [7855]
dctm#dm_dctm_docbase DETAIL: Role "dctm" does not exist.
Connection matched pg_hba.conf line 99: "host all all 172
.16.0.0/16 md5" 2021-07-20 11:53:49.056 UTC [7857]
postgres#postgres ERROR: zero-length delimited identifie
r at or near """" at character 17 2021-07-20 11:53:49.056 UTC [7857]
postgres#postgres STATEMENT: GRANT "dctm" TO ""; 2021-07-20
11:53:49.145 UTC [7857] postgres#postgres ERROR: zero-length
delimited identifie
r at or near """" at character 20 2021-07-20 11:53:49.145 UTC [7857]
postgres#postgres STATEMENT: REVOKE "dctm" FROM ""; password
authentication fail ed for user "dctm" 2021-07-20
11:53:46.434 UTC [7854] dctm#dm_dctm_docbase DETAIL: Role "dctm" does
not exist.
I am using PostgreSQL superuser login and is working fine with cli.
updated the value in heml chart i.e values.yaml for documentum : content-server.
Every time you try to re-run the install you should completely delete everything. It looks like Postgres has the tablespace already created and that's why the test is failing

pgpool second standby shows "down" in status

I have setup pgpool with 3 nodes. Lets say node1 as primary and node2,node3 and secondary. When I stop the database service on node1, the node2 becomes primary as expected but node3 is being shown as "down" in show pool_nodes command. It is showing that streaming is live on node3 and I can connect to psql as well on node3 but status is down in pgpool
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
2020-05-07 05:58:39.969 EDT [23820] LOG: record with incorrect prev-link 0/7D000028 at 0/81000060
2020-05-07 05:58:39.976 EDT [25653] FATAL: could not connect to the primary server: could not connect to server: Connection refused
Is the server running on host "node1" (10.32.5.121) and accepting
TCP/IP connections on port 5432?
Permission denied, please try again.
Permission denied, please try again.
2020-05-13 08:02:45.738 EDT [3159] ERROR: pgpool_remote_start failed
2020-05-13 08:02:45.738 EDT [3159] STATEMENT: SELECT pgpool_remote_start('node3', '/var/lib/pgsql/11/data')
2020-05-13 08:03:37.917 EDT [3032] LOG: unexpected EOF on standby connection
On executing pcp_recovery_node command, the following error occassionally occurs but node recovers with same command after 1 or 2 more retries
ERROR: executing remote start failed with error: "ERROR: pgpool_remote_start failed
I then have to attach the node with pcp_attach_node to change status to "up"

pg_Ctl -data directory has wrong ownership

I am unable to start the Postgres server and whenever I use pg_ctl I am getting the following error - can some one help me to fix this. I changed the folder permissions using CHmod and tried running with Sudo -s also but still the problem exists.
one error I did was, I deleted the Postmaster.pid when the server was running- post this I am getting this issue when ever I try to start the server through pg_ctl and another error when I use the pgadmin.
Any suggestions here will be really helpful- thanks.
Using Macos Shell command :
'pg_ctl start -D /Library/PostgreSQL/12/data waiting for server to start....2020-05-05 11:40:04.838 IST [1216] FATAL: data directory "/Library/PostgreSQL/12/data" has wrong ownership 2020-05-05 11:40:04.838 IST [1216] HINT: The server must be started by the user that owns the data directory. stopped waiting pg_ctl: could not start server Examine the log output.'
Using pgadmin the error is as follows :
'could not connect to server: Connection refused Is the server running on host "localhost" (::1) and accepting TCP/IP connections on port 5434? could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5434?'
p.s. : I modified the hba.conf and also the postgres.conf files to allow connection from the local ip
Error received on 5May
waiting for server to start....2020-05-05 19:54:13.029 IST [7274] LOG: starting PostgreSQL 12.2 on x86_64-apple-darwin, compiled by Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn), 64-bit
2020-05-05 19:54:13.030 IST [7274] LOG: listening on IPv6 address "::", port 5433
2020-05-05 19:54:13.030 IST [7274] LOG: listening on IPv4 address "0.0.0.0", port 5433
2020-05-05 19:54:13.030 IST [7274] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2020-05-05 19:54:13.039 IST [7274] LOG: redirecting log output to logging collector process... 2020-05-05 19:54:13.039 IST [7274] HINT: Future log output will appear in directory "log" stopped waiting .. pg_ctl: could not start server
Examine the log output.
Log file details
2020-05-05 21:29:30.748 IST [8853] LOG: invalid authentication method "0.0.0.0/0"
2020-05-05 21:29:30.748 IST [8853] CONTEXT: line 80 of configuration file "/Library/PostgreSQL/12/data/pg_hba.conf"
2020-05-05 21:29:30.748 IST [8853] FATAL: could not load pg_hba.conf
2020-05-05 21:29:30.749 IST [8853] LOG: database system is shut down
Details of my pg_HBA conf
# "local" is for Unix domain socket connections only
local all all 0.0.0.0/0 md5
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all md5
host replication all 127.0.0.1/32 md5
host replication all ::1/128 md5
host all all 0.0.0.0/0 md5
host all all ::/0 md5
latest log file
bash-3.2$ cat postgresql-2020-05-05_221328.log
2020-05-05 22:13:28.794 IST [9834] LOG: database system was interrupted; last known up at 2020-05-05 22:13:09 IST
2020-05-05 22:13:28.872 IST [9834] LOG: database system was not properly shut down; automatic recovery in progress
2020-05-05 22:13:28.874 IST [9834] LOG: redo starts at 0/17742C8
2020-05-05 22:13:28.874 IST [9834] LOG: invalid record length at 0/1774300: wanted 24, got 0
2020-05-05 22:13:28.874 IST [9834] LOG: redo done at 0/17742C8
2020-05-05 22:13:28.881 IST [9832] LOG: database system is ready to accept connections
......
also I found this error while staring the server and the PID is chaning everytime..
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2020-05-05 22:09:21.941 IST [9746] FATAL: lock file "postmaster.pid" already exists
2020-05-05 22:09:21.941 IST [9746] HINT: Is another postmaster (PID 9735) running in data directory "/Library/PostgreSQL/12/data"?
stopped waiting
pg_ctl: could not start server
Examine the log output.
bash-3.2$ kill -9 9735
bash-3.2$ pg_ctl start -D /Library/PostgreSQL/12/data
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2020-05-05 22:09:35.829 IST [9758] FATAL: lock file "postmaster.pid" already exists
2020-05-05 22:09:35.829 IST [9758] HINT: Is another postmaster (PID 9747) running in data directory "/Library/PostgreSQL/12/data"?
stopped waiting
pg_ctl: could not start server
Examine the log output.
502 9833 9832 0 10:13PM ?? 0:00.00 postgres: logger
502 9835 9832 0 10:13PM ?? 0:00.00 postgres: checkpointer
502 9836 9832 0 10:13PM ?? 0:00.04 postgres: background writer
502 9837 9832 0 10:13PM ?? 0:00.01 postgres: walwriter
502 9838 9832 0 10:13PM ?? 0:00.01 postgres: autovacuum launcher
502 9839 9832 0 10:13PM ?? 0:00.01 postgres: stats collector
502 9840 9832 0 10:13PM ?? 0:00.00 postgres: logical replication launcher
0 9641 9504 0 10:03PM ttys000 0:00.02 sudo -u postgres -s /bin/bash
502 9904 9642 0 10:37PM ttys000 0:00.00 grep postgres
The data directory should be owned by the postgres user and have user-only access (700 or u+rwx)
Does this match what you have set up?
Thom Brown
Disclosure: I am an EnterpriseDB employee.
Try running this code
pg_ctl -D /usr/local/var/postgres start

pgpool HA + repmgr for Postgresql 9.6

I'm trying to conifure pgpool in my postgresql environment (2 postgresql servers + 1 pgpool) to do HA while repmgr is responsible for the replication.
I'm getting the next messages in the log :
017-12-03 19:27:07: pid 19033: DEBUG: pool_flush_it: flush size: 0
2017-12-03 19:27:07: pid 19033: DEBUG: pool_read: read 103 bytes from backend 1
2017-12-03 19:27:07: pid 19033: ERROR: failed to authenticate
2017-12-03 19:27:07: pid 19033: DETAIL: password authentication failed for user "nobody"
2017-12-03 19:27:07: pid 19033: DEBUG: find_primary_node: no primary node found
2017-12-03 19:27:08: pid 19033: LOG: find_primary_node: checking backend no 0
2017-12-03 19:27:08: pid 19033: DEBUG: SSL is requested but SSL support is not available
2017-12-03 19:34:27: pid 22132: ERROR: unable to read data from DB node 1
2017-12-03 19:34:27: pid 22132: DETAIL: EOF encountered with backend
2017-12-03 19:28:27: pid 19033: DEBUG: find_primary_node: no primary node found
The pool_hba.conf :
TYPE DATABASE USER CIDR-ADDRESS METHOD
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
In postgresql pg_hba.conf I enabled connection from pgpool server :
####pgpool####
host all all 172.22.13.170/32 trust
1.What can be the problem ?
2.If the repmgr is responsible for the replication should I set the parameter backend_flag to 'DISALLOW_TO_FAILOVER'?
Thanks.
I'm just getting up to speed on repmgr and pgpool, but I think there are multiple issues here:
1) Your pgpool.conf has some default settings for alive checking, and the user for that is 'nobody', so to get that to work you need to create a pgsql user with that name so that pgpool can query all hosts to find the current master.
2) pgpool executes scripts to change which is the master etc, and that script would normally just run repmgr commands to promote a new primary at failover, so I don't think DISALLOW_TO_FAILOVER is needed.
If repmgr would failover, then the 1 part of you question would make pgpool find which the new master is anyway, but in that case i would have repmgr configure to not failover automatically (since they could fight on who should do what.