Kubernetes Can't connect to database running on host machine - postgresql

Question

I have a problem. In my cluster I have a Ruby-On-Rails application which I want to map to the database on the hosted machine (Not containerized). It's a Postgres database, which listens to the following port when I run:
sudo netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1109/postgres
Then I created a service which maps the DB_HOST to the local machine like this:
apiVersion: v1
kind: Service
metadata:
name: external-postgres-svc
namespace: myapp-nm
spec:
ports:
- port: 5432
targetPort: 5432
protocol: TCP
I also added an endpoint:
apiVersion: v1
kind: Endpoints
metadata:
name: external-postgres-svc
namespace: myapp-nm
subsets:
- addresses:
- ip: "10.0.2.2"
ports:
- port: 5432
And in my configmap I have the following config:
apiVersion: v1
kind: ConfigMap
metadata:
name: myapp-config
namespace: myapp-nm
data:
db_host: "external-postgres-svc.myapp-nm.svc"
db_port: "5432"
db_username: "myuser"
db_password: "mypass"
But then when all resources are created and the migration job runs, it never completes. After like 2-3 minutes it crashes and gives the error:
connection to server at port 5432 failed: Operation timed out
I have added:
listen_addresses = '*'
to the /etc/postgresql/14/main/postgresql.conf, I added:
host all all 0.0.0.0/0 md5
to the /etc/postgresql/14/main/pg_hba.conf,
so I think it should listen to incoming traffic. I also ran
sudo ufw allow 5432/TCP
to allow the firewall port on my machine and I checked if the user was correct and it is, so what can be the problem?
I can connect to the database if I am not in the cluster using the
ip
port
username
password
What am I doing wrong?

Answer 1

Error: Operation Timed Out : indicates that your server failed to issue a complete response within the allowed time period.
Check below solutions :
1) Check If /etc directory owned by another user, /etc directory needs to be set as root user and also open 5432 port. (Add the hostname and IP details of the host in /etc/hosts. In the case of a multi-node cluster, the /etc/hosts on each machine has to be updated with the details of all cluster nodes).
2) Check again that there's no route to the database server because it's being blocked by a firewall. Make sure you set a rule in the firewall to allow the Ruby-On-Rails application to connect.
3) Check the setting if you are not doing the connection locally ONLY>, in the postgresql.conf file:
Connection Settings - #listen_addresses = 'localhost' >>>> This should be = '*' instead of localhost
Save the conf and restart the service.
4) Check Tunnel issue :
Verify that attributes of the ssh process match what customers provided: Look at the output of ps aux | grep ssh, the relevant part is:
-L number_1:string_or_number_1:number_2 ... KnownHostsFile=/dev/null string_or_number_2
*some_number1: Looker side port number
*string_or_number_1: Database host number or IP address
*number_2: Database side port number
*string_or_number_2: Tunnel server host or IP address
How to manually set the port:
Note : It is not recommended since it is still possible that the looker set the port back again,instead. Recommend you to open needed ports on the DB.
Create a tunnel through the Looker database connections UI
Update tunnel via the API(PATCH /api/4.0/connections/:connection_name) after it's been created
Set the desired local_host_port
Make sure db_connection and the following fields are set by using API GET /api/4.0/ssh_tunnel/:ssh_tunnel_id or check from go-ssh-sidecar:
*tunnel_id: id of the tunnel
*port: new local_host_port
*host: localhost
Also check for Improper Tunnel migration:
The following types of issues may be related to SSH Tunnels on newly migrated K8s
1.Tunnel is not migrated, or tunnels are partially migrated
2.Tunnel is migrated with incorrect information
3.IPs have changed, and are not documented or announced to customers
4.Public keys have changed (specific to migrated tunnels)