I used to be able to run docker-compose using 5432:5432 port mapping, but am recently getting this error:
Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
I've gone through all of the older posts relating to this issue but have yet to be able to solve this.
I've been driving myself mad with this error for the past week trying to figure out how to stop whatever process is running with no avail.
I've tried running
lsof -i tcp:5432
which returns no results (as long as my pgadmin/server is not running)
I've also seen other posts mention running
netstat -anp tcp | grep 5432
which does return two lines:
tcp4 0 0 *.5432 *.* LISTEN
tcp6 0 0 *.5432 *.* LISTEN
But I'm not entirely sure what to do with that output?
Run lsof -i tcp:5432 with sudo rights:
sudo lsof -i tcp:5432
A process (very likely PostgreSQL) is listening on port 5432, preventing another to listen on that port. Stop the process, e.g. with systemctl, then you will be able to start your docker PostgreSQL.
There is a possibility that this might be happening due to multiple instances of postgres running on your machine.
In my case, I had previously installed postgres which was always running on the machine because of which docker-based postgres was unable to start due to the above error. Uninstalling previously installed postgres helped to resolve the issue.
Related
I am using a Windows 10 Pro machine.
When I run netstat, it is showing kubernetes:port as foreign address in active connections.
What does this mean? I have checked and there is no kubernetes cluster running on my machine.
How do I close these connections?
Minikube status:
$ minikube status
host:
kubelet:
apiserver:
kubectl:
That happens because of the way netstat renders output. It has nothing to do with actual Kubernetes.
I have Docker Desktop for Windows and it adds this to the hosts file:
# Added by Docker Desktop
192.168.43.196 host.docker.internal
192.168.43.196 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
There is a record which maps 127.0.0.1 to kubernetes.docker.internal. When netstat renders its output, it resolves foreign address and it looks at the hosts file and sees this record. It says kubernetes and that is what you see in the console. You can try to change it to
127.0.0.1 tomato.docker.internal
With this, netstat will print:
Proto Local Address Foreign Address State
TCP 127.0.0.1:6940 tomato:6941 ESTABLISHED
TCP 127.0.0.1:6941 tomato:6940 ESTABLISHED
TCP 127.0.0.1:8080 tomato:40347 ESTABLISHED
TCP 127.0.0.1:8080 tomato:40348 ESTABLISHED
TCP 127.0.0.1:8080 tomato:40349 ESTABLISHED
So what actually happens is there are connections from localhost to localhost (netstat -b will show apps that create them). Nothing to do with Kubernetes.
It seems that Windows docker changed your hosts file.
So, if you want to get rid of these connections, just comment out the corresponding lines in the hosts file.
The hosts file on Windows 10 is located in C:\Windows\System32\drivers\etc and
the records may look something like 127.0.0.1 kubernetes.docker.internal.
I am pretty sure it will disrupt your docker service on Windows (yet, I am not an expert), so don't forget to uncomment these lines whenever you need to get the docker service back.
OK, it looks like your minikube instance is definitely deleted. Keep in mind that in Linux or other nix-based systems it is totally normal that many processes use network sockets to communicate between each other e.g. you will see many established connections with both local and foreign addresses set to localhost:
tcp 0 0 localhost:45402 localhost:2379 ESTABLISHED
tcp 0 0 localhost:45324 localhost:2379 ESTABLISHED
tcp 0 0 localhost:2379 localhost:45300 ESTABLISHED
tcp 0 0 localhost:45414 localhost:2379 ESTABLISHED
tcp 0 0 localhost:2379 localhost:45388 ESTABLISHED
tcp 0 0 localhost:40600 localhost:8443 ESTABLISHED
kubernetes in your case is nothing more than hostname of one of your machines/VMs/instances. Maybe the one on top of which you ran your minikube you called kubernetes and that's why this hostname appears currently in your active network connections. Basically it has nothing to do with running kubernetes cluster.
To make it clearer you may cat the content of your /etc/hosts file and look for the entry kubernetes. Then you can compare them with your network interfaces addresses (run ip -4 a). Most probably kubernetes entry in /etc/hosts is mapped to one of them.
Let me know if it clarifies your doubts.
EDIT:
I've reproduced it on Minikube on my linux instance and noticed exactly the same behaviour, but it looks like the ESTABLISHED connections are showing only after successfull minikube stop. After minikube delete they're gone. It looks like those connections indeed belong to various components of kubernetes, but for some reason are not terminated. Basically closing established network connections is responsibility of the application which creates them and it looks like for some reason minikube is not terminating them.
If you run:
sudo netstat -ntp ### important: it must be run as superuser
it shows additionally PID/Program name column in which you can see by which program specific connection was established. You will see a lot of ESTABLISHED network connections belonging to etcd and kube-apiserver.
First I try to reboot the whole instance. It obviously close all the connections but then I verified a few times and it looks like successfully performed minikube delete also closes all connections.
Additionally you may want to check available docker containers by running:
docker ps
or:
docker container ls
After stopping the minikube instance it still shows those containers and it looks like the reason why a lot of connections with certain kubernetes components** are still shown by netstat command.
However after minikube delete neither containers nor ESTABLISHED connections with kubernetes cluster components are available any more.
EDIT 2: After a long time, solved! See answer below.
EDIT: I am sorry to say that the the problems went away "on their own" between yesterday and today, without me having done anything. Great non-deterministic lesson to learn here...
Bonus fun: the "could not bind IPv6 socket" error still appears in the error logs, so this was probably not even the problem in the first place.
I have a problem with a previously functioning docker PGSQL image. Until an uninspired rebuild yesterday ( :-D ), I've used this build successfully for the last 5+ months.
My system:
Ubuntu 17.04 64b
PGSQL 9.6.4
Docker version 17.11.0-ce, build 1caf76c
I am mapping host port 5433 to container port 5432
The problem (snippet from the PGSQL logs):
...
LOG: could not bind IPv6 socket: Cannot assign requested address
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
...
This is what ss is telling me (when the container is running, obviously):
$ docker exec -it db ss -atune
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp ESTAB 0 0 127.0.0.1:45876 127.0.0.1:45876 uid:999 ino:147509 sk:00000001 <->
tcp LISTEN 0 128 127.0.0.1:5432 *:* uid:999 ino:147500 sk:00000002 <->
I've cleaned all docker containers / images, I've reinstalled docker, nothing helped. Who can possibly be using the 5432 port in the container?
For that matter, am I reading this correctly, that PGSQL is complaining about the 5432 port being already used in the docker container?
Even if you have no solution, a basic idea of how to proceed with debugging this would be a great help.
EDIT:
postgres.docker file
FROM postgres:9.6.4
ADD bin/postgres-setup.sh /docker-entrypoint-initdb.d/postgres-setup.sh
RUN chmod 755 /docker-entrypoint-initdb.d/postgres-setup.sh && \
apt-get update && \
apt-get install -y --no-install-recommends postgresql-plpython3-9.6 python3-pip postgresql-9.6-pldebugger && \
pip3 install pyexcel pyexcel-xls pyexcel-xlsx pyexcel-xlsxw
After a long time, we finally figured out what was the problem -- adding explanation here, in case it helps others.
Since pgsql listens only to localhost by default, when it is running in the docker container, where we have the port mapping configuration, the external API was not able to connect to the pgsql server.
The solution is to allow pgsql to listen to all IP addresses:
Connect to db container shell: $ docker exec -ti db bash
Change the configuration file /var/lib/postgresql/data/postgresql.conf to allow pgsql to listen to all IPs: listen_addresses = '*'
Certain edits have to be made to the postgres.conf and pg_hba.conf files in order for Postgres inside the container to listen to connections from the host:
See the Gotchas section at the cityseer/postgis repo.
Check that your postgresql.conf file has the listen_addresses item uncommented and set to listen to all ports, i.e. listen_addresses = '*';
Check that your pg_hba.conf file allows the docker container's Postgres to provide local and host access per the following two lines:
local all all trust
host all all 0.0.0.0/0 trust
I've been struggling to fix this solution. I've followed multiple guidelines none of them seem to work for me..
So I open PostGres app from my applications folder
It throws following error in the log
FATAL: could not open lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
LOG: database system is shut down
I tried removing the following folder however it says the PGSQL.5432.lock file does not exist. I've done my research and none of them seem to be working for me!
I also tried running psql
psql: could not connect to server: Permission denied
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Threw me that error.
In psql if you didn't specify the user and run psql all alone, psql will take the current user of your system like :
[root#bdd ~]# psql
psql will take root as user.
So you need to tell psql which user to log with :
[root#bdd ~]# psql -U postgres
as you can see postgres is the default user if you didn't change it during installation.
If the problem maintain you need to find which port postgresql is using (you might change it during installation )
run :
[root#bdd ~]# netstat -tulpn | grep postgres
you will find something like this :
tcp 0 0 0.0.0.0:5533 0.0.0.0:* LISTEN 20856/postgres
tcp6 0 0 :::5533 :::* LISTEN 20856/postgres
In my case the port is 5533 so i need to do this :
[root#bdd ~]# psql -U postgres -p 5533
I recently installed postgresql on my ubuntu 14.04 vm, but I can't seem to be able to connect to my database from localhost: postgres://127.0.0.1:5432/mydb?sslmode=disable is not found, although psql mydb brings me to the correct terminal, sudo netstat -plunt |grep postgres returns nothing, but su postgres -c "netstat -plunt |grep postgres" prints:
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 31/postgres
tcp6 0 0 ::1:5432 :::* LISTEN 31/postgres
So it's clearly there, just somehow under the postgres ubuntu user (but with the database belonging to root). I know that there is a way to reconfigure my postgresql installation such that it can be reached through localhost (postgres://127.0.0.1:5432/mydb?sslmode=disable works), and if anyone could tell me what I need to do, that would be awesome. Thank you!
Please read before replying it as duplicate (as it perhaps can happen). I am running my postmaster (postgres) server. See below for 'sudo netstat -anp|grep 5432' output?
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 29606/postmaster
unix 2 [ ACC ] STREAM LISTENING 1650581 29606/postmaster /var/run/postgresql/.s.PGSQL.5432
unix 2 [ ACC ] STREAM LISTENING 1650582 29606/postmaster /tmp/.s.PGSQL.5432
I am able to connect from localhost using
psql -h localhost (OR 127.0.0.1) -d <DB> -U user -W
But when I try to connect from other hosts using tcp, by specifying
psql -h ip_add_postmaster -d <DB> -U user -W
It throws:
psql: could not connect to server: Connection refused
Is the server running on host XXXXXX and accepting TCP/IP connections on port 5432?
What's wrong here?
pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all peer
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
In postgresql.conf,
listen_addresses = 'localhost, 127.0.0.1, ip_add_postmaster'
Note: ip_add_postmaster is same as my Elastic IP and not public DNS. If this information
matters.
What am I doing wrong here? Machine is hosted on Amazon EC2 and have open the port 5432.
As your netstat output indicates, it's listening at 127.0.0.1:5432 which is localhost. That is only connectable from localhost ;)
Set listen_addresses='*' in your config and it will work.
[edit]
Other things to check:
is the amazon firewall blocking anything?
is iptables blocking anything?
But first make sure the listening address is correct, your netstat output shows that it won't work like this.
listen_addresses='localhost, private_ip' fixed the issue. I was not able to start postmaster server on elastic IPs. Once postgres server started o localhost and private IPs, I was able to connect.
One other issue I have found was if you end up with two Postgres installations, the second one can choose non-default port (in my case it was 5433 i/o 5432). So checking the port in postgresql.conf might be a good idea.
I ran into this issue and tried all sorts of fixes I found across SO, and want to add a simple solution that worked for me after realizing it had to do with permissions in my case.
Simply, if you're running a psql server on Windows, you are initially restricted to the default postgres superuser for logging in, launching the server, and so on.
So, first try running from the command line:
psql -U postgres -h localhost -p 5432
and enter your password at the prompt. If you've managed to login and the server is up, then it was a permissions issues. From here, you can create a role for yourself that has login privileges to whatever database you are trying to run.
If the error persists, then consider checking postgresql.conf as mentioned above, to make sure default IP is set to * or localhost, and the port set to 5432 or whatever port you want as default.
I also ran into the same issue. On debugging, it was nothing related to the port, but due to some missing directories in the Postgres folder.
While updating Mac OS (from 10.13.1 -> 10.13.13), some folders in the directory /usr/local/var/postgres/ gets deleted. The fix was the adding the missing directories:
mkdir /usr/local/var/postgres/pg_tblspc
mkdir /usr/local/var/postgres/pg_twophase
mkdir /usr/local/var/postgres/pg_stat
mkdir /usr/local/var/postgres/pg_stat_tmp
mkdir /usr/local/var/postgres/pg_replslot
mkdir /usr/local/var/postgres/pg_snapshots
mkdir /usr/local/var/postgres/pg_logical/{snapshots,mappings}