Can't connect to postgres in docker with port mapping using sequelize - postgresql

I am running a postgres image in a docker container and am trying to connect to it from another container using docker-compose. When I use the standard port of 5432 I am able to connect fine, but when I try to use a nonstandard port along with a port mapping I am getting a ECONNREFUSED error.
Here is my compose file:
networks:
production-net:
driver: bridge
services:
conn-test:
depends_on:
- db
environment:
DB_DIALECT: postgres
DB_HOST: db
DB_NAME: db-name
DB_PASSWORD: pass
DB_PORT: '54321'
DB_USER: user
image: my-image
networks:
production-net: null
db:
environment:
PGDATA: /pgdata
POSTGRES_DB: db-name
POSTGRES_PASSWORD: pass
POSTGRES_USER: user
image: postgres
networks:
production-net: null
ports:
- 54321:5432/tcp
version: '3.0'
And here is the output when I run docker-compose up
Creating network "composer_production-net" with driver "bridge"
Creating composer_db_1 ... done
Creating composer_conn-test_1 ... done
Attaching to composer_db_1, composer_conn-test_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /pgdata ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
conn-test_1 | Connecting to database
db_1 | syncing data to disk ...
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /pgdata -l logfile start
db_1 |
db_1 | waiting for server to start....2019-02-28 20:52:02.208 UTC [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-02-28 20:52:02.534 UTC [42] LOG: database system was shut down at 2019-02-28 20:51:58 UTC
db_1 | 2019-02-28 20:52:02.618 UTC [41] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down....2019-02-28 20:52:04.355 UTC [41] LOG: received fast shutdown request
db_1 | 2019-02-28 20:52:04.432 UTC [41] LOG: aborting any active transactions
db_1 | 2019-02-28 20:52:04.436 UTC [41] LOG: background worker "logical replication launcher" (PID 48) exited with exit code 1
db_1 | 2019-02-28 20:52:04.436 UTC [43] LOG: shutting down
db_1 | 2019-02-28 20:52:04.853 UTC [41] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2019-02-28 20:52:04.941 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-02-28 20:52:04.941 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-02-28 20:52:05.091 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-02-28 20:52:05.342 UTC [59] LOG: database system was shut down at 2019-02-28 20:52:04 UTC
db_1 | 2019-02-28 20:52:05.419 UTC [1] LOG: database system is ready to accept connections
conn-test_1 | Error during start up process
conn-test_1 | connect ECONNREFUSED 172.26.0.2:54321
conn-test_1 | { SequelizeConnectionRefusedError: connect ECONNREFUSED 172.26.0.2:54321
conn-test_1 | at connection.connect.err (/app/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:116:24)
conn-test_1 | at Connection.connectingErrorHandler (/app/node_modules/pg/lib/client.js:140:14)
conn-test_1 | at emitOne (events.js:116:13)
conn-test_1 | at Connection.emit (events.js:211:7)
conn-test_1 | at Socket.reportStreamError (/app/node_modules/pg/lib/connection.js:71:10)
conn-test_1 | at emitOne (events.js:116:13)
conn-test_1 | at Socket.emit (events.js:211:7)
conn-test_1 | at emitErrorNT (internal/streams/destroy.js:66:8)
conn-test_1 | at _combinedTickCallback (internal/process/next_tick.js:139:11)
conn-test_1 | at process._tickCallback (internal/process/next_tick.js:181:9)
conn-test_1 | name: 'SequelizeConnectionRefusedError',
conn-test_1 | parent:
conn-test_1 | { Error: connect ECONNREFUSED 172.26.0.2:54321
conn-test_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
conn-test_1 | errno: 'ECONNREFUSED',
conn-test_1 | code: 'ECONNREFUSED',
conn-test_1 | syscall: 'connect',
conn-test_1 | address: '172.26.0.2',
conn-test_1 | port: 54321 },
conn-test_1 | original:
conn-test_1 | { Error: connect ECONNREFUSED 172.26.0.2:54321
conn-test_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
conn-test_1 | errno: 'ECONNREFUSED',
conn-test_1 | code: 'ECONNREFUSED',
conn-test_1 | syscall: 'connect',
conn-test_1 | address: '172.26.0.2',
conn-test_1 | port: 54321 } }
conn-test_1 | Shutting down the application ...
I've confirmed that the docker host is resolving to the correct IP address, but it won't connect.
Can anyone point out what I am missing here?

When you're communicating directly between containers, you use the port number the service inside the target container is listening on. If you have a ports: declaration or a docker run -p option, it's the second port number; but in this setup, unless you want to access the service from outside of Docker space, that setting is strictly optional.
So you should set DB_PORT: '5432' to point at the "normal" PostgreSQL port, even though you've published it to the host on a different port number.

Related

Docker Compose + TypeORM: No Connection

I know there are plenty of questions around this topic, but I can't make it work for my (fairly straightforward setup). What am I missing? I guess we can neglect pgadmin out of the equation, but I kept in there for full disclosure.
docker-compose.yml
version: '3.8'
services:
postgres:
image: postgres:14.2-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
POSTGRES_DB: mydatabase
PGDATA: /data/mydatabase
volumes:
- pgstore:/var/lib/postgresql/data
ports:
- '5432:5432'
networks:
- postgres
restart: unless-stopped
pgadmin:
image: dpage/pgadmin4:6.5
environment:
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL:-admin#admin.com}
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD:-admin}
volumes:
- pgadmin:/root/.pgadmin
ports:
- '5050:80'
networks:
- postgres
depends_on:
- postgres
restart: unless-stopped
networks:
postgres:
driver: bridge
volumes:
pgstore:
pgadmin:
Starting it with docker-compose up yields:
Creating network "my-app_postgres" with driver "bridge"
Creating my-app_postgres_1 ... done
Creating my-app_pgadmin_1 ... done
Attaching to my-app_postgres_1, my-app_pgadmin_1
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /data/mydatabase ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default time zone ... UTC
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... sh: locale: not found
postgres_1 | 2022-09-06 14:03:48.965 UTC [32] WARNING: no usable system locales were found
postgres_1 | ok
postgres_1 | syncing data to disk ... initdb: warning: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | ok
postgres_1 |
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /data/mydatabase -l logfile start
postgres_1 |
postgres_1 | waiting for server to start....2022-09-06 14:03:49.890 UTC [38] LOG: starting PostgreSQL 14.2 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
postgres_1 | 2022-09-06 14:03:49.891 UTC [38] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-09-06 14:03:49.894 UTC [39] LOG: database system was shut down at 2022-09-06 14:03:49 UTC
postgres_1 | 2022-09-06 14:03:49.896 UTC [38] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | waiting for server to shut down....2022-09-06 14:03:50.195 UTC [38] LOG: received fast shutdown request
postgres_1 | 2022-09-06 14:03:50.196 UTC [38] LOG: aborting any active transactions
postgres_1 | 2022-09-06 14:03:50.197 UTC [38] LOG: background worker "logical replication launcher" (PID 45) exited with exit code 1
postgres_1 | 2022-09-06 14:03:50.197 UTC [40] LOG: shutting down
postgres_1 | 2022-09-06 14:03:50.204 UTC [38] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2022-09-06 14:03:50.307 UTC [1] LOG: starting PostgreSQL 14.2 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
postgres_1 | 2022-09-06 14:03:50.308 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2022-09-06 14:03:50.308 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2022-09-06 14:03:50.310 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-09-06 14:03:50.313 UTC [52] LOG: database system was shut down at 2022-09-06 14:03:50 UTC
postgres_1 | 2022-09-06 14:03:50.315 UTC [1] LOG: database system is ready to accept connections
pgadmin_1 | NOTE: Configuring authentication for SERVER mode.
pgadmin_1 |
pgadmin_1 | [2022-09-06 14:03:56 +0000] [1] [INFO] Starting gunicorn 20.1.0
pgadmin_1 | [2022-09-06 14:03:56 +0000] [1] [INFO] Listening at: http://[::]:80 (1)
pgadmin_1 | [2022-09-06 14:03:56 +0000] [1] [INFO] Using worker: gthread
pgadmin_1 | [2022-09-06 14:03:56 +0000] [89] [INFO] Booting worker with pid: 89
In my application, I initialize TypeORM with the following options:
const options = {
default: {
type: process.env.DATABASE_TYPE,
host: process.env.DATABASE_HOST,
port: process.env.DATABASE_PORT,
username: process.env.DATABASE_USERNAME,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABASE_NAME,
synchronize: true,
logging: false,
},
};
Whereas my .env holds the following information:
DATABASE_TYPE=postgres
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USERNAME=postgres
DATABASE_PASSWORD=postgres
DATABASE_NAME=mydatabase
I can access the container on the command line with:
docker exec -it my-app psql -U postgres mydatabase
and list the \l the databases and see mydatabase. But I get Did not find any relations. for \d because the application does not connect to the database and therefore there are no relations yet.
More information:
the application is running outside of the container
the application is not throwing an error, it just tries to connect indefinitely
I am using Docker container on MacOS
What am I missing here? Any help appreciated!

Docker postgres 9.4 password authentication failed for user "postgres"

I am running very basic docker-compose with following docker-compose.yml and getting following error.
docker-compose.yml
services:
redis:
image: redis
db:
image: postgres:9.4
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
vote:
image: voting-app
ports:
- 5000:80
depends_on:
- redis
worker:
image: worker-app
depends_on:
- redis
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
result:
image: result-app
ports:
- 5001:80
depends_on:
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
Error generated,
db_1 | vacuuming database template1 ... ok
db_1 | copying template1 to template0 ... ok
db_1 | copying template1 to postgres ... ok
db_1 | syncing data to disk ...
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | postgres -D /var/lib/postgresql/data
db_1 | or
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2021-03-03 12:45:13 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | waiting for server to shut down....LOG: shutting down
db_1 | LOG: database system is shut down
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2021-03-03 12:45:15 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
I checked on various stackoverflow existing questions but none of them are rectifying actual issue. Some suggesting to run postgres docker and bash it to update the password but its a workaround. Anyone has any concrete solution to this?
I figured it out finally and would be good to share what mistake I had done.
I changed the database configuration by providing environment variable in the docker-compose.yml file in db service as mentioned in the question already,
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
But the same needs to be updated in all the other custom services (open source github) which are trying to connect to this depende-on postgres database with same user/password/database which was not done.
However the error given by docker-compose is not explaining the situation clearly.
In All, I updated the applications with above username/password/database in connection property and the containers are up.

App couldn't connect to Postgresql Database with docker-compose

I'm trying to get my web application running using docker-compose.
My Dockerfile
FROM golang:1.15
WORKDIR /goprojects/vnlist
COPY . .
RUN go get github.com/revel/revel
RUN go get github.com/revel/cmd/revel
RUN go get -d -v ./...
EXPOSE 9000
CMD [ "revel", "run", "-a", "vnlist" ]
My docker-compose.yml
version: '3.3'
services:
web:
build: .
ports:
- "9000:9000"
db:
image: "postgres"
environment:
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "postgresql123"
POSTGRES_DB: "vnlist_db"
volumes:
- db-data:/foo
volumes:
db-data:
The DBsetup file
package models
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
var DB *gorm.DB
var (
DB_HOST = "localhost"
DB_PORT = "5432"
DB_USER = "postgres"
DB_PASSWORD = "postgresql123"
DB_NAME = "vnlist_db"
DB_TIMEZONE = "Asia/Jakarta"
)
func InitDB() {
dsn := fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=disable",
DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME)
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
fmt.Println("DB Error: ", err)
}
db.AutoMigrate(&VN{}, &User{})
DB = db
}
And then, I ran docker-compose up --build. Here's the result of the command:
Successfully built 1239cf1fceea
Successfully tagged vnlist_web:latest
Creating vnlist_db_1 ... done
Creating vnlist_web_1 ... done
Attaching to vnlist_db_1, vnlist_web_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting default timezone ... Etc/UTC
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
web_1 | Revel executing: run a Revel application
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | waiting for server to start....2021-01-18 14:25:35.070 UTC [47] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2021-01-18 14:25:35.437 UTC [48] LOG: database system was shut down at 2021-01-18 14:25:30 UTC
db_1 | 2021-01-18 14:25:35.530 UTC [47] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
web_1 | WARN 14:25:35 harness.go:175: No http.addr specified in the app.conf listening on localhost interface only. This will not allow external access
to your application
web_1 | Changed detected, recompiling
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down....2021-01-18 14:25:39.271 UTC [47] LOG: received fast shutdown request
db_1 | 2021-01-18 14:25:39.420 UTC [47] LOG: aborting any active transactions
db_1 | 2021-01-18 14:25:39.424 UTC [47] LOG: background worker "logical replication launcher" (PID 54) exited with exit code 1
db_1 | 2021-01-18 14:25:39.424 UTC [49] LOG: shutting down
db_1 | .2021-01-18 14:25:41.059 UTC [47] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2021-01-18 14:25:41.178 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2021-01-18 14:25:41.178 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2021-01-18 14:25:41.445 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2021-01-18 14:25:41.906 UTC [75] LOG: database system was shut down at 2021-01-18 14:25:40 UTC
db_1 | 2021-01-18 14:25:42.031 UTC [1] LOG: database system is ready to accept connections
web_1 | Parsing packages, (may require download if not cached)... Completed
web_1 | INFO 14:27:12 app run.go:35: Running revel server
web_1 | INFO 14:27:12 app plugin.go:9: Go to /#tests to run the tests.
web_1 |
web_1 | 2021/01/18 14:27:12 /goprojects/vnlist/app/models/DbSetup.go:24
web_1 | [error] failed to initialize database, got error failed to connect to `host=localhost user=postgres database=vnlist_db`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)
web_1 | DB Error: failed to connect to `host=localhost user=postgres database=vnlist_db`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)
web_1 |
web_1 | 2021/01/18 14:27:12 /root/go/pkg/mod/gorm.io/driver/postgres#v1.0.6/migrator.go:157 failed to connect to `host=localhost user=postgres database=vnlist_db`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)
web_1 | [0.659ms] [rows:0] CREATE TABLE "vns" ("id" bigserial,"created_at" timestamptz,"updated_at" timestamptz,"deleted_at" timestamptz,"title" text,"genre" text,"original_language" text,"translated" boolean,PRIMARY KEY ("id"))
web_1 |
web_1 | 2021/01/18 14:27:12 /root/go/pkg/mod/gorm.io/driver/postgres#v1.0.6/migrator.go:130 failed to connect to `host=localhost user=postgres database=vnlist_db`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)
web_1 | [1.093ms] [rows:0] CREATE INDEX "idx_vns_deleted_at" ON "vns" ("deleted_at")
web_1 | Revel engine is listening on.. localhost:35495
web_1 |
web_1 | Time to recompile 1m36.9527518s
web_1 | Revel proxy is listening, point your browser to : 9000
As we can see on the last few lines, the database failed to connect.
I've tried to change the DB's credential but the problem persists.
My question is, how to fix this?
Your DB_HOST would be the name of the database service DB_HOST = "db" instead of "localhost"

Docker Compose with PostgreSQL and Prometheus PostgreSQL Exporter refusing to connect?

I'm trying to solve this question regarding the PostgreSQL exporter to Prometheus (https://github.com/wrouesnel/postgres_exporter), Prometheus PostgreSQL server exporter example not working on MacOS?, running postgres and postgres_exporter containers connected to a user-defined bridge network rather than the host network, which seems not to work on Docker Desktop for Mac. I've created the following docker-compose.yml:
version: "3"
services:
db:
image: postgres
environment:
POSTGRES_PASSWORD: mypassword
networks:
- mynetwork
exporter:
image: wrouesnel/postgres_exporter
environment:
DATA_SOURCE_NAME: "postgresql://postgres:mypassword#db:5432/postgres?sslmode=disable"
ports:
- "9187:9187"
networks:
- mynetwork
networks:
mynetwork:
I am having two issues with this, however. Firstly, the logs from the exporter service show an error with this message:
"Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused"
Here is the full output:
> docker-compose up
Creating network "postgres-performance-testing_mynetwork" with the default driver
Creating postgres-performance-testing_db_1 ... done
Creating postgres-performance-testing_exporter_1 ... done
Attaching to postgres-performance-testing_db_1, postgres-performance-testing_exporter_1
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Established new database connection to \"db:5432\"." source="postgres_exporter.go:778"
exporter_1 | time="2019-10-03T20:16:56Z" level=error msg="Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused" source="postgres_exporter.go:1348"
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1459"
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting default timezone ... Etc/UTC
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | waiting for server to start....2019-10-03 20:16:57.084 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.095 UTC [43] LOG: database system was shut down at 2019-10-03 20:16:56 UTC
db_1 | 2019-10-03 20:16:57.101 UTC [42] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down...2019-10-03 20:16:57.180 UTC [42] LOG: received fast shutdown request
db_1 | .2019-10-03 20:16:57.182 UTC [42] LOG: aborting any active transactions
db_1 | 2019-10-03 20:16:57.186 UTC [42] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
db_1 | 2019-10-03 20:16:57.186 UTC [44] LOG: shutting down
db_1 | 2019-10-03 20:16:57.197 UTC [42] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-10-03 20:16:57.299 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.310 UTC [51] LOG: database system was shut down at 2019-10-03 20:16:57 UTC
db_1 | 2019-10-03 20:16:57.314 UTC [1] LOG: database system is ready to accept connections
Is this because of the Prometheus exporter trying to connect to the database before it is ready to accept connections? (This would seem to be the case from the sequence of the logging).
Another problem is that if I browse to localhost:9187, the result is that I download a file containing HTML, rather than actually seeing a link to /metrics that I can follow:
Any idea how I can resolve these issues?
Without knowing how your images are built, I see two likely culprits that need to be addressed:
It seems that the exporter container is starting up first, and attempting to connect to the db container before postgres is even up and running. You may need to add depends_on: db in the exporter section.
You may need to edit pg_hba.conf on the db container before exporter can connect to it. By default, only local connections are allowed, so you'll have to append exporter's IP address (or use a /0 netmask, if this is a development-only environment)
If you fix these two, things should start working for you.
Disclosure: I am an EnterpriseDB (EDB) employee
postgres-exporter:
image: wrouesnel/postgres_exporter:v0.8.0
restart: always
environment:
#- DATA_SOURCE_NAME=postgresql://postgres:password#postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_URI=postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_USER=postgres
- DATA_SOURCE_PASS=password
ports:
- "9187:9187"
networks:
- postgres-prometheus
depends_on:
- postgres-db
Note in this example that the value for DATA_SOURCE_NAME has no double-quotes and that the value for DATA_SOURCE_URI (in case it's your preference) has no "postgresql://". It starts directly from the hostname.

Troubles with connect to database in Docker Compose

In one folder I have 4 files: base.py, Dockerfile, docker-compose.yml and wait-for-it.sh.
base.py:
import psycopg2
print("JJJJJJJJ")
conn = psycopg2.connect("dbname='base123' user='postgres' host='db' password='pw1234'")
cur = conn.cursor()
cur.execute("CREATE TABLE test(id serial PRIMARY KEY, num int);")
Dockerfile:
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -y install python-pip
RUN apt-get update
RUN pip install --upgrade pip
RUN pip install psycopg2-binary
COPY base.py base.py
COPY wait-for-it.sh wait-for-it.sh
RUN chmod +x /wait-for-it.sh
CMD ["python","base.py"]
docker-compose.yml
version: '3'
services:
db:
image: 'postgres:latest'
expose:
- "5432"
ports:
- "5559:5432"
environment:
POSTGRES_PASSWORD: pw1234
POSTGRES_DB: base123
aprrka:
build: .
depends_on:
- db
command: ["./wait-for-it.sh", "db:5432", "--", "python", "base.py"]
wait-fot-it.sh:
link to code in github
After docker-compose up I have this OUTPUT:
...
Creating postgres_db_1 ... done
Creating postgres_aprrka_1 ... done
Attaching to postgres_db_1, postgres_aprrka_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
aprrka_1 | wait-for-it.sh: waiting 15 seconds for db:5432
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....2018-08-10 09:59:13.529 UTC [38] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:13.582 UTC [39] LOG: database system was shut down at 2018-08-10 09:59:12 UTC
db_1 | 2018-08-10 09:59:13.599 UTC [38] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | 2018-08-10 09:59:14.585 UTC [38] LOG: received fast shutdown request
db_1 | waiting for server to shut down....2018-08-10 09:59:14.593 UTC [38] LOG: aborting any active transactions
db_1 | 2018-08-10 09:59:14.613 UTC [38] LOG: worker process: logical replication launcher (PID 45) exited with exit code 1
db_1 | 2018-08-10 09:59:14.613 UTC [40] LOG: shutting down
db_1 | 2018-08-10 09:59:14.660 UTC [38] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2018-08-10 09:59:14.729 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:14.806 UTC [65] LOG: database system was shut down at 2018-08-10 09:59:14 UTC
db_1 | 2018-08-10 09:59:14.829 UTC [1] LOG: database system is ready to accept connections
db_1 | 2018-08-10 09:59:15.217 UTC [72] LOG: incomplete startup packet
aprrka_1 | wait-for-it.sh: db:5432 is available after 5 seconds
aprrka_1 | JJJJJJJJ
postgres_aprrka_1 exited with code 0
When I type command: psql -U postgres -h localhost -p 5559 in another terminal I can connect to base1234, but when type `\dt' i have no any relations, so my py file don't create table in postgres database. I think that no connection exist with postgres db. Please, help me
You need to close the cursor and commit all pending transactions to the database for the changes to be effective.
Add the following 2 lines at the end of your base.py:
cur.close()
conn.commit()