Docker Compose with PostgreSQL and Prometheus PostgreSQL Exporter refusing to connect? - postgresql

Question

I'm trying to solve this question regarding the PostgreSQL exporter to Prometheus (https://github.com/wrouesnel/postgres_exporter), Prometheus PostgreSQL server exporter example not working on MacOS?, running postgres and postgres_exporter containers connected to a user-defined bridge network rather than the host network, which seems not to work on Docker Desktop for Mac. I've created the following docker-compose.yml:
version: "3"
services:
db:
image: postgres
environment:
POSTGRES_PASSWORD: mypassword
networks:
- mynetwork
exporter:
image: wrouesnel/postgres_exporter
environment:
DATA_SOURCE_NAME: "postgresql://postgres:mypassword#db:5432/postgres?sslmode=disable"
ports:
- "9187:9187"
networks:
- mynetwork
networks:
mynetwork:
I am having two issues with this, however. Firstly, the logs from the exporter service show an error with this message:
"Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused"
Here is the full output:
> docker-compose up
Creating network "postgres-performance-testing_mynetwork" with the default driver
Creating postgres-performance-testing_db_1 ... done
Creating postgres-performance-testing_exporter_1 ... done
Attaching to postgres-performance-testing_db_1, postgres-performance-testing_exporter_1
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Established new database connection to \"db:5432\"." source="postgres_exporter.go:778"
exporter_1 | time="2019-10-03T20:16:56Z" level=error msg="Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused" source="postgres_exporter.go:1348"
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1459"
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting default timezone ... Etc/UTC
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | waiting for server to start....2019-10-03 20:16:57.084 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.095 UTC [43] LOG: database system was shut down at 2019-10-03 20:16:56 UTC
db_1 | 2019-10-03 20:16:57.101 UTC [42] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down...2019-10-03 20:16:57.180 UTC [42] LOG: received fast shutdown request
db_1 | .2019-10-03 20:16:57.182 UTC [42] LOG: aborting any active transactions
db_1 | 2019-10-03 20:16:57.186 UTC [42] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
db_1 | 2019-10-03 20:16:57.186 UTC [44] LOG: shutting down
db_1 | 2019-10-03 20:16:57.197 UTC [42] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-10-03 20:16:57.299 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.310 UTC [51] LOG: database system was shut down at 2019-10-03 20:16:57 UTC
db_1 | 2019-10-03 20:16:57.314 UTC [1] LOG: database system is ready to accept connections
Is this because of the Prometheus exporter trying to connect to the database before it is ready to accept connections? (This would seem to be the case from the sequence of the logging).
Another problem is that if I browse to localhost:9187, the result is that I download a file containing HTML, rather than actually seeing a link to /metrics that I can follow:
Any idea how I can resolve these issues?

Answer 1

Without knowing how your images are built, I see two likely culprits that need to be addressed:
It seems that the exporter container is starting up first, and attempting to connect to the db container before postgres is even up and running. You may need to add depends_on: db in the exporter section.
You may need to edit pg_hba.conf on the db container before exporter can connect to it. By default, only local connections are allowed, so you'll have to append exporter's IP address (or use a /0 netmask, if this is a development-only environment)
If you fix these two, things should start working for you.
Disclosure: I am an EnterpriseDB (EDB) employee

Answer 2

postgres-exporter:
image: wrouesnel/postgres_exporter:v0.8.0
restart: always
environment:
#- DATA_SOURCE_NAME=postgresql://postgres:password#postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_URI=postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_USER=postgres
- DATA_SOURCE_PASS=password
ports:
- "9187:9187"
networks:
- postgres-prometheus
depends_on:
- postgres-db
Note in this example that the value for DATA_SOURCE_NAME has no double-quotes and that the value for DATA_SOURCE_URI (in case it's your preference) has no "postgresql://". It starts directly from the hostname.