Docker postgres 9.4 password authentication failed for user "postgres" - postgresql

I am running very basic docker-compose with following docker-compose.yml and getting following error.
docker-compose.yml
services:
redis:
image: redis
db:
image: postgres:9.4
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
vote:
image: voting-app
ports:
- 5000:80
depends_on:
- redis
worker:
image: worker-app
depends_on:
- redis
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
result:
image: result-app
ports:
- 5001:80
depends_on:
- db
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
Error generated,
db_1 | vacuuming database template1 ... ok
db_1 | copying template1 to template0 ... ok
db_1 | copying template1 to postgres ... ok
db_1 | syncing data to disk ...
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | postgres -D /var/lib/postgresql/data
db_1 | or
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2021-03-03 12:45:13 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | waiting for server to shut down....LOG: shutting down
db_1 | LOG: database system is shut down
worker_1 | Waiting for db
result_1 | Waiting for db
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2021-03-03 12:45:15 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
db_1 | FATAL: password authentication failed for user "postgres"
db_1 | DETAIL: Connection matched pg_hba.conf line 95: "host all all all md5"
I checked on various stackoverflow existing questions but none of them are rectifying actual issue. Some suggesting to run postgres docker and bash it to update the password but its a workaround. Anyone has any concrete solution to this?

I figured it out finally and would be good to share what mistake I had done.
I changed the database configuration by providing environment variable in the docker-compose.yml file in db service as mentioned in the question already,
environment:
- POSTGRES_USER=udcoker
- POSTGRES_PASSWORD=udcoker
- POSTGRES_DB=docker
But the same needs to be updated in all the other custom services (open source github) which are trying to connect to this depende-on postgres database with same user/password/database which was not done.
However the error given by docker-compose is not explaining the situation clearly.
In All, I updated the applications with above username/password/database in connection property and the containers are up.

Related

Connection refused: Is the server running on host "db" (172.21.0.2) and accepting web_1 TCP/IP connections on port 5432?

I know that it is a typical issue, however I need a community help to resolve it. When I run docker-compose I get Connection refused: Is the server running on host "db" (172.21.0.2) and accepting web_1 TCP/IP connections on port 5432? Web portion of the docker-compose fails, but db runs. I can connect to the db using localhost:5432. It is not a wait issue, because tiangolo/uvicorn-gunicorn:python3.8-alpine3.10 implemented a wait mechanism - dockerize. Does anyone know where the issue is? Or perhaps could anyone just point me the right direction?
So my Dockerfile is
FROM tiangolo/uvicorn-gunicorn:python3.8-alpine3.10
# copy requirements file
COPY ./requirements.txt /usr/src/app/requirements.txt
# install dependencies
RUN set -eux \
&& apk add --no-cache --virtual .build-deps build-base \
libressl-dev libffi-dev gcc musl-dev python3-dev \
postgresql-dev openssl \
&& pip install --upgrade pip setuptools wheel \
&& pip install -r /usr/src/app/requirements.txt \
&& rm -rf /root/.cache/pip
# copy project
COPY . /app
My docker-compose is:
version: '3.7'
services:
db:
image: postgres:13-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
ports:
- 5432:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=api
web:
build: ./src
ports:
- 80:80
depends_on:
- db
environment:
- DATABASE_URL=postgresql://postgres:postgres#db/api
volumes:
postgres_data:
I added a wait using dockerize, however the web portion fails:
Attaching to gunicorn_db_1, gunicorn_web_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
web_1 | 2020/08/30 15:33:51 Waiting for: tcp://db:5432
web_1 | 2020/08/30 15:33:51 Problem with dial: dial tcp 172.21.0.2:5432: connect: connection refused. Sleeping 30s
db_1 | selecting default time zone ... Etc/UTC
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... initdb: warning: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ok
db_1 |
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....2020-08-30 15:33:53.037 UTC [45] LOG: starting PostgreSQL 13beta3 (Debian 13~beta3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1 | 2020-08-30 15:33:53.044 UTC [45] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2020-08-30 15:33:53.059 UTC [46] LOG: database system was shut down at 2020-08-30 15:33:52 UTC
db_1 | 2020-08-30 15:33:53.066 UTC [45] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | 2020-08-30 15:33:53.399 UTC [45] LOG: received fast shutdown request
db_1 | waiting for server to shut down....2020-08-30 15:33:53.404 UTC [45] LOG: aborting any active transactions
db_1 | 2020-08-30 15:33:53.407 UTC [45] LOG: background worker "logical replication launcher" (PID 52) exited with exit code 1
db_1 | 2020-08-30 15:33:53.409 UTC [47] LOG: shutting down
db_1 | 2020-08-30 15:33:53.438 UTC [45] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2020-08-30 15:33:53.530 UTC [1] LOG: starting PostgreSQL 13beta3 (Debian 13~beta3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1 | 2020-08-30 15:33:53.531 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2020-08-30 15:33:53.531 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2020-08-30 15:33:53.539 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2020-08-30 15:33:53.549 UTC [63] LOG: database system was shut down at 2020-08-30 15:33:53 UTC
db_1 | 2020-08-30 15:33:53.559 UTC [1] LOG: database system is ready to accept connections
web_1 | 2020/08/30 15:34:21 Connected to tcp://db:5432
gunicorn_web_1 exited with code 0
In first look your both files seemed good to me, actually adding a depends on should fix this but, according to documentation that doesn't works that way. it express dependency between containers but that does not mean that a container will wait to other to be ready.
For older versions of Compose we were able to add a health check like this
healthcheck:
test: ["-U postgres"]
interval: 3s
timeout: 30s
retries: 1
But since it's not supported anymore, best option to add a manual wait/sleep to that service.
There is a widely used bash script (wait for it) that you can use to test and wait on the availability of a TCP host and port. It's also recommended solution from the Compose Documentation. You can copy wait for it into your files and starting using it right away.
web:
command: /wait-for-it.sh db:5432
Also you might want to check this out: Wait for it usage with docker
Just change the Version of your Docker-Compose to version: "3.5"
and that solved the problem

Docker Compose with PostgreSQL and Prometheus PostgreSQL Exporter refusing to connect?

I'm trying to solve this question regarding the PostgreSQL exporter to Prometheus (https://github.com/wrouesnel/postgres_exporter), Prometheus PostgreSQL server exporter example not working on MacOS?, running postgres and postgres_exporter containers connected to a user-defined bridge network rather than the host network, which seems not to work on Docker Desktop for Mac. I've created the following docker-compose.yml:
version: "3"
services:
db:
image: postgres
environment:
POSTGRES_PASSWORD: mypassword
networks:
- mynetwork
exporter:
image: wrouesnel/postgres_exporter
environment:
DATA_SOURCE_NAME: "postgresql://postgres:mypassword#db:5432/postgres?sslmode=disable"
ports:
- "9187:9187"
networks:
- mynetwork
networks:
mynetwork:
I am having two issues with this, however. Firstly, the logs from the exporter service show an error with this message:
"Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused"
Here is the full output:
> docker-compose up
Creating network "postgres-performance-testing_mynetwork" with the default driver
Creating postgres-performance-testing_db_1 ... done
Creating postgres-performance-testing_exporter_1 ... done
Attaching to postgres-performance-testing_db_1, postgres-performance-testing_exporter_1
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Established new database connection to \"db:5432\"." source="postgres_exporter.go:778"
exporter_1 | time="2019-10-03T20:16:56Z" level=error msg="Error opening connection to database (postgresql://postgres:PASSWORD_REMOVED#db:5432/postgres?sslmode=disable): dial tcp 172.18.0.2:5432: connect: connection refused" source="postgres_exporter.go:1348"
exporter_1 | time="2019-10-03T20:16:56Z" level=info msg="Starting Server: :9187" source="postgres_exporter.go:1459"
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting default timezone ... Etc/UTC
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | waiting for server to start....2019-10-03 20:16:57.084 UTC [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.095 UTC [43] LOG: database system was shut down at 2019-10-03 20:16:56 UTC
db_1 | 2019-10-03 20:16:57.101 UTC [42] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down...2019-10-03 20:16:57.180 UTC [42] LOG: received fast shutdown request
db_1 | .2019-10-03 20:16:57.182 UTC [42] LOG: aborting any active transactions
db_1 | 2019-10-03 20:16:57.186 UTC [42] LOG: background worker "logical replication launcher" (PID 49) exited with exit code 1
db_1 | 2019-10-03 20:16:57.186 UTC [44] LOG: shutting down
db_1 | 2019-10-03 20:16:57.197 UTC [42] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-10-03 20:16:57.297 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-10-03 20:16:57.299 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-10-03 20:16:57.310 UTC [51] LOG: database system was shut down at 2019-10-03 20:16:57 UTC
db_1 | 2019-10-03 20:16:57.314 UTC [1] LOG: database system is ready to accept connections
Is this because of the Prometheus exporter trying to connect to the database before it is ready to accept connections? (This would seem to be the case from the sequence of the logging).
Another problem is that if I browse to localhost:9187, the result is that I download a file containing HTML, rather than actually seeing a link to /metrics that I can follow:
Any idea how I can resolve these issues?
Without knowing how your images are built, I see two likely culprits that need to be addressed:
It seems that the exporter container is starting up first, and attempting to connect to the db container before postgres is even up and running. You may need to add depends_on: db in the exporter section.
You may need to edit pg_hba.conf on the db container before exporter can connect to it. By default, only local connections are allowed, so you'll have to append exporter's IP address (or use a /0 netmask, if this is a development-only environment)
If you fix these two, things should start working for you.
Disclosure: I am an EnterpriseDB (EDB) employee
postgres-exporter:
image: wrouesnel/postgres_exporter:v0.8.0
restart: always
environment:
#- DATA_SOURCE_NAME=postgresql://postgres:password#postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_URI=postgres-db:5432/postgres?sslmode=disable
- DATA_SOURCE_USER=postgres
- DATA_SOURCE_PASS=password
ports:
- "9187:9187"
networks:
- postgres-prometheus
depends_on:
- postgres-db
Note in this example that the value for DATA_SOURCE_NAME has no double-quotes and that the value for DATA_SOURCE_URI (in case it's your preference) has no "postgresql://". It starts directly from the hostname.

How to run a *.sh file on Postgres Docker-Compose startup

I am trying to create a new DB + User on Postgres when starting a new Docker-Compose Postgres image. For some reason, the init-user-db.sh file is not being copied inside docker-entrypoint-initdb.d folder on the container.
Any idea why?
Thanks
docker-compose.yml:
version: '3'
services:
db:
image: postgres:11
volumes:
- ./db/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
ports:
- '5432:5432'
init-user-db.sh:
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" <<-EOSQL
CREATE USER bh_ideas_user;
CREATE DATABASE bh_ideas ENCODING UTF8;
GRANT ALL PRIVILEGES ON DATABASE bh_ideas TO bh_ideas_user;
ALTER USER bh_ideas_user WITH PASSWORD 'password123';
ALTER USER bh_ideas_user WITH SUPERUSER;
EOSQL
This is the output I receive:
$ docker-compose up && exit 0
Creating ideas-api_db_1 ... done
Attaching to ideas-api_db_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | ****************************************************
db_1 | WARNING: No password has been set for the database.
db_1 | This will allow anyone with access to the
db_1 | Postgres port to access your database. In
db_1 | Docker's default configuration, this is
db_1 | effectively any other container on the same
db_1 | system.
db_1 |
db_1 | Use "-e POSTGRES_PASSWORD=password" to set
db_1 | it in "docker run".
db_1 | ****************************************************
db_1 | waiting for server to start....2019-01-19 10:28:21.673 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-01-19 10:28:21.692 UTC [49] LOG: database system was shut down at 2019-01-19 10:28:21 UTC
db_1 | 2019-01-19 10:28:21.699 UTC [48] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | 2019-01-19 10:28:21.763 UTC [48] LOG: received fast shutdown request
db_1 | waiting for server to shut down....2019-01-19 10:28:21.766 UTC [48] LOG: aborting any active transactions
db_1 | 2019-01-19 10:28:21.767 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
db_1 | 2019-01-19 10:28:21.769 UTC [50] LOG: shutting down
db_1 | 2019-01-19 10:28:21.794 UTC [48] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2019-01-19 10:28:21.903 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2019-01-19 10:28:21.903 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2019-01-19 10:28:21.910 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2019-01-19 10:28:21.930 UTC [57] LOG: database system was shut down at 2019-01-19 10:28:21 UTC
db_1 | 2019-01-19 10:28:21.937 UTC [1] LOG: database system is ready to accept connections
From the docker inspect, the binding is properly done:
"Binds": ["/host_mnt/d/Projects/ideas-api/db/initdb.d:/docker-entrypoint-initdb.d:rw"],

Troubles with connect to database in Docker Compose

In one folder I have 4 files: base.py, Dockerfile, docker-compose.yml and wait-for-it.sh.
base.py:
import psycopg2
print("JJJJJJJJ")
conn = psycopg2.connect("dbname='base123' user='postgres' host='db' password='pw1234'")
cur = conn.cursor()
cur.execute("CREATE TABLE test(id serial PRIMARY KEY, num int);")
Dockerfile:
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get -y install python-pip
RUN apt-get update
RUN pip install --upgrade pip
RUN pip install psycopg2-binary
COPY base.py base.py
COPY wait-for-it.sh wait-for-it.sh
RUN chmod +x /wait-for-it.sh
CMD ["python","base.py"]
docker-compose.yml
version: '3'
services:
db:
image: 'postgres:latest'
expose:
- "5432"
ports:
- "5559:5432"
environment:
POSTGRES_PASSWORD: pw1234
POSTGRES_DB: base123
aprrka:
build: .
depends_on:
- db
command: ["./wait-for-it.sh", "db:5432", "--", "python", "base.py"]
wait-fot-it.sh:
link to code in github
After docker-compose up I have this OUTPUT:
...
Creating postgres_db_1 ... done
Creating postgres_aprrka_1 ... done
Attaching to postgres_db_1, postgres_aprrka_1
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
aprrka_1 | wait-for-it.sh: waiting 15 seconds for db:5432
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....2018-08-10 09:59:13.529 UTC [38] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:13.582 UTC [39] LOG: database system was shut down at 2018-08-10 09:59:12 UTC
db_1 | 2018-08-10 09:59:13.599 UTC [38] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 | CREATE DATABASE
db_1 |
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | 2018-08-10 09:59:14.585 UTC [38] LOG: received fast shutdown request
db_1 | waiting for server to shut down....2018-08-10 09:59:14.593 UTC [38] LOG: aborting any active transactions
db_1 | 2018-08-10 09:59:14.613 UTC [38] LOG: worker process: logical replication launcher (PID 45) exited with exit code 1
db_1 | 2018-08-10 09:59:14.613 UTC [40] LOG: shutting down
db_1 | 2018-08-10 09:59:14.660 UTC [38] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2018-08-10 09:59:14.726 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2018-08-10 09:59:14.729 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2018-08-10 09:59:14.806 UTC [65] LOG: database system was shut down at 2018-08-10 09:59:14 UTC
db_1 | 2018-08-10 09:59:14.829 UTC [1] LOG: database system is ready to accept connections
db_1 | 2018-08-10 09:59:15.217 UTC [72] LOG: incomplete startup packet
aprrka_1 | wait-for-it.sh: db:5432 is available after 5 seconds
aprrka_1 | JJJJJJJJ
postgres_aprrka_1 exited with code 0
When I type command: psql -U postgres -h localhost -p 5559 in another terminal I can connect to base1234, but when type `\dt' i have no any relations, so my py file don't create table in postgres database. I think that no connection exist with postgres db. Please, help me
You need to close the cursor and commit all pending transactions to the database for the changes to be effective.
Add the following 2 lines at the end of your base.py:
cur.close()
conn.commit()

In a Docker Compose multi-container application, how to prevent Postgres from running statements previously issued

I'm running a Scrapy project with an Item Pipeline which writes data to a PostgreSQL database. It is a Docker Compose multi-container application with the following docker-compose.yml:
version: '3'
services:
privoxy:
build: ./privoxy
links:
- tor
tor:
build:
context: ./tor
args:
password: "foo"
scraper:
build: ./scraper
environment:
- http_proxy=http://privoxy:8118
- PGPASSWORD=iperpassword
links:
- tor
- privoxy
- db
volumes:
- scraper-data:/scraper/apkmirror_scraper/data
- scraper-jobdir:/scraper/apkmirror_scraper/crawls
db:
image: postgres
environment:
- POSTGRES_PASSWORD=iperpassword
volumes:
- db-data:/var/lib/postgresql/data
volumes:
scraper-data:
scraper-jobdir:
db-data:
I'm using the postgres image from the official repository (https://hub.docker.com/_/postgres/), and have made the /var/lib/postgresql/data directory into a volume following https://docs.docker.com/compose/compose-file/#volume-configuration-reference.
The problem is that when I start the application, the db service outputs logs and errors pertaining to things I tried previously, not what I'm trying to do now. This is what I see:
apkmirrorscrapercompose_tor_1 is up-to-date
apkmirrorscrapercompose_db_1 is up-to-date
apkmirrorscrapercompose_privoxy_1 is up-to-date
Recreating apkmirrorscrapercompose_scraper_1
Attaching to apkmirrorscrapercompose_tor_1, apkmirrorscrapercompose_db_1, apkmirrorscrapercompose_privoxy_1, apkmirrorscrapercompose_scraper_1
scraper_1 | List of databases
scraper_1 | Name | Owner | Encoding | Collate | Ctype | Access privileges
scraper_1 | -----------+----------+----------+------------+------------+-----------------------
scraper_1 | postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
scraper_1 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | (3 rows)
scraper_1 |
scraper_1 | Postgres is up - executing command
scraper_1 | Tor appears to be working. Proceeding with command...
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: apkmirror_scraper)
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'apkmirror_scraper', 'NEWSPIDER_MODULE': 'apkmirror_scraper.spiders', 'SPIDER_MODULES': ['apkmirror_scraper.spiders']}
scraper_1 | 2017-06-30 12:32:59 [scrapy.middleware] INFO: Enabled extensions:
scraper_1 | ['scrapy.extensions.corestats.CoreStats',
scraper_1 | 'scrapy.extensions.telnet.TelnetConsole',
scraper_1 | 'scrapy.extensions.memusage.MemoryUsage',
scraper_1 | 'scrapy.extensions.closespider.CloseSpider',
scraper_1 | 'scrapy.extensions.logstats.LogStats']
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2017-06-29 13:40:40 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | done
db_1 | server started
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | waiting for server to shut down....LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2017-06-29 13:40:41 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:41:53 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:43:22 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:52:02 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | ERROR: column "developer" of relation "apks" does not exist at character 31
db_1 | STATEMENT: INSERT INTO apks (url, title, developer, app, version_name, version_code, architectures, package, apk_file_size, android_min_version, android_target_version, supported_dpis, md5_signature, time_uploaded, time_scraped, image_urls, images, file_urls, files, created_on, updated_on) VALUES ('http://www.apkmirror.com/apk/sony-mobile-communications/backup-restore-unknown-developer/backup-restore-unknown-developer-1-2-a-1-23-release/backup-restore-1-2-a-1-23-android-apk-download/', 'Backup & restore 1.2.A.1.23', 'Sony Mobile Communications', 'Backup & restore', '1.2.A.1.23', '2360343', ARRAY['arm'], 'com.sonymobile.synchub', 9541452, '6.0', '7.1', ARRAY['nodpi'], 'eda39fedfc33ea939a5bb4926b3980c1', '2017-04-25T16:30:00+00:00'::timestamptz, '2017-06-29T14:10:39+00:00'::timestamptz, ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/ap_resize/ap_resize.php?src=http%3A%2F%2Fwww.apkmirror.com%2Fwp-content%2Fuploads%2F2017%2F04%2F58ff36aea8e0c.png&w=96&h=96&q=100'], '{}', ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/download.php?id=203068'], '{}', '2017-06-29T14:19:21.661193'::timestamp, '2017-06-29T14:19:21.661207'::timestamp) RETURNING apks.id
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 16:28:54 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
Although I'm not showing details on the application, my main point is that I'm positive that data shown in the error message is from a previous run, and not from the APK download page that I'm currently trying to scrape.
In short, it seems like PostgreSQL is 'caching' previous statements and trying to re-run them even after I stop the application, do docker-compose build and docker-compose up. It seems to me that this behavior must somehow be driven by the data in /var/lib/postgresql/data, since this volume is the only thing that is persisted across restarting the container.
Is this indeed the case? If so, how can I disable this 'statement caching' behavior?
This just means that docker-compose is re-starting your previously stopped container (or re-attaching to it, if it is already running), and because of there is no need to re-build the image:
apkmirrorscrapercompose_db_1 is up-to-date
So, what your are seeing "again" in the logs is the docker logs of that container which includes all the life-generated logs of postgres.
Simply remove the container and start again (the db data is persisted in the volume, so it will survive):
docker-compose rm db