Unable to run Airflow and acess it - postgresql

I downloaded Airflow's docker-compose.yaml from his official website and putted into my folder and runned sudo docker compose up airflow-init that worked perfectly. And when I runned sudo docker compose up to run the server and tried to acess in localhost:8080, i could'nt. This is my logs when I run sudo docker compose up:
WARN[0000] The "AIRFLOW_UID" variable is not set. Defaulting to a blank string.
WARN[0000] The "AIRFLOW_UID" variable is not set. Defaulting to a blank string.
[+] Running 7/0
⠿ Container airflow-docker-redis-1 Created 0.0s
⠿ Container airflow-docker-postgres-1 Created 0.0s
⠿ Container airflow-docker-airflow-init-1 Created 0.0s
⠿ Container airflow-docker-airflow-scheduler-1 Created 0.0s
⠿ Container airflow-docker-airflow-webserver-1 Created 0.0s
⠿ Container airflow-docker-airflow-triggerer-1 Created 0.0s
⠿ Container airflow-docker-airflow-worker-1 Created 0.0s
Attaching to airflow-docker-airflow-init-1, airflow-docker-airflow-scheduler-1, airflow-docker-airflow-triggerer-1, airflow-docker-airflow-webserver-1, airflow-docker-airflow-worker-1, airflow-docker-postgres-1, airflow-docker-redis-1
airflow-docker-redis-1 | 1:C 12 Oct 2022 15:20:30.213 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
airflow-docker-redis-1 | 1:C 12 Oct 2022 15:20:30.213 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
airflow-docker-redis-1 | 1:C 12 Oct 2022 15:20:30.213 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.214 * monotonic clock: POSIX clock_gettime
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.217 * Running mode=standalone, port=6379.
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.217 # Server initialized
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.217 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * Loading RDB produced by version 7.0.5
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * RDB age 6 seconds
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * RDB memory usage when created 0.85 Mb
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * Done loading RDB, keys loaded: 0, keys expired: 0.
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * DB loaded from disk: 0.001 seconds
airflow-docker-redis-1 | 1:M 12 Oct 2022 15:20:30.218 * Ready to accept connections
airflow-docker-postgres-1 |
airflow-docker-postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
airflow-docker-postgres-1 |
airflow-docker-postgres-1 | 2022-10-12 15:20:30.319 UTC [1] LOG: starting PostgreSQL 13.8 (Debian 13.8-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
airflow-docker-postgres-1 | 2022-10-12 15:20:30.320 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
airflow-docker-postgres-1 | 2022-10-12 15:20:30.320 UTC [1] LOG: listening on IPv6 address "::", port 5432
airflow-docker-postgres-1 | 2022-10-12 15:20:30.323 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
airflow-docker-postgres-1 | 2022-10-12 15:20:30.330 UTC [26] LOG: database system was shut down at 2022-10-12 15:20:24 UTC
airflow-docker-postgres-1 | 2022-10-12 15:20:30.340 UTC [1] LOG: database system is ready to accept connections
airflow-docker-airflow-init-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
airflow-docker-airflow-init-1 | FutureWarning,
airflow-docker-airflow-init-1 |
airflow-docker-airflow-init-1 | WARNING!!!: AIRFLOW_UID not set!
airflow-docker-airflow-init-1 | If you are on Linux, you SHOULD follow the instructions below to set
airflow-docker-airflow-init-1 | AIRFLOW_UID environment variable, otherwise files will be owned by root.
airflow-docker-airflow-init-1 | For other operating systems you can get rid of the warning with manually created .env file:
airflow-docker-airflow-init-1 | See: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user
airflow-docker-airflow-init-1 |
airflow-docker-airflow-init-1 | The container is run as root user. For security, consider using a regular user account.
airflow-docker-airflow-init-1 |
airflow-docker-airflow-init-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
airflow-docker-airflow-init-1 | FutureWarning,
airflow-docker-airflow-init-1 | DB: postgresql+psycopg2://airflow:***#postgres/airflow
airflow-docker-airflow-init-1 | Performing upgrade with database postgresql+psycopg2://airflow:***#postgres/airflow
airflow-docker-airflow-init-1 | [2022-10-12 15:20:40,747] {migration.py:204} INFO - Context impl PostgresqlImpl.
airflow-docker-airflow-init-1 | [2022-10-12 15:20:40,748] {migration.py:211} INFO - Will assume transactional DDL.
airflow-docker-airflow-init-1 | [2022-10-12 15:20:40,847] {db.py:1531} INFO - Creating tables
airflow-docker-airflow-init-1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
airflow-docker-airflow-init-1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
airflow-docker-airflow-init-1 | Upgrades done
airflow-docker-airflow-init-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
airflow-docker-airflow-init-1 | FutureWarning,
airflow-docker-airflow-init-1 | [2022-10-12 15:20:55,367] {providers_manager.py:211} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
airflow-docker-airflow-init-1 | [2022-10-12 15:20:57,136] {providers_manager.py:211} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
airflow-docker-airflow-init-1 | airflow already exist in the db
airflow-docker-airflow-init-1 | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
airflow-docker-airflow-init-1 | FutureWarning,
airflow-docker-airflow-init-1 | 2.4.1
airflow-docker-airflow-init-1 exited with code 0
I've already searched and tried some solutions, but nothing did well. I just want to run and acess at port 8080. What can I do?

You need to run the following at first time before running the docker compose.
echo -e "AIRFLOW_UID=$(id -u)" > .env
See initialize environment at this link

Related

Problem with postgres for 2 concurrent docker deployments on the same host

I'm having issues with postgres for 2 docker deployments on the same host. I need to deploy the same app with docker compose on the same host. I have one docker-compose file and two .env for each instance. The app is built with Ruby and Hanami web framework.
When deploying the first instance (production) it runs fine. With the second instance (staging) I get this error during docker-compose setup:
PostgreSQL Database directory appears to contain a database; Skipping initialization
.
.
.
2022-07-30 16:13:48.978 UTC [33] FATAL: password authentication failed for user "some_user"
postgres_1 | 2022-07-30 16:13:48.978 UTC [33] DETAIL: Role "some_user" does not exist.
I also see the port being used is 5432, which is the one used for production.
For staging, I've specified the other_user and a different port as you can see in the file below.
Here are the files in question with which I wanted to separate these DB instances:
docker-compose
version: '3'
services:
postgres:
image: postgres
environment:
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_DB=${POSTGRES_DB}
ports:
- ${POSTGRES_PORT}:${POSTGRES_PORT}
volumes:
- postgres:/var/lib/postgresql/data
web:
build: .
command: >
bash -c "bundle exec hanami db migrate
&& bundle exec rake initial_settings:add_default_language
&& bundle exec rake initial_settings:add_session_validity
&& bundle exec rake import_user:create
&& bundle exec rake super_admin:create
&& bundle exec hanami assets precompile
&& bundle exec hanami server
&& cp -r apps/wordrocket/assets/webfonts public/webfonts
&& cp -r apps/wordrocket/assets/webfonts public/assets/webfonts
&& cp -r apps/wordrocket/assets/images/sort*.png public/assets
&& cp -r apps/wordrocket/assets/images/sort*.png public
&& cp -r apps/wordrocket/assets/images/ui-icons*.png public/assets/wordrocket
&& mkdir public/assets/images
&& cp -r apps/wordrocket/assets/images/sort*.png public/assets/images"
volumes:
- ./hanami_log/hanami_app.log:/usr/src/app/hanami_log/hanami_app.log
links:
- postgres
depends_on:
- postgres
nginx:
image: nginx:alpine
restart: unless-stopped
tty: true
ports:
- "${NGINX_PORT}:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- ./nginx_log:/var/log/nginx
depends_on:
- web
volumes:
postgres:
web:
nginx:
Production .env
POSTGRES_USER=some_user
POSTGRES_PASSWORD=some_password
POSTGRES_DB=app_production
NGINX_PORT=7080
COMPOSE_PROJECT_NAME=myAPP
POSTGRES_PORT=5432
Staging .env.staging
POSTGRES_USER=other_user
POSTGRES_PASSWORD=other_password
POSTGRES_DB=app_staging
NGINX_PORT=7081
COMPOSE_PROJECT_NAME=myAPP_staging
POSTGRES_PORT=5433
Makefile
production:
docker-compose -f docker-compose.yml --env-file .env up
staging:
docker-compose -f docker-compose.yml --env-file .env.staging up
I'm guessing I don't separate the postgres instances/dbs correctly, but I'm not sure. Also, is this a Docker or host problem?
+++ EDIT +++
I've made a separate docker-compose file for staging in order to differentiate between volumes.
production is the same:
volumes:
- postgres:/var/lib/postgresql/data
.
.
.
volumes:
postgres:
web:
nginx:
staging:
volumes:
- postgres_staging:/var/lib/postgresql/data
.
.
.
volumes:
postgres_staging:
web:
nginx:
This is the startup log up to the first error:
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default time zone ... Etc/UTC
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... ok
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 | initdb: warning: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2022-07-31 10:03:12.510 UTC [47] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
postgres_1 | 2022-07-31 10:03:12.513 UTC [47] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-07-31 10:03:12.524 UTC [48] LOG: database system was shut down at 2022-07-31 10:03:12 UTC
postgres_1 | 2022-07-31 10:03:12.529 UTC [47] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | waiting for server to shut down...2022-07-31 10:03:12.745 UTC [47] LOG: received fast shutdown request
postgres_1 | .2022-07-31 10:03:12.748 UTC [47] LOG: aborting any active transactions
postgres_1 | 2022-07-31 10:03:12.749 UTC [47] LOG: background worker "logical replication launcher" (PID 54) exited with exit code 1
postgres_1 | 2022-07-31 10:03:12.749 UTC [49] LOG: shutting down
postgres_1 | 2022-07-31 10:03:12.771 UTC [47] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2022-07-31 10:03:12.865 UTC [1] LOG: starting PostgreSQL 14.4 (Debian 14.4-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
postgres_1 | 2022-07-31 10:03:12.865 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2022-07-31 10:03:12.865 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2022-07-31 10:03:12.871 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2022-07-31 10:03:12.879 UTC [61] LOG: database system was shut down at 2022-07-31 10:03:12 UTC
postgres_1 | 2022-07-31 10:03:12.885 UTC [1] LOG: database system is ready to accept connections
postgres_1 | 2022-07-31 10:03:13.014 UTC [68] FATAL: password authentication failed for user "some_user"
Now this is the only instance being started. The production is not running even and it still tries to authenticate the production user.
Is it ignoring the .env.staging file?
+++ END EDIT +++
Best, Seba

docker-entrypoint-initdb.d/* files ignored on docker-compose first run

I try to setup a Postgresql db in my docker-compose and it worked as expected for a moment. But now, my migration sql scripts aren't runned anymore, even after a fresh new build (sudo docker-compose build or sudo docker-compose up --build) and after removing the persistent volume created locally (sudo rm -rf db-data).
$ sudo docker-compose up
[+] Running 14/14
⠿ db Pulled 10.7s
⠿ a2abf6c4d29d Pull complete 2.6s
⠿ e1769f49f910 Pull complete 2.8s
⠿ 33a59cfee47c Pull complete 2.9s
⠿ 461b2090c345 Pull complete 3.1s
⠿ 8ed8ab6290ac Pull complete 3.4s
⠿ 495e42c822a0 Pull complete 3.5s
⠿ 18e858c71c58 Pull complete 3.6s
⠿ 594792c80d5f Pull complete 3.7s
⠿ 794976979956 Pull complete 8.2s
⠿ eb5e1a73c3ca Pull complete 8.3s
⠿ 6d6360292cba Pull complete 8.3s
⠿ 131e916e1a28 Pull complete 8.4s
⠿ 757a73507e2e Pull complete 8.5s
[+] Running 2/2
⠿ Container db-1 Created 0.2s
⠿ Container app-1 Created 0.1s
Attaching to app-1, db-1
db-1 | The files belonging to this database system will be owned by user "postgres".
db-1 | This user must also own the server process.
db-1 |
db-1 | The database cluster will be initialized with locale "en_US.utf8".
db-1 | The default database encoding has accordingly been set to "UTF8".
db-1 | The default text search configuration will be set to "english".
db-1 |
db-1 | Data page checksums are disabled.
db-1 |
db-1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db-1 | creating subdirectories ... ok
db-1 | selecting dynamic shared memory implementation ... posix
db-1 | selecting default max_connections ... 100
db-1 | selecting default shared_buffers ... 128MB
db-1 | selecting default time zone ... Etc/UTC
db-1 | creating configuration files ... ok
db-1 | running bootstrap script ... ok
db-1 | performing post-bootstrap initialization ... ok
db-1 | syncing data to disk ... ok
db-1 |
db-1 |
db-1 | Success. You can now start the database server using:
db-1 |
db-1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db-1 |
db-1 | initdb: warning: enabling "trust" authentication for local connections
db-1 | You can change this by editing pg_hba.conf or using the option -A, or
db-1 | --auth-local and --auth-host, the next time you run initdb.
db-1 | waiting for server to start....2022-01-26 14:57:29.498 UTC [49] LOG: starting PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db-1 | 2022-01-26 14:57:29.506 UTC [49] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1 | 2022-01-26 14:57:29.529 UTC [50] LOG: database system was shut down at 2022-01-26 14:57:29 UTC
db-1 | 2022-01-26 14:57:29.542 UTC [49] LOG: database system is ready to accept connections
db-1 | done
db-1 | server started
db-1 | CREATE DATABASE
db-1 |
db-1 |
db-1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/00000000000000_diesel_initial_setup
db-1 |
db-1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/2021-10-21-140200_initial_schemes
db-1 |
db-1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/2022-01-24-120647_fill_tables
db-1 |
db-1 | waiting for server to shut down...2022-01-26 14:57:29.858 UTC [49] LOG: received fast shutdown request
db-1 | .2022-01-26 14:57:29.865 UTC [49] LOG: aborting any active transactions
db-1 | 2022-01-26 14:57:29.868 UTC [49] LOG: background worker "logical replication launcher" (PID 56) exited with exit code 1
db-1 | 2022-01-26 14:57:29.868 UTC [51] LOG: shutting down
db-1 | 2022-01-26 14:57:29.914 UTC [49] LOG: database system is shut down
db-1 | done
db-1 | server stopped
db-1 |
db-1 | PostgreSQL init process complete; ready for start up.
db-1 |
db-1 | 2022-01-26 14:57:29.993 UTC [1] LOG: starting PostgreSQL 14.1 (Debian 14.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
db-1 | 2022-01-26 14:57:29.994 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db-1 | 2022-01-26 14:57:29.994 UTC [1] LOG: listening on IPv6 address "::", port 5432
db-1 | 2022-01-26 14:57:30.008 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1 | 2022-01-26 14:57:30.026 UTC [63] LOG: database system was shut down at 2022-01-26 14:57:29 UTC
db-1 | 2022-01-26 14:57:30.038 UTC [1] LOG: database system is ready to accept connections
As you can see, my docker-entrypoint-initdb.d/* files are all being ignored. Notice that my 3 migrations scripts run fine on a local pg instance.
My docker-compose looks lik this:
version: '3.7'
services:
db:
image: postgres
restart: always
environment:
- POSTGRES_USER=***
- POSTGRES_PASSWORD=***
- POSTGRES_DB=***
ports:
- '5433:5432'
volumes:
- ./db-data:/var/lib/postgresql/data
- ./migrations:/docker-entrypoint-initdb.d/ # copy the sql migrations
[...]
volumes:
db-data:
According to poatgres image entrypoint only below file type executed:
*.sh
*.sql
*.sql.gz
*.sql.xz

Postgres, RHEL and Docker

I've been battling with this issue for the last 4 days of my life and it's driving me crazy.
I'm trying to deploy a service that uses a postgres DB in RHEL 8. In order to do so, I'm deploying them both using docker-compose.
The problem is that from the service container I can ping the postgres container, but the service is not able to connect the DB...
I simplified the use case, and used a docker compose that uses the adminer docker image to connect any of the listed DB managers including postgres. It works great on my machine and on the test server that uses ubuntu 20, but when I try it in RHEL 8, I can't get access to the DB either!
This are the docker and docker-compose versions:
Docker:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:36 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:11 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Docker-compose
docker-compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
The OS
Red Hat Enterprise Linux release 8.1 (Ootpa)
The actual docker-compose that I'm using is this:
version: '3.1'
services:
db:
image: postgres
restart: always
environment:
POSTGRES_PASSWORD: example
adminer:
image: adminer
restart: always
ports:
- 8080:8080
Logs for the postgres container seem to be fine when compared to the logs that I get in the other two systems:
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting default time zone ... Etc/UTC
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 | syncing data to disk ... ok
db_1 |
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | initdb: warning: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | waiting for server to start....2020-10-12 08:18:28.489 UTC [46] LOG: starting PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1 | 2020-10-12 08:18:28.499 UTC [46] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2020-10-12 08:18:28.545 UTC [47] LOG: database system was shut down at 2020-10-12 08:18:25 UTC
db_1 | 2020-10-12 08:18:28.557 UTC [46] LOG: database system is ready to accept connections
db_1 | done
db_1 | server started
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | waiting for server to shut down...2020-10-12 08:18:28.572 UTC [46] LOG: received fast shutdown request
db_1 | .2020-10-12 08:18:28.581 UTC [46] LOG: aborting any active transactions
db_1 | 2020-10-12 08:18:28.582 UTC [46] LOG: background worker "logical replication launcher" (PID 53) exited with exit code 1
db_1 | 2020-10-12 08:18:28.583 UTC [48] LOG: shutting down
db_1 | 2020-10-12 08:18:28.648 UTC [46] LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | 2020-10-12 08:18:28.693 UTC [1] LOG: starting PostgreSQL 12.4 (Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1 | 2020-10-12 08:18:28.694 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2020-10-12 08:18:28.694 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2020-10-12 08:18:28.712 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2020-10-12 08:18:28.751 UTC [55] LOG: database system was shut down at 2020-10-12 08:18:28 UTC
db_1 | 2020-10-12 08:18:28.764 UTC [1] LOG: database system is ready to accept connections
Did anybody encountered this problem before? Do you have nay suggestions that I could try?
EDIT:
The adminer container logs are all the same as with the other machines. They just throw this error when trying to connect, instead an ok message:
Is the server running on host "postgres" (10.10.10.2) and accepting TCP/IP connections on port 5432?
I'm guessing you installed docker from some site. For RHEL 8, try using Podman and Buildah - OCI compatible alternatives that don't require a daemon. This article explains it. These are the default container tools in RHEL 8.
Check the logs from the adminer container while in your RHEL 8 environment, you will be able to spot the issue from there.
The command will be docker logs adminer-container-name

Can't Access Web (Flask) Application from Google Cloud Platform's VM SSH Link

My goal is to run a docker-compose cluster on a VM from Google Cloud Platform. I have successfully installed docker and docker-compose:
$ uname -a
Linux instance-6 4.15.0-1083-gcp #94~16.04.1-Ubuntu SMP Sat Sep 5 22:53:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker -v
Docker version 19.03.13, build 4484c46d9d
$ docker-compose -v
docker-compose version 1.27.3, build 4092ae5d
I am following the basic tutorial to create a docker-compose cluster using: https://docs.docker.com/compose/gettingstarted/ (Steps #1-#4).
My app.py file is:
import time
import redis
from flask import Flask
app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)
def get_hit_count():
retries = 5
while True:
try:
return cache.incr('hits')
except redis.exceptions.ConnectionError as exc:
if retries == 0:
raise exc
retries -= 1
time.sleep(0.5)
#app.route('/')
def hello():
count = get_hit_count()
return 'Hello World! I have been seen {} times.\n'.format(count)
My requirements.txt file is:
flask
redis
My Dockerfile is:
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP app.py
ENV FLASK_RUN_HOST 0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
EXPOSE 5000
COPY . .
CMD ["flask", "run"]
And, my docker-compose.yml is:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
redis:
image: "redis:alpine"
Running docker-compose up gives me the correct output. One of the outputs points to where the web_1 is running.
$ docker-compose up
...
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
...
After pressing the link http://0.0.0.0:5000/, GCP doesn't connect. It tries to go to the URL: https://ssh.cloud.google.com/devshell/proxy?authuser=2&devshellProxyPath=%2F&port=5000&environment_name&environment_id, but then it gives the error: 500. That’s an error. There was an error. Please try again later. That’s all we know.
Going to the external IP address and putting in port 5000 also doesn't return anything. (http://IPAddress:500)
I checked the ports:
$ sudo docker-compose ps
Name Command State Ports
composetest_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp
composetest_web_1 flask run Up 0.0.0.0:5000->5000/tcp
I'm not sure what the reason it. I'm guessing it is the firewall configuration from GCP. Everything is just the default settings. I also allowed HTTP and HTTPS requests in the Compute Engine VM Instance settings. Would really appreciate more guidance on what to do. Thanks in advance!
See below for full output:
$ sudo docker-compose up
Starting composetest_redis_1 ... done
Starting composetest_web_1 ... done
Attaching to composetest_redis_1, composetest_web_1
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 23 Sep 2020 21:40:27.818 * Running mode=standalone, port=6379.
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # Server initialized
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * Loading RDB produced by version 6.0.8
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * RDB age 27 seconds
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * RDB memory usage when created 0.77 Mb
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * DB loaded from disk: 0.000 seconds
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * Ready to accept connections
web_1 | * Serving Flask app "app.py"
web_1 | * Environment: production
web_1 | WARNING: This is a development server. Do not use it in a production deployment.
web_1 | Use a production WSGI server instead.
web_1 | * Debug mode: off
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Need to properly allow traffic through that port by:
Creating a firewall rule with a tag
Including that tag in the VM's Network settings
Link here: Network Tags

Docker | Postgres Database is uninitialized and superuser password is not specified

I am using docker-compose.yml to create multiple running containers but failing to start Postgres docker server, with following logs and yes I have searched many related SO posts, but they didn't helped me out.
Creating network "complex_default" with the default driver
Creating complex_server_1 ... done
Creating complex_redis_1 ... done
Creating complex_postgres_1 ... done
Attaching to complex_postgres_1, complex_redis_1, complex_server_1
postgres_1 | Error: Database is uninitialized and superuser password is not specified.
postgres_1 | You must specify POSTGRES_PASSWORD to a non-empty value for the
postgres_1 | superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
postgres_1 |
postgres_1 | You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
postgres_1 | connections without a password. This is *not* recommended.
postgres_1 |
postgres_1 | See PostgreSQL documentation about "trust":
postgres_1 | https://www.postgresql.org/docs/current/auth-trust.html
complex_postgres_1 exited with code 1
below is my docker-compose configuration:
version: '3'
services:
postgres:
image: 'postgres:11-alpine'
redis:
image: 'redis:latest'
server:
build:
dockerfile: Dockerfile.dev
context: ./server
volumes:
- /app/node_modules
- ./server:/app
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- PGUSER=postgres
- PGHOST=postgres
- PGDATABASE=postgres
- PGPASSWORD=postgres_password
- PGPORT=5432
as well as package.json inside server directory is following:
{
"dependencies": {
"body-parser": "^1.19.0",
"cors": "^2.8.4",
"express": "^4.16.3",
"nodemon": "^2.0.4",
"pg": "7.4.3",
"redis": "^2.8.0"
},
"scripts": {
"dev": "nodemon",
"start": "node index.js"
}
}
also for better consideration, I have attached my hands-on project structure:
A year ago it were actually working fine, Does anyone have any idea, what's going wrong here inside my docker-compose file now.
A year ago it were actually working fine, Does anyone have any idea, what's going wrong here inside my docker-compose file now.
Seems like you pulled the fresh image, where in the new image you should specify Postgres user password. You can look into Dockerhub, the image is update one month ago
postgress-11-alpine
db:
image: postgres:11-alpine
restart: always
environment:
POSTGRES_PASSWORD: example
As the error message is self expalinatory
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
or POSTGRES_HOST_AUTH_METHOD=trust use this which is not recommended.
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is *not* recommended.
POSTGRES_PASSWORD
This environment variable is required for you to use the PostgreSQL image. It must not be empty or undefined. This environment variable sets the superuser password for PostgreSQL. The default superuser is defined by the POSTGRES_USER environment variable.
Environment Variables
#Adiii yes, you are nearly right, so I have to explicitly mentioned the environment also for the postgres image but with no db parent tag.
So here, I am explicitly mentioning the docker-compose.yaml config to help others for better understanding, also now I am using recent stable postgres image version 12-alpine, currently latest is postgres:12.3
version: '3'
services:
postgres:
image: 'postgres:12-alpine'
environment:
POSTGRES_PASSWORD: postgres_password
redis:
image: 'redis:latest'
server:
build:
dockerfile: Dockerfile.dev
context: ./server
volumes:
- /app/node_modules
- ./server:/app
environment:
- REDIS_HOST=redis
- REDIS_PORT=6379
- PGUSER=postgres
- PGHOST=postgres
- PGDATABASE=postgres
- PGPASSWORD=postgres_password
- PGPORT=5432
and so after docker-compose up the creating and running logs were as following:
PS E:\docker\complex> docker-compose up
Creating network "complex_default" with the default driver
Creating complex_postgres_1 ... done
Creating complex_redis_1 ... done
Creating complex_server_1 ... done
Attaching to complex_redis_1, complex_postgres_1, complex_server_1
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default time zone ... UTC
redis_1 | 1:C 05 Aug 2020 14:24:48.692 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 05 Aug 2020 14:24:48.692 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 05 Aug 2020 14:24:48.692 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
postgres_1 | creating configuration files ... ok
redis_1 | 1:M 05 Aug 2020 14:24:48.693 * Running mode=standalone, port=6379.
redis_1 | 1:M 05 Aug 2020 14:24:48.693 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 05 Aug 2020 14:24:48.694 # Server initialized
redis_1 | 1:M 05 Aug 2020 14:24:48.694 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1 | 1:M 05 Aug 2020 14:24:48.694 * Ready to accept connections
postgres_1 | running bootstrap script ... ok
server_1 |
server_1 | > # dev /app
server_1 | > nodemon
server_1 |
postgres_1 | performing post-bootstrap initialization ... sh: locale: not found
postgres_1 | 2020-08-05 14:24:50.153 UTC [29] WARNING: no usable system locales were found
server_1 | [nodemon] 2.0.4
server_1 | [nodemon] to restart at any time, enter `rs`
server_1 | [nodemon] watching path(s): *.*
server_1 | [nodemon] watching extensions: js,mjs,json
server_1 | [nodemon] starting `node index.js`
postgres_1 | ok
server_1 | Listening
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 | initdb: warning: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 | waiting for server to start....2020-08-05 14:24:51.634 UTC [34] LOG: starting PostgreSQL 12.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
postgres_1 | 2020-08-05 14:24:51.700 UTC [34] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-08-05 14:24:51.981 UTC [35] LOG: database system was shut down at 2020-08-05 14:24:50 UTC
postgres_1 | 2020-08-05 14:24:52.040 UTC [34] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1 |
postgres_1 | waiting for server to shut down....2020-08-05 14:24:52.121 UTC [34] LOG: received fast shutdown request
postgres_1 | 2020-08-05 14:24:52.186 UTC [34] LOG: aborting any active transactions
postgres_1 | 2020-08-05 14:24:52.188 UTC [34] LOG: background worker "logical replication launcher" (PID 41) exited with exit code 1
postgres_1 | 2020-08-05 14:24:52.188 UTC [36] LOG: shutting down
postgres_1 | 2020-08-05 14:24:52.669 UTC [34] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2020-08-05 14:24:52.832 UTC [1] LOG: starting PostgreSQL 12.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
postgres_1 | 2020-08-05 14:24:52.832 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-08-05 14:24:52.832 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-08-05 14:24:52.954 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-08-05 14:24:53.136 UTC [43] LOG: database system was shut down at 2020-08-05 14:24:52 UTC
postgres_1 | 2020-08-05 14:24:53.194 UTC [1] LOG: database system is ready to accept connections
Hope this would help manyone.
Adding on ArifMustafa's answer, This worked for me.
postgres:
image: 'postgres:12-alpine'
environment:
POSTGRES_PASSWORD: mypassword
expose:
- 5432
volumes:
- postgres_data:/var/lib/postgres/data/
volumes:
postgres_data:
In my case there was an error about the POSTGRES_PASSWORD (in docker-compose.yml) and the DATABASE settings for PASSWORD in the project_level settings.py file.
After providing a password in docker-compose.yml
db: # For the PostgreSQL database
image: postgres:11
environment:
- POSTGRES_PASSWORD=example
It is necessary to provide the same password to enable access to the POSTGRES database between the two containers (in my case the web application and the database it depended on). In the project_level settings.py:
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'postgres',
'USER': 'postgres',
'PASSWORD': 'example',
'HOST': 'db',
'PORT': 5432
}
}
To resolve the error when using the command,
docker pull postgres
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
or POSTGRES_HOST_AUTH_METHOD=trust use this which is not recommended.
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all connections without a password. This is not recommended.
Here comes the solution for this:
$ docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres
The default postgres user and database are created in the entrypoint with initdb.
The postgres database is a default database meant for use by users, utilities and third party applications.
postgresql.org/docs