I'm using FastAPI and postgresql to build a simple python api backend. When I try to build the database migration file (using Alembic) this error occures:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "db" to address: Name or service not known
As I searched a lot about this problem, it seems to be because of my docker config, but I couldn't figure it out. Here is my docker-compose code:
version: '3.7'
services:
web:
build: ./src
command: sh -c "alembic upgrade head && uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000"
volumes:
- ./src/:/usr/src/app/
ports:
- 8002:8000
environment:
- DATABASE_URL=postgresql://user:pass#db/my_db
db:
image: postgres:12.1-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=pass
- POSTGRES_DB=my_db
- POSTGRES_HOST_AUTH_METHOD=trust
ports:
- 5432:5432
volumes:
postgres_data:
I'm also suspicious to my other project that is using docker in my local machine (same setup with FastAPI, postgres, and docker), and it works fine and doesn't have any similar problem.
What should I check and change to fix the problem?
P.S: I'm a beginner in docker.
Here's what's happening:
Your web container is trying to connect to db, but db is not up yet, in simple words web cannot "see" db.
The short answer: start db first: docker-compose up -d db then start web: docker-compose up web or whatever you are running.
Now, the long answer. If you proceed with the short answer, that's ok, but it's cumbersome if you ask me. You could try changing the version to 2.x eg 2.4 and adding depends_on to web, example:
version: '2.4'
services:
web:
...
depends_on:
- db
db:
image: ...
...
If you just run docker-compose up ..., docker will start db first.
Now, you may hit another problem: postgres may not be ready to receive connections. In this case you will have to wait for it to be ready. You can achieve this by retrying the db connection on error (I don't remember the exact error, you will have to check the docs) or use something like pg_isready. Assuming this is a dev environment, you can add it to your Dockerfile by installing postgresql-client and changing your cmd to:
command: >
sh -c "
until pg_isready -q -h db; do sleep 1; done
&&
alembic upgrade head
&&
uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000"
I've searched over the web but couldn't find my answer anywhere.
I'm trying to run an API web service, using NestJS framework.
I'm running docker-compose that spins up the API server, a MongoDB instance, and a mongocryptd instance to allow Client-Side Field Level Encryption on my app.
I'm able to connect to the MongoDB instance, but not to the mongocryptd instance.
Docker-Compose file:
version: "3.7"
services:
api:
build:
context: .
dockerfile: Dockerfile
labels:
env: dev
args:
APP: appname
APP_PORT: 3000
ports:
- "3000:3000"
command: ["sh", "-c", "npm run start:app:dev"]
volumes:
- .:/app
mongodb:
build:
context: .
dockerfile: docker/MongoEP-Dockerfile
labels:
env: dev
args:
MONGO_PACKAGE: mongodb-enterprise
MONGO_REPO: repo.mongodb.com
image: mongo-enterprise:4.2.5
command: ["--auth"]
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: usr
MONGO_INITDB_ROOT_PASSWORD: pwd
ports:
- "27017:27017"
volumes: ["/private/var/services/mongodb:/data/db"]
mongocryptd:
build:
context: .
dockerfile: docker/MongoEP-Dockerfile
labels:
env: dev
args:
MONGO_PACKAGE: mongodb-enterprise
MONGO_REPO: repo.mongodb.com
image: mongo-enterprise:4.2.5
entrypoint: mongocryptd
restart: always
ports:
- "27020:27020"
volumes: ["/private/var/services/mongodb:/data/db"]
The used dockerfile is mongo's official dockerfile, but supplied with args to build an enterprise version of the image which includes the enterprise features.
When trying to connect to the database from the app, I'm running:
MongooseModule.forRoot(`mongodb://usr:pwd#mongodb:27017`, {
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
retryAttempts: 2,
autoEncryption: {
keyVaultNamespace,
kmsProviders,
extraOptions: {
mongocryptdURI: `mongodb://mongocryptd:27020`,
mongocryptdBypassSpawn: true
}
} as any
})
** This is the NestJS version of supplying the configs. it's similar to mongoose - the first argument is the URI and the second is the settings object
Without the autoEncryption options, I'm able to connect without any problems. That means that my database address is correct.
With the autoEncyption options, I'm getting MongooseServerSelectionError: connect ECONNREFUSED 172.25.0.4:27020 (mongocryptd address). That means that the IP is correct (DNS resolved), but the connection is refused. As I showed before, the port (27020) is being published by the docker-compose file, and I even tried to add an EXPOSE step in the build itself.
BUT when I map the network of the containers to host (network_mode: "host"), the application is able to connect without any problems (changing the connections DNS to localhost:27017 and 27020 of course). So that must mean it's a docker-related problem.
Additional things I've tried && a recap of what I tried:
Attach a volume to replace /etc/mongod.conf.orig with the following network configurations:
net:
port: 27017
bindIp: 0.0.0.0
bindIpAll: true
Instead of attaching a volume, replacing it ^ at the build step before launching the mongo service.
I also tried changing the bindIp to the specific application IP that was given by the docker network.
All types of connection strings with & without user credentials, auth source, and default database.
Port 27020 is published in docker-compose & exposed on docker file.
I ran out of ideas. Any help is appreciated! :)
EDIT:
After more debugging, I can see that mongod is running with --bind_ip_all by default so changing the conf file shouldn't have an effect.
Tried also running mongocryptd with mongods docker-entrypoint.sh entrypoint instead of overriding it.
Verify mongocryptd is running (ps awwxu, etc.)
Verify you can connect to it from bash on the same container where it is running using mongo.
Verify you can connect to it from host system using mongo.
Check mongocryptd logs (it's basically a mongod with some extra functionality).
I had similar issues with mongocryptd, but I was installing php node instead of npm. Tried different solutions but didn't managed to succeed. Even tried similar Asaf Kfir docker-composer.yaml with pre-installed mongodb-enterprise-cryptd lib, but had same issue. Keep in mind, in php node via Dockerfile I already been able to install libmongocrypt-dev and
mongodb-enterprise-cryptd. (I will leave php Dockerfile below)
Those three containers I managed to link under the same ip address, tested with ncat and I was able to reach. But when I tried to run tests from php node, it start throwing:
MongoDB\Driver\Exception\BulkWriteException: Bulk write failed due to previous MongoDB\Driver\Exception\RuntimeException: key vault error: Invalid reply to find command.
and had this issue for two weeks. Basically didn't know how to resolve it.
P.S. remember these words: The automatic feature of field level encryption is only available in MongoDB Enterprise 4.2 or later
At that time my docker-compose.yaml file looked like that:
version: '3'
services:
#PHP Service
php:
image: local-base-php
container_name: app
restart: unless-stopped
tty: true
ports:
- "27017:27017"
- "27020:27020"
environment:
SERVICE_NAME: app
SERVICE_TAGS: dev
working_dir: /var/www
volumes:
- ./:/var/www/projects
- ./php/local.ini:/usr/local/etc/php/conf.d/local.ini
#MongoDB Service
mongodb:
image: local-mongo-db
container_name: mongodb
restart: unless-stopped
tty: true
environment:
MONGO_INITDB_DATABASE: test
MONGO_INITDB_USERNAME: root
MONGO_INITDB_PASSWORD: rootpassword
network_mode: service:php
volumes: ["/tmp/mongodb:/data/db"]
#MongoDB Service
mongocryptd:
image: local-mongocryptd
container_name: mongocryptd
entrypoint: mongocryptd
restart: unless-stopped
tty: true
network_mode: service:php
volumes: ["/tmp/mongodb:/data/db"]
volumes:
dbdata:
driver: local
Mongo images where build from here: https://docs.mongodb.com/manual/tutorial/install-mongodb-enterprise-with-docker/
I managed to solve this issue like this:
Instead of building mongo-enterprise image I accidentally build up with official mongo image mongo:4.2 and everything worked well. I don't know why mongo says that enterprise is needed for encryption. Because for me mongo-enterprise encryption didn't worked. The original mongo:4.2 image worked perfectly.
working docker-compose.yaml:
version: '3'
services:
#PHP Service
php:
image: local-base-php
container_name: app
restart: always
tty: true
ports:
- "27017:27017"
environment:
SERVICE_NAME: app
SERVICE_TAGS: dev
working_dir: /var/www
volumes:
- ./:/var/www/projects
- ./php/local.ini:/usr/local/etc/php/conf.d/local.ini
#MongoDB Service
mongodb:
image: mongo:4.2
container_name: mongodb
restart: always
tty: true
environment:
MONGO_INITDB_DATABASE: test
MONGO_INITDB_USERNAME: root
MONGO_INITDB_PASSWORD: rootpassword
network_mode: service:php
php node Dockerfile:
FROM php:7.4
RUN apt-get update && apt-get install -y zip unzip libzip-dev git mercurial zlib1g-dev libicu-dev libcurl4-gnutls-dev libssl-dev libssh2-1-dev libgmp-dev libpng-dev uuid-dev
RUN cd /tmp && git clone https://github.com/php/pecl-networking-ssh2 && cd /tmp/pecl-networking-ssh2 \
&& phpize && ./configure && make && make install \
&& echo "extension=ssh2.so" > /usr/local/etc/php/conf.d/ext-ssh2.ini \
&& rm -rf /tmp/ssh2
RUN docker-php-ext-configure gmp
RUN docker-php-ext-install zip json pdo pdo_mysql curl opcache bcmath sockets gmp gd
RUN docker-php-ext-install -j$(nproc) intl
RUN pecl install uuid pcov redis mongodb
RUN docker-php-ext-enable uuid pcov redis mongodb
RUN curl -sS https://get.symfony.com/cli/installer | bash -s -- --install-dir /usr/local/bin
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
RUN curl -L -sS "https://github.com/splitsh/lite/releases/download/v1.0.1/lite_linux_amd64.tar.gz" | tar xvz -C /usr/local/bin
RUN apt-get update
RUN apt-get install -y curl gpg wget
RUN sh -c 'curl -s https://www.mongodb.org/static/pgp/libmongocrypt.asc | gpg --dearmor >/etc/apt/trusted.gpg.d/libmongocrypt.gpg'
RUN echo "deb https://libmongocrypt.s3.amazonaws.com/apt/ubuntu bionic/libmongocrypt/1.0 universe" | tee /etc/apt/sources.list.d/libmongocrypt.list
RUN wget -qO - mongodb.org/static/pgp/server-4.2.asc | apt-key add -
RUN echo "deb http://repo.mongodb.com/apt/debian stretch/mongodb-enterprise/4.2 main" | tee /etc/apt/sources.list.d/mongodb-enterprise.list
RUN apt-get update
RUN apt-get install -y libmongocrypt-dev && apt-get install --no-install-recommends -y mongodb-enterprise-cryptd
I hope I helped. Cheers!
UPDATE:
My tests are running without libmongocrypt-dev lib, so I guess you only need mongodb-enterprise-cryptd.
I decided to use socat forwarding traffic
Dockerfile:
# Build stage
FROM ubuntu:focal
ENV ENTRY_FILE=docker-entrypoint.sh
ENV MONGODB_PATH=/usr/src/mongodb
ENV ENTRY_POINT=$MONGODB_PATH/$ENTRY_FILE
RUN apt-get update && apt-get install sudo
RUN sudo apt-get install -y curl telnet vim socat libcurl4 libgssapi-krb5-2 libldap-2.4-2 libwrap0 libsasl2-2 libsasl2-modules libsasl2-modules-gssapi-mit snmp openssl liblzma5
RUN curl -k -o mongodb.tgz "https://downloads.mongodb.com/linux/mongodb-linux-$(arch)-enterprise-ubuntu2004-5.0.10.tgz"
RUN tar -xf mongodb.tgz --strip-components=1
RUN sudo ln -s $MONGODB_PATH/bin/* /usr/local/bin/
RUN sudo mkdir -p /data/db
RUN sudo mkdir -p /data/log
RUN sudo chown `whoami` /data/db
RUN sudo chown `whoami` /data/log
COPY ./$ENTRY_FILE $ENTRY_POINT
RUN chmod +x $ENTRY_POINT
ENTRYPOINT $ENTRY_POINT
docker-entrypoint.sh:
#!/bin/sh
socat -d -d TCP-LISTEN:27017,fork,bind=$(hostname -I | awk '{print $1}') TCP:127.0.0.1:17017 &
socat -d -d TCP-LISTEN:27018,fork,bind=$(hostname -I | awk '{print $1}') TCP:127.0.0.1:17018 &
socat -d -d TCP-LISTEN:27019,fork,bind=$(hostname -I | awk '{print $1}') TCP:127.0.0.1:17019 &
socat -d -d TCP-LISTEN:27020,fork,bind=$(hostname -I | awk '{print $1}') TCP:127.0.0.1:17020 &
./bin/mongod --port 17017 --dbpath /data/db --logpath /data/log/mongod.log &
./bin/mongocryptd --port 17020 --logpath /data/log/mongocryptd.log
$(hostname -I | awk '{print $1}') is the remote ip(my docker host ip, 172.2.x.x), you can change to your another container ip.
docker-compose.yml:
version: "3.8"
networks:
ABC:
external: false
name: ABC
services:
mongo-database:
container_name: MongoDB
privileged: true
build:
context: .docker/db
dockerfile: Dockerfile
volumes:
- .mongo/db:/data/db
- .mongo/log:/data/log
ports:
- 27017-27020:27017-27020
networks:
- ABC
I have a scrapy application which I'm trying to containerized it. Basically, this is my docker-compose.yml file:
version: '3'
services:
scrapper:
container_name: scrapper
build: .
ports:
- 80:80
depends_on:
- db
links:
- db
db:
volumes:
- ./scrapper/sql:/docker-entrypoint-initdb.d
image: postgres
container_name: postgres
restart: always
ports:
- 5432:5432
And this is my Dockerfile:
FROM python:3
WORKDIR /usr/app
COPY requirements.txt .
RUN pip3 install -r requirements.txt
COPY . .
But when I try to execute my application using the following command: docker run -it scrapper_scrapper scrapy crawl angeloni, I'm receiving this message:
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 88, in crawl
yield self.engine.open_spider(self.spider, start_requests)
psycopg2.OperationalError: could not translate host name "db" to address: Name or service not known
Why this is happening? When I execute docker-compose ps command, it shows:
Name Command State Ports
--------------------------------------------------------------------------
postgres docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
scrapper python3 Exit 0
When running docker-compose up to start db, that container will run under its network that is also created by docker compose. As such, running docker run ... will not be able to connect to that instance, since it is not running on the same network. But you can specify it with:
docker run --network $network_name
To get the docker networks available, you can run:
docker networks ls
I think you have to explicitly define a user network and put your containers on it:
https://docs.docker.com/network/bridge/
Under the section:
User-defined bridges provide automatic DNS resolution between containers.
I have an image (gepick:latest) with node app created from Dockerfile:
FROM centos:7
# Create app directory
WORKDIR /usr/src/app
RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
RUN yum install -y nodejs
RUN curl --silent --location https://dl.yarnpkg.com/rpm/yarn.repo | tee /etc/yum.repos.d/yarn.repo
RUN rpm --import https://dl.yarnpkg.com/rpm/pubkey.gpg
RUN yum install -y yarn
RUN yarn
COPY . .
EXPOSE 8080
CMD [ "yarn", "test-matches-collecting-job"]
My goal is run tests in docker. But it requires mongodb
docker run gepick:latest :
...
Mongoose default connection error: MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]
...
I tried link mongo:4 images container docker run --link 0d24c3a35d5a gepick:latest but get same error.
When you launch your container using a docker-compose yaml file Docker bridges the containers together and allows you to have it launch the mongo container before other containers which rely on mongo to be active ... try something like this
cat my-docker-compose.yml
version: '3'
services:
my-gepick:
image: gepick:latest
container_name: blah_gepick
restart: always
depends_on:
- loudmongo
volumes:
- /cryptdata5/var/log/blobs:/blobs
- /webapp/enduser/bundle:/tmp
environment:
- MONGO_SERVICE_HOST=loudmongo
- MONGO_SERVICE_PORT=$GKE_MONGO_PORT
- MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
- METEOR_SETTINGS=${METEOR_SETTINGS}
- MAIL_URL=smtp://support#${GKE_DOMAIN_NAME}:blah#loudmail:587/
links:
- loudmongo
ports:
- 127.0.0.1:3000:3000
working_dir: /tmp
command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
loudmongo:
image: mongo
container_name: loud_mongo
restart: always
ports:
- 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
volumes:
- /cryptdata7/var/data/db:/data/db
so your launch sequence may look like
docker-compose -f /somedir/my-docker-compose.yml pull
docker-compose -f /somedir/my-docker-compose.yml up -d
I am using postgresql with django in my project. I've got them in different containers and the problem is that i need to wait for postgres before running django. At this time i am doing it with sleep 5 in command.sh file for django container. I also found that netcat can do the trick but I would prefer way without additional packages. curl and wget can't do this because they do not support postgres protocol.
Is there a way to do it?
I've spent some hours investigating this problem and I got a solution.
Docker depends_on just consider service startup to run another service. Than it happens because as soon as db is started, service-app tries to connect to ur db, but it's not ready to receive connections. So you can check db health status in app service to wait for connection. Here is my solution, it solved my problem. :)
Important: I'm using docker-compose version 2.1.
version: '2.1'
services:
my-app:
build: .
command: su -c "python manage.py runserver 0.0.0.0:8000"
ports:
- "8000:8000"
depends_on:
db:
condition: service_healthy
links:
- db
volumes:
- .:/app_directory
db:
image: postgres:10.5
ports:
- "5432:5432"
volumes:
- database:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
volumes:
database:
In this case it's not necessary to create a .sh file.
This will successfully wait for Postgres to start. (Specifically line 6). Just replace npm start with whatever command you'd like to happen after Postgres has started.
services:
practice_docker:
image: dockerhubusername/practice_docker
ports:
- 80:3000
command: bash -c 'while !</dev/tcp/db/5432; do sleep 1; done; npm start'
depends_on:
- db
environment:
- DATABASE_URL=postgres://postgres:password#db:5432/practicedocker
- PORT=3000
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=password
- POSTGRES_DB=practicedocker
If you have psql you could simply add the following code to your .sh file:
RETRIES=5
until psql -h $PG_HOST -U $PG_USER -d $PG_DATABASE -c "select 1" > /dev/null 2>&1 || [ $RETRIES -eq 0 ]; do
echo "Waiting for postgres server, $((RETRIES--)) remaining attempts..."
sleep 1
done
The simplest solution is a short bash script:
while ! nc -z HOST PORT; do sleep 1; done;
./run-smth-else;
Problem with your solution tiziano is that curl is not installed by default and i wanted to avoid installing additional stuff. Anyway i did what bereal said. Here is the script if anyone would need it.
import socket
import time
import os
port = int(os.environ["DB_PORT"]) # 5432
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
while True:
try:
s.connect(('myproject-db', port))
s.close()
break
except socket.error as ex:
time.sleep(0.1)
In your Dockerfile add wait and change your start command to use it:
ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.7.3/wait /wait
RUN chmod +x /wait
CMD /wait && npm start
Then, in your docker-compose.yml add a WAIT_HOSTS environment variable for your api service:
services:
api:
depends_on:
- postgres
environment:
- WAIT_HOSTS: postgres:5432
postgres:
image: postgres
ports:
- "5432:5432"
This has the advantage that it supports waiting for multiple services:
environment:
- WAIT_HOSTS: postgres:5432, mysql:3306, mongo:27017
For more details, please read their documentation.
wait-for-it small wrapper scripts which you can include in your application’s image to poll a given host and port until it’s accepting TCP connections.
can be cloned in Dockerfile by below command
RUN git clone https://github.com/vishnubob/wait-for-it.git
docker-compose.yml
version: "2"
services:
web:
build: .
ports:
- "80:8000"
depends_on:
- "db"
command: ["./wait-for-it/wait-for-it.sh", "db:5432", "--", "npm", "start"]
db:
image: postgres
Why not curl?
Something like this:
while ! curl http://$POSTGRES_PORT_5432_TCP_ADDR:$POSTGRES_PORT_5432_TCP_PORT/ 2>&1 | grep '52'
do
sleep 1
done
It works for me.
I have managed to solve my issue by adding health check to docker-compose definition.
db:
image: postgres:latest
ports:
- 5432:5432
healthcheck:
test: "pg_isready --username=postgres && psql --username=postgres --list"
timeout: 10s
retries: 20
then in the dependent service you can check the health status:
my-service:
image: myApp:latest
depends_on:
kafka:
condition: service_started
db:
condition: service_healthy
source: https://docs.docker.com/compose/compose-file/compose-file-v2/#healthcheck
If the backend application itself has a PostgreSQL client, you can use the pg_isready command in an until loop. For example, suppose we have the following project directory structure,
.
├── backend
│ └── Dockerfile
└── docker-compose.yml
with a docker-compose.yml
version: "3"
services:
postgres:
image: postgres
backend:
build: ./backend
and a backend/Dockerfile
FROM alpine
RUN apk update && apk add postgresql-client
CMD until pg_isready --username=postgres --host=postgres; do sleep 1; done \
&& psql --username=postgres --host=postgres --list
where the 'actual' command is just a psql --list for illustration. Then running docker-compose build and docker-compose up will give you the following output:
Note how the result of the psql --list command only appears after pg_isready logs postgres:5432 - accepting connections as desired.
By contrast, I have found that the nc -z approach does not work consistently. For example, if I replace the backend/Dockerfile with
FROM alpine
RUN apk update && apk add postgresql-client
CMD until nc -z postgres 5432; do echo "Waiting for Postgres..." && sleep 1; done \
&& psql --username=postgres --host=postgres --list
then docker-compose build followed by docker-compose up gives me the following result:
That is, the psql command throws a FATAL error that the database system is starting up.
In short, using an until pg_isready loop (as also recommended here) is the preferable approach IMO.
There are couple of solutions as other answers mentioned.
But don't make it complicated, just let it fail-fast combined with restart: on-failure. Your service will open connection to the db and may fail at the first time. Just let it fail. Docker will restart your service until it green. Keep your service simple and business-focused.
version: '3.7'
services:
postgresdb:
hostname: postgresdb
image: postgres:12.2
ports:
- "5432:5432"
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=secret
- POSTGRES_DB=Ceo
migrate:
image: hanh/migration
links:
- postgresdb
environment:
- DATA_SOURCE=postgres://user:secret#postgresdb:5432/Ceo
command: migrate sql --yes
restart: on-failure # will restart until it's success
Check out restart policies.
None of other solution worked, except for the following:
version : '3.8'
services :
postgres :
image : postgres:latest
environment :
- POSTGRES_DB=mydbname
- POSTGRES_USER=myusername
- POSTGRES_PASSWORD=mypassword
healthcheck :
test: [ "CMD", "pg_isready", "-q", "-d", "mydbname", "-U", "myusername" ]
interval : 5s
timeout : 5s
retries : 5
otherservice:
image: otherserviceimage
depends_on :
postgres:
condition: service_healthy
Thanks to this thread: https://github.com/peter-evans/docker-compose-healthcheck/issues/16
Sleeping until pg_isready returns true unfortunately is not always reliable. If your postgres container has at least one initdb script specified, postgres restarts after it is started during it's bootstrap procedure, and so it might not be ready yet even though pg_isready already returned true.
What you can do instead, is to wait until docker logs for that instance return a PostgreSQL init process complete; ready for start up. string, and only then proceed with the pg_isready check.
Example:
start_postgres() {
docker-compose up -d --no-recreate postgres
}
wait_for_postgres() {
until docker-compose logs | grep -q "PostgreSQL init process complete; ready for start up." \
&& docker-compose exec -T postgres sh -c "PGPASSWORD=\$POSTGRES_PASSWORD PGUSER=\$POSTGRES_USER pg_isready --dbname=\$POSTGRES_DB" > /dev/null 2>&1; do
printf "\rWaiting for postgres container to be available ... "
sleep 1
done
printf "\rWaiting for postgres container to be available ... done\n"
}
start_postgres
wait_for_postgres
You can use the manage.py command "check" to check if the database is available (and wait 2 seconds if not, and check again).
For instance, if you do this in your command.sh file before running the migration, Django has a valid DB connection while running the migration command:
...
echo "Waiting for db.."
python manage.py check --database default > /dev/null 2> /dev/null
until [ $? -eq 0 ];
do
sleep 2
python manage.py check --database default > /dev/null 2> /dev/null
done
echo "Connected."
# Migrate the last database changes
python manage.py migrate
...
PS: I'm not a shell expert, please suggest improvements.
#!/bin/sh
POSTGRES_VERSION=9.6.11
CONTAINER_NAME=my-postgres-container
# start the postgres container
docker run --rm \
--name $CONTAINER_NAME \
-e POSTGRES_PASSWORD=docker \
-d \
-p 5432:5432 \
postgres:$POSTGRES_VERSION
# wait until postgres is ready to accept connections
until docker run \
--rm \
--link $CONTAINER_NAME:pg \
postgres:$POSTGRES_VERSION pg_isready \
-U postgres \
-h pg; do sleep 1; done
An example for Nodejs and Postgres api.
#!/bin/bash
#entrypoint.dev.sh
echo "Waiting for postgres to get up and running..."
while ! nc -z postgres_container 5432; do
# where the postgres_container is the hos, in my case, it is a Docker container.
# You can use localhost for example in case your database is running locally.
echo "waiting for postgress listening..."
sleep 0.1
done
echo "PostgreSQL started"
yarn db:migrate
yarn dev
# Dockerfile
FROM node:12.16.2-alpine
ENV NODE_ENV="development"
RUN mkdir -p /app
WORKDIR /app
COPY ./package.json ./yarn.lock ./
RUN yarn install
COPY . .
CMD ["/bin/sh", "./entrypoint.dev.sh"]
If you want to run it with a single line command. You can just connect to the container and check if postgres is running
docker exec -it $DB_NAME bash -c "\
until psql -h $HOST -U $USER -d $DB_NAME-c 'select 1'>/dev/null 2>&1;\
do\
echo 'Waiting for postgres server....';\
sleep 1;\
done;\
exit;\
"
echo "DB Connected !!"
Inspired by #tiziano answer and the lack of nc or pg_isready, it seems that in a recent docker python image (python:3.9 here) that curl is installed by default and I have the following check running in my entrypoint.sh:
postgres_ready() {
$(which curl) http://$DBHOST:$DBPORT/ 2>&1 | grep '52'
}
until postgres_ready; do
>&2 echo 'Waiting for PostgreSQL to become available...'
sleep 1
done
>&2 echo 'PostgreSQL is available.'
Trying with a lot of methods, Dockerfile, docker compose yaml, bash script. Only last of method help me: with makefile.
docker-compose up --build -d postgres
sleep 2
docker-compose up --build -d app