I want to run a webapp with docker-compose. I have a centos7 host running postgresql and the docker-engine. My docker-compose includes a postgresql-image and should run with the PGDATA from the host system. But everytime I run the docker-compose I get the error:
initdb: directory "/var/lib/docker-postgresql" exists but is not empty
The docker-compose part for the postgresql-database looks like:
db:
build: ./postgres/
container_name: ps01
volumes:
- ./postgres:/tmp
- /var/lib/pgsql/9.4:/var/lib/docker-postgresql
expose:
- "5432"
environment:
PGDATA: /var/lib/docker-postgresql
I mount the /var/lib/pgsql/9.4 inside the Docker-Postgresql-App to /var/lib/docker-postgresql and set this path to the PGDATA-ENV.
The Dockerfile in ./postgres/ looks like:
FROM postgres:latest
ENV POSTGIS_MAJOR 2.3
ENV POSTGIS_VERSION 2.3.1+dfsg-1.pgdg80+1
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION \
postgis=$POSTGIS_VERSION \
&& rm -rf /var/lib/apt/lists/*
What should I do to share my Postgres-Data from the host?
Related
I have an application in which I use a postgresql and celery database. Each element is running in a different container, in the celery container I am already connected to the postgres database, however I don't know how I could configure tortoise-orm to start in the celery container, since I have a task in which I want to interact with the database using tortoise.
This is my docker compose:
version: '3.8'
services:
web:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: fastapi_celery_example_web
# '/start' is the shell script used to run the service
command: /start
# this volume is used to map the files and folders on the host to the container
# so if we change code on the host, code in the docker container will also be changed
volumes:
- .:/app
ports:
- 8010:8000
env_file:
- .env/.dev-sample
depends_on:
- redis
- db
db:
image: postgres:14-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_DB=fastapi_celery
- POSTGRES_USER=fastapi_celery
- POSTGRES_PASSWORD=fastapi_celery
redis:
image: redis:7-alpine
celery_worker:
build:
context: .
dockerfile: ./compose/local/fastapi/Dockerfile
image: fastapi_celery_example_celery_worker
command: /start-celeryworker
volumes:
- .:/app
env_file:
- .env/.dev-sample
depends_on:
- redis
- db
This is my dockerfile:
FROM python:3.10-slim-buster
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
RUN apt-get update \
# dependencies for building Python packages
&& apt-get install -y build-essential \
# psycopg2 dependencies
&& apt-get install -y libpq-dev \
# Additional dependencies
&& apt-get install -y telnet netcat \
# cleaning up unused files
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
&& rm -rf /var/lib/apt/lists/*
# Requirements are installed here to ensure they will be cached.
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
COPY ./compose/local/fastapi/entrypoint /entrypoint
RUN sed -i 's/\r$//g' /entrypoint
RUN chmod +x /entrypoint
COPY ./compose/local/fastapi/start /start
RUN sed -i 's/\r$//g' /start
RUN chmod +x /start
COPY ./compose/local/fastapi/celery/worker/start /start-celeryworker
RUN sed -i 's/\r$//g' /start-celeryworker
RUN chmod +x /start-celeryworker
COPY ./compose/local/fastapi/celery/beat/start /start-celerybeat
RUN sed -i 's/\r$//g' /start-celerybeat
RUN chmod +x /start-celerybeat
COPY ./compose/local/fastapi/celery/flower/start /start-flower
RUN sed -i 's/\r$//g' /start-flower
RUN chmod +x /start-flower
WORKDIR /app
ENTRYPOINT ["/entrypoint"]
The task:
#shared_task()
def task_send_welcome_email(user_pk):
from project.users.models import User
user = User.filter(id=user_pk).first()
logger.info(f'send email to {user.email} {user.id}')
I tried installing pg_cron on Postgres running inside a Docker container but getting this error could not access file "pg_cron": No such file or directory. Any ideas on how to resolve?
Based on https://stackoverflow.com/a/51797554, I tried the following:
docker-compose.yml
version: '3.7'
services:
pg:
container_name: pg-container
image: postgres:11.5
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
- pgstorage
ports:
- "5432:5432"
volumes:
pgstorage:
002-setup.sh
#!/bin/sh
# Remove last line "shared_preload_libraries='citus'"
sed -i '$ d' ${PGDATA}/postgresql.conf
cat <<EOT >> ${PGDATA}/postgresql.conf
shared_preload_libraries='pg_cron'
cron.database_name='${POSTGRES_DB:-postgres}'
EOT
# Required to load pg_cron
pg_ctl restart
003-main.sql
CREATE EXTENSION pg_cron;
From what I can see you are not installing pg_cron anywhere. Since it is not packaged with the default Postgres Docker image you will have to care of that.
For example by extending the Image and using a build entry in your docker-compose.yml.
# Dockerfile relative to docker-compose.yml
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
version: '3.7'
services:
pg:
container_name: pg-container
build: .
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
This worked for me - it probably needs some more optimization.
The proposed solution didn't work with a newly created container for me. So, I did it like this:
Docker file
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
RUN cd / && \
rm -rf /pg_cron && \
apt-get remove -y git build-essential postgresql-server-dev-11 && \
apt-get autoremove --purge -y && \
apt-get clean && \
apt-get purge
COPY init-db /docker-entrypoint-initdb.d
init-db/pg-cron.sh
#!/usr/bin/env bash
# use same db as the one from env
dbname="$POSTGRES_DB"
# create custom config
customconf=/var/lib/postgresql/data/custom-conf.conf
echo "" > $customconf
echo "shared_preload_libraries = 'pg_cron'" >> $customconf
echo "cron.database_name = '$dbname'" >> $customconf
chown postgres $customconf
chgrp postgres $customconf
# include custom config from main config
conf=/var/lib/postgresql/data/postgresql.conf
found=$(grep "include = '$customconf'" $conf)
if [ -z "$found" ]; then
echo "include = '$customconf'" >> $conf
fi
Also, you can place other init files into init-db directory.
Docker compose file
version: '3.7'
services:
postgres:
container_name: your-container
build: .
environment:
POSTGRES_DB: "your_db"
POSTGRES_USER: "your_user"
POSTGRES_PASSWORD: "your_user"
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
pgdata:
driver: local
For those who are looking for a ready image, please try the following:
docker pull ramazanpolat/postgres_cron:11
I have this docker-compose
version: '3.7'
services:
app-db:
build:
context: ./
dockerfile: Dockerfile-pg
image: app-pg:1.0.0
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app
volumes:
- ./docker-entrypoint-initdb.d/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh
- v-app-pgdata:/var/lib/postgresql
- v-app-pglog:/data/var/log/postgresql
- v-app-pgconf:/etc/postgresql
app-main:
build:
context: ./
dockerfile: Dockerfile-tar-cp
image: app-main:1.0.0
restart: always
ports:
- 80:80
volumes:
v-app-pgdata:
name: v-app-pgdata
v-app-pglog:
name: v-app-pglog
v-app-pgconf:
name: v-app-pgconf
so an app container and a postgres derived container:
#docker build -t app-pg:1.0.0 -f Dockerfile-pg .
#docker run -d --name appC-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres app-pg:1.0.0
FROM postgres:12.1
MAINTAINER xxx
#ARG A_DB_USER='postgres'
#ARG A_DB_PASS='postgres'
ARG A_DB_NAME='app'
ARG A_TZ='Europe/Zurich'
#ENV DB_USER=${A_DB_USER}
#ENV DB_PASS=${A_DB_PASS}
ENV DB_NAME=${A_DB_NAME}
ENV TZ=${A_TZ}
# Adjusting Timezone in the system
RUN echo $TZ > /etc/timezone && \
apt-get update && apt-get install -y tzdata && \
rm /etc/localtime && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
# install postgis
RUN apt-get update && \
apt-get install -y postgis && \
apt-get clean
USER postgres
#Add password "postgres" to user postgres, create db, add .sql
#RUN /etc/init.d/postgresql start && \
# psql --command "ALTER USER ${DB_USER} WITH PASSWORD '${DB_PASS}'; SET TIME ZONE '${TZ}';" && \
# createdb -O ${DB_USER} ${DB_NAME} -E UTF8 && \
# psql -d ${DB_NAME} -c 'CREATE EXTENSION postgis'
EXPOSE 5432
My problem is that postgres default dockerfile have this line:
VOLUME /var/lib/postgresql/data
So even if I create a named Volumes with the same folder, my docker-compose create 4 and not 3 volumes, one unnamed due to that line.
How is it possible solve this issue?
I had the same problem as you and I solved it by editing the docker-compose.yml the following way:
Create a volume for the data
volumes:
v-app-pgdata:
In your declaration of the database service, in the volumes clause, you need to change - v-app-pgdata:/var/lib/postgresql to:
- v-app-pgdata:/var/lib/postgresql/data
The problem was that you where not mounting the volume to the correct place, and therefore, it had to create another volume that is declared in the base image VOLUME /var/lib/postgresql/data. If you run this container without attaching a volume to this mount point, it will automatically create it, and with a random name.
However, it has been pointed out to me that if the OP strictly needs a volume in /var/lib/postgresql then this solution wont work.
Hope this helps, as it worked for me.
Trying to dockerise my Symfony 4 app, running with PostgreSQL
But when I'm running :
$ sudo docker-compose build
I'm having this error :
In AbstractPostgreSQLDriver.php line 73:
An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server : Connection refused :
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Here's my docker-compose.yml file :
version: '3.7'
services:
db:
image: ${POSTGRES_IMAGE}
restart: always
environment:
POSTGRES_DB: ${DATABASE_NAME}
POSTGRES_USER: ${DATABASE_USER}
POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
php:
build:
context: .
dockerfile: ./docker/php/Dockerfile
restart: on-failure
user: 1000:1000
volumes:
- ./docker/php/uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
- .:/var/www/symfony
working_dir: /var/www/symfony
depends_on:
- db
Content of my .env file
DATABASE_NAME=db
DATABASE_HOST=127.0.0.1
DATABASE_PORT=5432
DATABASE_USER=postgres
DATABASE_PASSWORD=root
## Docker images (name and version)
PHP_IMAGE=php:7.3-fpm
POSTGRES_IMAGE=postgres
Also FYI :
$ sudo docker-compose config
debian#debian:~/dev/symfony$ sudo docker-compose config
services:
db:
environment:
POSTGRES_DB: mrd
POSTGRES_PASSWORD: root
POSTGRES_USER: postgres
image: postgres
restart: always
php:
build:
context: /home/debian/dev/symfony
dockerfile: ./docker/php/Dockerfile
depends_on:
- db
restart: on-failure
user: 1000:1000
volumes:
- /home/debian/dev/symfony:/var/www/symfony:rw
working_dir: /var/www/symfony
version: '3.7'
My Dockerfile:
FROM php:7.3-fpm
WORKDIR /var/www/symfony
RUN apt-get update
# Install Postgres PDO
RUN apt-get install -y libpq-dev \
&& apt-get install -y zip \
&& docker-php-ext-install pgsql pdo_pgsql \
&& apt-get install -y git
RUN pecl install apcu
RUN php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" \
&& php -r "if (hash_file('SHA384', 'composer-setup.php') === 'a5c698ffe4b8e849a443b120cd5ba38043260d5c4023dbf93e1558871f1f07f58274fc6f4c93bcfd858c6bd0775cd8d1') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" \
&& php composer-setup.php --filename=composer \
&& php -r "unlink('composer-setup.php');" \
&& mv composer /usr/local/bin/composer
RUN ls -larth
COPY . /var/www/symfony
RUN PATH=$PATH:/var/www/symfony/vendor/bin:bin
RUN pwd \
&& ls \
&& composer install --no-interaction --no-ansi --optimize-autoloader\
&& php bin/console doctrine:database:create \
&& php bin/console doctrine:schema:update --no-interaction \
&& php bin/console doctrine:fixtures:load --no-interaction
Has anybody a clue on why it does this ? And how to solve it ?
Searched a lot, and couldn't find anything that worked. Thought the depends_on: db would do the trick, but no.
Try to run:
docker-compose ps
see Container name
Set DATABASE_HOST=Container name
DATABASE_NAME=db
DATABASE_HOST=## paste Container name
DATABASE_PORT=5432
DATABASE_USER=postgres
DATABASE_PASSWORD=root
## Docker images (name and version)
PHP_IMAGE=php:7.3-fpm
POSTGRES_IMAGE=postgres
The pastes in your question indicate a number of things to be addressed with your docker-compose.yml file:
You are missing an env_file: entry for your .env file
You are missing an image for the php service
You specify . as your build context (which should be a directory that indicates where your Dockerfile is located)
You specify a path as your Dockerfile (which should be a filename, not a directory) -- I guess this might all work, but it's a bit confusing to read
Indentation for -db and - /home/debian/dev/symfony:/var/www/symfony:rw is off -- this might not be a problem, but it's still difficult to read
From your question about build failing with an error regarding inability to connect to a database -- could you update your question and share your Dockerfile for symfony? I suspect that you need to remove a reference to a database connection.
I want to start using Flyway to version our database changes. I am trying to create a Postgres Docker container with seeded data, TAG and publish that docker image to be used in automated testing.
I tried using docker-compose, however I haven't figured a way to TAG and publish after Flyway runs.
Repository with test project
https://github.com/bigboy1122/flyway_postgres
Here is a docker-compose I created
version: '3.7'
services:
flyway:
image: boxfuse/flyway
restart: always
command: -url=jdbc:postgresql://db:5432/foo -user='postgres' -password='test123' -schemas='bar' migrate
volumes:
- .:/flyway/sql
depends_on:
- db
db:
image: tgalati1122/flyway_seeded_postgres
build:
context: .
dockerfile: ./Dockerfile
restart: always
environment:
POSTGRES_PASSWORD: 'test123'
POSTGRES_USER: 'postgres'
POSTGRES_DB: 'foo'
ports:
- 5432:5432
adminer:
image: adminer
restart: always
ports:
- 8080:8080
depends_on:
- db
Here is trying to use Docker multi build feature.
In the example below, the database I think spins up but I can't get flyway to access it.
FROM postgres:10.5-alpine as donar
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
RUN /docker-entrypoint.sh --help
FROM debian:stretch-slim as build_tools
ENV FLWAY_VERSION='5.2.0
RUN set -ex; \
if ! command -v gpg > /dev/null; then \
apt-get update; \
apt-get install -y --no-install-recommends \
gnupg \
dirmngr \
wget \
; \
rm -rf /var/lib/apt/lists/*; \
fi
VOLUME flyway/sql
RUN wget --no-check-certificate https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/5.2.0/flyway-commandline-5.2.0-linux-x64.tar.gz -O - | tar -xz
RUN pwd; \
ls -l; \
cd flyway-5.2.0; \
pwd; \
ls -l; \
sh ./flyway -url=jdbc:postgresql://localhost:5432/optins -user='postgres' -password='test123' -schemas='bar' migrate; \
FROM postgres:10.5-alpine
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
COPY --chown=postgres:postgres --from=donor /pgdata /pgdata
The idea I am going for as database changes occur, I want to automatically build a new lightweight test database as well as update the persisted databases throughout the enterprise.