Rename volume in docker compose, with docker derived from postgres - postgresql

I have this docker-compose
version: '3.7'
services:
app-db:
build:
context: ./
dockerfile: Dockerfile-pg
image: app-pg:1.0.0
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app
volumes:
- ./docker-entrypoint-initdb.d/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh
- v-app-pgdata:/var/lib/postgresql
- v-app-pglog:/data/var/log/postgresql
- v-app-pgconf:/etc/postgresql
app-main:
build:
context: ./
dockerfile: Dockerfile-tar-cp
image: app-main:1.0.0
restart: always
ports:
- 80:80
volumes:
v-app-pgdata:
name: v-app-pgdata
v-app-pglog:
name: v-app-pglog
v-app-pgconf:
name: v-app-pgconf
so an app container and a postgres derived container:
#docker build -t app-pg:1.0.0 -f Dockerfile-pg .
#docker run -d --name appC-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres app-pg:1.0.0
FROM postgres:12.1
MAINTAINER xxx
#ARG A_DB_USER='postgres'
#ARG A_DB_PASS='postgres'
ARG A_DB_NAME='app'
ARG A_TZ='Europe/Zurich'
#ENV DB_USER=${A_DB_USER}
#ENV DB_PASS=${A_DB_PASS}
ENV DB_NAME=${A_DB_NAME}
ENV TZ=${A_TZ}
# Adjusting Timezone in the system
RUN echo $TZ > /etc/timezone && \
apt-get update && apt-get install -y tzdata && \
rm /etc/localtime && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
# install postgis
RUN apt-get update && \
apt-get install -y postgis && \
apt-get clean
USER postgres
#Add password "postgres" to user postgres, create db, add .sql
#RUN /etc/init.d/postgresql start && \
# psql --command "ALTER USER ${DB_USER} WITH PASSWORD '${DB_PASS}'; SET TIME ZONE '${TZ}';" && \
# createdb -O ${DB_USER} ${DB_NAME} -E UTF8 && \
# psql -d ${DB_NAME} -c 'CREATE EXTENSION postgis'
EXPOSE 5432
My problem is that postgres default dockerfile have this line:
VOLUME /var/lib/postgresql/data
So even if I create a named Volumes with the same folder, my docker-compose create 4 and not 3 volumes, one unnamed due to that line.
How is it possible solve this issue?

I had the same problem as you and I solved it by editing the docker-compose.yml the following way:
Create a volume for the data
volumes:
v-app-pgdata:
In your declaration of the database service, in the volumes clause, you need to change - v-app-pgdata:/var/lib/postgresql to:
- v-app-pgdata:/var/lib/postgresql/data
The problem was that you where not mounting the volume to the correct place, and therefore, it had to create another volume that is declared in the base image VOLUME /var/lib/postgresql/data. If you run this container without attaching a volume to this mount point, it will automatically create it, and with a random name.
However, it has been pointed out to me that if the OP strictly needs a volume in /var/lib/postgresql then this solution wont work.
Hope this helps, as it worked for me.

Related

Installing and using pg_cron extension on Postgres running inside of Docker container

I tried installing pg_cron on Postgres running inside a Docker container but getting this error could not access file "pg_cron": No such file or directory. Any ideas on how to resolve?
Based on https://stackoverflow.com/a/51797554, I tried the following:
docker-compose.yml
version: '3.7'
services:
pg:
container_name: pg-container
image: postgres:11.5
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
- pgstorage
ports:
- "5432:5432"
volumes:
pgstorage:
002-setup.sh
#!/bin/sh
# Remove last line "shared_preload_libraries='citus'"
sed -i '$ d' ${PGDATA}/postgresql.conf
cat <<EOT >> ${PGDATA}/postgresql.conf
shared_preload_libraries='pg_cron'
cron.database_name='${POSTGRES_DB:-postgres}'
EOT
# Required to load pg_cron
pg_ctl restart
003-main.sql
CREATE EXTENSION pg_cron;
From what I can see you are not installing pg_cron anywhere. Since it is not packaged with the default Postgres Docker image you will have to care of that.
For example by extending the Image and using a build entry in your docker-compose.yml.
# Dockerfile relative to docker-compose.yml
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
version: '3.7'
services:
pg:
container_name: pg-container
build: .
environment:
POSTGRES_DB: "pgdb"
POSTGRES_USER: "pguser"
POSTGRES_PASSWORD: "pgpass"
volumes:
- ./:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
This worked for me - it probably needs some more optimization.
The proposed solution didn't work with a newly created container for me. So, I did it like this:
Docker file
FROM postgres:11.5
RUN apt-get update && apt-get -y install git build-essential postgresql-server-dev-11
RUN git clone https://github.com/citusdata/pg_cron.git
RUN cd pg_cron && make && make install
RUN cd / && \
rm -rf /pg_cron && \
apt-get remove -y git build-essential postgresql-server-dev-11 && \
apt-get autoremove --purge -y && \
apt-get clean && \
apt-get purge
COPY init-db /docker-entrypoint-initdb.d
init-db/pg-cron.sh
#!/usr/bin/env bash
# use same db as the one from env
dbname="$POSTGRES_DB"
# create custom config
customconf=/var/lib/postgresql/data/custom-conf.conf
echo "" > $customconf
echo "shared_preload_libraries = 'pg_cron'" >> $customconf
echo "cron.database_name = '$dbname'" >> $customconf
chown postgres $customconf
chgrp postgres $customconf
# include custom config from main config
conf=/var/lib/postgresql/data/postgresql.conf
found=$(grep "include = '$customconf'" $conf)
if [ -z "$found" ]; then
echo "include = '$customconf'" >> $conf
fi
Also, you can place other init files into init-db directory.
Docker compose file
version: '3.7'
services:
postgres:
container_name: your-container
build: .
environment:
POSTGRES_DB: "your_db"
POSTGRES_USER: "your_user"
POSTGRES_PASSWORD: "your_user"
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
pgdata:
driver: local
For those who are looking for a ready image, please try the following:
docker pull ramazanpolat/postgres_cron:11

Using Flyway in CI/CD to populate/seed a Postgres DB in Docker then TAG and publish the new docker image to be used for testing

I want to start using Flyway to version our database changes. I am trying to create a Postgres Docker container with seeded data, TAG and publish that docker image to be used in automated testing.
I tried using docker-compose, however I haven't figured a way to TAG and publish after Flyway runs.
Repository with test project
https://github.com/bigboy1122/flyway_postgres
Here is a docker-compose I created
version: '3.7'
services:
flyway:
image: boxfuse/flyway
restart: always
command: -url=jdbc:postgresql://db:5432/foo -user='postgres' -password='test123' -schemas='bar' migrate
volumes:
- .:/flyway/sql
depends_on:
- db
db:
image: tgalati1122/flyway_seeded_postgres
build:
context: .
dockerfile: ./Dockerfile
restart: always
environment:
POSTGRES_PASSWORD: 'test123'
POSTGRES_USER: 'postgres'
POSTGRES_DB: 'foo'
ports:
- 5432:5432
adminer:
image: adminer
restart: always
ports:
- 8080:8080
depends_on:
- db
Here is trying to use Docker multi build feature.
In the example below, the database I think spins up but I can't get flyway to access it.
FROM postgres:10.5-alpine as donar
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
RUN /docker-entrypoint.sh --help
FROM debian:stretch-slim as build_tools
ENV FLWAY_VERSION='5.2.0
RUN set -ex; \
if ! command -v gpg > /dev/null; then \
apt-get update; \
apt-get install -y --no-install-recommends \
gnupg \
dirmngr \
wget \
; \
rm -rf /var/lib/apt/lists/*; \
fi
VOLUME flyway/sql
RUN wget --no-check-certificate https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/5.2.0/flyway-commandline-5.2.0-linux-x64.tar.gz -O - | tar -xz
RUN pwd; \
ls -l; \
cd flyway-5.2.0; \
pwd; \
ls -l; \
sh ./flyway -url=jdbc:postgresql://localhost:5432/optins -user='postgres' -password='test123' -schemas='bar' migrate; \
FROM postgres:10.5-alpine
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
COPY --chown=postgres:postgres --from=donor /pgdata /pgdata
The idea I am going for as database changes occur, I want to automatically build a new lightweight test database as well as update the persisted databases throughout the enterprise.

How to disable docker container restart

Postgres docker is restarting with changed name after stopping it.
How to disable restart?
I've tried
docker update --restart=no my-container-ID
but when i stop container its starting again with new Container ID
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
53e52dfc9015 postgres:latest "docker-entrypoint.s…" 5 hours ago Up 5 hours 5432/tcp startmarketplace_db.1.o2i5ig3cn0tba5a64r4vkrb8n
$docker stop 53e52dfc9015
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a75d1587c66d postgres:latest "docker-entrypoint.s…" 46 seconds ago Up 39 seconds 5432/tcp startmarketplace_db.1.5ukdrwdo1bc0tssf4rzdkjrta
Source code of Dockerfile:
FROM php:7.2-apache
RUN apt-get update \
&& apt-get install -y \
curl git unzip vim \
libpng-dev libpq-dev \
&& docker-php-ext-install gd pdo pdo_pgsql pgsql
# Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# xDebug
RUN yes | pecl install xdebug \
&& echo "zend_extension=$(find /usr/local/lib/php/extensions/ -name xdebug.so)" > /usr/local/etc/php/conf.d/xdebug.ini \
&& echo "xdebug.remote_enable=on" >> /usr/local/etc/php/conf.d/xdebug.ini \
&& echo "xdebug.remote_autostart=on" >> /usr/local/etc/php/conf.d/xdebug.ini
# PHP
ADD ./php.ini /usr/local/etc/php
# Apache
ADD ./virtualhost.conf /etc/apache2/sites-enabled
RUN a2enmod rewrite
COPY ./entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Source code of docker-compose.yml:
version: '3.1'
services:
web:
build: ./xxx
ports:
- "9001:80"
volumes:
- ./app/xxx:/var/www/html
environment:
XDEBUG_CONFIG: >
remote_host=172.18.0.1
idekey=xxx
PHP_IDE_CONFIG: serverName=xxx
links:
- db
db:
image: postgres
environment:
POSTGRES_DB: xxx
POSTGRES_USER: xxx
POSTGRES_PASSWORD: xxx
ports:
- "5432:5432"
That's running in swarm mode. You need to stop the service or remove the entire stack.
For just the service:
docker service rm startmarketplace_db
For the entire stack:
docker stack rm startmarketplace

PostgreSQL PGDATA from host in Docker-System

I want to run a webapp with docker-compose. I have a centos7 host running postgresql and the docker-engine. My docker-compose includes a postgresql-image and should run with the PGDATA from the host system. But everytime I run the docker-compose I get the error:
initdb: directory "/var/lib/docker-postgresql" exists but is not empty
The docker-compose part for the postgresql-database looks like:
db:
build: ./postgres/
container_name: ps01
volumes:
- ./postgres:/tmp
- /var/lib/pgsql/9.4:/var/lib/docker-postgresql
expose:
- "5432"
environment:
PGDATA: /var/lib/docker-postgresql
I mount the /var/lib/pgsql/9.4 inside the Docker-Postgresql-App to /var/lib/docker-postgresql and set this path to the PGDATA-ENV.
The Dockerfile in ./postgres/ looks like:
FROM postgres:latest
ENV POSTGIS_MAJOR 2.3
ENV POSTGIS_VERSION 2.3.1+dfsg-1.pgdg80+1
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts=$POSTGIS_VERSION \
postgis=$POSTGIS_VERSION \
&& rm -rf /var/lib/apt/lists/*
What should I do to share my Postgres-Data from the host?

How to persist data using a postgres database, Docker, and Kubernetes?

I am trying to mount a persistent disk on my container which runs a Postgres custom image. I am using Kubernetes and following this tutorial.
This is my db_pod.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: lp-db
labels:
name: lp-db
spec:
containers:
- image: my_username/my-db
name: my-db
ports:
- containerPort: 5432
name: my-db
volumeMounts:
- name: pg-data
mountPath: /var/lib/postgresql/data
volumes:
- name: pg-data
gcePersistentDisk:
pdName: my-db-disk
fsType: ext4
I create the disk using the command gcloud compute disks create --size 200GB my-db-disk.
However, when I run the pod, delete it, and then run it again (like in the tutorial) my data is not persisted.
I tried multiple versions of this file, including with PersistentVolumes and PersistentVolumeClaims, I tried changing the mountPath, but to no success.
Edit
Dockerfile for creating the Postgres image:
FROM ubuntu:trusty
RUN rm /bin/sh && \
ln -s /bin/bash /bin/sh
# Get Postgres
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
RUN apt-get update && \
apt-get install -y wget
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Install virtualenv (will be needed later)
RUN apt-get update && \
apt-get install -y \
libjpeg-dev \
libpq-dev \
postgresql-9.4 \
python-dev \
python-pip \
python-virtualenv \
strace \
supervisor
# Grab gosu for easy step-down from root
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates wget && rm -rf /var/lib/apt/lists/* \
&& wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture)" \
&& wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture).asc" \
&& gpg --verify /usr/local/bin/gosu.asc \
&& rm /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& apt-get purge -y --auto-remove ca-certificates wget
# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
# Adjust PostgreSQL configuration so that remote connections to the database are possible.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.4/main/pg_hba.conf
# And add ``listen_addresses`` to ``/etc/postgresql/9.4/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.4/main/postgresql.conf
RUN echo "log_directory='/var/log/postgresql'" >> /etc/postgresql/9.4/main/postgresql.conf
# Add all code from the project and all config files
WORKDIR /home/projects/my-project
COPY . .
# Add VOLUMEs to allow backup of config, logs and databases
ENV PGDATA /var/lib/postgresql/data
VOLUME /var/lib/postgresql/data
# Expose an entrypoint and a port
RUN chmod +x scripts/sh/*
EXPOSE 5432
ENTRYPOINT ["scripts/sh/entrypoint-postgres.sh"]
And entrypoint script:
echo " I am " && gosu postgres whoami
gosu postgres /etc/init.d/postgresql start && echo 'Started postgres'
gosu postgres psql --command "CREATE USER myuser WITH SUPERUSER PASSWORD 'mypassword';" && echo 'Created user'
gosu postgres createdb -O myuser mydb && echo 'Created db'
# This just keeps the container alive.
tail -F /var/log/postgresql/postgresql-9.4-main.log
In the end, it seems that the real problem was the fact that I was trying to create the database from my entrypoint script.
Things such as creating a db or a user should be done at container creation time so I ended up using the standard Postgres image, which actually provides a simple and easy way to create an user and a db.
This is the fully functional configuration file for Postgres.
apiVersion: v1
kind: Pod
metadata:
name: postgres
labels:
name: postgres
spec:
containers:
- name: postgres
image: postgres
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
- name: POSTGRES_USER
value: myuser
- name: POSTGRES_PASSWORD
value: mypassword
- name: POSTGRES_DB
value: mydb
ports:
- containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pg-data
volumes:
- name: pg-data
persistentVolumeClaim:
claimName: pg-data-claim
Thanks to all those who helped me :)
does your custom postgresql persist data at /var/lib/postgresql/data?
are you able to get logs from your postgresql container and spot anything interesting?
when your pod is running, can you see the mountpoints inside your container and check the persistent disk is there?
I followed this scenario and I was able to persist my data by changing the mountPath to /var/lib/postgresql and also reproduced using cassandra (i.e. /var/lib/cassandra for mountPath)
I was able to delete/restart pods from different nodes/hosts and still see my "users" table and the data I previously entered. However, I was not using a custom image, I just used standard docker images.