Error Converting Docker Commands to docker-compose.yml - postgresql

When trying to translate the following 2 docker commands into a docker-compose.yml using Compose version 3
docker run​ \
--name timescaledb \
--network timescaledb-net \
-e POSTGRES_PASSWORD=insecure \
-e POSTGRES_INITDB_WALDIR=/var/lib/postgresql/data/pg_wal \
-e PGDATA=/var/lib/postgresql/data/pg_data \
timescale/timescaledb:latest-pg11 postgres \
-cwal_level=archive \
-carchive_mode=on \
-carchive_command="/usr/bin/wget wale/wal-push/%f -O -" \
-carchive_timeout=600 \
-ccheckpoint_timeout=700 \
-cmax_wal_senders=1
and
docker run​ \
--name wale \
--network timescaledb-net \
--volumes-from timescaledb \
-v ./backups:/backups \
-e WALE_LOG_DESTINATION=stderr \
-e PGWAL=/var/lib/postgresql/data/pg_wal \
-e PGDATA=/var/lib/postgresql/data/pg_data \
-e PGHOST=timescaledb \
-e PGPASSWORD=insecure \
-e PGUSER=postgres \
-e WALE_FILE_PREFIX=file://localhost/backups \
timescale/timescaledb-wale:latest
we get the following error when running docker-compose up:
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.wale: 'volumes_from'
How can we translate the 2 Docker commands correctly to use Compose version 3? We will need to be able to specify the location of the volumes on the host (i.e. ./timescaledb).
Using Mac OS X 10.15.3, Docker 19.03.8, Docker Compose 1.25.4
docker-compose.yml
version: '3.3'
services:
timescaledb:
image: timescale/timescaledb:latest-pg11
container_name: timescaledb
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=insecure
- POSTGRES_INITDB_WALDIR=/var/lib/postgresql/data/pg_wal
- PGDATA=/var/lib/postgresql/data/pg_data
command: -cwal_level=archive -carchive_mode=on -carchive_command="/usr/bin/wget wale/wal-push/%f -O -" -carchive_timeout=600 -ccheckpoint_timeout=700 -cmax_wal_senders=1
volumes:
- ./timescaledb:/var/lib/postgresql/data
networks:
- timescaledb-net
wale:
image: timescale/timescaledb-wale:latest
container_name: wale
environment:
- WALE_LOG_DESTINATION=stderr
- PGWAL=/var/lib/postgresql/data/pg_wal
- PGDATA=/var/lib/postgresql/data/pg_data
- PGHOST=timescaledb
- PGPASSWORD=insecure
- PGUSER=postgres
- WALE_FILE_PREFIX=file://localhost/backups
volumes_from:
- tsdb
volumes:
- ./backups:/backups
networks:
- timescaledb-net
depends_on:
- timescaledb
networks:
timescaledb-net:

In the container timescaledb you are actually mounting the /var/lib/postgresql/data to ./timescaledb and hence, if you want to use the same volume for the wale container, you can edit the wale container like:
...
volumes:
- ./backups:/backups
- ./timescaledb:/var/lib/postgresql/data
...
In this case, both of the containers will be able to read and write from the same mounted volume to your local machine.
Also, remember to remove this part as it is not a valid command in docker-compose:
volumes_from:
- tsdb

Related

Docker compose fails on Raspberry Pi

Running $ docker compose up -d on the following docker-compose.yml
version: '3'
services:
web:
platform: linux/arm/v6
build: ./web
restart: always
environment:
DATABASE_URL: ${DATABASE_URL}
SECRET_KEY: ${SECRET_KEY}
TZ: Europe/Amsterdam
ports:
- ${PORT}:5000
depends_on:
- db
volumes:
- web-data:/data
db:
platform: linux/arm/v6
image: arm32v6/postgres:15.1-alpine
restart: always
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
ports:
- ${POSTGRES_PORT}:5432
volumes:
- postgres-data:/var/lib/postgresql/data
volumes:
postgres-data:
web-data:
should create two docker containers. However, it produces
[+] Building 6.5s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.5s
=> => transferring dockerfile: 32B 0.2s
=> [internal] load .dockerignore 0.2s
=> => transferring context: 2B 0.1s
=> ERROR resolve image config for docker.io/docker/dockerfile:1 2.4s
------
> resolve image config for docker.io/docker/dockerfile:1:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend
dockerfile.v0: failed to solve with frontend gateway.v0: no match for platform in manifest
sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc: not found
I verified that both services (web and db) in my docker-compose file can be build using docker build, which excludes any platform issues from the services.
What does no match for platform in manifest mean? More importantly, how can I fix the error?
A temporary solution is to replace docker-compose.yml with explicit docker commands.
# Create the network and volumes
docker network create randpion-app
docker volume create postgres-data
docker volume create web-data
# Build the web service
docker build -t randpion/web web
# Load the environment variables from the .env file
export $(cat .env | xargs)
# Start the database service
docker run -d \
--network randpion-app --network-alias db --name randpion-db \
--platform "linux/arm/v6" \
-v postgres-data:/var/lib/postgresql/data \
-e POSTGRES_USER=${POSTGRES_USER} \
-e POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
-e POSTGRES_DB=${POSTGRES_DB} \
-p ${POSTGRES_PORT}:5432 \
arm32v6/postgres:15.1-alpine
# Start the web service
docker run -d \
--network randpion-app --network-alias web --name randpion-web \
--platform "linux/arm/v6" \
-v web-data:/data \
-e DATABASE_URL=${DATABASE_URL} \
-e SECRET_KEY=${SECRET_KEY} \
-e TZ=Europe/Amsterdam \
-p ${PORT}:5000 \
randpion/web
This runs fine for now, but the solution is not ideal. For example, the containers must restart after every reboot.

Rename volume in docker compose, with docker derived from postgres

I have this docker-compose
version: '3.7'
services:
app-db:
build:
context: ./
dockerfile: Dockerfile-pg
image: app-pg:1.0.0
restart: always
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app
volumes:
- ./docker-entrypoint-initdb.d/init-user-db.sh:/docker-entrypoint-initdb.d/init-user-db.sh
- v-app-pgdata:/var/lib/postgresql
- v-app-pglog:/data/var/log/postgresql
- v-app-pgconf:/etc/postgresql
app-main:
build:
context: ./
dockerfile: Dockerfile-tar-cp
image: app-main:1.0.0
restart: always
ports:
- 80:80
volumes:
v-app-pgdata:
name: v-app-pgdata
v-app-pglog:
name: v-app-pglog
v-app-pgconf:
name: v-app-pgconf
so an app container and a postgres derived container:
#docker build -t app-pg:1.0.0 -f Dockerfile-pg .
#docker run -d --name appC-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres app-pg:1.0.0
FROM postgres:12.1
MAINTAINER xxx
#ARG A_DB_USER='postgres'
#ARG A_DB_PASS='postgres'
ARG A_DB_NAME='app'
ARG A_TZ='Europe/Zurich'
#ENV DB_USER=${A_DB_USER}
#ENV DB_PASS=${A_DB_PASS}
ENV DB_NAME=${A_DB_NAME}
ENV TZ=${A_TZ}
# Adjusting Timezone in the system
RUN echo $TZ > /etc/timezone && \
apt-get update && apt-get install -y tzdata && \
rm /etc/localtime && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
# install postgis
RUN apt-get update && \
apt-get install -y postgis && \
apt-get clean
USER postgres
#Add password "postgres" to user postgres, create db, add .sql
#RUN /etc/init.d/postgresql start && \
# psql --command "ALTER USER ${DB_USER} WITH PASSWORD '${DB_PASS}'; SET TIME ZONE '${TZ}';" && \
# createdb -O ${DB_USER} ${DB_NAME} -E UTF8 && \
# psql -d ${DB_NAME} -c 'CREATE EXTENSION postgis'
EXPOSE 5432
My problem is that postgres default dockerfile have this line:
VOLUME /var/lib/postgresql/data
So even if I create a named Volumes with the same folder, my docker-compose create 4 and not 3 volumes, one unnamed due to that line.
How is it possible solve this issue?
I had the same problem as you and I solved it by editing the docker-compose.yml the following way:
Create a volume for the data
volumes:
v-app-pgdata:
In your declaration of the database service, in the volumes clause, you need to change - v-app-pgdata:/var/lib/postgresql to:
- v-app-pgdata:/var/lib/postgresql/data
The problem was that you where not mounting the volume to the correct place, and therefore, it had to create another volume that is declared in the base image VOLUME /var/lib/postgresql/data. If you run this container without attaching a volume to this mount point, it will automatically create it, and with a random name.
However, it has been pointed out to me that if the OP strictly needs a volume in /var/lib/postgresql then this solution wont work.
Hope this helps, as it worked for me.

How to access exposed containers with docker-compose and Hyperledger Sawtooth?

I'm trying to run a bare bones version of Hyperledger Sawtooth using Docker CE on a Mac. The docker-compose.yaml has containers running the base images from Sawtooth.
I'm unable to access the Sawtooth REST API from the host machine even though there are ports published for it when I run docker ps. The docker-compose file has worked on other Macs running Docker CE so I'm suspecting it may be a configuration or setup issue.
The contents of the docker-compose.yaml are below:
version: '2.1'
services:
settings-tp:
image: 'hyperledger/sawtooth-settings-tp:1.1.3'
container_name: sawtooth-settings-tp
depends_on:
- validator
entrypoint: settings-tp --connect tcp://validator:4004
identity-tp:
image: 'hyperledger/sawtooth-identity-tp:1.1.3'
container_name: sawtooth-identity-tp
depends_on:
- validator
entrypoint: identity-tp -vv --connect tcp://validator:4004
rest-api:
image: 'hyperledger/sawtooth-rest-api:1.1.3'
container_name: sawtooth-rest-api
ports:
- '8008:8008'
depends_on:
- validator
entrypoint: sawtooth-rest-api --connect tcp://validator:4004 --bind rest-api:8008
validator:
image: 'hyperledger/sawtooth-validator:1.1.3'
container_name: sawtooth-validator
ports:
- '4004:4004'
command: |
bash -c "
if [ ! -f /etc/sawtooth/keys/validator.priv ]; then
sawadm keygen
sawtooth keygen my_key
sawset genesis -k /root/.sawtooth/keys/my_key.priv
sawset proposal create \
-k /root/.sawtooth/keys/my_key.priv \
sawtooth.consensus.algorithm.name=Devmode \
sawtooth.consensus.algorithm.version=0.1 \
-o config.batch && \
sawadm genesis config-genesis.batch config.batch
fi;
sawtooth-validator -vvv \
--endpoint tcp://validator:8800 \
--bind component:tcp://eth0:4004 \
--bind network:tcp://eth0:8800 \
--bind consensus:tcp://eth0:5050 \
"
devmode-engine:
image: 'hyperledger/sawtooth-devmode-engine-rust:1.1.3'
container_name: sawtooth-devmode-engine-rust-default
depends_on:
- validator
entrypoint: devmode-engine-rust -C tcp://validator:5050
If you cannot access the port from the host, the container must not be running correctly. Look for error messages for that container when starting docker-compose
What does docker ps -a show?
Can you connect to the port? Try something like telnet localhost 8008

Using Flyway in CI/CD to populate/seed a Postgres DB in Docker then TAG and publish the new docker image to be used for testing

I want to start using Flyway to version our database changes. I am trying to create a Postgres Docker container with seeded data, TAG and publish that docker image to be used in automated testing.
I tried using docker-compose, however I haven't figured a way to TAG and publish after Flyway runs.
Repository with test project
https://github.com/bigboy1122/flyway_postgres
Here is a docker-compose I created
version: '3.7'
services:
flyway:
image: boxfuse/flyway
restart: always
command: -url=jdbc:postgresql://db:5432/foo -user='postgres' -password='test123' -schemas='bar' migrate
volumes:
- .:/flyway/sql
depends_on:
- db
db:
image: tgalati1122/flyway_seeded_postgres
build:
context: .
dockerfile: ./Dockerfile
restart: always
environment:
POSTGRES_PASSWORD: 'test123'
POSTGRES_USER: 'postgres'
POSTGRES_DB: 'foo'
ports:
- 5432:5432
adminer:
image: adminer
restart: always
ports:
- 8080:8080
depends_on:
- db
Here is trying to use Docker multi build feature.
In the example below, the database I think spins up but I can't get flyway to access it.
FROM postgres:10.5-alpine as donar
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
RUN /docker-entrypoint.sh --help
FROM debian:stretch-slim as build_tools
ENV FLWAY_VERSION='5.2.0
RUN set -ex; \
if ! command -v gpg > /dev/null; then \
apt-get update; \
apt-get install -y --no-install-recommends \
gnupg \
dirmngr \
wget \
; \
rm -rf /var/lib/apt/lists/*; \
fi
VOLUME flyway/sql
RUN wget --no-check-certificate https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/5.2.0/flyway-commandline-5.2.0-linux-x64.tar.gz -O - | tar -xz
RUN pwd; \
ls -l; \
cd flyway-5.2.0; \
pwd; \
ls -l; \
sh ./flyway -url=jdbc:postgresql://localhost:5432/optins -user='postgres' -password='test123' -schemas='bar' migrate; \
FROM postgres:10.5-alpine
ENV PGDATA=/pgdata
ENV POSTGRES_PASSWORD='test123'
ENV POSTGRES_USER='postgres'
ENV POSTGRES_DB='foo'
EXPOSE 5432:5432
COPY --chown=postgres:postgres --from=donor /pgdata /pgdata
The idea I am going for as database changes occur, I want to automatically build a new lightweight test database as well as update the persisted databases throughout the enterprise.

Docker Volumes on Mac with Postgres

I'm attempting to run the following command on Mac OSX:
docker run --rm --name kong-database \
--network=kong-net \
-v /Volumes/docker/postgres:/var/lib/postgresql/data \
-p 5432:5432 \
-e "POSTGRES_USER=kong" \
-e "POSTGRES_DB=kong" \
postgres:9.6
With the intention that it will launch postgres in in a docker container, and persist the database data on my local file system in /Volumes/docker/postgres. However, I get the following error, and the container fails to run:
chown: changing ownership of '/var/lib/postgresql/data': Operation not permitted
Any idea what I can do to get this going?
Turns out that /Volumes was owned by root. When I switched this to something like /Users/JoshuaErney/Volumes, this ran successfully. Here's an example:
docker run --rm --name kong-database \
--network=kong-net \
-v /Users/JoshuaErney/Volumes/docker/postgres:/var/lib/postgresql/data \
-p 5432:5432 \
-e "POSTGRES_USER=kong" \
-e "POSTGRES_DB=kong" \
postgres:9.6
Answer by jerney is correct, here is what it looks like with docker-compose.yaml:
services:
test-postgres-compose :
...
volumes :
- pgdata:/var/lib/postgresql/data
...
volumes :
pgdata :
Docker compose volumes info