Customize postgres start command in default docker postgres image - postgresql

I want to change the start command of postgres to support SSL in the default docker image
db.Docerfile
FROM postgres:14.5-alpine
COPY ./.docker/dev/init-database.sh /docker-entrypoint-initdb.d/
COPY ./.docker/dev/migrations/database_schema.tar ./
COPY ./.docker/dev/certs/out/server.key /var/lib/postgresql
COPY ./.docker/dev/certs/out/server.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crl /var/lib/postgresql
COPY ./.docker/dev/certs/out/news_user.key ./
COPY ./.docker/dev/certs/out/news_user.crt ./
RUN chown 0:70 /var/lib/postgresql/server.key && chmod 640 /var/lib/postgresql/server.key
RUN chown 0:70 /var/lib/postgresql/server.crt && chmod 640 /var/lib/postgresql/server.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crt && chmod 640 /var/lib/postgresql/myCA.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crl && chmod 640 /var/lib/postgresql/myCA.crl
RUN chown 0:70 ./news_user.key && chmod 640 ./news_user.key
RUN chown 0:70 ./news_user.crt && chmod 640 ./news_user.crt
RUN chown postgres:postgres /docker-entrypoint-initdb.d/init-database.sh
EXPOSE 5432
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
When I run this image, I get an error saying
"root" execution of the PostgreSQL server is not permitted.
news_database | The server must be started under an unprivileged user ID to prevent
news_database | possible system security compromise. See the documentation for
news_database | more information on how to properly start the server.
The default postgres image comes with a user named postgres so I tried adding a user postgres line before CMD
Now it gives me a new error saying
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
Can someone kindly tell me how to fix this so that the postgres command works?
UPDATE 1
I am not running the dockerfile directly, I run it via a docker-compose.yml file since I have a python script that accesses the database in another container
docker.compose.yml
version: "3.8"
services:
news_database:
build:
context: ../..
dockerfile: ./.docker/dev/db.Dockerfile
container_name: news_database
restart: unless-stopped
env_file:
- .env
ports:
- "5432:5432"
volumes:
- news_db:/var/lib/postgresql/data
# https://stackoverflow.com/a/55835081/5371505
# https://stackoverflow.com/questions/72213661/test-connection-to-postgres-with-ssl-with-the-command-line
healthcheck:
test: ["CMD-SHELL", "pg_isready -d 'hostaddr=$DATABASE_HOST user=$DATABASE_USER port=$DATABASE_PORT dbname=$DATABASE_NAME'"]
interval: 5s
timeout: 5s
retries: 5
# Dont add a 'restart' policy to the app because we run it as a cronjob regardless of whether it succeeds or fails
news_app:
# https://stackoverflow.com/questions/65594752/docker-compose-how-to-reference-files-in-other-directories
build:
context: ../..
dockerfile: ./.docker/dev/app.Dockerfile
env_file:
- .env
image: news_app
container_name: news_app
depends_on:
news_database:
condition: service_healthy
volumes:
news_db:
driver: local
I run the above file with this command
docker compose -f ".docker/dev/docker-compose.yml" up -d --build news_database && docker compose -f ".docker/dev/docker-compose.yml" logs --follow

I see a couple of problems with your Dockerfile. First, I've stripped out a few bits to reduce the complexity to something I can reproduce locally, so I have:
FROM postgres:14.5-alpine
COPY ./certs/out/server.key /var/lib/postgresql
COPY ./certs/out/server.crt /var/lib/postgresql
RUN chown 0:70 /var/lib/postgresql/server.key && chmod 640 /var/lib/postgresql/server.key
RUN chown 0:70 /var/lib/postgresql/server.crt && chmod 640 /var/lib/postgresql/server.crt
EXPOSE 5432
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
If we build a pgtest image from this Dockerfile and run it, we see:
$ docker build -t pgtest .
...
$ docker run --rm pgtest
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise. See the documentation for
more information on how to properly start the server.
What's going on here? It turns out this is caused by how you've written your CMD directive. You need to use the "exec style", which means passing in a JSON array rather than a string:
CMD ["postgres", \
"-c", "ssl=on", \
"-c", "ssl_cert_file=/var/lib/postgresql/server.crt", \
"-c", "ssl_key_file=/var/lib/postgresql/server.key"]
This is necessary because there's a check in the entrypoint script for
the image that looks like this:
if [ "$1" = 'postgres' ] && ! _pg_want_help "$#"; then
When you write:
CMD postgres -c ...
This wraps the command in a call to sh -c, so $1 will be sh,
thus skipping the important database initialization logic. When we use
the exec format of the CMD statement, $1 will be postgres, as
expected.
With that fix, the image runs correctly:
$ docker run --rm -e POSTGRES_PASSWORD=secret pgtest
.
.
.
PostgreSQL init process complete; ready for start up.
2022-08-28 12:48:48.784 UTC [1] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2022-08-28 12:48:48.784 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-28 12:48:48.784 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-28 12:48:48.786 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-28 12:48:48.789 UTC [48] LOG: database system was shut down at 2022-08-28 12:48:48 UTC
2022-08-28 12:48:48.792 UTC [1] LOG: database system is ready to accept connections

Related

Docker 'backup' process container not seeing Database container postgres

I have a simple docker-compose.yml & associated Dockerfiles that give me a simple dev and prod environment for a nginx-uvicorn-django-postgres stack. I want to add an optional 'backup' container that just runs cron to periodically connect to the 'postgres' container.
# backup container - derived from [this blog][1]
ARG DOCKER_REPO
ARG ALPINE_DOCKER_IMAGE # ALPINE
ARG ALPINE_DOCKER_TAG # LATEST
FROM ${DOCKER_REPO}${ALPINE_DOCKER_IMAGE}:${ALPINE_DOCKER_TAG}
ARG DB_PASSWORD
ARG DB_HOST # "db"
ARG DB_PORT # "5432"
ARG DB_NAME # "ken"
ARG DB_USERNAME # "postgres"
ENV PGPASSWORD=${DB_PASSWORD} HOST=${DB_HOST} PORT=${DB_PORT} PSQL_DB_NAME=${DB_NAME} \
USERNAME=${DB_USERNAME}
RUN printenv
RUN mkdir /output && \
mkdir /output/backups && \
mkdir /scripts && \
chmod a+x /scripts
COPY ./scripts/ /scripts/
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/15min/${DB_NAME}_15
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/daily/${DB_NAME}_day
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/weekly/${DB_NAME}_week
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/monthly/${DB_NAME}_month
RUN apk update && \
apk upgrade && \
apk add --no-cache postgresql-client && \
chmod a+x /etc/periodic/15min/${DB_NAME}_15 && \
chmod a+x /etc/periodic/daily/${DB_NAME}_day && \
chmod a+x /etc/periodic/weekly/${DB_NAME}_week && \
chmod a+x /etc/periodic/monthly/${DB_NAME}_month
The django container is derived from the official Python image and connects (through psycopg2) with values (as ENV value) for host, dbname, username, password and port. The 'backup' container has these same values, but I get this error from the command line:
> pg_dump --host="$HOST" --port="$PORT" --username="$USERNAME" --dbname="$PSQL_DB_NAME"
pg_dump: error: could not translate host name "db" to address: Name does not resolve
Is Alpine missing something relevant that is present in the official Python?
Edit:
I am running with a system of shell scripts that take care of housekeeping for different configurations. so
> ./ken.sh dev_server
will set up the environment variables and then run docker-compose for the project and the containers
docker-compose.yml doesn't explicitly create a network.
I don't know what "db" should resolve to beyond just 'db://'? - its what the django container gets and it is able to resolve a connection to the 'db' service.
service:
db:
image: ${DOCKER_REPO}${DB_DOCKER_IMAGE}:${DB_DOCKER_TAG} #postgres: 14
container_name: ${PROJECT_NAME}_db
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- PGPASSWORD
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
command: ["postgres", "-c", "log_statement=all"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h db"]
interval: 2s
timeout: 5s
retries: 25
This is the 'dev_server' script run by the parent ken.sh script
function dev_server() {
trap cleanup EXIT
wait_and_launch_browser &
docker-compose -p "${PROJECT_NAME}" up -d --build db nginx web pgadmin backup
echo "Generate static files and copy them into static and file volumes."
source ./scripts/generate_static_files.sh
docker-compose -p "${PROJECT_NAME}" logs -f web nginx backup
}
Update: Worked through "Reasons why docker containers can't talk to each other" and found that all the containers are on a ken_default network, from 170.20.0.2 to 170.20.0.6.
I can docker exec ken_backup backup ken_db -c2, but not from db to backup, because the db container doesn't include ping.
From a shell on backup I cannot ping ken_db - ken_db doesn't resolve, nor does 'db'.
I can't make much of that and I'm not sure what to try next.
You are running the backup container as a separate service.
Docker-compose creates a unique network for each service (docker-compose.yml file).
You need to get the DB and your backup container on the same docker network.
See this post

Docker Postgres file illegal option -

I want to put variables inside my CMD of a Dockerfile that has a Postgres container with certificates needed for SSL. I am using this Dockerfile as build context from a docker-compose.yml file that this database as one service and an app
db.Dockerfile
FROM postgres:14.5-alpine
ENV EXT_KEY .key
COPY ./.docker/dev/init-database.sh /docker-entrypoint-initdb.d/
COPY ./.docker/dev/migrations/database_schema.tar ./
COPY ./.docker/dev/certs/out/postgresdb$EXT_KEY /var/lib/postgresql
COPY ./.docker/dev/certs/out/postgresdb.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crl /var/lib/postgresql
COPY ./.docker/dev/certs/out/news_user$EXT_KEY ./
COPY ./.docker/dev/certs/out/news_user.crt ./
RUN chown 0:70 /var/lib/postgresql/postgresdb$EXT_KEY && chmod 640 /var/lib/postgresql/postgresdb$EXT_KEY
RUN chown 0:70 /var/lib/postgresql/postgresdb.crt && chmod 640 /var/lib/postgresql/postgresdb.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crt && chmod 640 /var/lib/postgresql/myCA.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crl && chmod 640 /var/lib/postgresql/myCA.crl
RUN chown 0:70 ./news_user$EXT_KEY && chmod 640 ./news_user$EXT_KEY
RUN chown 0:70 ./news_user.crt && chmod 640 ./news_user.crt
RUN chown postgres:postgres /docker-entrypoint-initdb.d/init-database.sh
EXPOSE 5432
USER postgres
ENTRYPOINT ["docker-entrypoint.sh"]
CMD [ "-c", "ssl=on" , "-c", "ssl_cert_file=/var/lib/postgresql/postgresdb.crt", "-c",\
"ssl_key_file=/var/lib/postgresql/postgresdb.${EXT_KEY}", "-c",\
"ssl_ca_file=/var/lib/postgresql/myCA.crt", "-c", "ssl_crl_file=/var/lib/postgresql/myCA.crl" ]
docker-compose.yml
version: "3.8"
services:
news_database:
build:
context: ../..
dockerfile: ./.docker/dev/db.Dockerfile
container_name: news_database
restart: unless-stopped
env_file:
- .env
ports:
- "5432:5432"
volumes:
- news_db:/var/lib/postgresql/data
news_app:
...
volumes:
news_db:
driver: local
When I run this the variable is not present in the CMD and therefore the container fails
Attempt 1
I tried changing the final command from array to string format
CMD -c ssl=on -c ssl_cert_file=/var/lib/postgresql/postgresdb.crt -c ssl_key_file=/var/lib/postgresql/postgresdb.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
It gives me an /bin/sh: illegal option - error
Attempt 2
I removed the entrypoint completely and tried directly calling postgres with a CMD
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/postgresdb.crt -c ssl_key_file=/var/lib/postgresql/postgresdb.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
It immediately gives me another error when I run it via docker-compose
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
All I want is to have variables inside that CMD, can someone kindly tell me a way to make this work?
Your ENTRYPOINT instruction is specified as JSON array, meaning it's in exec form - no shell will be invoked for the execution of docker-entrypoint.sh. As there is no shell invoked, there won't be any environment variable expansion.
To make it work, try this:
ENTRYPOINT [ "sh", "-c", \
"docker-entrypoint.sh \
-c ssl=on \
-c ssl_cert_file=/var/lib/postgresql/postgresdb.crt \
-c ssl_key_file=/var/lib/postgresql/postgresdb${EXT_KEY} \
-c ssl_ca_file=/var/lib/postgresql/myCA.crt \
-c ssl_crl_file=/var/lib/postgresql/myCA.crl \
${0} ${#}" ]

Run docker-compose command without exiting

I am using a postgres image and i need to start ssh service on start.
The problem is that if I run a command in docker-compose file the proccess exits with code 0.
How can I start ssh service but keep postgres serice active too?
DOCKER FILE:
FROM postgres:13
RUN apt update && apt install openssh-server sudo -y
RUN echo 'root:password' | chpasswd
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/g' /etc/ssh/sshd_config
DOCKER-COMPOSE FILE:
postgres:
container_name: db_postgres
command: sh -c "service ssh start "
image: postgresc
build:
context: ../backend_apollo_server_express
dockerfile: Dockerfile.database
environment:
- "POSTGRES_USER=lims"
- "POSTGRES_PASSWORD=lims"
volumes:
- /home/javier/lims/dockerVolumes/db:/var/lib/postgresql/data
- "/etc/timezone:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
ports:
- 5434:5432
You can try to use run postgres after you command
command: sh -c "service ssh start & postgres"
Try
command: sh -c "nohup service ssh start && service postgres start &"
In order to leave the process running in the background. This way the process won't exit

docker-compose can't find file to inject to Postgresql when running a command

I'm trying to restore a .backup file to a Postgresql database. For that, I use a docker-compose file to launch a postgres docker container:
docker-compose.yml
postgresql:
image: postgres
restart: always
ports:
- "5432:5432"
environment:
- POSTGRES_USER:postgres
# - PGDATA:/var/lib/postgresql/data/pgdata/
volumes:
- ${PWD}/project/data/MX/bkup/data:/var/lib/postgresql
command:
- pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
When I run my docker-compose file using the command:
docker-compose up postgresql
I get the error:
(virtual) med#nid:~/projects/project/pkg$ docker-compose up postgresql
Recreating pkg_postgresql_1 ...
Recreating pkg_postgresql_1 ... done
Attaching to pkg_postgresql_1
postgresql_1 | /usr/local/bin/docker-entrypoint.sh: line 176: /pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup: No such file or directory
pkg_postgresql_1 exited with code 127
This happens even though the file is inside the volume
med#nid:~/projects/project/pkg$ docker-compose exec postgresql bash
root#ab7dbe2b0232:/# cd /var/lib/postgresql/
root#ab7dbe2b0232:/var/lib/postgresql# ls -l
total 1054780
drwx------ 19 postgres postgres 4096 Oct 2 08:51 data
-r--r--r-- 1 postgres ssl-cert 1080082803 Sep 27 15:40 ph.backup
I tried to use the -h argument of pg_restore in the docker-compose command:
pg_restore -h tcp://`docker-machine ip default`:5432 -U postgres -d postgres /var/lib/postgresql/ph.backup
What works:
If I comment the command target in the docker-compose.yml, launch the docker container and run the command inside it I get to have the data injected!
Is there a fix for this, Meaning, is there a way to make the command work directly from the docker-compose.yml file?
There are two forms of Docker Compose command:. You should move the command up on to the same line
command: pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
The form you have individually spells out each argument (in a YAML list); for example
command:
- /bin/ls
- -l
- -r
- -t
(Also consider just installing the PostgreSQL client tools on your host and running this outside of Docker; use localhost as the host name and the first number from the database container's ports: as the port number.)
The error comes from the form of the command configuration. If you inspect the finished postgres container (docker inspect <container-id>) the entrypoint and command look like this:
"Cmd": [
"pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup"
],
"Entrypoint": [
"docker-entrypoint.sh"
]
That practically means that the default entrypoint script docker-entrypoint
is executed with one argument which is the pg_restore command. At line 176 the script execs the passed in arguments (exec "$#").
The exec command needs a command and a list of arguments
exec [command [arguments]]
but in this case the command is the full string formed by pg_restore and its arguments. This obviously is not a valid file
Now, if you change command in docker-compose.yml to:
command: pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup
inspecting the container shows the following:
"Cmd": [
"pg_restore",
"-U",
"postgres",
"-d",
"postgres",
"/var/lib/postgresql/ph.backup"
]
that means that exec will run pg_restore as command passing the rest as arguments and everything works as expected.
In alternative you could override the entrypoint in the docker-compose file to execute the command in a shell:
entrypoint: /bin/bash -c
command:
- pg_restore -U postgres -d postgres /var/lib/postgresql/ph.backup

How restore a postgres database with fig?

I'm trying to use FIG (http://www.fig.sh/) for a django app. I can't recreate the database from a dump, I try:
fig run db pg_restore -d DBNAME < backup.sql
And get:
socket.error: [Errno 104] Connection reset by peer
But this run (still not see the tables in the db):
fig run db pg_restore < backup.sql
This is the dockerfile:
FROM python:3.4
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
ADD backup.sql /code/
RUN pip install -r requirements.txt
RUN pg_restore -d postgres backup.sql
ADD . /code/
And fig.yml:
db:
image: postgres
ports:
- 5432
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
When you run
fig run db pg_restore -d DBNAME < backup.sql
postgresd is not running. You've replaced the startup of the daemon with the pg_restore command.
I would suggest doing something like this:
Move backup.sql to dockerfiles/db/backup.sql
Create a dockerfiles/db/Dockerfile
change your fig.yml to use build for the db instead
Dockerfile
FROM postgres
ADD . /files
WORKDIR /files
RUN /etc/init.d/postgresql start && \
pg_restore -d DBNAME < backup.sql && \
/etc/init.d/postgresql stop
fig.yml
db:
build: dockerfiles/db
Now when you run any fig commands your database should be ready to go