I'm trying to use FIG (http://www.fig.sh/) for a django app. I can't recreate the database from a dump, I try:
fig run db pg_restore -d DBNAME < backup.sql
And get:
socket.error: [Errno 104] Connection reset by peer
But this run (still not see the tables in the db):
fig run db pg_restore < backup.sql
This is the dockerfile:
FROM python:3.4
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
ADD backup.sql /code/
RUN pip install -r requirements.txt
RUN pg_restore -d postgres backup.sql
ADD . /code/
And fig.yml:
db:
image: postgres
ports:
- 5432
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
When you run
fig run db pg_restore -d DBNAME < backup.sql
postgresd is not running. You've replaced the startup of the daemon with the pg_restore command.
I would suggest doing something like this:
Move backup.sql to dockerfiles/db/backup.sql
Create a dockerfiles/db/Dockerfile
change your fig.yml to use build for the db instead
Dockerfile
FROM postgres
ADD . /files
WORKDIR /files
RUN /etc/init.d/postgresql start && \
pg_restore -d DBNAME < backup.sql && \
/etc/init.d/postgresql stop
fig.yml
db:
build: dockerfiles/db
Now when you run any fig commands your database should be ready to go
Related
I have a simple docker-compose.yml & associated Dockerfiles that give me a simple dev and prod environment for a nginx-uvicorn-django-postgres stack. I want to add an optional 'backup' container that just runs cron to periodically connect to the 'postgres' container.
# backup container - derived from [this blog][1]
ARG DOCKER_REPO
ARG ALPINE_DOCKER_IMAGE # ALPINE
ARG ALPINE_DOCKER_TAG # LATEST
FROM ${DOCKER_REPO}${ALPINE_DOCKER_IMAGE}:${ALPINE_DOCKER_TAG}
ARG DB_PASSWORD
ARG DB_HOST # "db"
ARG DB_PORT # "5432"
ARG DB_NAME # "ken"
ARG DB_USERNAME # "postgres"
ENV PGPASSWORD=${DB_PASSWORD} HOST=${DB_HOST} PORT=${DB_PORT} PSQL_DB_NAME=${DB_NAME} \
USERNAME=${DB_USERNAME}
RUN printenv
RUN mkdir /output && \
mkdir /output/backups && \
mkdir /scripts && \
chmod a+x /scripts
COPY ./scripts/ /scripts/
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/15min/${DB_NAME}_15
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/daily/${DB_NAME}_day
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/weekly/${DB_NAME}_week
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/monthly/${DB_NAME}_month
RUN apk update && \
apk upgrade && \
apk add --no-cache postgresql-client && \
chmod a+x /etc/periodic/15min/${DB_NAME}_15 && \
chmod a+x /etc/periodic/daily/${DB_NAME}_day && \
chmod a+x /etc/periodic/weekly/${DB_NAME}_week && \
chmod a+x /etc/periodic/monthly/${DB_NAME}_month
The django container is derived from the official Python image and connects (through psycopg2) with values (as ENV value) for host, dbname, username, password and port. The 'backup' container has these same values, but I get this error from the command line:
> pg_dump --host="$HOST" --port="$PORT" --username="$USERNAME" --dbname="$PSQL_DB_NAME"
pg_dump: error: could not translate host name "db" to address: Name does not resolve
Is Alpine missing something relevant that is present in the official Python?
Edit:
I am running with a system of shell scripts that take care of housekeeping for different configurations. so
> ./ken.sh dev_server
will set up the environment variables and then run docker-compose for the project and the containers
docker-compose.yml doesn't explicitly create a network.
I don't know what "db" should resolve to beyond just 'db://'? - its what the django container gets and it is able to resolve a connection to the 'db' service.
service:
db:
image: ${DOCKER_REPO}${DB_DOCKER_IMAGE}:${DB_DOCKER_TAG} #postgres: 14
container_name: ${PROJECT_NAME}_db
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- PGPASSWORD
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
command: ["postgres", "-c", "log_statement=all"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h db"]
interval: 2s
timeout: 5s
retries: 25
This is the 'dev_server' script run by the parent ken.sh script
function dev_server() {
trap cleanup EXIT
wait_and_launch_browser &
docker-compose -p "${PROJECT_NAME}" up -d --build db nginx web pgadmin backup
echo "Generate static files and copy them into static and file volumes."
source ./scripts/generate_static_files.sh
docker-compose -p "${PROJECT_NAME}" logs -f web nginx backup
}
Update: Worked through "Reasons why docker containers can't talk to each other" and found that all the containers are on a ken_default network, from 170.20.0.2 to 170.20.0.6.
I can docker exec ken_backup backup ken_db -c2, but not from db to backup, because the db container doesn't include ping.
From a shell on backup I cannot ping ken_db - ken_db doesn't resolve, nor does 'db'.
I can't make much of that and I'm not sure what to try next.
You are running the backup container as a separate service.
Docker-compose creates a unique network for each service (docker-compose.yml file).
You need to get the DB and your backup container on the same docker network.
See this post
I want to change the start command of postgres to support SSL in the default docker image
db.Docerfile
FROM postgres:14.5-alpine
COPY ./.docker/dev/init-database.sh /docker-entrypoint-initdb.d/
COPY ./.docker/dev/migrations/database_schema.tar ./
COPY ./.docker/dev/certs/out/server.key /var/lib/postgresql
COPY ./.docker/dev/certs/out/server.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crl /var/lib/postgresql
COPY ./.docker/dev/certs/out/news_user.key ./
COPY ./.docker/dev/certs/out/news_user.crt ./
RUN chown 0:70 /var/lib/postgresql/server.key && chmod 640 /var/lib/postgresql/server.key
RUN chown 0:70 /var/lib/postgresql/server.crt && chmod 640 /var/lib/postgresql/server.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crt && chmod 640 /var/lib/postgresql/myCA.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crl && chmod 640 /var/lib/postgresql/myCA.crl
RUN chown 0:70 ./news_user.key && chmod 640 ./news_user.key
RUN chown 0:70 ./news_user.crt && chmod 640 ./news_user.crt
RUN chown postgres:postgres /docker-entrypoint-initdb.d/init-database.sh
EXPOSE 5432
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
When I run this image, I get an error saying
"root" execution of the PostgreSQL server is not permitted.
news_database | The server must be started under an unprivileged user ID to prevent
news_database | possible system security compromise. See the documentation for
news_database | more information on how to properly start the server.
The default postgres image comes with a user named postgres so I tried adding a user postgres line before CMD
Now it gives me a new error saying
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
Can someone kindly tell me how to fix this so that the postgres command works?
UPDATE 1
I am not running the dockerfile directly, I run it via a docker-compose.yml file since I have a python script that accesses the database in another container
docker.compose.yml
version: "3.8"
services:
news_database:
build:
context: ../..
dockerfile: ./.docker/dev/db.Dockerfile
container_name: news_database
restart: unless-stopped
env_file:
- .env
ports:
- "5432:5432"
volumes:
- news_db:/var/lib/postgresql/data
# https://stackoverflow.com/a/55835081/5371505
# https://stackoverflow.com/questions/72213661/test-connection-to-postgres-with-ssl-with-the-command-line
healthcheck:
test: ["CMD-SHELL", "pg_isready -d 'hostaddr=$DATABASE_HOST user=$DATABASE_USER port=$DATABASE_PORT dbname=$DATABASE_NAME'"]
interval: 5s
timeout: 5s
retries: 5
# Dont add a 'restart' policy to the app because we run it as a cronjob regardless of whether it succeeds or fails
news_app:
# https://stackoverflow.com/questions/65594752/docker-compose-how-to-reference-files-in-other-directories
build:
context: ../..
dockerfile: ./.docker/dev/app.Dockerfile
env_file:
- .env
image: news_app
container_name: news_app
depends_on:
news_database:
condition: service_healthy
volumes:
news_db:
driver: local
I run the above file with this command
docker compose -f ".docker/dev/docker-compose.yml" up -d --build news_database && docker compose -f ".docker/dev/docker-compose.yml" logs --follow
I see a couple of problems with your Dockerfile. First, I've stripped out a few bits to reduce the complexity to something I can reproduce locally, so I have:
FROM postgres:14.5-alpine
COPY ./certs/out/server.key /var/lib/postgresql
COPY ./certs/out/server.crt /var/lib/postgresql
RUN chown 0:70 /var/lib/postgresql/server.key && chmod 640 /var/lib/postgresql/server.key
RUN chown 0:70 /var/lib/postgresql/server.crt && chmod 640 /var/lib/postgresql/server.crt
EXPOSE 5432
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
If we build a pgtest image from this Dockerfile and run it, we see:
$ docker build -t pgtest .
...
$ docker run --rm pgtest
"root" execution of the PostgreSQL server is not permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromise. See the documentation for
more information on how to properly start the server.
What's going on here? It turns out this is caused by how you've written your CMD directive. You need to use the "exec style", which means passing in a JSON array rather than a string:
CMD ["postgres", \
"-c", "ssl=on", \
"-c", "ssl_cert_file=/var/lib/postgresql/server.crt", \
"-c", "ssl_key_file=/var/lib/postgresql/server.key"]
This is necessary because there's a check in the entrypoint script for
the image that looks like this:
if [ "$1" = 'postgres' ] && ! _pg_want_help "$#"; then
When you write:
CMD postgres -c ...
This wraps the command in a call to sh -c, so $1 will be sh,
thus skipping the important database initialization logic. When we use
the exec format of the CMD statement, $1 will be postgres, as
expected.
With that fix, the image runs correctly:
$ docker run --rm -e POSTGRES_PASSWORD=secret pgtest
.
.
.
PostgreSQL init process complete; ready for start up.
2022-08-28 12:48:48.784 UTC [1] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2022-08-28 12:48:48.784 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-28 12:48:48.784 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-28 12:48:48.786 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-28 12:48:48.789 UTC [48] LOG: database system was shut down at 2022-08-28 12:48:48 UTC
2022-08-28 12:48:48.792 UTC [1] LOG: database system is ready to accept connections
I want to put variables inside my CMD of a Dockerfile that has a Postgres container with certificates needed for SSL. I am using this Dockerfile as build context from a docker-compose.yml file that this database as one service and an app
db.Dockerfile
FROM postgres:14.5-alpine
ENV EXT_KEY .key
COPY ./.docker/dev/init-database.sh /docker-entrypoint-initdb.d/
COPY ./.docker/dev/migrations/database_schema.tar ./
COPY ./.docker/dev/certs/out/postgresdb$EXT_KEY /var/lib/postgresql
COPY ./.docker/dev/certs/out/postgresdb.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crt /var/lib/postgresql
COPY ./.docker/dev/certs/out/myCA.crl /var/lib/postgresql
COPY ./.docker/dev/certs/out/news_user$EXT_KEY ./
COPY ./.docker/dev/certs/out/news_user.crt ./
RUN chown 0:70 /var/lib/postgresql/postgresdb$EXT_KEY && chmod 640 /var/lib/postgresql/postgresdb$EXT_KEY
RUN chown 0:70 /var/lib/postgresql/postgresdb.crt && chmod 640 /var/lib/postgresql/postgresdb.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crt && chmod 640 /var/lib/postgresql/myCA.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crl && chmod 640 /var/lib/postgresql/myCA.crl
RUN chown 0:70 ./news_user$EXT_KEY && chmod 640 ./news_user$EXT_KEY
RUN chown 0:70 ./news_user.crt && chmod 640 ./news_user.crt
RUN chown postgres:postgres /docker-entrypoint-initdb.d/init-database.sh
EXPOSE 5432
USER postgres
ENTRYPOINT ["docker-entrypoint.sh"]
CMD [ "-c", "ssl=on" , "-c", "ssl_cert_file=/var/lib/postgresql/postgresdb.crt", "-c",\
"ssl_key_file=/var/lib/postgresql/postgresdb.${EXT_KEY}", "-c",\
"ssl_ca_file=/var/lib/postgresql/myCA.crt", "-c", "ssl_crl_file=/var/lib/postgresql/myCA.crl" ]
docker-compose.yml
version: "3.8"
services:
news_database:
build:
context: ../..
dockerfile: ./.docker/dev/db.Dockerfile
container_name: news_database
restart: unless-stopped
env_file:
- .env
ports:
- "5432:5432"
volumes:
- news_db:/var/lib/postgresql/data
news_app:
...
volumes:
news_db:
driver: local
When I run this the variable is not present in the CMD and therefore the container fails
Attempt 1
I tried changing the final command from array to string format
CMD -c ssl=on -c ssl_cert_file=/var/lib/postgresql/postgresdb.crt -c ssl_key_file=/var/lib/postgresql/postgresdb.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
It gives me an /bin/sh: illegal option - error
Attempt 2
I removed the entrypoint completely and tried directly calling postgres with a CMD
CMD postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/postgresdb.crt -c ssl_key_file=/var/lib/postgresql/postgresdb.key -c ssl_ca_file=/var/lib/postgresql/myCA.crt -c ssl_crl_file=/var/lib/postgresql/myCA.crl
It immediately gives me another error when I run it via docker-compose
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
All I want is to have variables inside that CMD, can someone kindly tell me a way to make this work?
Your ENTRYPOINT instruction is specified as JSON array, meaning it's in exec form - no shell will be invoked for the execution of docker-entrypoint.sh. As there is no shell invoked, there won't be any environment variable expansion.
To make it work, try this:
ENTRYPOINT [ "sh", "-c", \
"docker-entrypoint.sh \
-c ssl=on \
-c ssl_cert_file=/var/lib/postgresql/postgresdb.crt \
-c ssl_key_file=/var/lib/postgresql/postgresdb${EXT_KEY} \
-c ssl_ca_file=/var/lib/postgresql/myCA.crt \
-c ssl_crl_file=/var/lib/postgresql/myCA.crl \
${0} ${#}" ]
I've being trying to setup gitlab CI with my django project. The project uses postgis extension. After all this setup I still get the error that postgis.control file could not be found
$ export PGPASSWORD=$POSTGRES_PASSWORD
$ psql -c "CREATE EXTENSION IF NOT EXISTS postgis;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
ERROR: could not open extension control file "/usr/share/postgresql/11/extension/postgis.control": No such file or directory
ERROR: Job failed: exit code 1
Here is my .gitlab-ci.yml file
image: python:3.6
stages:
- test
services:
- mdillon/postgis
- postgres
variables:
POSTGRES_DB: my_db
POSTGRES_USER: my_user
POSTGRES_PASSWORD: ""
TESTFOLDER: "myapp/apps/api myapp/apps/logger"
DATABASE_URL: "postgres://my_user:#mdillon-postgis/my_db"
test:
stage: test
image: mdillon/postgis
before_script:
- apt-get update -qy
- export PGPASSWORD=$POSTGRES_PASSWORD
- psql -c "CREATE EXTENSION IF NOT EXISTS postgis;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
- psql -c "CREATE EXTENSION IF NOT EXISTS postgis_topology;" -d $POSTGRES_DB -U $POSTGRES_USER -h "postgres"
- apt-get install -y openjdk-8-jre-headless libjpeg-dev zlib1g-dev software-properties-common ghostscript libxslt1-dev binutils libproj-dev libgdal-dev gdal-bin memcached libmemcached-dev
- export DEBIAN_FRONTEND=noninteractive;
- pip install --upgrade pip
- pip install -r requirements/base.pip
- pip install flake8
script:
- python manage.py test $TESTFOLDER --noinput --settings=myapp.settings.gitlab_ci --parallel 4 --verbosity=2
only:
- master
In my case, I discovered that it was the host I was using to connect to the database that was the cause of the problem.
After reading through the GitLab documentation, I discovered GitLab uses the name of the service as the host for the connection. So in my case, when connecting from my Python application, I used mdillon-postgis as my host.
You can find more details here https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services
You probably don't have the postgis installed in the database server. you need ro run: sudo apt-get install postgis
I am using the official postgresql docker image (version 9.4). I have extended the Dockerfile, so I can alter the settings in the postgresql.conf etc, using a bash script. It successfully adds and runs the script on entrypoint, for a single sed command. But when I put 2 or more sed commands, I get the following error:
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/config.sh
: No such file or directoryread
/var/lib/postgresql/data/postgresql.conf
I am trying on Windows 10, in combination with Vagrant and VirtualBox, using NFS file system on shared folders, via the vagrant-winnfsd plugin.
Why is this happening? How can I alter my bash script in order to work with more configuration settings? Is there a better way?
Dockerfile:
FROM postgres:9.4
RUN echo "Europe/Athens" > /etc/timezone \
&& dpkg-reconfigure -f noninteractive tzdata
RUN localedef -i el_GR -c -f UTF-8 -A /usr/share/locale/locale.alias el_GR.UTF-8
ADD config.sh /docker-entrypoint-initdb.d/
RUN chmod 755 /docker-entrypoint-initdb.d/config.sh
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
config.sh:
#!/bin/bash
sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
database.yml
postgres:
container_name: postgres-9.4
image: ***/postgres-9.4
volumes_from:
- postgres_data
ports:
- 5432:5432
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
- USERMAP_UID=999
- USERMAP_GID=999
postgres_data:
container_name: postgres_data
image: ***/postgres-9.4
volumes:
- ./services/postgres:/etc/postgresql
- ./services/postgres:/var/lib/postgresql
- ./services/postgres/logs:/var/log/postgresql
command: "true"
You might want to try using a RUN statement to execute your bash script or just run sed directly with both commands combined with a semicolon:
RUN sed -i -e 's/^#\(logging_collector = \).*/\1on/; s/^\(max_connections = \).*/\11000/' \
/var/lib/postgresql/data/postgresql.conf
A more scalable solution would be to put the sed program in an external file, then use these statements:
ADD postgres-edit.sed /var/local
RUN sed -i -f /var/local/postgres-edit.sed /var/lib/postgresql/data/postgresql.conf
postgres-edit.sed:
# sed script to edit postgresql configuration
s/^#\(logging_collector = \).*/\1on/
s/^\(max_connections = \).*/\11000/
Seems like a duplicate of How to customize the configuration file of the official PostgreSQL docker image?.
Copy-paste of my answer at https://stackoverflow.com/a/40598124/385548.
Inject custom postgresql.conf into postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize PostgreSQL configuration in Docker consistently, I suggest using config_file postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the postgres server with psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;