Docker mongodb config file - mongodb

There is a way to link /data/db directory of the container to your localhost. But I can not find anything about configuration. How to link /etc/mongo.conf to anything from my local file system. Or maybe some other approach is used. Please share your experience.

I'm using the mongodb 3.4 official docker image. Since the mongod doesn't read a config file by default, this is how I start the mongod service:
docker run -d --name mongodb-test -p 37017:27017 \
-v /home/sa/data/mongod.conf:/etc/mongod.conf \
-v /home/sa/data/db:/data/db mongo --config /etc/mongod.conf
removing -d will show you the initialization of the container
Using a docker-compose.yml:
version: '3'
services:
mongodb_server:
container_name: mongodb_server
image: mongo:3.4
env_file: './dev.env'
command:
- '--auth'
- '-f'
- '/etc/mongod.conf'
volumes:
- '/home/sa/data/mongod.conf:/etc/mongod.conf'
- '/home/sa/data/db:/data/db'
ports:
- '37017:27017'
then
docker-compose up

When you run docker container using this:
docker run -d -v /var/lib/mongo:/data/db \
-v /home/user/mongo.conf:/etc/mongo.conf -p port:port image_name
/var/lib/mongo is a host's mongo folder.
/data/db is a folder in docker container.

I merely wanted to know the command used to specify a config for mongo through the docker run command.
First you want to specify the volume flag with -v to map a file or directory from the host to the container. So if you had a config file located at /home/ubuntu/ and wanted to place it within the /etc/ folder of the container you would specify it with the following:
-v /home/ubuntu/mongod.conf:/etc/mongod.conf
Then specify the command for mongo to read the config file after the image like so:
mongo -f /etc/mongod.conf
If you put it all together, you'll get something like this:
docker run -d --net="host" --name mongo-host -v /home/ubuntu/mongod.conf:/etc/mongod.conf mongo -f /etc/mongod.conf

For some reason I should use MongoDb with VERSION:3.0.1
Now : 2016-09-13 17:42:06
That is what I found:
#first step: run mongo 3.0.1 without conf
docker run --name testmongo -p 27017:27017 -d mongo:3.0.1
#sec step:
docker exec -it testmongo cat /entrypoint.sh
#!/bin/bash
set -e
if [ "${1:0:1}" = '-' ]; then
set -- mongod "$#"
fi
if [ "$1" = 'mongod' ]; then
chown -R mongodb /data/db
numa='numactl --interleave=all'
if $numa true &> /dev/null; then
set -- $numa "$#"
fi
exec gosu mongodb "$#"
fi
exec "$#"
I find that there are two ways to start a mongod service.
What I try:
docker run --name mongo -d -v your/host/dir:/container/dir mongo:3.0.1 -f /container/dir/mongod.conf
the last -f is the mongod parameter, you can also use --config instead.
make sure the path like your/host/dir exists and the file mongod.conf in it.

Related

Docker 'backup' process container not seeing Database container postgres

I have a simple docker-compose.yml & associated Dockerfiles that give me a simple dev and prod environment for a nginx-uvicorn-django-postgres stack. I want to add an optional 'backup' container that just runs cron to periodically connect to the 'postgres' container.
# backup container - derived from [this blog][1]
ARG DOCKER_REPO
ARG ALPINE_DOCKER_IMAGE # ALPINE
ARG ALPINE_DOCKER_TAG # LATEST
FROM ${DOCKER_REPO}${ALPINE_DOCKER_IMAGE}:${ALPINE_DOCKER_TAG}
ARG DB_PASSWORD
ARG DB_HOST # "db"
ARG DB_PORT # "5432"
ARG DB_NAME # "ken"
ARG DB_USERNAME # "postgres"
ENV PGPASSWORD=${DB_PASSWORD} HOST=${DB_HOST} PORT=${DB_PORT} PSQL_DB_NAME=${DB_NAME} \
USERNAME=${DB_USERNAME}
RUN printenv
RUN mkdir /output && \
mkdir /output/backups && \
mkdir /scripts && \
chmod a+x /scripts
COPY ./scripts/ /scripts/
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/15min/${DB_NAME}_15
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/daily/${DB_NAME}_day
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/weekly/${DB_NAME}_week
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/monthly/${DB_NAME}_month
RUN apk update && \
apk upgrade && \
apk add --no-cache postgresql-client && \
chmod a+x /etc/periodic/15min/${DB_NAME}_15 && \
chmod a+x /etc/periodic/daily/${DB_NAME}_day && \
chmod a+x /etc/periodic/weekly/${DB_NAME}_week && \
chmod a+x /etc/periodic/monthly/${DB_NAME}_month
The django container is derived from the official Python image and connects (through psycopg2) with values (as ENV value) for host, dbname, username, password and port. The 'backup' container has these same values, but I get this error from the command line:
> pg_dump --host="$HOST" --port="$PORT" --username="$USERNAME" --dbname="$PSQL_DB_NAME"
pg_dump: error: could not translate host name "db" to address: Name does not resolve
Is Alpine missing something relevant that is present in the official Python?
Edit:
I am running with a system of shell scripts that take care of housekeeping for different configurations. so
> ./ken.sh dev_server
will set up the environment variables and then run docker-compose for the project and the containers
docker-compose.yml doesn't explicitly create a network.
I don't know what "db" should resolve to beyond just 'db://'? - its what the django container gets and it is able to resolve a connection to the 'db' service.
service:
db:
image: ${DOCKER_REPO}${DB_DOCKER_IMAGE}:${DB_DOCKER_TAG} #postgres: 14
container_name: ${PROJECT_NAME}_db
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- PGPASSWORD
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
command: ["postgres", "-c", "log_statement=all"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h db"]
interval: 2s
timeout: 5s
retries: 25
This is the 'dev_server' script run by the parent ken.sh script
function dev_server() {
trap cleanup EXIT
wait_and_launch_browser &
docker-compose -p "${PROJECT_NAME}" up -d --build db nginx web pgadmin backup
echo "Generate static files and copy them into static and file volumes."
source ./scripts/generate_static_files.sh
docker-compose -p "${PROJECT_NAME}" logs -f web nginx backup
}
Update: Worked through "Reasons why docker containers can't talk to each other" and found that all the containers are on a ken_default network, from 170.20.0.2 to 170.20.0.6.
I can docker exec ken_backup backup ken_db -c2, but not from db to backup, because the db container doesn't include ping.
From a shell on backup I cannot ping ken_db - ken_db doesn't resolve, nor does 'db'.
I can't make much of that and I'm not sure what to try next.
You are running the backup container as a separate service.
Docker-compose creates a unique network for each service (docker-compose.yml file).
You need to get the DB and your backup container on the same docker network.
See this post

How to restore mongoDb collection BSON backup files to docker abernix/meteord:node-8.4.0-base?

I have made a backup from a mongoDb collection with the following command
mongodump -h 127.0.0.1 --port 9001 -d meteor -c products
I have copied the dump folder recursively to my server with the following command
scp -r dump root#66.204.148.25:/root
I can not restore with the following command
docker exec -i mongodb mongorestore -d mew -c audioQuestions_Joker dump/meteor
the files are there but I get the following message
2020-01-12T11:38:10.863+0000 Failed: mongorestore target 'dump/meteor' invalid: stat dump/meteor: no such file or directory
What would be the correct command to restore the collection backup from the BSON files?
FYI docker abernix/meteord:node-8.4.0-base
Thanks
You need to have sudo privileges.
1 Modify docker-compose.* file for MongoDB container and add new volume
Suppose you have something like this:
mongo:
image: mongo:4.2
container_name: mongodb
ports:
- 27017:27017
volumes:
- "./local/path:/data/db"
- "./local/tmp/path:/home"
restart: always
command: --auth
2 You need to build
sudo docker-compose -f docker-compose.yml up -d
3 Copy inside ./local/tmp/path you dump folder and check if MongoDB container has access to it.
sudo docker ps
sudo docker exec -it mongodb_CONTAINER ID /bin/bash
# ls /home - If you see there dump folder, continue step 4
4 Now, execute MongoDB restore (you may do it inside MongoDB container)
mongorestore -d mew -c audioQuestions_Joker /home/dump/meteor
5 If you have restored successfully, update again docker-compose and remove - "./local/tmp/path:/home" and build.

Restoring the database dump of an older version of mongo to a new version of mongo

Currently, I have an older version of mongo, i.e 2.6 running on my system. I already have my site in production and have a lot of client data. I am planning an upgrade to mongo 3.2.
So, my question is whether mongorestore of mongo v3.2 work with data dump of v2.6? Or, is it known to create problems?
Any answers will be invaluable! Thanks
I asked this same question on the official MongoDB mailing list. They said not to upgrade more than 1 major version at a time. (Major versions being: 2.2, 2.4, 2.6, 3.0, 3.2, 3.4)
I didn't want to follow the normal upgrade process of installing every version Just to launch mongod and then shut it down. That feels to me like it would leave cruft behind and I like to have my infrastructure building scripted and version controlled. So, I decided to launch new EC2 instances with the latest Ubuntu (since my Mongo v2.4 servers were also 2 LTS versions behind) and the latest MongoDB. I used docker images of intermediate versions of MongoDB to do the data upgrades.
https://gist.github.com/RichardBronosky/2d04c7c2e9a5bea67cd9760a35415a3f#file-uat_mongodb_upgrade_from_prod-sh
The bulk of the solution is this:
# mongo.conf is using the default dbPath: /var/lib/mongodb
# this path is for temporary use by the mongo docker container
mkdir -p /data/db/dump
# see: https://hub.docker.com/_/mongo/ (search for /data/db)
# see: https://github.com/docker-library/mongo/blob/30d09dbd6343d3cbd1bbea2d6afde49f5d9a9295/3.4/Dockerfile#L59
cd /data/db
mongodump -h prodmongo.int
# Get major versions from https://hub.docker.com/r/library/mongo/tags/
step=0
for major_version in 2.6.12 3.0.14 3.2.11 3.4.1; do
sudo docker stop some-mongo || true
sudo docker rm some-mongo || true
sudo docker run --name some-mongo -v /data/db:/data/db -d mongo:$major_version
false; while [[ $? > 0 ]]; do
sleep 0.5
sudo docker exec -it some-mongo mongo --eval 'printjson((new Mongo()).getDBNames())'
done
if (( $step == 0 )); then
sudo docker exec -it some-mongo mongorestore /data/db/dump
fi
((step += 1))
done
# Finish up with docker
sudo rm -rf /data/db/dump/*
sudo docker exec -it some-mongo bash -c 'cd /data/db; mongodump'
sudo docker stop some-mongo
sudo docker rm some-mongo
# Load upgraded data into latest version of MongoDB (WiredTiger storage engine will be used)
mongorestore /data/db/dump
sudo rm -rf /data
As you have data from mongo 2.6, index field limitation is already fulfilled.
Mongo 3.2 will restore this backup without any issue.
The other way you can upgrade your db (if you have replica set) is to replace one 2.6 member with 3.2 and wait for sync, then other one... This will give you business continuity :-)
Bruno's answer worked for me up to version 3.4, but then there was a problem each update required setting a feature compatibility version, then at 4.2 you can't start mongo if you're still using MMAPv1, you need to switch to WiredTiger.
So, thank you very much Bruno, you got me most of the way there.
Below is my updated script, hope it helps someone out!
#!/bin/bash
# Log commands to stdout
set -o xtrace
# Exit on error
set -o errexit
# Exit on use of unset variables
set -o nounset
# Exit and report on pipe failure
set -o pipefail
# mongo.conf is using the default dbPath: /var/lib/mongodb
# this path is for temporary use by the mongo docker container
mkdir -p /data/db/dump
# see: https://hub.docker.com/_/mongo/ (search for /data/db)
# see: https://github.com/docker-library/mongo/blob/30d09dbd6343d3cbd1bbea2d6afde49f5d9a9295/3.4/Dockerfile#L59
cd /data/db
# our docker host holding the database
mongodump -h $PROD_MONGO_IP
# Get major versions from https://hub.docker.com/r/library/mongo/tags/
step=0
prev_real_major=""
for major_version in 2.6.12 3.0.15 3.2.21 3.4.24 3.6.23 4.0.27 4.2.17 4.4.10 5.0.3; do
real_major=`echo $major_version | cut -f1,2 -d "."`
sudo docker stop some-mongo || true
sudo docker rm some-mongo || true
docker run --rm -d --name some-mongo -v /data/db:/data/db mongo:$major_version
set +o errexit
false; while [[ $? > 0 ]]; do
sleep 0.5
docker exec -it some-mongo mongo --eval 'printjson((new Mongo()).getDBNames())'
if [[ $real_major > 3.3 ]]; then
docker exec -it some-mongo mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: \"$real_major\" } )"
fi
done
set -o errexit
if (( $step == 0 )); then
docker exec -it some-mongo mongorestore /data/db/dump
fi
# upgrade to WiredTiger
if [[ $real_major == 4.0 ]]; then
# delete the database dump from earlier
rm -rf /data/db/dump/
# dump the database again
docker exec -w /data/db -it some-mongo mongodump
# stop the existing mongo container
docker stop some-mongo
# delete everything in /data/db except /data/db/dump
find /data/db -mindepth 1 ! -regex '^/data/db/dump\(/.*\)?' -delete
# run the 4.0 mongo container again
docker run --rm -d --name some-mongo -v /data/db:/data/db mongo:$major_version
# restore the database, which automatically makes it wiretiger.
docker exec -it some-mongo mongorestore /data/db/dump
fi
((step += 1))
done
# Finish up with docker
sudo rm -rf /data/db/dump/*
docker exec -it some-mongo bash -c 'cd /data/db; mongodump'
docker stop some-mongo
# Commented these out because I did them manually, you decide what you want to do here
# Load upgraded data into latest version of MongoDB (WiredTiger storage engine will be used)
# mongorestore /data/db/dump
# sudo rm -rf /data

Customize the configuration of the official PostgreSQL docker image

I am using the official postgresql docker image (version 9.4). I have extended the Dockerfile, so I can alter the settings in the postgresql.conf etc, using a bash script. It successfully adds and runs the script on entrypoint, for a single sed command. But when I put 2 or more sed commands, I get the following error:
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/config.sh
: No such file or directoryread
/var/lib/postgresql/data/postgresql.conf
I am trying on Windows 10, in combination with Vagrant and VirtualBox, using NFS file system on shared folders, via the vagrant-winnfsd plugin.
Why is this happening? How can I alter my bash script in order to work with more configuration settings? Is there a better way?
Dockerfile:
FROM postgres:9.4
RUN echo "Europe/Athens" > /etc/timezone \
&& dpkg-reconfigure -f noninteractive tzdata
RUN localedef -i el_GR -c -f UTF-8 -A /usr/share/locale/locale.alias el_GR.UTF-8
ADD config.sh /docker-entrypoint-initdb.d/
RUN chmod 755 /docker-entrypoint-initdb.d/config.sh
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
config.sh:
#!/bin/bash
sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
database.yml
postgres:
container_name: postgres-9.4
image: ***/postgres-9.4
volumes_from:
- postgres_data
ports:
- 5432:5432
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
- USERMAP_UID=999
- USERMAP_GID=999
postgres_data:
container_name: postgres_data
image: ***/postgres-9.4
volumes:
- ./services/postgres:/etc/postgresql
- ./services/postgres:/var/lib/postgresql
- ./services/postgres/logs:/var/log/postgresql
command: "true"
You might want to try using a RUN statement to execute your bash script or just run sed directly with both commands combined with a semicolon:
RUN sed -i -e 's/^#\(logging_collector = \).*/\1on/; s/^\(max_connections = \).*/\11000/' \
/var/lib/postgresql/data/postgresql.conf
A more scalable solution would be to put the sed program in an external file, then use these statements:
ADD postgres-edit.sed /var/local
RUN sed -i -f /var/local/postgres-edit.sed /var/lib/postgresql/data/postgresql.conf
postgres-edit.sed:
# sed script to edit postgresql configuration
s/^#\(logging_collector = \).*/\1on/
s/^\(max_connections = \).*/\11000/
Seems like a duplicate of How to customize the configuration file of the official PostgreSQL docker image?.
Copy-paste of my answer at https://stackoverflow.com/a/40598124/385548.
Inject custom postgresql.conf into postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize PostgreSQL configuration in Docker consistently, I suggest using config_file postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the postgres server with psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;

How to customize the configuration file of the official PostgreSQL Docker image?

I'm using the official Postgres Docker image, trying to customize its configuration. For this purpose, I use the command sed to change max_connections for example:
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
I tried two methods to apply this configuration:
The first is by adding the commands to a script and copying it within the init folder: /docker-entrypoint-initdb.d.
The second method is by running the commands directly within my Dockerfile with the "RUN" command (this method worked fine with a non-official PostgreSQL image with a different path to the configuration file /etc/postgres/...).
In both cases the changes fail because the configuration file is missing (I think it's not created yet).
How should I change the configuration?
Here is the Dockerfile used to create the image:
# Database (http://www.cs3c.ma/)
FROM postgres:9.4
MAINTAINER Sabbane <contact#cs3c.ma>
ENV TERM=xterm
RUN apt-get update
RUN apt-get install -y nano
ADD scripts /scripts
# ADD scripts/setup-my-schema.sh /docker-entrypoint-initdb.d/
# Allow connections from anywhere.
RUN sed -i -e"s/^#listen_addresses =.*$/listen_addresses = '*'/" /var/lib/postgresql/data/postgresql.conf
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
# Configure logs
RUN sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_directory = 'pg_log'.*$/log_directory = '\/var\/log\/postgresql'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_filename = 'postgresql-\%Y-\%m-\%d_\%H\%M\%S.log'.*$/log_filename = 'postgresql_\%a.log'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_file_mode = 0600.*$/log_file_mode = 0644/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_truncate_on_rotation = off.*$/log_truncate_on_rotation = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_rotation_age = 1d.*$/log_rotation_age = 1d/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_min_duration_statement = -1.*$/log_min_duration_statement = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_checkpoints = off.*$/log_checkpoints = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_connections = off.*$/log_connections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_disconnections = off.*$/log_disconnections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^log_line_prefix = '\%t \[\%p-\%l\] \%q\%u#\%d '.*$/log_line_prefix = '\%t \[\%p\]: \[\%l-1\] user=\%u,db=\%d'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_lock_waits = off.*$/log_lock_waits = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_temp_files = -1.*$/log_temp_files = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#statement_timeout = 0.*$/statement_timeout = 1800000 # in milliseconds, 0 is disabled (current 30min)/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^lc_messages = 'en_US.UTF-8'.*$/lc_messages = 'C'/" /var/lib/postgresql/data/postgresql.conf
# Performance Tuning
RUN sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^shared_buffers =.*$/shared_buffers = 16GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#effective_cache_size = 128MB.*$/effective_cache_size = 48GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#work_mem = 1MB.*$/work_mem = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#maintenance_work_mem = 16MB.*$/maintenance_work_mem = 2GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_segments = .*$/checkpoint_segments = 32/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_completion_target = 0.5.*$/checkpoint_completion_target = 0.7/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#wal_buffers =.*$/wal_buffers = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#default_statistics_target = 100.*$/default_statistics_target = 100/" /var/lib/postgresql/data/postgresql.conf
VOLUME ["/var/lib/postgresql/data", "/var/log/postgresql"]
CMD ["postgres"]
With this Dockerfile, the build process produces an error:
sed: can't read /var/lib/postgresql/data/postgresql.conf: No such file or directory
With Docker Compose
When working with Docker Compose, you can use command: postgres -c option=value in your docker-compose.yml to configure Postgres.
For example, this makes Postgres log to a file:
command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs
Adapting Vojtech Vitek's answer, you can use
command: postgres -c config_file=/etc/postgresql.conf
to change the config file Postgres will use. You'd mount your custom config file with a volume:
volumes:
- ./customPostgresql.conf:/etc/postgresql.conf
Here's the docker-compose.yml of my application, showing how to configure Postgres:
# Start the app using docker-compose pull && docker-compose up to make sure you have the latest image
version: '2.1'
services:
myApp:
image: registry.gitlab.com/bullbytes/myApp:latest
networks:
- myApp-network
db:
image: postgres:9.6.1
# Make Postgres log to a file.
# More on logging with Postgres: https://www.postgresql.org/docs/current/static/runtime-config-logging.html
command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs
environment:
# Provide the password via an environment variable. If the variable is unset or empty, use a default password
# Explanation of this shell feature: https://unix.stackexchange.com/questions/122845/using-a-b-for-variable-assignment-in-scripts/122848#122848
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-4WXUms893U6j4GE&Hvk3S*hqcqebFgo!vZi}
# If on a non-Linux OS, make sure you share the drive used here. Go to Docker's settings -> Shared Drives
volumes:
# Persist the data between container invocations
- postgresVolume:/var/lib/postgresql/data
- ./logs:/logs
networks:
myApp-network:
# Our application can communicate with the database using this hostname
aliases:
- postgresForMyApp
networks:
myApp-network:
driver: bridge
# Creates a named volume to persist our data. When on a non-Linux OS, the volume's data will be in the Docker VM
# (e.g., MobyLinuxVM) in /var/lib/docker/volumes/
volumes:
postgresVolume:
Permission to write to the log directory
Note that when on Linux, the log directory on the host must have the right permissions.
Otherwise you'll get the slightly misleading error
FATAL: could not open log file
"/logs/postgresql-2017-02-04_115222.log": Permission denied
I say misleading, since the error message suggests that the directory in the container has the wrong permission, when in reality the directory on the host doesn't permit writing.
To fix this, I set the correct permissions on the host using
chgroup ./logs docker && chmod 770 ./logs
The postgres:9.4 image you've inherited from declares a volume at /var/lib/postgresql/data. This essentially means you can't copy any files to that path in your image; the changes will be discarded.
You have a few choices:
You could just add your own configuration files as a volume at run-time with docker run -v postgresql.conf:/var/lib/postgresql/data/postgresql.conf .... However, I'm not sure exactly how that will interact with the existing volume.
You could copy the file over when the container is started. To do that, copy your file into the build at a location which isn't underneath the volume then call a script from the entrypoint or cmd which will copy the file to the correct location and start Postgres.
Clone the project behind the Postgres official image and edit the Dockerfile to add your own config file in before the VOLUME is declared (anything added before the VOLUME instruction is automatically copied in at run-time).
Pass all config changes in command option in docker-compose file
Like this:
services:
postgres:
...
command:
- "postgres"
- "-c"
- "max_connections=1000"
- "-c"
- "shared_buffers=3GB"
- "-c"
...
When you run the official entrypoint (i.e, when you launch the container), it runs initdb in $PGDATA (/var/lib/postgresql/data by default), and then it stores two files in that directory:
postgresql.conf with default manual settings.
postgresql.auto.conf with settings overriden automatically with ALTER SYSTEM commands.
The entrypoint also executes any /docker-entrypoint-initdb.d/*.{sh,sql} files.
All this means you can supply a shell/SQL script in that folder that configures the server for the next boot (which will be immediately after the DB initialization, or the next time you boot the container).
Example:
conf.sql file:
ALTER SYSTEM SET max_connections = 6;
ALTER SYSTEM RESET shared_buffers;
Dockerfile file:
FROM posgres:9.6-alpine
COPY *.sql /docker-entrypoint-initdb.d/
RUN chmod a+r /docker-entrypoint-initdb.d/*
And then you will have to execute conf.sql manually in the already existing databases. Since configuration is stored in the volume, it will survive rebuilds.
An alternative is to pass the -c option as many times as you wish:
docker container run -d postgres -c max_connections=6 -c log_lock_waits=on
This way, you don't need to build a new image, and you don't need to care about already existing or not databases; all will be affected.
Inject custom postgresql.conf into Postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running the Postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize the PostgreSQL configuration in Docker consistently, I suggest using the config_file Postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the Postgres server with the psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;
You can put your custom postgresql.conf in a temporary file inside the container, and overwrite the default configuration at runtime.
To do that:
Copy your custom postgresql.conf inside your container
Copy the updateConfig.sh file in /docker-entrypoint-initdb.d/
Dockerfile
FROM postgres:9.6
COPY postgresql.conf /tmp/postgresql.conf
COPY updateConfig.sh /docker-entrypoint-initdb.d/_updateConfig.sh
updateConfig.sh
#!/usr/bin/env bash
cat /tmp/postgresql.conf > /var/lib/postgresql/data/postgresql.conf
At runtime, the container will execute the script inside /docker-entrypoint-initdb.d/ and overwrite the default configuration with your custom one.
I looked through all the answers and there is another option left: You can change your CMD value in the Dockerfile (it is not the best one, but still a possible way to achieve your goal).
Basically we need to:
Copy the config file into the Docker container
Override Postgres start options
Dockerfile example:
FROM postgres:9.6
USER postgres
# Copy Postgres config file into container
COPY postgresql.conf /etc/postgresql
# Override default Postgres config file
CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
Though I think using command: postgres -c config_file=/etc/postgresql/postgresql.conf in your docker-compose.yml file as proposed by Matthias Braun is the best option.
I was also using the official image (FROM postgres)
and I was able to change the config by executing the following commands.
The first thing is to locate the PostgreSQL config file.
This can be done by executing this command in your running database.
SHOW config_file;
I my case it returns /data/postgres/postgresql.conf.
The next step is to find out what is the hash of your running PostgreSQL docker container.
docker ps -a
This should return a list of all the running containers. In my case it looks like this.
...
0ba35e5427d9 postgres "docker-entrypoint.s…" ....
...
Now you have to switch to the bash inside your container by executing:
docker exec -it 0ba35e5427d9 /bin/bash
Inside the container check if the config is at the correct path and display it.
cat /data/postgres/postgresql.conf
I wanted to change the max connections from 100 to 1000 and the shared buffer from 128MB to 3GB.
With the sed command I can do a search and replace with the corresponding variables ins the config.
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /data/postgres/postgresql.conf
sed -i -e"s/^shared_buffers = 128MB.*$/shared_buffers = 3GB/" /data/postgres/postgresql.conf
The last thing we have to do is to restart the database within the container.
Find out which version you of PostGres you are using.
cd /usr/lib/postgresql/
ls
In my case its 12
So you can now restart the database by executing the following command with the correct version in place.
su - postgres -c "PGDATA=$PGDATA /usr/lib/postgresql/12/bin/pg_ctl -w restart"
A fairly low-tech solution to this problem seems to be to declare the service (I'm using swarm on AWS and a yaml file) with your database files mounted to a persisted volume (here AWS EFS as denoted by the cloudstor:aws driver specification).
version: '3.3'
services:
database:
image: postgres:latest
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
volumes:
postgresql:
driver: "cloudstor:aws"
postgresql_data:
driver: "cloudstor:aws"
The db comes up as initialized with the image default settings.
You edit the conf settings inside the container, e.g if you want to increase the maximum number of concurrent connections that requires a restart
stop the running container (or scale the service down to zero and then back to one)
the swarm spawns a new container, which this time around picks up your persisted configuration settings and merrily applies them.
A pleasant side-effect of persisting your configuration is that it also persists your databases (or was it the other way around) ;-)
My solution is for colleagues who needs to make changes in config before launching docker-entrypoint-initdb.d
I was needed to change 'shared_preload_libraries' setting so during it's work postgres already has new library preloaded and code in docker-entrypoint-initdb.d can use it.
So I just patched postgresql.conf.sample file in Dockerfile:
RUN echo "shared_preload_libraries='citus,pg_cron'" >> /usr/share/postgresql/postgresql.conf.sample
RUN echo "cron.database_name='newbie'" >> /usr/share/postgresql/postgresql.conf.sample
And with this patch it become possible to add extension in .sql file in docker-entrypoint-initdb.d/:
CREATE EXTENSION pg_cron;
Using docker compose you can mount a volume with postgresql.auto.conf.
Example:
version: '2'
services:
db:
image: postgres:10.9-alpine
volumes:
- postgres:/var/lib/postgresql/data:z
- ./docker/postgres/postgresql.auto.conf:/var/lib/postgresql/data/postgresql.auto.conf
ports:
- 5432:5432