Multiple databases in docker and docker-compose - postgresql

I have a project consisting of two main Java apps that use eight Postgres databases, so is there a way in docker-compose to build eight different databases so that each one has a different owner and password? Can I even do that in docker-compose?
Example:
services:
postgresql:
build: db/.
ports:
- "5432:5432"
environment:
- POSTGRES_DB=database1
- POSTGRES_USER=database1
- POSTGRES_PASSWORD=database1
I know I can put all the .sql files in the docker-entrypoint-initdb.d and Postgres will make them automatically, but how do I declare what .sql file goes in what database?

Usually when I need more than one database in a docker project it's a test database. I find it easier to simply spin up a second docker container, without worrying about scripts or volume separation.
The main trick is to not conflict the default ports (e.g. 5432 for postgres) and you're good to go.
Then docker-compose can be something as simple as this:
version: '3.0'
services:
db:
image: postgres
environment:
- POSTGRES_DB
- POSTGRES_USER
- POSTGRES_PASSWORD
ports:
- ${POSTGRES_DEV_PORT}:5432
volumes:
- app-volume:/var/lib/postgresql/data
db-test:
image: postgres
environment:
- POSTGRES_DB
- POSTGRES_USER
- POSTGRES_PASSWORD
ports:
- ${POSTGRES_TEST_PORT}:5432
# Notice I don't even use a volume here since I don't care to persist test data between runs
volumes:
app-volume: #
Caveat: Obviously, more containers will typically imply in a higher memory footprint

According to this Github issue might be possible to achieve multiple databases by using bash scripts which you will have to pass in your Dockerfile
EDIT:
To create multiple Databases you could use the following script:
https://github.com/mrts/docker-postgresql-multiple-databases
or
https://github.com/MartinKaburu/docker-postgresql-multiple-databases
Which suggest that you have to clone one of the above git repos and mount it as a volume to: /docker-entrypoint-initdb.d then you would be able to pass multiple database names by using: POSTGRES_MULTIPLE_DATABASES variable

Well - take a look at this Github project: https://github.com/mrts/docker-postgresql-multiple-databases
According to official postgres docker image documentation:
If you would like to do additional initialization in an image derived from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files and source any *.sh scripts found in that directory to do further initialization before starting the service.
You will find the prepared script on that repo which you could use.

Related

Postgres running via docker not persisting data after initialization script

I'm using docker for the first time to set up a test database that my team can then use. I'm having some trouble getting my data on DBeaver after running my docker-compose file. The issue I'm facing is that my database does not show up in DBeaver (along with relevant Schemas and Tables that I also create/populate in my initialization sql script).
Here is my docker-compose.yml
version: "3"
services:
test_database:
image: postgres:latest
build:
context: ./
dockerfile: Dockerfile
restart: always
ports:
- 5432:5432
environment:
- POSTGRES_USER=dev
- POSTGRES_PASSWORD=test1234
- POSTGRES_DB=testdb
container_name: test_database
In this, I specify the docker file I want it to use for building. Here is the dockerfile:
# syntax = docker/dockerfile:1.3
FROM postgres:latest
ADD test_data.tar .
COPY init_test_db.sql /docker-entrypoint-initdb.d/
Now, when I run docker-compose build and docker-compose up, I can see through the logs that my SQL commands (CREATE, COPY, etc.) do get executed and the rows do get added. But when I connect to this instance through DBeaver, I can't see this at all. In fact, the only database on there is the default Postgres database, even through the logs say I'm connected to test_database.
I followed some other solutions and used docker volume prune as well, but that didn't affect anything (I read some solutions about clearing up volumes, and at that point, I had volumes: /tmp:/tmp as well). Any ideas?
Wow, this wasn't an error after all. All I had to do was go on the connection settings on DBeaver and check 'Show all databases' under the Postgres tab. Hope this can help someone :)

Mongodb authentication issue when using same volume

I am using mongo db in the docker container. I am using docker compose to spin up mongo. Now we have old mongo containers running and authentication is not enforced. In order to use authentication enforced and start up script I am using .env filein my docker compose file as below. But .env file and startup script are taking place only if I change the volume. By using same volume both .env file and startup script is not taking any effect. Is there any way to use the same volume and create users using .env and also use start up script.
services:
mongo:
image: mongo
container_name: mongot
restart: always
env_file:
- .env
ports:
- 27019:27017
volumes:
- /data/db8/configdb:/data/configdb
- /data/db8/db:/data/db
- $PWD/mongoentry/:/docker-entrypoint-initdb.d/
network_mode: "bridge"
command: mongod
Environment files and startup scripts are only used when creating a new database. If a database already exists, they aren't used.
In your case, you already have a database in the volume, so they aren't used. But if you change the volume so no database exists, Mongo creates a new one using your values and scripts.
From the docs:
When you start the mongo image, you can adjust the initialization of
the MongoDB instance by passing one or more environment variables on
the docker run command line. Do note that none of the variables below
will have any effect if you start the container with a data directory
that already contains a database

Can't keep postgres data persistent using Github CodeSpaces with Docker-Compose

I set up a Github codespaces environment using devcontainer.json and docker-compose.yaml. Everything works fine, but the postgres database defined in docker-compose.yml loses its data every time the container needs to be re-built.
Here's the bottom part of the docker-compose.yml
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
as you can see, I am trying to map the postgres data volume into a postgres-data volume, but this doesn't work for some reason.
What am I doing wrong that's preventing postgres data from persisting between container builds?
Another option would be to look into using Spawn. (Disclaimer - I'm one of the devs working on it).
We've written some documentation about exactly how to use Spawn-hosted databases with GitHub codespaces here: https://docs.spawn.cc/blog/2021/08/01/spawn-and-codespaces
This will allow you to provision a database thats independent from the GitHub codespace and preserve data between restarts.
You get some extra features with Spawn like arbitrary save points, resets and loading back to saved revisions with Spawn - but the key functionality of spinning up a database for a GitHub codespace and preserving data is one of the things it works extremely well for.
according to https://docs.github.com/en/codespaces/customizing-your-codespace/configuring-codespaces-for-your-project#dockerfile ,
only docker images can be pulled from source and set-up, nowhere they mention that volume persistence is guaranteed.
and after going through this https://code.visualstudio.com/docs/remote/devcontainerjson-reference looks like mounts and few other features related to volumes are not supported for codespaces.
workspaceMount : Not yet supported in Codespaces or when using Clone Repository in Container Volume.
workaround :
in .devcontainer folder where your dockerfile is present add a line like this
RUN curl https://<your_public_cloud>/your_volume.vol -O
here <your_public_cloud> can be google drive, aws or any endpoint where you have access to download the volume. its also the volume you needed to be persist.
and once its downloaded you can mount the volume to postgres service or make a hotswap.
and when you want to save, just upload the volume to your cloud storage provider.
repeat the process every time you build, and save and upload before "unbuild" or dismissing your codespace whatever you like to call.
hope that eases your issue, happy coding!
As long as you don't remove the volume with docker-compose down --volumes as an example, the data should persist.
I had the same issue; and it turned out that I had put a crontab running docker system prune -af every 15 minutes!
You could just mount a host directory, instead of using a docker volume:
volumes:
- /home/me/postgres_data:/var/lib/postgresql/data
This guarantees that no volume cleanup (accidental or deliberate) nukes your database.
Indeed the postgres docs do this in their examples. See the PGDATA environment variable.
As you don't have access to VM, maybe the directory containing your docker-compose.yml changes.
In that case, volume name may change too.
Indeed, by default, your volume name would be the following :
<directory_name>_postgres-data
Could you try a named volume (starting with compose 3.4):
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
external: false
name: postgres-data
documentation of docker-compose can be found here :
https://docs.docker.com/compose/compose-file/compose-file-v3/#name
EDIT 1
If your VM is created at each build, docker dependencies too.
volumes, networks, etc...
A persistent volume is needed somewhere (surviving VM builds).
You may have to create a directory in your local workspace, like:
/local/workspace/postgres-data/
which become in codespaces according to my understanding :
./postgres-data
Check permissions, your user may not exist in the container.
As a result your compose file become:
db:
image: postgres:latest
restart: unless-stopped
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass

creating mutilple databases inside postgres container

My Application needs 3 databases. I am using PostgreSQL.
How can I launch all the 3 databases in single container at one shot. All 3 have different tables & scripts. For each of those, *.sql files are being executed by copying in the Dockerfile.
I tried in conventional way . Didn't work.
Dockerfile:
FROM postgres
ENV POSTGRES_USER postgres
ENV POSTGRES_PASSWORD postgres
ENV POSTGRES_DB my_db_dev
COPY /devdb.sql /docker-entrypoint-initdb.d/
ENV POSTGRES_DB my_db_test
COPY /testdb.sql /docker-entrypoint-initdb.d/
ENV POSTGRES_DB my_db_prod
COPY /proddb.sql /docker-entrypoint-initdb.d/
Here, only the last DB (my_db_prod) is launching up.
How Can I make all 3 up at once ?
The ENV Dockerfile command sets an environment variable at build-time.
The COPY is also executed at build-time, basically overwriting the docker-entrypoint-initdb.d
When you invoke these actions, the latest executed overrides the previous one, thus the image final state will have POSTGRES_DB=my_db_prod and the docker-entrypoint-initdb.d will have the contents of proddb.sql.
The database is not created at build-time. Instead, it is created at run-time, following the instructions from /docker-entrypoint-initdb.d/ (/proddb.sql ) and POSTGRES_DB my_db_prod, hence this is the state in which the succession of commands from build-time left the image.
To create multiple databases, you can merge the scripts of the 3 entrypoints in a single one, or even better, have one script for each DB and have the scripts read different ENVs:
COPY ./create_second_db.sql /docker-entrypoint-initdb.d/create_second_db.sql
COPY ./create_third_db.sql /docker-entrypoint-initdb.d/create_third_db.sql
Here is a complete example that can save some time.
i follow up this way and it working for me.
my docker-compose.yml :
version: '3.7'
db:
image: postgres:10.5
container_name: pg
restart: always
environment:
POSTGRES_PASSWORD: postgres
volumes:
- /initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
i mount one folder initdb from docker host to inside postgres container like above.
inside initdb have 2 sql files. and it successful ran for both script sql.

Using docker-compose to create tables in postgresql database

I am using docker-compose to deploy a multicontainer python Flask web application. I'm having difficulty understanding how to create tables in the postgresql database during the build so I don't have to add them manually with psql.
My docker-compose.yml file is:
web:
restart: always
build: ./web
expose:
- "8000"
links:
- postgres:postgres
volumes:
- /usr/src/flask-app/static
env_file: .env
command: /usr/local/bin/gunicorn -w 2 -b :8000 app:app
nginx:
restart: always
build: ./nginx/
ports:
- "80:80"
volumes:
- /www/static
volumes_from:
- web
links:
- web:web
data:
restart: always
image: postgres:latest
volumes:
- /var/lib/postgresql
command: "true"
postgres:
restart: always
image: postgres:latest
volumes_from:
- data
ports:
- "5432:5432"
I dont want to have to enter psql in order to type in:
CREATE DATABASE my_database;
CREATE USER this_user WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE "my_database" to this_user;
\i create_tables.sql
I would appreciate guidance on how to create the tables.
It didn't work for me with the COPY approach in Dockerfile. But I managed to run my init.sql file by adding the following to docker-compose.yml:
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
init.sql was in the same directory as my docker-compose.yml.
I picked the solution from this gist. Check this article for more information.
I dont want to have to enter psql in order to type in
You can simply use container's built-in init mechanism:
COPY init.sql /docker-entrypoint-initdb.d/10-init.sql
This makes sure that your sql is executed after DB server is properly booted up.
Take a look at their entrypoint script. It does some preparations to start psql correctly and looks into /docker-entrypoint-initdb.d/ directory for files ending in .sh, .sql and .sql.gz.
10- in filename is because files are processed in ASCII order. You can name your other init files like 20-create-tables.sql and 30-seed-tables.sql.gz for example and be sure that they are processed in order you need.
Also note that invoking command does not specify the database. Keep that in mind if you are, say, migrating to docker-compose and your existing .sql files don't specify DB either.
Your files will be processed at container's first start instead of build stage though. Since Docker Compose stops images and then resumes them, there's almost no difference, but if it's crucial for you to init the DB at build stage I suggest still using built-in init method by calling /docker-entrypoint.sh from your dockerfile and then cleaning up at /docker-entrypoint-initdb.d/ directory.
I would create the tables as part of the build process. Create a new Dockerfile in a new directory ./database/
FROM postgres:latest
COPY . /fixtures
WORKDIR /fixtures
RUN /fixtures/setup.sh
./database/setup.sh would look something like this:
#!/bin/bash
set -e
/etc/init.d/postgresql start
psql -f create_fixtures.sql
/etc/init.d/postgresql stop
Put your create user, create database, create table sql (and any other fixture data) into a create_fixtures.sql file in the ./database/ directory.
and finally your postgres service will change to use build:
postgres:
build: ./database/
...
Note: Sometimes you'll need a sleep 5 (or even better a script to poll and wait for postgresql to start) after the /etc/init.d/postgresql start line. In my experience either the init script or the psql client handles this for you, but I know that's not the case with mysql, so I thought I'd call it out.