Is a new postgres database created upon every docker-compose up? - postgresql

Let's say I have the following setup in my docker-compose.yml.
services:
postgres:
image: postgres:11.6
env_file:
- .local.env
volumes:
- ./database/:/docker-entrypoint-initdb.d
ports:
- 5432:5432
...
where ./database contains some SQL files that initialize the database. Here's my question... is initdb run every single time the stopped postgres container starts running again (via $ docker-compose up).
Thus, is it fair to say that every time I restart my postgres container, it builds the entire database from scratch all over again?
My guess is 'yes' as in the documentation it says
The default postgres user and database are created in the entrypoint with initdb.

The answer is no, when you stop your container it is not deleted, only stopped, you can start it when it is stopped the same when you stop your computer it will not vanish from your desk :)
You can even restart it when it is running, same as you would do with your computer.
However when you remove/delete the container with
docker rm -f containername
or
docker-compose rm
then it is truly deleted, equivalent of making your computer vanish from your desk.
But even than you can still persist your data with volume mounts, for example in your compose file your ./database directory will not be deleted from your host machine even when you delete the containers using it. It is the equivalent of using an external usb drive in your computer, so when you make your computer vanish from your desk with deleting it, you still have your usb drive with the data on it that was there when you still had your computer.
So you can persist your database files with the same technique in a volume mount like this:
services:
postgres:
image: postgres:11.6
env_file:
- .local.env
volumes:
- ./database/:/docker-entrypoint-initdb.d
- ./postgres-data/data:/var/lib/postgresql/data
ports:
- 5432:5432
...
This way when you delete your container(s) and do "docker-compose up" again for the same compose file, postgres will not run its init scirpt because the /var/lib/postgresql directory is already populated in it.
However, my computer analogy is valid only in this context, please do not think of containers being mini computers or mini virtual machines, they are not! But that's an other discussion.

Related

Postgres running via docker not persisting data after initialization script

I'm using docker for the first time to set up a test database that my team can then use. I'm having some trouble getting my data on DBeaver after running my docker-compose file. The issue I'm facing is that my database does not show up in DBeaver (along with relevant Schemas and Tables that I also create/populate in my initialization sql script).
Here is my docker-compose.yml
version: "3"
services:
test_database:
image: postgres:latest
build:
context: ./
dockerfile: Dockerfile
restart: always
ports:
- 5432:5432
environment:
- POSTGRES_USER=dev
- POSTGRES_PASSWORD=test1234
- POSTGRES_DB=testdb
container_name: test_database
In this, I specify the docker file I want it to use for building. Here is the dockerfile:
# syntax = docker/dockerfile:1.3
FROM postgres:latest
ADD test_data.tar .
COPY init_test_db.sql /docker-entrypoint-initdb.d/
Now, when I run docker-compose build and docker-compose up, I can see through the logs that my SQL commands (CREATE, COPY, etc.) do get executed and the rows do get added. But when I connect to this instance through DBeaver, I can't see this at all. In fact, the only database on there is the default Postgres database, even through the logs say I'm connected to test_database.
I followed some other solutions and used docker volume prune as well, but that didn't affect anything (I read some solutions about clearing up volumes, and at that point, I had volumes: /tmp:/tmp as well). Any ideas?
Wow, this wasn't an error after all. All I had to do was go on the connection settings on DBeaver and check 'Show all databases' under the Postgres tab. Hope this can help someone :)

Unable to login to Postgres

I am not able to login into my postgres databse deployed in docker. PFB my docker-compose.yml
discountdb:
image: postgres
docker-compose.override.yml
discountdb:
container_name: discountdb
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=admin1234
- POSTGRES_DB=DiscountDb
restart: always
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data/
When I am trying to create server from pgAdmin4, I am getting the following error
Following is the db logs
What did I miss?
The environment variables you set are only used if Postgres can't find an existing database when then container starts.
Since you map a docker volume to /var/lib/postgresql/data/, chances are that you already have a database with existing users defined.
Try removing the volume mapping, so you're sure that Postgres creates a fresh database. If that solves it, then you have two options:
If you don't need the data in the volume, you can delete the postgres_data volume so Postgres creates a new database
If you need the data, you need to find out what userid/password you need to use to access the existing database in the volume
Nothing is wrong here but the POSTGRES_PASSWORD is taken into account the first time you start the container I mean when your postgres_data folder is still empty. If you changed the password but postgres_data is not empty the new password is ignored, you must log in with the first one.
The issue was with my postgres_data volume. I removed the volume using the command
docker volume rm -f discountdb
then ran the docker-compose again which resolved the issue.

How to make sure docker-compose will not remove my volume with postgres data

I am running a simple django webapp with docker-compose. I define both a web service and a db service in a docker-compose.yml file:
version: "3.8"
services:
db:
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
env_file:
- ./.env.dev
depends_on:
- db
volumes:
postgres_data:
I start the service by running:
docker-compose up -d
I can load some data in there with a custom django command that I wrote for my app. Everything is running fine (with data) on localhost:8000.
However, when I run
docker-compose down
(so without -v) and then again
docker-compose up -d
the database is empty again. The volume was not persisted. From what I read in the docker-compose docs and also in several posts here at SO, persisting the volume and reusing it when you start a new container should be the default behavior (which, if I understand it correctly, you can disable by using the --renew-anon-volumes flag).
However in my case, the volume is not persisted. Or maybe it is, but my data is gone.
By doing docker volume ls I can see that my volume (I'll use the name my_volume here) still exists after the docker-compose down command. However, the CreatedAt value has been changed. This makes me think it's a different volume with the same name, and my data is already gone, but I don't know how to confirm that.
This SO answer suggests to mount the volume on /var/lib/postgresql instead of /var/lib/postgresql/data. However, I've seen other resources (like this one) where the opposite is suggested. I've tried both, but neither option works.
Thanks for any advice.
It turns out that the Dockerfile of my app was using an entrypoint in which the following command was executed: python manage.py flush which clears all data in the database. As this gets executed every time the app container starts, it clears all data. It had nothing to do with docker-compose.

Postgres AWS EBS volume doesn't persist when updating service

I deploy a service on a standard Docker for AWS stack (using this template).
I deploy using docker stack deploy -c docker-compose.yml pos with this compose file:
version: "3.2"
services:
postgres_vanilla:
image: postgres
volumes:
- db-data:/var/lib/postgresql
volumes:
db-data:
driver: "cloudstor:aws"
driver_opts:
size: "6"
ebstype: "gp2"
backing: "relocatable"
I then change some data in the db and force an update of the service with docker service update --force pos_postgres_vanilla
Problem is that the data I change doesn't persist after the update.
I've noticed that postgres initdb script runs every time I update, so I assume it's related.
Is there something i'm doing wrong?
Issue was that cloudstor:aws creates the volume with a lost+found under it, so when postgres starts it finds that the data directory isn't empty and complains about it. To fix that I changed the volume to be mounted one directory above the data directory, at /var/lib/postgresql, but that caused postgres to not find the PGVERSION file, which in turn caused it to run initdb every time the container starts (https://github.com/docker-library/postgres/blob/master/11/docker-entrypoint.sh#L57).
So to work around it, instead of changing the volume to be mounted one directory above the data directory, I changed the data directory to be one level below the volume mount by overriding environment variable PGDATA (to something like /var/lib/postgresql/data/db/).

Need some advice dockerizing MongoDB

I am playing with MongoDB and Docker and at this point I am trying to create a useful image for myself to use at work. I have created the following Dockerfile:
FROM mongo:2.6
VOLUME /data/db /data/configdb
CMD ["mongod"]
EXPOSE 27017
And I have added it to my docker-compose.yml file:
version: '2'
services:
### PHP/Apache Container
php-apache:
container_name: "php55-dev"
image: reynierpm/php55-dev
ports:
- "80:80"
environment:
PHP_ERROR_REPORTING: 'E_ALL & ~E_DEPRECATED & ~E_NOTICE'
volumes:
- ~/mmi:/var/www
- ~/data:/data
links:
- mongodb
### MongoDB Container
mongodb:
container_name: "mongodb"
build: ./mongo
environment:
MONGODB_USER: "xxxx"
MONGODB_DATABASE: "xxxx"
MONGODB_PASS: "xxxx"
ports:
- "27017:27017"
volumes:
- ~/data/mongo:/data/db
I have some questions regarding this setup I have made:
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
I am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
VOLUME is not required when you are mounting a host directory but it is helpful as metadata. VOLUME does provide some special "copy data on volume creation" semantics when mounting a Docker volume (non host dir) which will impact your data initialisation method choice.
am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
MONGO_USER, MONGO_DATABASE and MONGO_PASS do not do anything in the official mongo Docker image or to mongod itself.
The mongo image has added support for similar environment variables:
MONGO_INITDB_ROOT_USERNAME
MONGO_INITDB_ROOT_PASSWORD
MONGO_INITDB_DATABASE
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Whether you initialise data at build or runtime is up to your usage. As mentioned previously, Docker can copy data from a specified VOLUME into a volume it creates. If you are mounting a host directory you probably need to do the initialisation at run time.
mongorestore requires a running server to restore to. During a build you would need to launch the server and restore in the same RUN step. At runtime you might need to include a startup script that checks for existence of your database.
Mongo is able to initialise any empty directory into a blank mongo instance so you don't need to be worried about mongo not starting.