New to Docker and I'm trying to set Postgres and pgadmin4 to run as a single service on docker for Mac inside a virtual machine. Everything works but as soon as I stop the service my data is gone. I'm using a named volume to persist data but probably doing something wrong. What is it?
Here's my setup:
# create my VM
docker-machine create dbvm
# set the right environment
eval $(docker-machine env dbvm)
Here's my docker-compose.yaml file:
version: '3'
services:
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=my_db
volumes:
- pgdata:/pgdata
ports:
- 5432:5432
pgadmin:
image: fenglc/pgadmin4
ports:
- 5050:5050
volumes:
- pgadmindata:/pgadmindata
volumes:
pgdata:
pgadmindata:
With docker-compose.yaml, I run:
docker stack deploy -c docker-compose.yaml dbstack
I can do everything on this setup, but if I run docker stack rm dbstack the data is gone after this, but the volumes still exist.
$ docker volume ls
DRIVER VOLUME NAME
local 0c15b0b22c6b850e8768c14045da166253424dda4df8d2e13df75fd54d833412
local 22bab81d9d1de0e07de97363596b096f944752eba617ff574a0ab525239227f5
local 6da6e29fb98ad0f66d7da6a75dc76066ce014b26ea43567c55ed318fda707105
local dbstack_pgadmindata
local dbstack_pgdata
What am I missing?
Unless you have it in some config not shown, I believe you need to map to the default data location inside the container e.g., pgdata:/var/lib/postgresql/data
#Idg is partially correct. postgres data lives at /var/lib/postgresql/data per the Docker Hub readme.
But for it to work in your named volume, you can't use a path on the left side, so correct value would be:
volumes:
- pgdata:/var/lib/postgresql/data
Then the postgres data will stay in that named volume, on the node it was created on.
Related
When I created volume in Docker using command:
docker volume create pg-data
Then I set up basic postgresql database from postgres image:
docker run --rm -v pg-data:/var/lib/postgresql/data --name pg-docker -e POSTGRES_PASSWORD=docker -p 5433:5432 postgres
Everything worked fine. Database persist and I can even access it directly from the host. I created several roles here like app_user_1.
Now then I wanted to spin up postgresql in container using docker-compose. I shutdown the above postgresql container beforehand.
There I have this settting:
version: '3.7'
services:
db:
image: postgres
volumes:
- pg-data:/var/lib/postgresql/data/
expose:
- 5432
restart: always
environment:
- POSTGRES_PASSWORD=docker
- POSTGRES_USER=postgres
web:
build: .
volumes:
- ./app:/app
ports:
- 8001:8000
environment:
- ENVIRONMENT=dev
- TESTING=0
depends_on:
- db
volumes:
pg-data:
However it seems that even though I mapped the same volume and used same env settings as in docker run command the postgresql instance in container created with docker-compose has no databases and no roles at all.
I get the following error:
psql: error: FATAL: role "postgres" does not exist
or
psql: error: FATAL: role "app_user_1" does not exist
So it seems it behaves as though as it is different instance of postgresql.
When I restarted the first container with docker run everything was there (all the databases and roles).
Any idea why this is happening? How can I reuse the databases from the first container in the docker-compose?
You need to define the volume you wish to use (the one you created manually with docker volume create as external to docker-compose as it was created externally
This is because the volumes created by docker-compose are 'internal' to it, so using ones created by just docker are 'external'. =)
Ref the offical docs at https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose
The change to your compose file would be as follows:
...
volumes:
pg-data:
external: true
(Just that last line)
Hope that helps! =)
Additional Note
You can confirm this, by performing a docker volume ls | grep pg-data command which will list all volumes, then only show you the ones referencing 'pg-data'.
On my system where I was testing before I gave my answer, I get the following:
docker volume ls | grep pg-data
local pg-data
local postgresstackoverflow_pg-data
As you can see, the docker volume create one is listed first, as a local volume called 'pg-data', then the docker-compose.yml created one is next prefixed by the naming convention of docker-compose with the directory name that it was in at the time.
I create an Dockerfile with Postgresql with this code:
FROM postgres:9.4
MAINTAINER Fabio Ebner
ENV POSTGRES_PASSWORD="dna44100"
ENV POSTGRES_PORT=5432
EXPOSE ${POSTGRES_PORT}
COPY init.sql /docker-entrypoint-initdb.d/
so How can I specify to always save my db data in my user Machine? cause with this code everty time I stop the container my data are lost
You will need to mount a volume. pointing your host machine to the container's directory /var/lib/postgresql
Source: docker mounting volumes on host
You need to mount a volume to the data directory of PostgreSQL.
You can use the following, using the docker-compose file:
version: "3"
services:
test-postgresql:
image: postgres:9.4
container_name: test-postgresql
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: dna44100
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
- ./folder-on-host:/var/lib/postgresql/data
With the docker-compose file you can start the container with docker-compose up and stop the container with docker-compose down. The database and settings are saved on the specified directory (./folder-on-host).
If you want to remove the volume you can use the command: docker-compose down -v
You can also use the docker run to mount a volume, using the -v or -volume option:
docker run -v ./folder-on-host:/var/lib/postgresql/data yourimagename
I'm trying to create a couple of containers with Docker. One is a postgres:latest image and the other is ubuntu:latest image with postgresql-client installed.
I have an existing database cluster in my localhost that I've used before install docker, of course. Now I want to use that cluster in my PostgreSQL container. The path in my computer is /Users/Marco/Data.
I've created a volume too, with the command docker volume create --opt device=/Users/Marco/Data data_container to store the cluster in it.
Then tried to make a docker-compose.ymlfile with the following content:
version: '2'
services:
db:
image: postgres
environment:
- PGDATA=/var/lib/postgresql/data/
ports:
- "5432:5432"
volumes:
- data_container:/var/lib/postgresql/data/pgdata
shell:
image: ubuntu_pgsql
command: /bin/bash
tty: true
stdin_open: true
links:
- db
volumes:
data_container:
external: true
When I want to launch the containers with docker-compose up -d it shows me the following error:
ERROR: for db Cannot start service db: error while mounting volume '/var/lib/docker/volumes/data_container/_data': error while mounting volume with options: type='' device='/Users/Marco/Data' o='': no such device
ERROR: Encountered errors while bringing up the project.
What could be failing? Thanks.
If data container is a directory you should declare as:
volumes:
- ./data_container:/var/lib/postgresql/data/pgdata
If you used a data_container without ./ before, docker compose understand that is a file, not a directory.
I'm trying to persist postgres data in a docker container so that as soon as you docker-compose down and docker-compose up -d you don't lose data from your previous session. I haven't been able to make it do much of anything - pulling the container down and back up again routinely deletes the data.
Here's my current docker-compose.yml:
version: '2'
services:
api:
build: .
ports:
- '8245:8245'
volumes:
- .:/home/app/api
- /home/app/api/node_modules
- /home/app/api/public/src/bower_components
links:
- db
db:
build: ./database
env_file: .env
ports:
- '8246:5432'
volumes_from:
- dbdata
dbdata:
image: "postgres:9.5.2"
volumes:
- /var/lib/postgresql/data
Help?
According to the Dockment of Docker Compose, when you write something like:
volumes:
- /var/lib/postgresql/data
It creates a new docker volume and map it to /var/lib/postgresql/data inside the container.
Therefore, each time you run docker-compose up and docker-compose down, it creates new volume. You can confirm the behavior with docker volume ls.
To avoid it, you have two options:
(A) Map host directory into container
You can map directory of host into container using <HOST_PATH>:<CONTAINER_PATH>.
volumes:
- /path/to/your/host/directory:/var/lib/postgresql/data
The data of postgresql will be saved into /path/to/your/host/directory of the container host.
(B) Use external container
docker-compose has an option of external container.
When it is set to true, it won't always create volume.
Here's an example.
version: '2'
services:
dbdata:
image: postgres:9.5.2
volumes:
- mypostgresdb:/var/lib/postgresql/data
volumes:
mypostgresdb:
external: true
With external: true, docker-compose won't create the mypostgredb volume, so you have to create it by your own using following command:
docker volume create --name=mypostgredb
The data of postgresql will be saved into docker volume named mypostgredb. Read reference for more detail.
I have a main service in my docker-compose file that uses postgres's image and, though I seem to be successfully connecting to the database, the data that I'm writing to it is not being kept beyond the lifetime of the container (what I did is based on this tutorial).
Here's my docker-compose file:
main:
build: .
volumes:
- .:/code
links:
- postgresdb
command: python manage.py insert_into_database
environment:
- DEBUG=true
postgresdb:
build: utils/sql/
volumes_from:
- postgresdbdata
ports:
- "5432"
environment:
- DEBUG=true
postgresdbdata:
build: utils/sql/
volumes:
- /var/lib/postgresql
command: true
environment:
- DEBUG=true
and here's the Dockerfile I'm using for the postgresdb and postgresdbdata services (which essentially creates the database and adds a user):
FROM postgres
ADD make-db.sh /docker-entrypoint-initdb.d/
How can I get the data to stay after the main service has finished running, in order to be able to use it in the future (such as when I call something like python manage.py retrieve_from_database)? Is /var/lib/postgresql even the right directory, and would boot2docker have access to it given that it's apparently limited to /Users/?
Thank you!
The problem is that Compose creates a new version of the postgresdbdata container each time it restarts, so the old container and its data gets lost.
A secondary issue is that your data container shouldn't actually be running; data containers are really just a namespace for a volume that can be imported with --volumes-from, which still works with stopped containers.
For the time being the best solution is to take the postgresdbdata container out of the Compose config. Do something like:
$ docker run --name postgresdbdata postgresdb echo "Postgres data container"
Postgres data container
The echo command will run and the container will exit, but as long as don't docker rm it, you will still be able to refer to it in --volumes-from and your Compose application should work fine.