I have a Postgres database running on my host. The datafiles for the database is stored at /usr/local/var/postgresql#13.
To get the full system running easily I'd like to have a Docker with a Postgres service running for other Docker apps to connect to. I would however like to have the Docker Postgres service to use the existing datafiles on the host ...
How do I set up the volume correctly to point to the hosts database files?
Do have to have a user/password when running the Docker against existing datafiles?
I have the following but can get the volume to work ...
version: "3.9"
services:
web:
build: .
ports:
- 8081:3011
depends_on:
- db
environment:
- PGHOST=db
- PGDATABASE=loggingtestdb
- PGUSER=postgres
- PGPASSWORD=postgres
db:
image: postgres
ports:
- 5432:5432
volumes:
- /usr/local/var/postgresql#13 <--- Need help here.
How do I map the container pg datafile location to the hosts pg datafile location? 🙏
Update 1
This is the datafile folder for the db on the host
After comments I updated the volumes to below
volumes:
- /usr/local/var/postgresql#13:/var/lib/postgresql/data
But when running docker compose I only get
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /usr/local/var/postgresql#13
Update 2
/use/local works fine. But as soon as I add the /var folder to the path Docker for some reason can’t find it … What am I missing here?
Related
Can anyone tell me where I am going wrong. All I am trying to do is name a Mongo database using docker compose.
I have a docker compose file that looks like this:
version: "3"
services:
mongo-db:
image: mongo
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
- MONGO_INITDB_DATABASE=mydbname
ports:
- 27017:27017
volumes:
- mongo-db:/data/db
volumes:
mongo-db:
I run docker docker-compose -f docker-compose.yml up -d --build and it runs. I then open Robo 3T and connect to my container but every time I do the database is called test and not mydbname. Any ideas? TIA
The environment variables are only used to create a new database if no database already exists. You map a volume to /data/db and that volume probably contains an existing database named 'test'.
Find the volume using docker volume ls. It's called something like <directory name>_mongo-db. Then delete it using docker volume rm <volume name>.
Now Docker will create a new, empty volume and Mongo will create a new database when you start the container. And it'll use the values from the environment variables.
When I created volume in Docker using command:
docker volume create pg-data
Then I set up basic postgresql database from postgres image:
docker run --rm -v pg-data:/var/lib/postgresql/data --name pg-docker -e POSTGRES_PASSWORD=docker -p 5433:5432 postgres
Everything worked fine. Database persist and I can even access it directly from the host. I created several roles here like app_user_1.
Now then I wanted to spin up postgresql in container using docker-compose. I shutdown the above postgresql container beforehand.
There I have this settting:
version: '3.7'
services:
db:
image: postgres
volumes:
- pg-data:/var/lib/postgresql/data/
expose:
- 5432
restart: always
environment:
- POSTGRES_PASSWORD=docker
- POSTGRES_USER=postgres
web:
build: .
volumes:
- ./app:/app
ports:
- 8001:8000
environment:
- ENVIRONMENT=dev
- TESTING=0
depends_on:
- db
volumes:
pg-data:
However it seems that even though I mapped the same volume and used same env settings as in docker run command the postgresql instance in container created with docker-compose has no databases and no roles at all.
I get the following error:
psql: error: FATAL: role "postgres" does not exist
or
psql: error: FATAL: role "app_user_1" does not exist
So it seems it behaves as though as it is different instance of postgresql.
When I restarted the first container with docker run everything was there (all the databases and roles).
Any idea why this is happening? How can I reuse the databases from the first container in the docker-compose?
You need to define the volume you wish to use (the one you created manually with docker volume create as external to docker-compose as it was created externally
This is because the volumes created by docker-compose are 'internal' to it, so using ones created by just docker are 'external'. =)
Ref the offical docs at https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose
The change to your compose file would be as follows:
...
volumes:
pg-data:
external: true
(Just that last line)
Hope that helps! =)
Additional Note
You can confirm this, by performing a docker volume ls | grep pg-data command which will list all volumes, then only show you the ones referencing 'pg-data'.
On my system where I was testing before I gave my answer, I get the following:
docker volume ls | grep pg-data
local pg-data
local postgresstackoverflow_pg-data
As you can see, the docker volume create one is listed first, as a local volume called 'pg-data', then the docker-compose.yml created one is next prefixed by the naming convention of docker-compose with the directory name that it was in at the time.
New to Docker and I'm trying to set Postgres and pgadmin4 to run as a single service on docker for Mac inside a virtual machine. Everything works but as soon as I stop the service my data is gone. I'm using a named volume to persist data but probably doing something wrong. What is it?
Here's my setup:
# create my VM
docker-machine create dbvm
# set the right environment
eval $(docker-machine env dbvm)
Here's my docker-compose.yaml file:
version: '3'
services:
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=my_db
volumes:
- pgdata:/pgdata
ports:
- 5432:5432
pgadmin:
image: fenglc/pgadmin4
ports:
- 5050:5050
volumes:
- pgadmindata:/pgadmindata
volumes:
pgdata:
pgadmindata:
With docker-compose.yaml, I run:
docker stack deploy -c docker-compose.yaml dbstack
I can do everything on this setup, but if I run docker stack rm dbstack the data is gone after this, but the volumes still exist.
$ docker volume ls
DRIVER VOLUME NAME
local 0c15b0b22c6b850e8768c14045da166253424dda4df8d2e13df75fd54d833412
local 22bab81d9d1de0e07de97363596b096f944752eba617ff574a0ab525239227f5
local 6da6e29fb98ad0f66d7da6a75dc76066ce014b26ea43567c55ed318fda707105
local dbstack_pgadmindata
local dbstack_pgdata
What am I missing?
Unless you have it in some config not shown, I believe you need to map to the default data location inside the container e.g., pgdata:/var/lib/postgresql/data
#Idg is partially correct. postgres data lives at /var/lib/postgresql/data per the Docker Hub readme.
But for it to work in your named volume, you can't use a path on the left side, so correct value would be:
volumes:
- pgdata:/var/lib/postgresql/data
Then the postgres data will stay in that named volume, on the node it was created on.
I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.
I run:
docker-compose build
then
docker-compose up
And I receive the following error:
ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""
My full docker-compose.yml looks like this (note the service called db where the volume is defined):
version: '3'
services:
nginx:
image: nginx:latest
ports:
- 80:8000
volumes:
- ./src:/src
- ./config/nginx:/etc/nginx/conf.d
- ./src/static:/static
depends_on:
- web
web:
build: .
command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
depends_on:
- db
volumes:
- ./src:/src
- ./src/static:/static
expose:
- 8000
db:
image: postgres:latest
volumes:
- /private/var/lib/postgresql:/var/lib/postgresql
ports:
- 5432:5432
Any ideas for how to solve?
The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:
VOLUME /var/lib/postgresql/data
Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.
For clarity, here is the volume the docker engine created for you:
/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data
And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:
mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied
Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.
As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.
I'm trying to create a couple of containers with Docker. One is a postgres:latest image and the other is ubuntu:latest image with postgresql-client installed.
I have an existing database cluster in my localhost that I've used before install docker, of course. Now I want to use that cluster in my PostgreSQL container. The path in my computer is /Users/Marco/Data.
I've created a volume too, with the command docker volume create --opt device=/Users/Marco/Data data_container to store the cluster in it.
Then tried to make a docker-compose.ymlfile with the following content:
version: '2'
services:
db:
image: postgres
environment:
- PGDATA=/var/lib/postgresql/data/
ports:
- "5432:5432"
volumes:
- data_container:/var/lib/postgresql/data/pgdata
shell:
image: ubuntu_pgsql
command: /bin/bash
tty: true
stdin_open: true
links:
- db
volumes:
data_container:
external: true
When I want to launch the containers with docker-compose up -d it shows me the following error:
ERROR: for db Cannot start service db: error while mounting volume '/var/lib/docker/volumes/data_container/_data': error while mounting volume with options: type='' device='/Users/Marco/Data' o='': no such device
ERROR: Encountered errors while bringing up the project.
What could be failing? Thanks.
If data container is a directory you should declare as:
volumes:
- ./data_container:/var/lib/postgresql/data/pgdata
If you used a data_container without ./ before, docker compose understand that is a file, not a directory.