I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.
I run:
docker-compose build
then
docker-compose up
And I receive the following error:
ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""
My full docker-compose.yml looks like this (note the service called db where the volume is defined):
version: '3'
services:
nginx:
image: nginx:latest
ports:
- 80:8000
volumes:
- ./src:/src
- ./config/nginx:/etc/nginx/conf.d
- ./src/static:/static
depends_on:
- web
web:
build: .
command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
depends_on:
- db
volumes:
- ./src:/src
- ./src/static:/static
expose:
- 8000
db:
image: postgres:latest
volumes:
- /private/var/lib/postgresql:/var/lib/postgresql
ports:
- 5432:5432
Any ideas for how to solve?
The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:
VOLUME /var/lib/postgresql/data
Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.
For clarity, here is the volume the docker engine created for you:
/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data
And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:
mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied
Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.
As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.
Related
I have a Postgres database running on my host. The datafiles for the database is stored at /usr/local/var/postgresql#13.
To get the full system running easily I'd like to have a Docker with a Postgres service running for other Docker apps to connect to. I would however like to have the Docker Postgres service to use the existing datafiles on the host ...
How do I set up the volume correctly to point to the hosts database files?
Do have to have a user/password when running the Docker against existing datafiles?
I have the following but can get the volume to work ...
version: "3.9"
services:
web:
build: .
ports:
- 8081:3011
depends_on:
- db
environment:
- PGHOST=db
- PGDATABASE=loggingtestdb
- PGUSER=postgres
- PGPASSWORD=postgres
db:
image: postgres
ports:
- 5432:5432
volumes:
- /usr/local/var/postgresql#13 <--- Need help here.
How do I map the container pg datafile location to the hosts pg datafile location? 🙏
Update 1
This is the datafile folder for the db on the host
After comments I updated the volumes to below
volumes:
- /usr/local/var/postgresql#13:/var/lib/postgresql/data
But when running docker compose I only get
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /usr/local/var/postgresql#13
Update 2
/use/local works fine. But as soon as I add the /var folder to the path Docker for some reason can’t find it … What am I missing here?
When I created volume in Docker using command:
docker volume create pg-data
Then I set up basic postgresql database from postgres image:
docker run --rm -v pg-data:/var/lib/postgresql/data --name pg-docker -e POSTGRES_PASSWORD=docker -p 5433:5432 postgres
Everything worked fine. Database persist and I can even access it directly from the host. I created several roles here like app_user_1.
Now then I wanted to spin up postgresql in container using docker-compose. I shutdown the above postgresql container beforehand.
There I have this settting:
version: '3.7'
services:
db:
image: postgres
volumes:
- pg-data:/var/lib/postgresql/data/
expose:
- 5432
restart: always
environment:
- POSTGRES_PASSWORD=docker
- POSTGRES_USER=postgres
web:
build: .
volumes:
- ./app:/app
ports:
- 8001:8000
environment:
- ENVIRONMENT=dev
- TESTING=0
depends_on:
- db
volumes:
pg-data:
However it seems that even though I mapped the same volume and used same env settings as in docker run command the postgresql instance in container created with docker-compose has no databases and no roles at all.
I get the following error:
psql: error: FATAL: role "postgres" does not exist
or
psql: error: FATAL: role "app_user_1" does not exist
So it seems it behaves as though as it is different instance of postgresql.
When I restarted the first container with docker run everything was there (all the databases and roles).
Any idea why this is happening? How can I reuse the databases from the first container in the docker-compose?
You need to define the volume you wish to use (the one you created manually with docker volume create as external to docker-compose as it was created externally
This is because the volumes created by docker-compose are 'internal' to it, so using ones created by just docker are 'external'. =)
Ref the offical docs at https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose
The change to your compose file would be as follows:
...
volumes:
pg-data:
external: true
(Just that last line)
Hope that helps! =)
Additional Note
You can confirm this, by performing a docker volume ls | grep pg-data command which will list all volumes, then only show you the ones referencing 'pg-data'.
On my system where I was testing before I gave my answer, I get the following:
docker volume ls | grep pg-data
local pg-data
local postgresstackoverflow_pg-data
As you can see, the docker volume create one is listed first, as a local volume called 'pg-data', then the docker-compose.yml created one is next prefixed by the naming convention of docker-compose with the directory name that it was in at the time.
I am trying to figure out how to completely remove a docker container with a postgres database and rebuild using docker-compose?
I created a server and database container using docker-compose. The database did not get set up how I wanted, so I would like to remove the database and rebuild. I assumed the easiest solution, given it is brand new would be to stop the container from running, remove the container and then run docker-compose again.
I have followed those steps, do not see any of the containers. I do not see any volumes associated with the containers. However, when I run docker-compose it appears to be using the postgres database that was previously created?
Here is what my docker-compose files consists of with user/password/db name extracted.
services:
server:
image: "node:10"
user: "node"
working_dir: /home/node/app
volumes:
- ./:/home/node/app
ports:
- 3030:3030
command: "npm start"
depends_on:
- db
db:
image: postgres:latest
restart: always
environment:
POSTGRES_USER: [user]
POSTGRES_PASSWORD: [password]
POSTGRES_DB: [db_name]
volumes:
- ./data/postgres:/var/lib/postgresql/data
I expected that by using:
docker stop [container] to stop the container, then
docker rm [container] to remove the container
I could rebuild fresh with docker-compose up
You can list the volumes used by docker with this command:
docker volume ls
Then, if needed, you can inspect the volumes to find which one your database uses:
docker volume inspect xyzvolumename
After locating the volume used by your database, delete it for a fresh start:
docker volume rm locatedvolumename
Docker stop and docker rm will not work untill you remove bind mount volume from your docker-compose.
Remove this from your docker-compose
- ./data/postgres:/var/lib/postgresql/data
or delete everything from host directory inside
./data/postgres
I'm trying to create a couple of containers with Docker. One is a postgres:latest image and the other is ubuntu:latest image with postgresql-client installed.
I have an existing database cluster in my localhost that I've used before install docker, of course. Now I want to use that cluster in my PostgreSQL container. The path in my computer is /Users/Marco/Data.
I've created a volume too, with the command docker volume create --opt device=/Users/Marco/Data data_container to store the cluster in it.
Then tried to make a docker-compose.ymlfile with the following content:
version: '2'
services:
db:
image: postgres
environment:
- PGDATA=/var/lib/postgresql/data/
ports:
- "5432:5432"
volumes:
- data_container:/var/lib/postgresql/data/pgdata
shell:
image: ubuntu_pgsql
command: /bin/bash
tty: true
stdin_open: true
links:
- db
volumes:
data_container:
external: true
When I want to launch the containers with docker-compose up -d it shows me the following error:
ERROR: for db Cannot start service db: error while mounting volume '/var/lib/docker/volumes/data_container/_data': error while mounting volume with options: type='' device='/Users/Marco/Data' o='': no such device
ERROR: Encountered errors while bringing up the project.
What could be failing? Thanks.
If data container is a directory you should declare as:
volumes:
- ./data_container:/var/lib/postgresql/data/pgdata
If you used a data_container without ./ before, docker compose understand that is a file, not a directory.
I have a main service in my docker-compose file that uses postgres's image and, though I seem to be successfully connecting to the database, the data that I'm writing to it is not being kept beyond the lifetime of the container (what I did is based on this tutorial).
Here's my docker-compose file:
main:
build: .
volumes:
- .:/code
links:
- postgresdb
command: python manage.py insert_into_database
environment:
- DEBUG=true
postgresdb:
build: utils/sql/
volumes_from:
- postgresdbdata
ports:
- "5432"
environment:
- DEBUG=true
postgresdbdata:
build: utils/sql/
volumes:
- /var/lib/postgresql
command: true
environment:
- DEBUG=true
and here's the Dockerfile I'm using for the postgresdb and postgresdbdata services (which essentially creates the database and adds a user):
FROM postgres
ADD make-db.sh /docker-entrypoint-initdb.d/
How can I get the data to stay after the main service has finished running, in order to be able to use it in the future (such as when I call something like python manage.py retrieve_from_database)? Is /var/lib/postgresql even the right directory, and would boot2docker have access to it given that it's apparently limited to /Users/?
Thank you!
The problem is that Compose creates a new version of the postgresdbdata container each time it restarts, so the old container and its data gets lost.
A secondary issue is that your data container shouldn't actually be running; data containers are really just a namespace for a volume that can be imported with --volumes-from, which still works with stopped containers.
For the time being the best solution is to take the postgresdbdata container out of the Compose config. Do something like:
$ docker run --name postgresdbdata postgresdb echo "Postgres data container"
Postgres data container
The echo command will run and the container will exit, but as long as don't docker rm it, you will still be able to refer to it in --volumes-from and your Compose application should work fine.