Docker: postgres volume location and permissions - postgresql

I have the following docker-compose file
version: '3.7'
volumes:
postgres-data:
services:
postgres:
environment:
- POSTGRES_PASSWORD=mypwd
- POSTGRES_USER=randomuser
image: 'postgres:14'
restart: always
volumes:
- './postgres-data:/var/lib/postgresql/data'
I seem to have multiple issues regarding the volume:
A folder named postgres-data is created in the docker-compose file location when I run up, though it seems that for other images, they get placed in the /var/lib/docker/volumes folder instead (without creating such a folder). Is this expected ? Is it a good practice to have the volume folder created in the same location as the docker-compose file, instead of the /var/lib/docker/volumes folder ?
This folder has weird ownership, I can't get into it as my current user (though I am in the docker group).
I tried reading the image documentation, especially the "Arbitrary --user Notes", but didn't understand what to do with it. I also tried not setting the POSTGRES_USER (which then defaults to postgres), but the result is the same.
What's the correct way to create a volume using this image ?

Your volume mount is explicitly to a subdirectory of the current directory
volumes:
- './postgres-data:/var/lib/postgresql/data'
# ^^ (a slash before the colon always means a bind mount)
If you want to use a named volume you need to declare that at the top level of the Compose file, and refer to the volume name (without a slash) when you use it
volumes:
postgres-data:
services:
...
volumes:
- 'postgres-data:/var/lib/postgresql/data'
# ^^ (no slash)
One isn't really "better" than the other for this case. A bind-mounted host directory is much easier to back up; a named volume will be noticeably faster on MacOS or Windows; you can directly see and edit the files with a bind mount; you can use the Docker ecosystem to clean up named volumes. For a database in particular, seeing the data files isn't very useful and I might prefer a named volume, but that's not at all a strong preference.
File ownership for bind mounts is a frequent question. On native Linux, the numeric user ID is the only thing that matters for permission checking. This is resolved by the /etc/passwd file into a username, but the host and container have different copies of this file (and that's okay). The unusual owner you're seeing with ls -l from the host matches the numeric uid of the default user in the postgres image.
That image is well-designed, though, and the upshot of the section in the Docker Hub documentation is that you can specify any Compose user: you want, probably matching the host uid owning the directory.
sudo rm -rf ./postgres-data # with the wrong owner
id -u # what's my current numeric uid?
version: '3.8'
services:
postgres:
volumes: # using a host directory
- './postgres-data:/var/lib/postgresql/data'
user: 1000 # matches the `id -u` output

Related

Location of bind-mount volume provided in docker-compose file of mongodb

How to see the location of bind-mount volume provided in the docker-compose yml file. I have created an bind-mount to persist the mongodb.
It is working fine i.e. if container is shut, then also the data is present, but I want to know where is this location present in my computer.
version : "3"
services:
eswmongodb:
image: mongo:latest
container_name: mongocont
ports:
- "27017:27017"
volumes:
- "~/mongo/db:/data/db"
if the container is shut, the data is present
There would be no way for you to know this unless you've already found it stored on the host.
The location is what you've given - ~/mongo/db. Open a terminal and cd to the path
Keep in mind that in Windows, ~ is a special character and is sometimes hidden in the file explorer. If you're using it to get to your User folder, you should prefer using environment variables instead https://superuser.com/questions/332871/what-is-the-equivalent-of-linuxs-tilde-in-windows

docker-compose: how to map a file from a container into a volume

I'm trying to share a file from one container to another other. An essential detail is that (the machine running) my docker host does not have explicit access to the file: It pulls a git repo and doesn't know about the internal file organization of the repo (with the single exception of the docker-compose file). Hence, the approach of standard mapping <host-path>:<container-path> is not applicable; e.g. this is not possible for me: How to mount a single file in a volume
Below is the docker-compose file, stripped for increased readability. Say service_1 has the file /app/awesome.txt. We then want to mount it into service_2 as /opt/awesome.txt.
# docker-compose.yml
version: '3'
services:
service_1:
volumes:
- shared_vol:/public
# how to make service_1 map 'awesome.txt' into /public ?
service_2:
volumes:
- shared_vol/awesome.txt:/opt/awesome.txt
volumes:
shared_vol:
Working solutions that I have, but are not fond of,
running a script/cmd within service_1, copying the file into the shared volume: causes race-condition as service_2 needs the file upon startup
introducing a third service, which the other two depends_on, and does nothing but put the file in the shared volume
Any help, tips or guidance is most appreciated!
can you just do something like this
version: '3.5'
volumes:
xxx:
services:
service_1:
...
volumes:
- xxx:/app
service_2:
...
volumes:
- xxx:/opt

Postgres Dockerfile exploration - VOLUME statement usage

I am looking at sample dockerfile to see how VOLUME is used , I come across the following lines from - https://github.com/docker-library/postgres/blob/master/Dockerfile-alpine.template
ENV PGDATA /var/lib/postgresql/data
# this 777 will be replaced by 700 at runtime (allows semi-arbitrary "--user" values)
RUN mkdir -p "$PGDATA" && chown -R postgres:postgres "$PGDATA" && chmod 777 "$PGDATA"
VOLUME /var/lib/postgresql/data
What is the purpose of using a volume here , here is my understanding - please confirm
Create directory pointed by $PGDATA in image file system.
Map it with the VOLUME so that any content created later as part of populating the content thorough docker-entrypont.sh by exposing a predefined directory that could be used by the container.
What if the VOLUME instr is not defined ? It might more laborious for someone to figure out where to keep custom changes unless VOLUME is not defined
Volume is define here, so when you start a container ( out of this image ) a new anonymous volume is created.
The volume will hold your sensible data in this regard, so this is all you need to "persist" during normal/soft docker image lifecycled.
Usually when the maintainers of docker images are already aware where the data, which will be sensible to keep, is located ( like here ) there will decorate the folder using VOLUME in the Dockerfile. This will, as mentioned, create a anon-volume during runtime but also makes you aware ( using docker inspect or reading the Dockerfile ) where volumes for persistence are located.
In production you usually will used a named volume / path mount in your docker-compose file mounted to this very folder
docker-compose.yml as named volume
volumes:
mydbdata:/var/lib/postgresql/data
docker-compose.yml as path
volumes:
./local/path/data:/var/lib/postgresql/data
There are actually cons in defining such VOLUME definitions in the Dockerfile, which i will not elaborate here, but the main reason is "lifetime".
Having no VOLUME in the Dockerfile and running
docker-compose up -d
# do something, manipulate the data
docker-compose down
# all your data would be lost when starting again
docker-compose up -d
Would remove not only the running container, but all your DB data, which might not what you intended ( you just wanted to recreated the container ).
With VOLUME in the Dockerfile, the anon-volume would be persisted even over docker-compose down

How to store MongoDB data with docker-compose

I have this docker-compose:
version: "2"
services:
api:
build: .
ports:
- "3007:3007"
links:
- mongo
mongo:
image: mongo
volumes:
- /data/mongodb/db:/data/db
ports:
- "27017:27017"
The volumes, /data/mongodb/db:/data/db, is the first part (/data/mongodb/db) where the data is stored inside the image and the second part (/data/db) where it's stored locally?
It works on production (ubuntu) but when i run it on my dev-machine (mac) I get:
ERROR: for mongo Cannot start service mongo: error while creating mount source path '/data/mongodb/db': mkdir /data/mongodb: permission denied
Even if I run it as sudo. I've added the /data directory in the "File Sharing"-section in the docker-program on the mac.
Is the idea to use the same docker-compose on both production and development? How do I solve this issue?
Actually it's the other way around (HOST:CONTAINER), /data/mongodb/db is on your host machine and /data/db is in the container.
You have added the /data in the shared folders of your dev machine but you haven't created /data/mongodb/db, that's why you get a permission denied error. Docker doesn't have the rights to create folders.
I get the impression you need to learn a little bit more about the fundamentals of Docker to fully understand what you are doing. There are a lot of potential pitfalls running Docker in production, and my recommendation is to learn the basics really well so you know how to handle them.
Here is what the documentation says about volumes:
[...] specify a path on the host machine (HOST:CONTAINER)
So you have it the wrong way around. The first part is the past on the host, e.g. your local machine, and the second is where the volume is mounted within the container.
Regarding your last question, have a look at this article: Using Compose in production.
Since Docker-Compose syntax version 3.2, you can use a long syntax of the volume property to specify the type of volume. This allows you to create a "Bind" volume, which effectively links a folder from a container to a folder in your host.
Here is an example :
version : "3.2"
services:
mongo:
container_name: mongo
image: mongo
volumes:
- type: bind
source: /data
target: /data/db
ports:
- "42421:27017"
source is the folder in your host and target the folder in your container
More information avaliable here : https://docs.docker.com/compose/compose-file/#long-syntax

Need some advice dockerizing MongoDB

I am playing with MongoDB and Docker and at this point I am trying to create a useful image for myself to use at work. I have created the following Dockerfile:
FROM mongo:2.6
VOLUME /data/db /data/configdb
CMD ["mongod"]
EXPOSE 27017
And I have added it to my docker-compose.yml file:
version: '2'
services:
### PHP/Apache Container
php-apache:
container_name: "php55-dev"
image: reynierpm/php55-dev
ports:
- "80:80"
environment:
PHP_ERROR_REPORTING: 'E_ALL & ~E_DEPRECATED & ~E_NOTICE'
volumes:
- ~/mmi:/var/www
- ~/data:/data
links:
- mongodb
### MongoDB Container
mongodb:
container_name: "mongodb"
build: ./mongo
environment:
MONGODB_USER: "xxxx"
MONGODB_DATABASE: "xxxx"
MONGODB_PASS: "xxxx"
ports:
- "27017:27017"
volumes:
- ~/data/mongo:/data/db
I have some questions regarding this setup I have made:
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
I am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
VOLUME is not required when you are mounting a host directory but it is helpful as metadata. VOLUME does provide some special "copy data on volume creation" semantics when mounting a Docker volume (non host dir) which will impact your data initialisation method choice.
am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
MONGO_USER, MONGO_DATABASE and MONGO_PASS do not do anything in the official mongo Docker image or to mongod itself.
The mongo image has added support for similar environment variables:
MONGO_INITDB_ROOT_USERNAME
MONGO_INITDB_ROOT_PASSWORD
MONGO_INITDB_DATABASE
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Whether you initialise data at build or runtime is up to your usage. As mentioned previously, Docker can copy data from a specified VOLUME into a volume it creates. If you are mounting a host directory you probably need to do the initialisation at run time.
mongorestore requires a running server to restore to. During a build you would need to launch the server and restore in the same RUN step. At runtime you might need to include a startup script that checks for existence of your database.
Mongo is able to initialise any empty directory into a blank mongo instance so you don't need to be worried about mongo not starting.