Docker: change folder where to store docker volumes - mongodb

On my Ubuntu EC2 I host an application using docker containers. db data and upload data is being stored in volumes CaseBook-data-db and CaseBook-data-uploads which are being created with this commands:
docker volume create --name=CaseBook-data-db
docker volume create --name=CaseBook-data-uploads
Volumes being attached through docker-compose file:
version: '2'
services:
mongo:
container_name: "CaseBook-db"
restart: always
image: mongo:3.2.7
ports:
- "27017"
volumes:
- data_db:/data/db
labels:
- "ENVIRONMENT_TYPE=meteor"
app:
container_name: "CaseBook-app"
restart: always
image: "meteor/casebook"
build: .
depends_on:
- mongo
environment:
- MONGO_URL=mongodb://mongo:27017/CaseBook
ports:
- "80:3000"
volumes:
- data_uploads:/Meteor-CaseBook-Container/.uploads
labels:
- "ENVIRONMENT_TYPE=meteor"
volumes:
data_db:
external:
name: CaseBook-data-db
data_uploads:
external:
name: CaseBook-data-uploads
I need to store those docker volumes in different folder(for example /home/ubuntu/data/) of the host machine. How to change docker storage folder for volumes? Or there is a better way in doing this? Thank you in advance.

Named volumes will be stored inside docker's folder (/var/lib/docker). If you want to create a volume in a specific host folder, use a host volume with the following syntax:
docker run -v /home/ubuntu/data/app-data:/app-data my-image
Or from your compose file:
version: '2'
services:
mongo:
container_name: "CaseBook-db"
restart: always
image: mongo:3.2.7
ports:
- "27017"
volumes:
- /home/ubuntu/data/db:/data/db
labels:
- "ENVIRONMENT_TYPE=meteor"
app:
container_name: "CaseBook-app"
restart: always
image: "meteor/casebook"
build: .
depends_on:
- mongo
environment:
- MONGO_URL=mongodb://mongo:27017/CaseBook
ports:
- "80:3000"
volumes:
- /home/ubuntu/data/uploads:/Meteor-CaseBook-Container/.uploads
labels:
- "ENVIRONMENT_TYPE=meteor"
With host volumes, any contents of the volume inside the image will be overlaid with the exact contents of the host folder, including UID's of the host folder. An empty host folder is not initialized from the image the way an empty named volume is. UID mappings tend to be the most difficult part of using a host volume.
Edit: from the comments below, if you need a named volume that acts as a host volume, there is a local persist volume plugin that's listed on docker's plugin list. After installing the plugin, you can create volumes that point to host folders, with the feature that even after removing the named volume, the host directory is left behind. Sample usage from the plugin includes:
docker volume create -d local-persist -o mountpoint=/data/images --name=images
docker run -d -v images:/path/to/images/on/one/ one
docker run -d -v images:/path/to/images/on/two/ two
They also include a v2 compose file with the following volume example:
volumes:
data:
driver: local-persist
driver_opts:
mountpoint: /data/local-persist/data
One additional option that I've been made aware of in the past month is to use the local volume driver's mount options to manually create a bind mount. This is similar to a host volume in docker with the following differences:
If the directory doesn't exist, trying to start a container with a named volume pointing to a bind mount will fail. With host volumes, docker will initialize it to an empty directory owned by root.
If the directory is empty, a named volume will initialize the bind mount with the contents of the image at the mount location, including file and directory ownership/permissions. With a host volume, there is no initialization of the host directory contents.
To create a named volume as a bind mount, you can create it in advance with:
docker volume create --driver local \
--opt type=none \
--opt device=/home/user/test \
--opt o=bind \
test_vol
From a docker run command, this can be done with --mount:
docker run -it --rm \
--mount type=volume,dst=/container/path,volume-driver=local,volume-opt=type=none,volume-opt=o=bind,volume-opt=device=/home/user/test \
foo
Or in a compose file, you can create the named volume with:
volumes:
data:
driver: local
driver_opts:
type: none
o: bind
device: /home/user/test
My preference would be to use the named volume with the local driver instead of the local-persist 3rd party driver if you need the named volume features.

Another way with build-in driver local:
docker volume create --opt type=none --opt device=/home/ubuntu/data/ --opt o=bind data_db
(This use DimonVersace example with: data_db declared as external named volume in docker-compose and /home/ubuntu/data/ as the folder on the host machine)

Related

Docker Compose Mounting Volume Error: No Such File or Directory

I am trying to build a simple Drupal container using docker-compose, but I'm getting an error when trying to mount the named volume. Here is my docker-compose.yml file:
version: '2.1'
services:
web:
build: ./build
image: "myrmidon16/drupal-base:latest"
working_dir: "/var/www/html"
volumes:
- sites:/var/www/html/sites
links:
- "db"
ports:
- 8001:80
db:
image: "mysql:latest"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_USER=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=drupal-base
volumes:
- ".:/var/www/html"
ports:
- "3306"
drush:
image: "mparker17/mush"
working_dir: "/var/www/html"
volumes:
- .:/var/www/html
links:
- "db"
volumes:
sites:
driver: local
driver_opts:
type: bind
o: bind
device: ${PWD}/sites
But when I run docker-compose up -d I get the following error, even though the directory exists:
ERROR: for web Cannot create container for service web: error while mounting volume with options: type='bind' device='/mnt/f/Documents/sites' o='bind': no such file or directory
I'm running Docker for Windows Edge (18.05.0-ce-rc1) on Windows 10 Pro with Compose v 1.21.0. I'm also sharing the F drive through Docker for Windows.
Please let me know if you need any additonal information. Thank you.
On Windows, local driver does not support any options.
So in your docker-compose options o:bind is invalid.
According to: Documentation https://docs.docker.com/engine/reference/commandline/volume_create/#driver-specific-options
It does work on Windows, but you have to add a leading slash to the device option, e.g.
docker volume create \
--opt type=none \
--opt o=bind \
--opt device=/$(pwd)/grafana/dashboards \
--name grafana_dashboards
On Windows you should not create volume in the volume section. The one in service is enough to create a mounth to the Windows file system, as long as it is enabled in:
Docker->Settings->Shared Drives - the whole drive
and make sure you click the "Reset credentials" if you change windows password
The path c:\Temp\some_dir works and the current directory notation(.) I am not sure.

Docker with Postgresql

I create an Dockerfile with Postgresql with this code:
FROM postgres:9.4
MAINTAINER Fabio Ebner
ENV POSTGRES_PASSWORD="dna44100"
ENV POSTGRES_PORT=5432
EXPOSE ${POSTGRES_PORT}
COPY init.sql /docker-entrypoint-initdb.d/
so How can I specify to always save my db data in my user Machine? cause with this code everty time I stop the container my data are lost
You will need to mount a volume. pointing your host machine to the container's directory /var/lib/postgresql
Source: docker mounting volumes on host
You need to mount a volume to the data directory of PostgreSQL.
You can use the following, using the docker-compose file:
version: "3"
services:
test-postgresql:
image: postgres:9.4
container_name: test-postgresql
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: dna44100
volumes:
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
- ./folder-on-host:/var/lib/postgresql/data
With the docker-compose file you can start the container with docker-compose up and stop the container with docker-compose down. The database and settings are saved on the specified directory (./folder-on-host).
If you want to remove the volume you can use the command: docker-compose down -v
You can also use the docker run to mount a volume, using the -v or -volume option:
docker run -v ./folder-on-host:/var/lib/postgresql/data yourimagename

How to persist data on a volume when using docker swarm mode?

New to Docker and I'm trying to set Postgres and pgadmin4 to run as a single service on docker for Mac inside a virtual machine. Everything works but as soon as I stop the service my data is gone. I'm using a named volume to persist data but probably doing something wrong. What is it?
Here's my setup:
# create my VM
docker-machine create dbvm
# set the right environment
eval $(docker-machine env dbvm)
Here's my docker-compose.yaml file:
version: '3'
services:
db:
image: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=my_db
volumes:
- pgdata:/pgdata
ports:
- 5432:5432
pgadmin:
image: fenglc/pgadmin4
ports:
- 5050:5050
volumes:
- pgadmindata:/pgadmindata
volumes:
pgdata:
pgadmindata:
With docker-compose.yaml, I run:
docker stack deploy -c docker-compose.yaml dbstack
I can do everything on this setup, but if I run docker stack rm dbstack the data is gone after this, but the volumes still exist.
$ docker volume ls
DRIVER VOLUME NAME
local 0c15b0b22c6b850e8768c14045da166253424dda4df8d2e13df75fd54d833412
local 22bab81d9d1de0e07de97363596b096f944752eba617ff574a0ab525239227f5
local 6da6e29fb98ad0f66d7da6a75dc76066ce014b26ea43567c55ed318fda707105
local dbstack_pgadmindata
local dbstack_pgdata
What am I missing?
Unless you have it in some config not shown, I believe you need to map to the default data location inside the container e.g., pgdata:/var/lib/postgresql/data
#Idg is partially correct. postgres data lives at /var/lib/postgresql/data per the Docker Hub readme.
But for it to work in your named volume, you can't use a path on the left side, so correct value would be:
volumes:
- pgdata:/var/lib/postgresql/data
Then the postgres data will stay in that named volume, on the node it was created on.

Permission denied when running `mkdir` inside of a Docker container

I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.
I run:
docker-compose build
then
docker-compose up
And I receive the following error:
ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""
My full docker-compose.yml looks like this (note the service called db where the volume is defined):
version: '3'
services:
nginx:
image: nginx:latest
ports:
- 80:8000
volumes:
- ./src:/src
- ./config/nginx:/etc/nginx/conf.d
- ./src/static:/static
depends_on:
- web
web:
build: .
command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
depends_on:
- db
volumes:
- ./src:/src
- ./src/static:/static
expose:
- 8000
db:
image: postgres:latest
volumes:
- /private/var/lib/postgresql:/var/lib/postgresql
ports:
- 5432:5432
Any ideas for how to solve?
The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:
VOLUME /var/lib/postgresql/data
Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.
For clarity, here is the volume the docker engine created for you:
/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data
And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:
mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied
Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.
As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.

How to compose an external data directory in PostgreSQL image within Docker?

I'm trying to create a couple of containers with Docker. One is a postgres:latest image and the other is ubuntu:latest image with postgresql-client installed.
I have an existing database cluster in my localhost that I've used before install docker, of course. Now I want to use that cluster in my PostgreSQL container. The path in my computer is /Users/Marco/Data.
I've created a volume too, with the command docker volume create --opt device=/Users/Marco/Data data_container to store the cluster in it.
Then tried to make a docker-compose.ymlfile with the following content:
version: '2'
services:
db:
image: postgres
environment:
- PGDATA=/var/lib/postgresql/data/
ports:
- "5432:5432"
volumes:
- data_container:/var/lib/postgresql/data/pgdata
shell:
image: ubuntu_pgsql
command: /bin/bash
tty: true
stdin_open: true
links:
- db
volumes:
data_container:
external: true
When I want to launch the containers with docker-compose up -d it shows me the following error:
ERROR: for db Cannot start service db: error while mounting volume '/var/lib/docker/volumes/data_container/_data': error while mounting volume with options: type='' device='/Users/Marco/Data' o='': no such device
ERROR: Encountered errors while bringing up the project.
What could be failing? Thanks.
If data container is a directory you should declare as:
volumes:
- ./data_container:/var/lib/postgresql/data/pgdata
If you used a data_container without ./ before, docker compose understand that is a file, not a directory.