I'm trying to run a docker-compose on my Windows machine spinning up a MongoDB instance and a data-only container which proxies an attached volume containing the database files.
mongodata:
image: mongo:2.6.8
volumes:
- ./data/db:/data/db
command: --break-mongo
mongo:
image: mongo:2.6.8
volumes_from:
- mongodata
ports:
- "27017:27017"
command: --smallfiles --rest
*p.s. the --break-mongo command is there on purpose as it just needs to create the volume
To my understanding, using a data-only volume pattern would handle permission issues but I can see the following error during the Mongo container startup:
[0m2016-01-26T00:23:52.340+0000 [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file /data/db/journal/tempLatencyTest for writing errno:1 Operation not permitted; returning false
[0m2016-01-26T00:23:52.341+0000 [initandlisten] Unable to remove temporary file due to: boost::filesystem::remove: Text file busy: "/data/db/journal/tempLatencyTest"
[0m2016-01-26T00:23:52.344+0000 [initandlisten] exception in initAndListen: 13516 couldn't open file /data/db/journal/j._0 for writing errno:1 Operation not permitted, terminating
Therefore I'm unable to use MongoDb with an attached volume from my local machine. Is there any way around this issue?
The documentation states
If you are using Docker Machine on Mac or Windows, your Docker daemon
has only limited access to your OS X or Windows filesystem. Docker
Machine tries to auto-share your /Users (OS X) or C:\Users (Windows)
directory. So, you can mount files or directories on OS X using.
docker run -v /Users/<path>:/<container path> ...
On Windows, mount directories using:
docker run -v /c/Users/<path>:/<container path> ...
All other paths come from your virtual machine’s filesystem. For example, if you are
using VirtualBox some other folder available for sharing, you need to
do additional work. In the case of VirtualBox you need to make the
host folder available as a shared folder in VirtualBox. Then, you can
mount it using the Docker -v flag.
Basically, either try to give a full path beginning from your C:\Users folder as shown above, or if you can't have that, make the host folder a shared folder in Virtualbox.
Update
No need to give a full path. docker-compose will handle that. You have to make sure that your docker-compose.yml is inside (somewhere down the line) of your Users folder. It can't be in some root folder. If you are already doing that, then you will have to adjust your permissions. Just give full permissions to that folder.
Update: Check out the latest Docker for Windows and MacOS X.
Faster and more reliable: no more VirtualBox! The Docker engine is
running in an Alpine Linux distribution on top of an xhyve Virtual
Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is
managed by the Docker application. You don’t need docker-machine to
run Docker for Mac and Windows.
Note: if Windows, you need Windows 10 Pro to make it work as Hyper-V is not included in other releases.
For Docker Toolbox previously, it seems there is no solution at all on Windows and OS X due to VirtualBox. The image documentation indeed states:
WARNING (Windows & OS X): The default Docker setup on Windows and OS X
uses a VirtualBox VM to host the Docker daemon. Unfortunately, the
mechanism VirtualBox uses to share folders between the host system and
the Docker container is not compatible with the memory mapped files
used by MongoDB (see vbox bug, docs.mongodb.org and related
jira.mongodb.org bug). This means that it is not possible to run a
MongoDB container with the data directory mapped to the host
As an workaround I just copy from a folder before mongo deamon starts. Also, in my case I don't care of journal files, so i only copy database files.
I've used this command on my docker-compose.yml
command: bash -c "(rm /data/db/*.lock && cd /prev && cp *.* /data/db) && mongod"
And everytime before stoping the container I use:
docker exec <container_name> bash -c 'cd /data/db && cp $(ls *.* | grep -v *.lock) /prev'
Note: /prev is set as a volume. path/to/your/prev:/prev
Another workaround is to use mongodump and mongorestore.
in docker-compose.yml: command: bash -c "(sleep 30; mongorestore
--quiet) & mongod"
in terminal: docker exec <container_name> mongodump
Note: I use sleep because I want to make sure that mongo started, and it takes a while.
I know this involves manual work etc, but I am happy that at least I got mongo with existing data running on my Windows 10 machine, and still can work on my Macbook when I want.
(croscopy https://stackoverflow.com/a/42044756/1894856)
Related
There are some similar posts but this is specifically related to running Postgres with WSL2 backend on Docker desktop. WSL2 brings full Linux experience on Windows. Volumes can be mounted to both Windows and Linux file systems. But the best practice is to use Linux file system for performance reasons see docker documentation.
Performance is much higher when files are bind-mounted from the Linux filesystem, rather than remoted from the Windows host. Therefore avoid docker run -v /mnt/c/users:/users (where /mnt/c is mounted from Windows).
Instead, from a Linux shell use a command like docker run -v ~/my-project:/sources where ~ is expanded by the Linux shell to $HOME.
My WSL distro is Ubuntu 20.04 LTS. I'm bind mounting Postgres data directory to a directory on Linux filesystem and I'm also configuring the Postgres PGDATA to use a sub-directory because this is instructed on the official Docker image docs:
PGDATA
This optional variable can be used to define another location - like a subdirectory - for the database files. The default is /var/lib/postgresql/data. If the data volume you're using is a filesystem mountpoint (like with GCE persistent disks) or remote folder that cannot be chowned to the postgres user (like some NFS mounts), Postgres initdb recommends a subdirectory be created to contain the data.
So this is how I start Postgres with the volume mounting to WSL2 Ubuntu file system:
docker run -d \
--name some-postgres -e POSTGRES_PASSWORD=root \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v ~/custom/mount:/var/lib/postgresql/data \
postgres
I can exec into the running container and verify that the data folder exists and it's configured correctly:
Now from the host machine (WSL2 Linux) if I try to access that folder I get the permission denied:
I would appreciate if anyone can provide a solution. None of the existing posts worked to resolve the issue.
This has got nothing to do with PostgreSQL. Docker containers run as root and so any directory created by Docker will also belong to root.
When you attach to the container and list the directory under /var/lib/postgresql/data it shows postgres as the owner.
Check "Arbitrary --user Notes" section in the official documentation here
The second option "bind-mount /etc/passwd read-only from the host" worked for me.
Two things that were blocking us working with WSL2 on Windows were:
Folder c:\Program files\WindowsApps didn't have admin account listed as owner
McAfee was blocking the WSL. In order to disable blocking we had to remove following rule: Open McAfee -> Threat Prevention -> Show Advanced (button in Right upper corner) -> scroll down to Rules -> name of the rule is "Executing Subsystem for Linux"
I wish to store my persists data in my local D:\dockerData\postgres9.6. Below is my docker command
docker pull postgres
docker run -d -v /d/dockerData/postgres9.6:/var/lib/postgresql/data -p 5432:5432 postgres
It successful create a container and I can use pgAdmin to access and create database.
But I found out that there is no file in my D:\dockerData\postgres9.6. I exec bash into the container, there is at least 20+ files inside /var/lib/postgresql/data.
Anyone can point out which part goes wrong?
It depends what kind of Docker you are using on Windows:
Docker Toolbox with VirtualBox: only C:\Users\mylogin is shared by default. D:\ is not mounted.
Docker for Windows with HyperV: only C:\ is mounted by default. Make sure D:\ is a shared drive: see image
I think I don't get it. First, I created docker-machine:
$ docker-machine create -d virtualbox dev
$ eval $(docker-machine env dev)
Then I wrote Dockerfile and docker-compose.yml:
FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
version: '2'
services:
db:
image: postgres
web:
build: .
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
Finally, I built and started the image:
$ docker-compose build --no-cache
$ docker-compose start
I checked ip of my virtual machine
$ docker-machine ip dev
and successfully opened the site in my browser. But when I made some changes in my code - nothing happened. So I logged to the "dev" machine:
$ docker-machine ssh dev
and I didn't find my code! So I logged to the docker "web" image:
$ docker exec -it project_web_1 bash
and there was a code, but unchanged.
What is the docker-machine for? What is the sense? Why docker doesn't syncing files after changes? It looks like docker + docker-machine + docker-compose are pain in the a...s for local development :-)
Thanks.
Docker is the command-line tool that uses containerization to manage multiple images and containers and volumes and such -- a container is basically a lightweight virtual machine. See https://docs.docker.com/ for extensive documentation.
Until recently Docker didn't run on native Mac or Windows OS, so another tool was created, Docker-Machine, which creates a virtual machine (using yet another tool, e.g. Oracle VirtualBox), runs Docker on that VM, and helps coordinate between the host OS and the Docker VM.
Since Docker isn't running on your actual host OS, docker-machine needs to deal with IP addresses and ports and volumes and such. And its settings are saved in environment variables, which means you have to run commands like this every time you open a new shell:
eval $(docker-machine env default)
docker-machine ip default
Docker-Compose is essentially a higher-level scripting interface on top of Docker itself, making it easier (ostensibly) to manage launching several containers simultaneously. Its config file (docker-compose.yml) is confusing since some of its settings are passed down to the lower-level docker process, and some are used only at the higher level.
I agree that it's a mess; my advice is to start with a single Dockerfile and get it running either with docker-machine or with the new beta native Mac/Windows Docker, and ignore docker-compose until you feel more comfortable with the lower-level tools.
I want to start a Mongo container with a read-only fs for security reason according to 5.12.
I have the following docker-compose.yml:
version: '2'
services:
mongodb:
image: mongo:3.2
command: -f /etc/mongo.conf
volumes:
- ./mongo/mongo.conf:/etc/mongo.conf:ro
- /data/db
user: mongodb
read_only: true
On docker-compose up it fails with the error Failed to unlink socket file /tmp/mongodb-27017.sock errno:30 Read-only file system.
OK. No problem. I could add - /tmp to the volumes.
But is this good practice to add every path to the volumes? And are there some other paths to add? Like some log paths and so on?
Is there a list from Mongodb?
TL;DR case
You don't need have read-only container, you should only keep your user as non-root for host machine, mount only dirs that you really need and manage permisiions only for mounted dirs.
Full answer
From the official mongo docker image and best usage practices of docker much easier and convenient case is using gosu. In this case your MongoDB will be running by non-root user, that should be enough secure.
All directories that is not mounted from host to container could not be affected from container to host. As example, even if you remove root of you system inside of container where nothing is mounted, it will not affect host dirs (but it WILL affect all mounted dirs, so be careful if you decide to try it by yourself =)).
Also for MongoDB /data/db directory is where all db info stored it writes all info "schemas" etc., so while it is in read only mode, mongodb will not work in any case. This is why you could see chown -R mongodb:mongodb /data/db lines befor mongodb start in docker-entrypoint.sh from official mongodb docker image.
I am running Docker on windows and I have a database with some entries on disk at C:\data\db.
I want to add this database to my container. I have tried numerous ways to do this but failed.
I tried: docker run -p 27017:27017 -v //c/data/db:/data/db --name mongodb devops-mongodb
In my dockerfile I have:
RUN mkdir -p /data/db
VOLUME /data/db
But this doesn't add my current database on disk to the container. It creates a fresh /data/db directory and persists the data I add to it.
The docs here https://docs.docker.com/userguide/dockervolumes/ under 'Mount a host directory as a data volume' specifically told me to execute the -v //c/data/db:/data/db but this isn't working.
Any ideas?
You're using Boot2Docker (which runs inside a Virtual Machine). Boot2Docker uses VirtualBox guest additions to make directories on your Windows machine available to Docker running inside the Virtual Machine.
By default, only the C:\Users directory (on Windows), or /Users/ directory (on OS X) is shared with the virtual machine. Anything outside those directories is not shared with the Virtual Machine, which results in Docker creating an empty directory at the specified location for the volume.
To share directories outside C:\Users\ with the Virtual Machine, you have to manually configure Boot2Docker to share those. You can find the steps needed in the VirtualBox guest addition section of the README;
If some other path or share is desired, it can be mounted at run time by doing something like:
$ mount -t vboxsf -o uid=1000,gid=50 your-other-share-name /some/mount/location
It is also important to note that in the future, the plan is to have any share which is created in VirtualBox with the "automount" flag turned on be mounted during boot at the directory of the share name (ie, a share named home/jsmith would be automounted at /home/jsmith).
Please be aware that using VirtualBox guest additions have a really bad impact on performance (reading/writing to the volume will be really slow). Which could be fine for development, but should be used with caution.