I want to start a Mongo container with a read-only fs for security reason according to 5.12.
I have the following docker-compose.yml:
version: '2'
services:
mongodb:
image: mongo:3.2
command: -f /etc/mongo.conf
volumes:
- ./mongo/mongo.conf:/etc/mongo.conf:ro
- /data/db
user: mongodb
read_only: true
On docker-compose up it fails with the error Failed to unlink socket file /tmp/mongodb-27017.sock errno:30 Read-only file system.
OK. No problem. I could add - /tmp to the volumes.
But is this good practice to add every path to the volumes? And are there some other paths to add? Like some log paths and so on?
Is there a list from Mongodb?
TL;DR case
You don't need have read-only container, you should only keep your user as non-root for host machine, mount only dirs that you really need and manage permisiions only for mounted dirs.
Full answer
From the official mongo docker image and best usage practices of docker much easier and convenient case is using gosu. In this case your MongoDB will be running by non-root user, that should be enough secure.
All directories that is not mounted from host to container could not be affected from container to host. As example, even if you remove root of you system inside of container where nothing is mounted, it will not affect host dirs (but it WILL affect all mounted dirs, so be careful if you decide to try it by yourself =)).
Also for MongoDB /data/db directory is where all db info stored it writes all info "schemas" etc., so while it is in read only mode, mongodb will not work in any case. This is why you could see chown -R mongodb:mongodb /data/db lines befor mongodb start in docker-entrypoint.sh from official mongodb docker image.
Related
I have a docker-compose.yml which I am using to deploy to a remote host from my local (Mac) machine using docker context. The compose config is as follows:
database:
image: postgres:14.2
restart: on-failure
volumes:
- ./db-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
In order to persist data, I have defined a volume ./db-data:/var/lib/postgresql/data. This db-data folder does not exist in my local machine. I want delete this mount completely because I don't want any of the previously persisted data. I know I can define a new volume directory but I would like to use the same directory name (db-data). I have tried the following:
docker compose down --volume --remove-orphans - when I recreate new container, previously persisted data still exists
There is no folder called ./db-data in my Mac working directory.
I tried searching var/lib/docker in my Mac. But that directory does not exists.
Docker for Mac app doesn't list any volumes
There is no db-data in the remote host where the database is deployed
Running docker inspect <container-id> listed the mount directory for the container. The mount directory resembled absolute path of my local computer. For example it was like /Users/<user-name>/dir/db-data. When I saw this I assumed this had to be in the local computer due to the prefix Users/<user-name> but this path was actually found in the root of the remote machine.
Thats because the directory for docker volumes is in the docker vm for MACOS.
Where is /var/lib/docker on Mac/OS X
You would have to follow this to see the volume
The data in the database is intended to be surfaced by an API in another container. Previously, I have successfully loaded the database during run using this suggestion. However, my database is quite large (10gb) and ideally I would not have to load the database again each time I start a new container. I want the database to be loaded on build. To accomplish this, I tried the following for my Dockerfile:
FROM mongo:4.0.6-xenial
COPY dump /data/dump
RUN mongod --fork --logpath /var/log/mongod.log \
&& mongorestore /data/dump \
&& mongo --eval "db.getSiblingDB('db').createUser({user:'user',pwd:'pwd',roles:['readWrite']})" \
&& mongod --shutdown
I expected the database to be in the container when I ran this image, but it was not, nor does the user exist. However, the log file /var/log/mongod.log indicates that the database loaded successfully as far as I can tell. Why did this not work?
The official mongo Docker image writes the database data in a docker volume.
At run time (thus in a docker container), keep in mind that files written to volumes do not end up written on the container file system. This is done to persist your data so that it survives container deletion, but more importantly in the context of database, for performance reasons. To have good I/O performances with disks, disk operations must be done on a volume, not on the container file system itself.
At build time (thus when creating a docker image), if you happen to have RUN/ADD/COPY directives in your Dockerfile write files to a location which is already declared as a volume, those files will be discarded. However, if you write the files to a directory in your Dockerfile, and only after you declare that directory as a volume, then those the volume will keep those files unless you start your container specifying a volume with the docker run -v option.
This means that in the case your own Dockerfile is built FROM mongo, the /data location is already declared as a volume. Writing files to that location is pointless.
What can be done?
Make you own mongo image from scratch
Knowing how volumes works, you could copy the contents from the Dockerfile of the official mongo Docker image and insert a RUN/ADD/COPY directive to write the files you want to the /data/db location before the VOLUME /data/db /data/configdb directive.
Override the entrypoint
Assuming you have a tar archive named mongo-data-db.tar with the contents of the /data/db location from a mongo container having all the database and collections you want, you could use the following Dockerfile and copy-initial-data-entry-point.sh, you can build an image which will copy those data to the /data/db location every time the container is started. This only make sense in a use case where such a container is used for a test suite which requiers the very same initial data everytime such a container is started as previous data are replaced with the inital data at each start.
Dockerfile:
FROM mongo
COPY ./mongo-data-db.tar /mongo-data-db.tar
COPY ./copy-initial-data-entry-point.sh /
RUN chmod +x /copy-initial-data-entry-point.sh
ENTRYPOINT [ "/copy-initial-data-entry-point.sh"]
CMD ["mongod"]
copy-initial-data-entry-point.sh:
#!/bin/bash
set -e
tar xf /mongo-data-db.tar -C /
exec /usr/local/bin/docker-entrypoint.sh "$#"
In order to extract the contents of a /data/db from the volume of a mongo container named my-mongo-container, proceed as follow:
stop the mongo container: docker stop my-mongo-container
create a temporary container to produce the tar archive from the volume: docker run --rm --volumes-from my-mongo-container -v $(pwd):/out ubuntu tar cvf /out/mongo-data-db.tar
Note that this archive will be quite large as it contains the full contents of the mongo server data including indexes as described on the mongo documentation
I am playing with MongoDB and Docker and at this point I am trying to create a useful image for myself to use at work. I have created the following Dockerfile:
FROM mongo:2.6
VOLUME /data/db /data/configdb
CMD ["mongod"]
EXPOSE 27017
And I have added it to my docker-compose.yml file:
version: '2'
services:
### PHP/Apache Container
php-apache:
container_name: "php55-dev"
image: reynierpm/php55-dev
ports:
- "80:80"
environment:
PHP_ERROR_REPORTING: 'E_ALL & ~E_DEPRECATED & ~E_NOTICE'
volumes:
- ~/mmi:/var/www
- ~/data:/data
links:
- mongodb
### MongoDB Container
mongodb:
container_name: "mongodb"
build: ./mongo
environment:
MONGODB_USER: "xxxx"
MONGODB_DATABASE: "xxxx"
MONGODB_PASS: "xxxx"
ports:
- "27017:27017"
volumes:
- ~/data/mongo:/data/db
I have some questions regarding this setup I have made:
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
I am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Do I need VOLUME /data/db /data/configdb at the Dockerfile or would be enough to have this line ~/data/mongo:/data/configdb at docker-compose.yml?
VOLUME is not required when you are mounting a host directory but it is helpful as metadata. VOLUME does provide some special "copy data on volume creation" semantics when mounting a Docker volume (non host dir) which will impact your data initialisation method choice.
am assuming (and I took it from here) that as soon as I build the Mongo image I will be creating a database and giving full permissions to the user with password as it's on the environment variables? I am right? (I couldn't find anything helpful here)
MONGO_USER, MONGO_DATABASE and MONGO_PASS do not do anything in the official mongo Docker image or to mongod itself.
The mongo image has added support for similar environment variables:
MONGO_INITDB_ROOT_USERNAME
MONGO_INITDB_ROOT_PASSWORD
MONGO_INITDB_DATABASE
How do I import a current mongo backup (several JSON files) into the database that should be created on the mongo container? I believe I need to run mongorestore command but how? do I need to create an script and run it each time the container start? or should I run during image build? What's the best approach?
Whether you initialise data at build or runtime is up to your usage. As mentioned previously, Docker can copy data from a specified VOLUME into a volume it creates. If you are mounting a host directory you probably need to do the initialisation at run time.
mongorestore requires a running server to restore to. During a build you would need to launch the server and restore in the same RUN step. At runtime you might need to include a startup script that checks for existence of your database.
Mongo is able to initialise any empty directory into a blank mongo instance so you don't need to be worried about mongo not starting.
I'm trying to run a docker-compose on my Windows machine spinning up a MongoDB instance and a data-only container which proxies an attached volume containing the database files.
mongodata:
image: mongo:2.6.8
volumes:
- ./data/db:/data/db
command: --break-mongo
mongo:
image: mongo:2.6.8
volumes_from:
- mongodata
ports:
- "27017:27017"
command: --smallfiles --rest
*p.s. the --break-mongo command is there on purpose as it just needs to create the volume
To my understanding, using a data-only volume pattern would handle permission issues but I can see the following error during the Mongo container startup:
[0m2016-01-26T00:23:52.340+0000 [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file /data/db/journal/tempLatencyTest for writing errno:1 Operation not permitted; returning false
[0m2016-01-26T00:23:52.341+0000 [initandlisten] Unable to remove temporary file due to: boost::filesystem::remove: Text file busy: "/data/db/journal/tempLatencyTest"
[0m2016-01-26T00:23:52.344+0000 [initandlisten] exception in initAndListen: 13516 couldn't open file /data/db/journal/j._0 for writing errno:1 Operation not permitted, terminating
Therefore I'm unable to use MongoDb with an attached volume from my local machine. Is there any way around this issue?
The documentation states
If you are using Docker Machine on Mac or Windows, your Docker daemon
has only limited access to your OS X or Windows filesystem. Docker
Machine tries to auto-share your /Users (OS X) or C:\Users (Windows)
directory. So, you can mount files or directories on OS X using.
docker run -v /Users/<path>:/<container path> ...
On Windows, mount directories using:
docker run -v /c/Users/<path>:/<container path> ...
All other paths come from your virtual machine’s filesystem. For example, if you are
using VirtualBox some other folder available for sharing, you need to
do additional work. In the case of VirtualBox you need to make the
host folder available as a shared folder in VirtualBox. Then, you can
mount it using the Docker -v flag.
Basically, either try to give a full path beginning from your C:\Users folder as shown above, or if you can't have that, make the host folder a shared folder in Virtualbox.
Update
No need to give a full path. docker-compose will handle that. You have to make sure that your docker-compose.yml is inside (somewhere down the line) of your Users folder. It can't be in some root folder. If you are already doing that, then you will have to adjust your permissions. Just give full permissions to that folder.
Update: Check out the latest Docker for Windows and MacOS X.
Faster and more reliable: no more VirtualBox! The Docker engine is
running in an Alpine Linux distribution on top of an xhyve Virtual
Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is
managed by the Docker application. You don’t need docker-machine to
run Docker for Mac and Windows.
Note: if Windows, you need Windows 10 Pro to make it work as Hyper-V is not included in other releases.
For Docker Toolbox previously, it seems there is no solution at all on Windows and OS X due to VirtualBox. The image documentation indeed states:
WARNING (Windows & OS X): The default Docker setup on Windows and OS X
uses a VirtualBox VM to host the Docker daemon. Unfortunately, the
mechanism VirtualBox uses to share folders between the host system and
the Docker container is not compatible with the memory mapped files
used by MongoDB (see vbox bug, docs.mongodb.org and related
jira.mongodb.org bug). This means that it is not possible to run a
MongoDB container with the data directory mapped to the host
As an workaround I just copy from a folder before mongo deamon starts. Also, in my case I don't care of journal files, so i only copy database files.
I've used this command on my docker-compose.yml
command: bash -c "(rm /data/db/*.lock && cd /prev && cp *.* /data/db) && mongod"
And everytime before stoping the container I use:
docker exec <container_name> bash -c 'cd /data/db && cp $(ls *.* | grep -v *.lock) /prev'
Note: /prev is set as a volume. path/to/your/prev:/prev
Another workaround is to use mongodump and mongorestore.
in docker-compose.yml: command: bash -c "(sleep 30; mongorestore
--quiet) & mongod"
in terminal: docker exec <container_name> mongodump
Note: I use sleep because I want to make sure that mongo started, and it takes a while.
I know this involves manual work etc, but I am happy that at least I got mongo with existing data running on my Windows 10 machine, and still can work on my Macbook when I want.
(croscopy https://stackoverflow.com/a/42044756/1894856)
It is quite easy to run MongoDB containerised using docker. Though each time you start a new mongodb container, you will get new empty database.
What should I do in order to keep the database content between container restarts? I tried to bind external directory to container using -v option but without any success.
I tried using the ehazlett/mongodb image and it worked fine.
With this image, you can easily specify where mongo store its data with DATA_DIR env variable. I am sure it must not be very difficult to change on your image too.
Here is what I did:
mkdir test; docker run -v `pwd`/test:/tmp/mongo -e DATA_DIR=/tmp/mongo ehazlett/mongodb
notice the `pwd` in within the -v, as the server and the client might have different path, it is important to specify the absolute path.
With this command, I can run mongo as many time as I want and the database will always be store in the ./test directory I just created.
When using the official Mongo docker image, which is i.e. version mongo:4.2.2-bionic as writing this answer, and using docker-compose, you can achieve persistent data storage using this docker-compose.yml file example.
In the official mongo image, data is stored in the container under the root directory in the folder /data/db by default.
Map this folder to a folder in your local working directory called data (in this example).
Make sure ports are set and mapped, default 27017-27019:27017-27019.
Example of my docker-compose.yml:
version: "3.2"
services:
mongodb:
image: mongo:4.2.2-bionic
container_name: mongodb
restart: unless-stopped
ports:
- 27017-27019:27017-27019
volumes:
- ./data:/data/db
Run docker-compose up in the directory where the yml file is located to run the mongodb container with persistent storage. If you do not have the official image yet, it will pull it from Dockerhub first.
Old post but may be someone still need quick and easy solution...
The easiest way I found is using binding to volume.
Following that way you can easily attach existing MongoDB data; and it will live even after you destroying the container.
Create volume that points to your folder (may include existing db). In my case it's done under Windows, but you can do it on any file system:
docker volume create --opt type=none --opt o=bind --opt device=d:/data/db db
Create/run docker container with MongoDB using that volume binding:
docker run --name mongodb -d -p 27017:27017 -v db:/data/db mongo