Is there an easy way to clear a mongodb database running in docker?
Or possibly delete it all and create it again?
There are a number of ways:
The same way you would normally clear a mongodb database. You're running it in Docker, but it is a real mongodb after all. See this, and this.
If you've mounted your database's data as a volume on the host system using -v on the docker run command, you can just clear this folder out.
If you haven't, the data lives in the container. You can remove the container and recreate it (docker rm container_name). Or run a shell in the container and remove the data from the container's filesystem (docker exec -it container_name bash).
Regarding the last option, you shouldn't be in this scenario because your data should live on the host system and your container should be disposable.
docker exec -it mongodb_container bash -c "mongo db-name --eval 'db.dropDatabase()'"
I had a case with slightly different requirements:
docker-compose
authentication needed
bash variable expansion (the container was running with environemt variables MONGO_USER & MONGO_PASSWORD already set)
So, for anyone who might need a handy one-liner to authenticate and drop a mongo db with docker, here it is 👇:
docker-compose exec mongo_container_name /bin/bash -c 'mongo database_name -u $MONGO_USER -p $MONGO_PASSWORD --authenticationDatabase admin --eval "db.dropDatabase();"'
And if you add a Makefile to the "mix", it's exactly the same but it needs $$ to the variables:
db-reset:
docker-compose exec mongo_container_name /bin/bash -c 'mongo database_name -u $$MONGO_USER -p $$MONGO_PASSWORD --authenticationDatabase admin --eval "db.dropDatabase();"'
Related
I'm using docker and dockerfile to build images. I want to build a PostgreSQL image so I'm using this dockerfile:
ARG POSTGRES_USER=vetouz
ARG POSTGRES_PASSWORD=***
ARG POSTGRES_DB=vetouz_mediatheque
FROM postgres:latest
USER postgres
EXPOSE 5432
Then I run the image using this command
docker run -e POSTGRES_PASSWORD=vetouz -d --name postgres postgres:latest
When I do that the role vetouz, the password and the db vetouz_mediatheque are not created and I don't understand why. I know it because when I access my container with sudo docker exec -it postgres bash and then run psql -U vetouz I get the error role vetouz does not exist.
It works if I run my image with the following command:
docker run -e POSTGRES_PASSWORD=*** -e POSTGRES_USER=vetouz -e POSTGRES_DB=vetouz_mediatheque -d --name postgres postgres:latest
But I would rather define my variables in the dockerfile.
Any idea why it's not working?
Please use ENV instead of ARG. args are only available during build, envs are available during runtime as well.
Source
https://docs.docker.com/engine/reference/builder/#arg
https://docs.docker.com/engine/reference/builder/#env
YOUR PROBLEM
As already stated your are using ARG that is only available when building the Docker image, but using env variables to set sensitive information in a Docker image is not a safe approach, and I will explain why.
SECURITY CONCERNS
But I would rather define my variables in the dockerfile.
This is ok for information that is not sensitive, but not a best practice for sensitive information like passwords, because the database credentials will be stored in plain text in the Dockerfile, and even if you use ARG to set the ENV var they will be available in the docker image layers.
docker run -e POSTGRES_PASSWORD=*** -e POSTGRES_USER=vetouz -e POSTGRES_DB=vetouz_mediatheque -d --name postgres postgres:latest
This is also a bad practice in terms of security because now your database credentials are saved into the bash history.
In a Linux machine you can check with:
history | grep -i POSTGRES
A MORE SECURE APPROACH
Create an .env file:
POSTGRES_USER=vetouz
POSTGRES_PASSWORD=your-password-here
POSTGRES_DB=vetouz_mediatheque
Don't forget to add the .env file to .gitignore:
echo ".env" >> .gitignore
Running the Docker Container
Now run the docker container with:
docker run --env-file ./.env -d --name postgres postgres:latest
I'm running a dockerized mongo container.
I'd like to create a mongo image with some initialized data.
Any ideas?
A more self-contained approach:
create javascript files that initialize your database
create a derived MongoDB docker image that contains these files
There are many answers that use disposable containers or create volumes and link them, but this seems overly complicated. If you take a look at the mongo docker image's docker-entrypoint.sh, you see that line 206 executes /docker-entrypoint-initdb.d/*.js files on initialization using a syntax: mongo <db> <js-file>. If you create a derived MongoDB docker image that contains your seed data, you can:
have a single docker run command that stands up a mongo with seed data
have data is persisted through container stops and starts
reset that data with docker stop, rm, and run commands
easily deploy with runtime schedulers like k8s, mesos, swarm, rancher
This approach is especially well suited to:
POCs that just need some realistic data for display
CI/CD pipelines that need consistent data for black box testing
example deployments for product demos (sales engineers, product owners)
How to:
Create and test your initialization scripts (grooming data as appropriate)
Create a Dockerfile for your derived image that copies your init scripts
FROM mongo:3.4
COPY seed-data.js /docker-entrypoint-initdb.d/
Build your docker image
docker build -t mongo-sample-data:3.4 .
Optionally, push your image to a docker registry for others to use
Run your docker image
docker run \
--name mongo-sample-data \
-p 27017:27017 \
--restart=always \
-e MONGO_INITDB_DATABASE=application \
-d mongo-sample-data:3.4
By default, docker-entrypoint.sh will apply your scripts to the test db; the above run command env var MONGO_INITDB_DATABASE=application will apply these scripts to the application db instead. Alternatively, you could create and switch to different dbs in the js file.
I have a github repo that does just this - here are the relevant files.
with the latest release of mongo docker , something like this works for me.
FROM mongo
COPY dump /home/dump
COPY mongo_restore.sh /docker-entrypoint-initdb.d/
the mongo restore script looks like this.
#!/bin/bash
# Restore from dump
mongorestore --drop --gzip --db "<RESTORE_DB_NAME>" /home/dump
and you could build the image normally.
docker build -t <TAG> .
First create a docker volume
docker volume create --name mongostore
then create your mongo container
docker run -d --name mongo -v mongostore:/data/db mongo:latest
The -v switch here is responsible for mounting the volume mongostore at the /data/db location, which is where mongo saves its data. The volume is persistent (on the host). Even with no containers running you will see your mongostore volume listed by
docker volume ls
You can kill the container and create a new one (same line as above) and the new mongo container will pick up the state of the previous container.
Initializing the volume
Mongo initializes a new database if none is present. This is responsible for creating the initial data in the mongostore. Let's say that you want to create a brand new environment using a pre-seeded database. The problem becomes how to transfer data from your local environment (for instance) to the volume before creating the mongo container. I'll list two cases.
Local environment
You're using either Docker for Mac/Windows or Docker Toolbox. In this case you can easily mount a local drive to a temporary container to initialize the volume. Eg:
docker run --rm -v /Users/myname/work/mongodb:/incoming \
-v mongostore:/data alpine:3.4 cp -rp /incoming/* /data
This doesn't work for cloud storage. In that case you need to copy the files.
Remote environment (AWS, GCP, Azure, ...)
It's a good idea to tar/compress things up to speed the upload.
tar czf mongodata.tar.gz /Users/myname/work/mongodb
Then create a temporary container to untar and copy the files to the mongostore. the tail -f /dev/null just makes sure that the container doesn't exit.
docker run -d --name temp -v mongostore:/data alpine:3.4 tail -f /dev/null
Copy files to it
docker cp mongodata.tar.gz temp:.
Untar and move to the volume
docker exec temp tar xzf mongodata.tar.gz && cp -rp mongodb/* /data
Cleanup
docker rm temp
You could also copy the files to the remote host and mounting from there but I tend to avoid interacting with the remote host at all.
Disclaimer. I'm writing this from memory (no testing).
Here is how its done with docker-compose. I use an older image of mongo but the docker-entrypoint.sh accepts *.js and *.sh files for all versions of the image.
docker-compose.yaml
version: '3'
services:
mongo:
container_name: mongo
image: mongo:3.2.12
ports:
- "27017:27017"
volumes:
- mongo-data:/data/db:cached
- ./deploy/local/mongo_fixtures /fixtures
- ./deploy/local/mongo_import.sh:/docker-entrypoint-initdb.d/mongo_import.sh
volumes:
mongo-data:
driver: local
mongo_import.sh:
#!/bin/bash
# Import from fixtures
mongoimport --db wcm-local --collection clients --file /fixtures/properties.json && \
mongoimport --db wcm-local --collection configs --file /fixtures/configs.json
And my monogo_fixtures json files are the product of monogoexport which have the following format:
{"_id":"some_id","field":"value"}
{"_id":"another_id","field":"value"}
This should help those using this without a custom Dockefile, just using the image straight away with the right entrypoint setup right in your docker-compose file. Cheers!
I've found a way that is somehow easier for me.
Say you have a database in a docker container on your server, and you want to back it up, here’s what you could do.
What might differ from your setup to mine is the name of your mongo docker container [mongodb] (default when using elastic_spence). So make sure you start your container first with --name mongodb to match the following steps:
$ docker run \
--rm \
--link mongodb:mongo \
-v /root:/backup \
mongo \
bash -c ‘mongodump --out /backup --host $MONGO_PORT_27017_TCP_ADDR’
And to restore the database from a dump.
$ docker run \
--rm \
--link mongodb:mongo \
-v /root:/backup \
mongo \
bash -c ‘mongorestore /backup --host $MONGO_PORT_27017_TCP_ADDR’
If you need to download the dump from to your server you can use scp:
$ scp -r root#IP:/root/backup ./backup
Or upload it:
$ scp -r ./backup root#IP:/root/backup
P.S: Original source by Tim Brandin available at https://blog.studiointeract.com/mongodump-and-mongorestore-for-mongodb-in-a-docker-container-8ad0eb747c62
Thank you!
I deployed my app on a Ubuntu server using mup deploy (https://github.com/arunoda/meteor-up) with the option "setupMongo": true in the mup.json file.
Everything works fine, and I would like to save the mongoDB database daily to FTP or S3, or to set a mongoDB replica to another server (to avoid copying the whole database every time, but it seems more complicated).
If deployed with mup, you are in luck.
You can find the steps here: https://github.com/xpressabhi/mup-data-backup
Here are the steps again:
MongoDB Data Backup deployed via mup
These commands run well only if meteor deployed with mup tool. Mup creates docker for mongodb hence taking backup becomes easy with these commands.
Backup
Take backup of running app data from docker then copy to local folder out of docker.
docker exec -it mongodb mongodump --archive=/root/mongodump.gz --gzip
docker cp mongodb:/root/mongodump.gz mongodump_$(date +%Y-%m-%d_%H-%M-%S).gz
Copy backup to server
Move data to another server/local machine or a backup location
scp /path/to/dumpfile root#serverip:/path/to/backup
Delete old data from meteor deployment
Get into mongo console running in docker then drop current database before getting new data.
docker exec -it mongodb mongo appName
db.runCommand( { dropDatabase: 1 } )
Restore data to meteor docker
docker cp /path/to/dumpfile mongodb:/root/mongodump.gz
docker exec -it mongodb mongorestore --archive=/root/mongodump.gz --gzip
The best way is to mongodump it.
Assuming its running on the mup instance itself since it only listens to 127.0.0.1 you would have to ssh in and use mongodump.
If you simply run it:
mongodump
It will create a directory dump containing your backup.
If you want to do this remotely you would have to edit /etc/mongodb.conf to ensure it binds globally, you will have to create users though since it will be publicly accessible. Then set auth to true.
You could then mongodump from your own machine (you can download the mongodump binary from mongodb.org):
./mongodump --host <your server ip address> --username <username> --password <password>
This answer is inspired by:
sheharyar.me/blog/regular-mongo-backups-using-cron
It uses a script to: mongodump -> tar -> wput (ftp)
First, create a bash script:
#!/bin/bash
MONGO_DATABASE="your_db_name"
APP_NAME="your_app_name"
MONGO_HOST="127.0.0.1"
MONGO_PORT="27017"
TIMESTAMP=`date +%F-%H%M`
MONGODUMP_PATH="/usr/bin/mongodump"
BACKUPS_DIR="/home/username/backups/$APP_NAME"
BACKUP_NAME="$APP_NAME-$TIMESTAMP"
# mongo admin --eval "printjson(db.fsyncLock())"
# $MONGODUMP_PATH -h $MONGO_HOST:$MONGO_PORT -d $MONGO_DATABASE
$MONGODUMP_PATH -d $MONGO_DATABASE
# mongo admin --eval "printjson(db.fsyncUnlock())"
mkdir -p $BACKUPS_DIR
mv dump $BACKUP_NAME
tar -zcvf $BACKUPS_DIR/$BACKUP_NAME.tgz $BACKUP_NAME
rm -rf $BACKUP_NAME
wput $BACKUP_NAME.tgz ftp://login:password#ftp.domain.com/backups/
Save it as mongo_backup.sh and run:
chmod +x mongo_backup.sh
bash mongo_backup.sh
sudo su
crontab -e
And enter this new line:
00 00 * * * /bin/bash /home/username/scripts/mongo_backup.sh
That's it.
I have a data container with Dockerfile:
from ubuntu:latest
VOLUME ["/var/lib/postgresql/9.3/main"]
and a postgresql service container:
#install stuff
............
# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
then I start the data container and postgresql container:
docker run -i -t -d --name data docker:data
docker run -i -t -p 49131:5432 --name postgresql --volumes-from data --rm docker:postgresql
it says :
FATAL: data directory "/var/lib/postgresql/9.3/main" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
it seems like the /var/lib/postgresql/9.3/main folder belong to the root user in the data container. Then i attach to the container, add user and change owner of that folder to postgres:
docker attach data
useradd postgres -s /bin/bash
chown -R postgres:postgres /var/lib/postgresql
then try again with the same error.
what is the problem and what am i missing?
The postgres user probably hasn't been assigned the same UID in the separate containers.
There's no need to make it this difficult though, just use the postgres image to create your data container e.g:
docker run --name data-container postgres echo "Data Container"
That way all the permissions will be set up correctly. For more information see:
http://container42.com/2014/11/18/data-only-container-madness/ and
http://container-solutions.com/2014/12/understanding-volumes-docker/
I'm trying to run postgresql in docker container, but of course I need to have my database data to be persistent, so I'm trying to use data only container which expose volume to store database at this place.
So, my data container has such Dockerfile:
FROM ubuntu
# Create data directory
RUN mkdir -p /data/postgresql
# Create /data volume
VOLUME /data/postgresql
Which I run:
docker run --name postgresql_data lyapun/postgresql_data true
In my postgresql.conf I set:
data_directory = '/data/postgresql'
Then I run my postgresql container in such way:
docker run -d --name postgre --volumes-from postgresql_data lyapun/postgresql
And I got:
2014-07-04 07:45:57 GMT FATAL: data directory "/data/postgresql" has wrong ownership
2014-07-04 07:45:57 GMT HINT: The server must be started by the user that owns the data directory.
How to deal with this issue? I googled a lot to find some information about using postgresql with docker volumes, but I didn't found anything.
Thanks!
Ok, seems like I found workaround for this issue.
Instead of running postgres in such way:
CMD ["/usr/lib/postgresql/9.1/bin/postgres", "-D", "/var/lib/postgresql/9.1/main", "-c", "config_file=/etc/postgresql/9.1/main/postgresql.conf"]
I wrote bash script:
chown -Rf postgres:postgres /data/postgresql
chmod -R 700 /data/postgresql
sudo -u postgres /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
And replaced CMD in postgresql image to:
CMD ["bash", "/run.sh"]
It works!
You have to set ownership of directory /data/postgresql to the same user, under which you are running your postgresql binary. For example, in Ubuntu it is usually postgres user.
Then you have to use this command:
chown postgres.postgres /data/postgresql
A better way to solve that issue, assuming your postgres images is named "postgres" and that your backup is ./backup.tar:
First, add this to your postgres Dockerfile:
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
Then run:
docker run -it --name postgres -v $(pwd):/db postgres sh -c "tar xvf /db/backup.tar --no-overwrite-dir" && \
docker run -it --name data --volumes-from postgres busybox true && \
docker rm postgres && \
docker run -it --name postgres --volumes-from=data postgres
You don't have permission issues since the archive is extracted by the postgres user of your postgres image, so it is the owner of the extracted files.
You can then backup your data using the data container. The advantage of this solution is that you don't chmod/chown every time you run the image.
This type of errors is quite common when you link a NTFS directory into your docker container. NTFS directories don't support ext3 file & directory access control.
The only way to make it work is to link directory from a ext3 drive into your container.
I got a bit desperate when I played around Apache / PHP containers with linking the www folder. After I linked files reside on a ext3 filesystem the problem disappear.
I published a short Docker tutorial on youtube, may it helps to understand this problem: https://www.youtube.com/watch?v=eS9O05TTFjM