I have a docker-compose file that will bring up mongo and mongo-express containers in the same network "mynet".
I have created a network by:
docker network create mynet
I have created a volume named "demo-vol" externally by docker command.
docker volume create demo-vol
Inside the container, I have created a sample mongo collection.
When I do docker-compose up I'm able to see the container running but I'm not able to find the mongo data in that specified volume.
Below is my docker-compose.yaml file
version: '3'
services:
mongo:
image: mongo
container_name: mymongo
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: example
volumes:
- "/demo-vol:/data/db"
networks:
- mynet
ports:
- 27017:27017
mongoexpress:
image: mongo-express
container_name: mymongoexpress
ports:
- 8081:8081
volumes:
- "/demo-vol:/data/db"
environment:
ME_CONFIG_MONGODB_ADMINUSERNAME: root
ME_CONFIG_MONGODB_ADMINPASSWORD: example
depends_on:
- mongo
networks:
- mynet
volumes:
demo-vol:
external: true
networks:
mynet:
external: true
What I need is:
Even after deleting the container, I want my data to be persistent.
How to do that and please explain. Where i'm going wrong?
Note:I'm a beginner to Docker concepts.
Thanks in advance.
You can to use local driver for volume
volumes:
demo-vol:
driver: local
and try to remove slash
volumes:
- demo-vol:/data/db
Related
I have created a program and tested that works just fine. I decided to dockerize it, and it seems after maybe some hours or few days the data of mongoDB container get all deleted. The docker-compose.yml file:
version: '3'
services:
node:
restart: always
build: ./nodeServer
container_name: nodeserver
ports:
- 5000:5000
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.2
environment:
- TZ=Europe/Athens
database:
restart: always
build: ./mongoDump/database
container_name: mongodb
ports:
- 27017:27017
networks:
twitter_articles:
ipv4_address: 172.24.0.4
volumes:
- ./data:/data/db
environment:
- TZ=Europe/Athens
pythonscript:
restart: always
build: ./python
container_name: pythonscript
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.3
environment:
- TZ=Europe/Athens
networks:
twitter_articles:
ipam:
config:
- subnet: 172.24.0.0/24
And the three Dockerfile's that they are builded:
nodeserver:
FROM node:14.16.1
COPY package*.json ./
RUN npm install
COPY . ./
CMD [ "npm", "start"]
mongodb:
FROM mongo:5.0.3
CMD docker-entrypoint.sh mongod
pythonscript
FROM python:3.9
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY . ./
CMD [ "python", "-u", "./init2.py" ]
As mentioned before without Docker the app works just fine and there isn't that kind of behaviour of database getting wiped out. I have tried also internal Docker storage which also does the same thing. I have tried to check the logs and I saw that there is an error happening in pythonscript container each time database wipes out. I know that an error should happen in pythonscript but there is no such a code anywhere in the app to perform deletion of collections or databases (also without Docker this error still happens but nothing gets deleted).
Any ideas?
You can create an external volume and add the data of the mongoDB into it. That way your data doesn't get wiped even when you turn off your docker-compose.
version: '3'
services:
node:
restart: always
build: ./nodeServer
container_name: nodeserver
ports:
- 5000:5000
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.2
environment:
- TZ=Europe/Athens
database:
restart: always
build: ./mongoDump/database
container_name: mongodb
ports:
- 27017:27017
networks:
twitter_articles:
ipv4_address: 172.24.0.4
volumes:
- mongo_data:/data/db
environment:
- TZ=Europe/Athens
pythonscript:
restart: always
build: ./python
container_name: pythonscript
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.3
environment:
- TZ=Europe/Athens
networks:
twitter_articles:
ipam:
config:
- subnet: 172.24.0.0/24
volumes:
mongo_data:
external: true
now you have to create a volume in your docker using
docker volume create --name=mongo_data
then
docker-compose down
and
docker-compose up --build -d
I have been advised that it is always better idea to save data outside of docker container in separate volume. Look for this tutorial volumes.
You need to make an persistant volume for your database, because as you noted on your docker-compose.yml file you got:
restart: always
so everytime your python script got an error, it's stopped and it's depending on Mariadb, so it's restarted and data got wiped.
Make sure the data is stored outside the docker container because are treated like cattles and not pets. New containers are created freshly with no data from previous version.
I'd ensure that container user has a pre-configured ID with write access to the host folder targeted for db data persistence.
I'd use an absolute path on the host side too when mapping persistent data folders in Docker.
Referring to:
volumes:
- ./data:/data/db
I have the following docker-compose.yml file:
version: "3"
services:
pokerstats:
image: pokerstats
container_name: pokerstats
ports:
- 8080:8080
depends_on:
- db
db:
image: mongo
container_name: mongo
volumes:
- ./database:/data
ports:
- "27018:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: admin
MONGO_INITDB_DATABASE: pokerstats
My issue is that when I run docker-compose down all the data within my mongo database is lost.
How can I create a mongo volume that persists even when the mongo container goes down?
Per the image documentation the database volume needs to be /data/db. This is also seen in the Dockerfile volume.
Since the volume is defined in the Dockerfile, if you do not create a volume at that directory, even if you created a volume in the parent like /data, docker will create an anonymous volume at /data/db which will show up as a long guid volume name in docker volume ls. Depending on how the container is run, those may be left behind.
Therefore the fix is to adjust your volume mount to that path:
version: "3"
services:
pokerstats:
image: pokerstats
container_name: pokerstats
ports:
- 8080:8080
depends_on:
- db
db:
image: mongo
container_name: mongo
volumes:
- ./database:/data/db
ports:
- "27018:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: admin
MONGO_INITDB_DATABASE: pokerstats
Note that unless you need direct access to this data on the host, I'd recommend using a named volume instead. It includes initialization steps that helps with permission issues you may encounter with host volumes, particularly when running directly on a Linux host.
To use a named volume, that would look like:
version: "3"
services:
pokerstats:
image: pokerstats
container_name: pokerstats
ports:
- 8080:8080
depends_on:
- db
db:
image: mongo
container_name: mongo
volumes:
- dbdata:/data/db
ports:
- "27018:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: admin
MONGO_INITDB_DATABASE: pokerstats
volumes:
dbdata:
Try to change volumes for MongoDb container:
volumes:
- "./database:/data/db"
I have been trying to dockerize my spring boot application which depends on redis, kafka and mongodb.
Following is the docker-compose.yml:
version: '3.3'
services:
my-service:
image: my-service
build:
context: ../../
dockerfile: Dockerfile
restart: always
container_name: my-service
environment:
KAFKA_CONFLUENT_BOOTSTRAP_SERVERS: kafka:9092
MONGO_HOSTS: mongodb:27017
REDIS_HOST: redis
REDIS_PORT: 6379
volumes:
- /private/var/log/my-service/:/var/log/my-service/
ports:
- 8080:8090
- 1053:1053
depends_on:
- redis
- kafka
- mongodb
portainer:
image: portainer/portainer
command: -H unix:///var/run/docker.sock
restart: always
container_name: portainer
ports:
- 9000:9000
- 9001:8000
volumes:
- /var/run/docker.sock:/var/run/docker.sock
redis:
image: redis
container_name: redis
restart: always
ports:
- 6379:6379
zookeeper:
image: wurstmeister/zookeeper
ports:
- 2181:2181
container_name: zookeeper
kafka:
image: wurstmeister/kafka
ports:
- 9092:9092
container_name: kafka
environment:
KAFKA_CREATE_TOPICS: "cms.entity.change:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
mongodb:
image: mongo:latest
container_name: mongodb
environment:
MONGO_INITDB_ROOT_USERNAME:
MONGO_INITDB_ROOT_PASSWORD:
ports:
- 27017:27017
volumes:
- ./data/db:/data/db
The issue is that this starts up mongo as a STANDALONE instance. So the APIs in my service that persist data are failing as mongo needs to start as a REPLICA_SET.
How can I edit my docker-compose file to start mongo as a REPLICA_SET?
I had the same issue and ended up on this stackoverflow post.
We had a requirement of using official mongoDB docker image (https://hub.docker.com/_/mongo ) and couldn't use bitnami as suggested in Vahid's answer.
This answer isn't exactly what's needed by the question asked and coming in 6 months too late; but it should give directions to someone who need to use the mongoDb standalone replicaset throw away instance for integration testing purpose. If you need to use it in PROD then you'll have to provide environment variables for volumes and auth as per Vahid's answer.
version: '3.7'
services:
mongodb:
image: mongo:latest
container_name: myservice-mongodb
networks:
- myServiceNetwork
expose:
- 27017
command: --replSet singleNodeReplSet
mongodb-replicaset:
container_name: mongodb-replicaset-helper
depends_on:
- mongodb
networks:
- myServiceNetwork
image: mongo:latest
command: bash -c "sleep 5 && mongo --host myservice-mongodb --port 27017 --eval \"rs.initiate()\" && sleep 2 && mongo --host myservice-mongodb --port 27017 --eval \"rs.status()\" && sleep infinity"
my-service:
depends_on:
- mongodb-replicaset
image: myserviceimage
container_name: myservicecontainer
networks:
- myServiceNetwork
environment:
myservice__Database__ConnectionString: mongodb://myservice-mongodb:27017/?connect=direct&replicaSet=singleNodeReplSet&readPreference=primary
myservice__Database__Name: myserviceDb
networks:
myServiceNetwork:
driver: bridge
NOTE: Please look at the way how connection string is passed as env variable to the service depending on mongo replicaset instance. You'd have to ensure that the name used in setting up the mongodb replicaset (in my case singleNodeReplicaSet) is passed on to the service depending on it.
Edited:
my previous answer was far wrong so I changed it. I managed to make it work using 'bitnami/mongodb:4.0'. Not sure if that would help you or not, but maybe it gives you some idea. They have a docker-compose file ready for replicaset mode.
version: '3'
services:
mdb-primary:
image: 'bitnami/mongodb:4.0'
environment:
- MONGODB_REPLICA_SET_MODE=primary
- MONGODB_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-primary
mdb-secondary:
image: 'bitnami/mongodb:4.0'
depends_on:
- mdb-primary
environment:
- MONGODB_PRIMARY_HOST=mdb-primary
- MONGODB_REPLICA_SET_MODE=secondary
- MONGODB_PRIMARY_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-secondary
mdb-arbiter:
image: 'bitnami/mongodb:4.0'
depends_on:
- mdb-primary
environment:
- MONGODB_PRIMARY_HOST=mdb-primary
- MONGODB_REPLICA_SET_MODE=arbiter
- MONGODB_PRIMARY_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-arbiter
mongo-cli:
image: 'bitnami/mongodb:latest'
don't forget to add volumes and map it to /bitnami on the primary node
the last container, mongo-cli is for testing purposes. So you can connect to the replicaset using the cli, there is an argument about that here if you like to read about it.
$ docker-compose exec mongo-cli bash
$ mongo "mongodb://mdb-primary:27017/test?replicaSet=replicaset"
Here is my docker compose file:
version: "3.3"
services:
test:
image: test
networks:
- mongo_net
ports:
- 4000:80
depends_on:
- mongodb
links:
- mongodb
mongodb:
image: mongo:latest
networks:
- mongo_net
ports:
- 27017:27017
volumes:
- local_data:/data/db
volumes:
local_data:
networks:
mongo_net:
driver: bridge
The 'test' image cannot find the 'mongodb' instance.
My assumption is that the 'links' section would connect the two, but it is not happening.
What am I missing?
for your compose file try just using depends_on. links is deprecated and maybe thats why you are currently getting issues since this is a V3 compose file.
I want to use my app from a docker container with anothers docker container, one for postgres and one for solr.
My docker compose is:
version: '3'
services:
core:
build: ./core
ports:
- "8081:8081"
environment:
- "SPRING_PROFILES_ACTIVE=production"
links:
- postgresdb
- solrdb
postgresdb:
image: postgres:9.4
container_name: postgres
ports:
- "5432:5432"
environment:
- DB_DRIVER=org.postgresql.Driver
- DB_URL=jdbc:postgresql://localhost:5432/db
- DB_USERNAME=db
- DB_PASSWORD=db
networks:
default:
solrdb:
image: solr:5.5
container_name: solr
ports:
- "8983:8983"
environment:
- DB_URL=http://localhost:8984/solr
networks:
default:
networks:
default:
external:
name: mynet
And already I have containers for solr and postgres created, just I want to use it. How I can do it?
You have already exposed the ports for solrdb and the postgresdb. So in your other container access these Dbs by the container names and the exposed port.
For example, solrDb should be accessed via solrdb:8983 and
the postgresdb should be accessed via postgresdb:5432
Edit :
Make sure that both the containers are operating in the same network. You need to add this network field for all the containers.
postgresdb:
image: postgres:9.4
container_name: postgresdb
ports:
- "5432:5432"
environment:
- DB_DRIVER=org.postgresql.Driver
- DB_URL=jdbc:postgresql://localhost:5432/db
- DB_USERNAME=db
- DB_PASSWORD=db
networks:
default:
and in the end of all the docker file
networks:
default:
external:
name: <your-network-name>
And make sure that your predefined network is running prior to the start of these containers.
To create the network :
docker network create --driver overlay --scope global <your-network-name>
Note : Your container ('postgresdb') will be accessible by postgresdb:5432