My old docker-compose file config was
image: mongo:4.2-bionic
container_name: mongodb1
restart: on-failure
env_file: .env
environment:
- MONGO_INITDB_ROOT_USERNAME=$MONGO_USERNAME
- MONGO_INITDB_ROOT_PASSWORD=$MONGO_PASSWORD
ports:
- "$MONGO_PORT:27017"
volumes:
- data1:/data/db
networks:
- app_network
command: ["--replSet", "rs0", "--bind_ip_all"]
This was working fine. It would create the docker-container and then I would create replica inside docker-container. But then I upgraded the docker image to mongo:6.0.3 and now every time I try to start the container it throws error:
BadValue: security.keyFile is required when authorization is enabled
with replica sets
Related
I have created a program and tested that works just fine. I decided to dockerize it, and it seems after maybe some hours or few days the data of mongoDB container get all deleted. The docker-compose.yml file:
version: '3'
services:
node:
restart: always
build: ./nodeServer
container_name: nodeserver
ports:
- 5000:5000
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.2
environment:
- TZ=Europe/Athens
database:
restart: always
build: ./mongoDump/database
container_name: mongodb
ports:
- 27017:27017
networks:
twitter_articles:
ipv4_address: 172.24.0.4
volumes:
- ./data:/data/db
environment:
- TZ=Europe/Athens
pythonscript:
restart: always
build: ./python
container_name: pythonscript
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.3
environment:
- TZ=Europe/Athens
networks:
twitter_articles:
ipam:
config:
- subnet: 172.24.0.0/24
And the three Dockerfile's that they are builded:
nodeserver:
FROM node:14.16.1
COPY package*.json ./
RUN npm install
COPY . ./
CMD [ "npm", "start"]
mongodb:
FROM mongo:5.0.3
CMD docker-entrypoint.sh mongod
pythonscript
FROM python:3.9
COPY requirements.txt ./
RUN pip install -r requirements.txt
COPY . ./
CMD [ "python", "-u", "./init2.py" ]
As mentioned before without Docker the app works just fine and there isn't that kind of behaviour of database getting wiped out. I have tried also internal Docker storage which also does the same thing. I have tried to check the logs and I saw that there is an error happening in pythonscript container each time database wipes out. I know that an error should happen in pythonscript but there is no such a code anywhere in the app to perform deletion of collections or databases (also without Docker this error still happens but nothing gets deleted).
Any ideas?
You can create an external volume and add the data of the mongoDB into it. That way your data doesn't get wiped even when you turn off your docker-compose.
version: '3'
services:
node:
restart: always
build: ./nodeServer
container_name: nodeserver
ports:
- 5000:5000
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.2
environment:
- TZ=Europe/Athens
database:
restart: always
build: ./mongoDump/database
container_name: mongodb
ports:
- 27017:27017
networks:
twitter_articles:
ipv4_address: 172.24.0.4
volumes:
- mongo_data:/data/db
environment:
- TZ=Europe/Athens
pythonscript:
restart: always
build: ./python
container_name: pythonscript
depends_on:
- database
networks:
twitter_articles:
ipv4_address: 172.24.0.3
environment:
- TZ=Europe/Athens
networks:
twitter_articles:
ipam:
config:
- subnet: 172.24.0.0/24
volumes:
mongo_data:
external: true
now you have to create a volume in your docker using
docker volume create --name=mongo_data
then
docker-compose down
and
docker-compose up --build -d
I have been advised that it is always better idea to save data outside of docker container in separate volume. Look for this tutorial volumes.
You need to make an persistant volume for your database, because as you noted on your docker-compose.yml file you got:
restart: always
so everytime your python script got an error, it's stopped and it's depending on Mariadb, so it's restarted and data got wiped.
Make sure the data is stored outside the docker container because are treated like cattles and not pets. New containers are created freshly with no data from previous version.
I'd ensure that container user has a pre-configured ID with write access to the host folder targeted for db data persistence.
I'd use an absolute path on the host side too when mapping persistent data folders in Docker.
Referring to:
volumes:
- ./data:/data/db
I am trying to set up a docker-pod with laravel, mariadb, nginx, redis and phpmyadmin. The laravel webspace works finde but if i switch to port 10081 like configured in the docker-compose.yml i am not able to login with the root account.
it sais " mysqli::real_connect(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution"
i already tried to configure a "my-network" which links all of the container, but if i understand docker right there is already a "defaul" network which does this. It didnt change the error message anyway.
here is my full docker-compose file
version: "3.8"
services:
redis:
image: redis:6.0-alpine
expose:
- "6380"
db:
image: mariadb:10.4
ports:
- "3307:3306"
environment:
MYSQL_USERNAME: root
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: laravel
volumes:
- db-data:/var/lib/mysql
nginx:
image: nginx:1.19-alpine
build:
context: .
dockerfile: ./docker/nginx.Dockerfile
restart: always
depends_on:
- php
ports:
- "10080:80"
networks:
- default
environment:
VIRTUAL_HOST: cockpit.example.de
volumes:
- ./docker/nginx.conf:/etc/nginx/nginx.conf:ro
- ./public:/app/public:ro
php:
build:
target: dev
context: .
dockerfile: ./docker/php.Dockerfile
working_dir: /app
env_file: .env
restart: always
expose:
- "9000"
depends_on:
- composer
- redis
- db
volumes:
- ./:/app
- ./docker/www.conf:/usr/local/etc/php-fpm.d/www.conf:ro
links:
- db:mysql
phpmyadmin:
image: phpmyadmin/phpmyadmin:latest
ports:
- 10081:80
restart: always
environment:
PMA_HOST : db
MYSQL_USERNAME: root
MYSQL_ROOT_PASSWORD: secret
depends_on:
- db
#user: "109:115"
links:
- db:mysql
node:
image: node:12-alpine
working_dir: /app
volumes:
- ./:/app
command: sh -c "npm install && npm run watch"
composer:
image: composer:1.10
working_dir: /app
#environment:
#SSH_AUTH_SOCK: /ssh-auth.sock
volumes:
- ./:/app
#- "$SSH_AUTH_SOCK:/ssh-auth.sock"
- /etc/passwd:/etc/passwd:ro
- /etc/group:/etc/group:ro
command: composer install --ignore-platform-reqs --no-scripts
volumes:
db-data:
Make sure you have defined all attributes correctly for phpmyadmin container, in the current case there was the absence of -network definition
phpmyadmin:
image: phpmyadmin/phpmyadmin:latest
container_name: phpmyadmin
restart: always
ports:
# 8080 is the host port and 80 is the docker port
- 8080:80
environment:
- PMA_ARBITRARY:1
- PMA_HOST:mysql
- MYSQL_USERNAME:root
- MYSQL_ROOT_PASSWORD:secret
depends_on:
- mysql
networks:
# define your network where all containers are connected to each other
- laravel
volumes:
# define directory path where you shall store your persistent data and config
# files of phpmyadmin
- ./docker/phpmyadmin
Maybe your container cannot start because its volume contains incompatible data. It can happen if you downgrade the version of mysql or mariadb image.
You can resolve the problem if you remove the volume and import the database again. Maybe you have to create a backup first.
I have been trying to dockerize my spring boot application which depends on redis, kafka and mongodb.
Following is the docker-compose.yml:
version: '3.3'
services:
my-service:
image: my-service
build:
context: ../../
dockerfile: Dockerfile
restart: always
container_name: my-service
environment:
KAFKA_CONFLUENT_BOOTSTRAP_SERVERS: kafka:9092
MONGO_HOSTS: mongodb:27017
REDIS_HOST: redis
REDIS_PORT: 6379
volumes:
- /private/var/log/my-service/:/var/log/my-service/
ports:
- 8080:8090
- 1053:1053
depends_on:
- redis
- kafka
- mongodb
portainer:
image: portainer/portainer
command: -H unix:///var/run/docker.sock
restart: always
container_name: portainer
ports:
- 9000:9000
- 9001:8000
volumes:
- /var/run/docker.sock:/var/run/docker.sock
redis:
image: redis
container_name: redis
restart: always
ports:
- 6379:6379
zookeeper:
image: wurstmeister/zookeeper
ports:
- 2181:2181
container_name: zookeeper
kafka:
image: wurstmeister/kafka
ports:
- 9092:9092
container_name: kafka
environment:
KAFKA_CREATE_TOPICS: "cms.entity.change:1:1" # topic:partition:replicas
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_PORT: 9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
mongodb:
image: mongo:latest
container_name: mongodb
environment:
MONGO_INITDB_ROOT_USERNAME:
MONGO_INITDB_ROOT_PASSWORD:
ports:
- 27017:27017
volumes:
- ./data/db:/data/db
The issue is that this starts up mongo as a STANDALONE instance. So the APIs in my service that persist data are failing as mongo needs to start as a REPLICA_SET.
How can I edit my docker-compose file to start mongo as a REPLICA_SET?
I had the same issue and ended up on this stackoverflow post.
We had a requirement of using official mongoDB docker image (https://hub.docker.com/_/mongo ) and couldn't use bitnami as suggested in Vahid's answer.
This answer isn't exactly what's needed by the question asked and coming in 6 months too late; but it should give directions to someone who need to use the mongoDb standalone replicaset throw away instance for integration testing purpose. If you need to use it in PROD then you'll have to provide environment variables for volumes and auth as per Vahid's answer.
version: '3.7'
services:
mongodb:
image: mongo:latest
container_name: myservice-mongodb
networks:
- myServiceNetwork
expose:
- 27017
command: --replSet singleNodeReplSet
mongodb-replicaset:
container_name: mongodb-replicaset-helper
depends_on:
- mongodb
networks:
- myServiceNetwork
image: mongo:latest
command: bash -c "sleep 5 && mongo --host myservice-mongodb --port 27017 --eval \"rs.initiate()\" && sleep 2 && mongo --host myservice-mongodb --port 27017 --eval \"rs.status()\" && sleep infinity"
my-service:
depends_on:
- mongodb-replicaset
image: myserviceimage
container_name: myservicecontainer
networks:
- myServiceNetwork
environment:
myservice__Database__ConnectionString: mongodb://myservice-mongodb:27017/?connect=direct&replicaSet=singleNodeReplSet&readPreference=primary
myservice__Database__Name: myserviceDb
networks:
myServiceNetwork:
driver: bridge
NOTE: Please look at the way how connection string is passed as env variable to the service depending on mongo replicaset instance. You'd have to ensure that the name used in setting up the mongodb replicaset (in my case singleNodeReplicaSet) is passed on to the service depending on it.
Edited:
my previous answer was far wrong so I changed it. I managed to make it work using 'bitnami/mongodb:4.0'. Not sure if that would help you or not, but maybe it gives you some idea. They have a docker-compose file ready for replicaset mode.
version: '3'
services:
mdb-primary:
image: 'bitnami/mongodb:4.0'
environment:
- MONGODB_REPLICA_SET_MODE=primary
- MONGODB_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-primary
mdb-secondary:
image: 'bitnami/mongodb:4.0'
depends_on:
- mdb-primary
environment:
- MONGODB_PRIMARY_HOST=mdb-primary
- MONGODB_REPLICA_SET_MODE=secondary
- MONGODB_PRIMARY_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-secondary
mdb-arbiter:
image: 'bitnami/mongodb:4.0'
depends_on:
- mdb-primary
environment:
- MONGODB_PRIMARY_HOST=mdb-primary
- MONGODB_REPLICA_SET_MODE=arbiter
- MONGODB_PRIMARY_ROOT_PASSWORD=somepassword
- MONGODB_REPLICA_SET_KEY=replicasetkey
- MONGODB_ADVERTISED_HOSTNAME=mdb-arbiter
mongo-cli:
image: 'bitnami/mongodb:latest'
don't forget to add volumes and map it to /bitnami on the primary node
the last container, mongo-cli is for testing purposes. So you can connect to the replicaset using the cli, there is an argument about that here if you like to read about it.
$ docker-compose exec mongo-cli bash
$ mongo "mongodb://mdb-primary:27017/test?replicaSet=replicaset"
Hi I have an application where I am using docker-compose. I have large amount of data in the remote mongo DB server which is running on port 28107. How can I connect from my docker-compose to this remote server?
Below is my docker-compose.yml file:
version: '3'
services:
myapp:
# container_name: myapp
restart: always
build: .
ports:
- '52000:52000'
# - '8080:8080'
# - '4300:4300'
# - '4301:4301'
environment:
- MONGO_URL=mongodb://test:test#ip_address:28107/test
# command: ["./wait-for-it.sh", "mongo:28107", "--", "npm", "start"]
links:
- redis
- mongo
mongo:
# container_name: myapp-mongo
image: 'mongo:latest'
ports:
- '28107:28107'
# - '27017:27017'
volumes:
# - ~/Downloads/db_dump_09_01_2020:/data/db
- /data/db
# - /data/configdb
# command: mongod --auth
redis:
# container_name: myapp-redis
restart: always
image: 'redis:4.0.11'
# command: ["redis-server", "--appendonly", "yes"]
depends_on:
- helper
sysctls:
- net.core.somaxconn=511
ports:
- '6379:6379'
helper:
image: alpine
command: sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
privileged: true
In the above code, in the environment parameter, I have mentioned the remote mongo DB server url. I have all the data in that url. I don't want to export that data in my localhost and mount it in my docker container but instead, I would like to directly link my docker container to that remote mongo DB server.
How can I do it ? I am new to the docker concepts.
Seems like you are trying to connect to an external MongoDb server on a network different from your docker network. Then you shouldn't need.
mongo:
# container_name: myapp-mongo
image: 'mongo:latest'
ports:
- '28107:28107'
# - '27017:27017'
volumes:
# - ~/Downloads/db_dump_09_01_2020:/data/db
- /data/db
# - /data/configdb
# command: mongod --auth
You should only need to provide the necessary environment settings of the remote MongoDb server to myapp.
I'm building a dockerfile. But I meet a problem. It says that :
/bin/sh: 1: mongod: not found
My dockerfile:
FROM mongo:latest
FROM node
RUN mongod
COPY . .
RUN node ./scripts/import-data.js
Here is what happen when docker build:
Sending build context to Docker daemon 829.5MB
Step 1/8 : FROM rabbitmq
---> e8261c2af9fe
Step 2/8 : FROM portainer/portainer
---> 00ead811e8ae
Step 3/8 : FROM docker.elastic.co/elasticsearch/elasticsearch:6.5.1
---> 32f93c89076d
Step 4/8 : FROM mongo:latest
---> 5976dac61f4f
Step 5/8 : FROM node
---> b074182f4154
Step 6/8 : RUN mongod
---> Running in 0a4b66a77178
/bin/sh: 1: mongod: not found
The command '/bin/sh -c mongod' returned a non-zero code: 127
Any idea ?
The problem is that you are using two FROM instructions, which is referred to as a multi-stage build. The final image will be based on the node image that doesn't contain the mongo database.
* Edit *
here are more details about what is happening:
FROM mongo:latest
the base image is mongo:latest
FROM node
now the base image is node:latest. The previous image is just standing there...
RUN mongod
COPY . .
RUN node ./scripts/import-data.js
now you run mongod and the other commands in your final image that is based on node (which doesn't contain mongo)
It happens because multiple FROM instructions should be used for Multistage Build (check the documentation) and NOT for image creation contains all of present applications.
Multistage builds provide you possibility of delegation building process into container's environment without local application installation.
FROM rabbitmq
...some instructions require rabbitmq...
FROM mongo:latest
...some instructions require mongo...
In other words if you want to create an image with rabbitmq, mongo and other application you have to choose the image and install applications manually.
Use docker-compose (https://docs.docker.com/compose/install/) to run the images rather than attempting to build a new image from a collection of existing images. Your docker-compose.yml might look something like:
version: '3.7'
services:
portainer:
image: 'portainer/portainer'
container_name: 'portainer'
hostname: 'portainer'
domainname: 'example.com'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock'
- 'portainer_data:/data'
ports:
- '9000:9000'
rabbitmq:
image: 'rabbitmq'
container_name: 'rabbitmq'
hostname: 'rabbitmq'
domainname: 'example.com'
volumes:
- 'rabbitmq_data:/var/lib/rabbitmq'
elasticsearch:
image: 'elasticsearch:7.1.1'
container_name: 'elasticsearch'
hostname: 'elasticsearch'
domainname: 'example.com'
environment:
- 'discovery.type=single-node'
volumes:
- 'elasticsearch_data:/usr/share/elasticsearch/data'
ports:
- '9200:9200'
- '9300:9300'
node:
image: 'node:12'
container_name: 'node'
hostname: 'node'
domainname: 'example.com'
user: 'node'
working_dir: '/home/node/app'
environment:
- 'NODE_ENV=production'
volumes:
- './my-app:/home/node/app'
ports:
- '3000:3000'
command: 'npm start'
mongo:
image: 'mongo'
container_name: 'mongo'
hostname: 'mongo'
domainname: 'example.com'
restart: 'always'
environment:
- 'MONGO_INITDB_ROOT_USERNAME=root'
- 'MONGO_INITDB_ROOT_PASSWORD=example'
volumes:
- 'mongo_data:/data/db'
volumes:
portainer_data:
rabbitmq_data:
elasticsearch_data:
mongo_data:
I see, quite simple
step1. create this Dockerfile:
FROM mongo:latest
step2. create image from this Dockerfile:
docker build . -t my_mongo_build
This is equal to docker run ..... mongo:latest, used for some strange scenario