I have been trying all day to setup a replica set for MongoDB transactions automated by Docker Compose, but no success so far. Keep encountering the error where mongo1 (connected from mongo-setup) can't connect to mongo2 and mongo3 containers, saying connection refused.
The error message I got:
{
"ok" : 0,
"errmsg" : "replSetInitiate quorum check failed because not all proposed set members responded affirmatively: mongo2:27018 failed with Error connecting to mongo2:27018 (172.30.0.5:27018) :: caused by :: Connection refused, mongo3:27019 failed with Error connecting to mongo3:27019 (172.30.0.4:27019) :: caused by :: Connection refused",
"code" : 74,
"codeName" : "NodeNotFound"
}
uncaught exception: ReferenceError: EOF is not defined :
#(shell):1:1
uncaught exception: SyntaxError: unexpected token: string literal :
#(shell):1:5
My Docker Compose:
mongo3:
hostname: mongo3
image: mongo:5.0.5-focal
networks:
- shopnetwork
volumes:
- ./data/mongo3:/data/db
ports:
- 27019:27019
restart: always
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "shop-mongo-set" ]
mongo2:
hostname: mongo2
image: mongo:5.0.5-focal
networks:
- shopnetwork
volumes:
- ./data/mongo2:/data/db
ports:
- 27018:27018
restart: always
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "shop-mongo-set" ]
mongo1:
hostname: mongo1
image: mongo:5.0.5-focal
networks:
- shopnetwork
volumes:
- ./data/mongo1:/data/db
ports:
- 27017:27017
depends_on:
- mongo2
- mongo3
restart: always
entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "shop-mongo-set" ]
mongo-setup:
image: mongo:5.0.5-focal
networks:
- shopnetwork
volumes:
- ./script:/script
depends_on:
- mongo1
- mongo2
- mongo3
entrypoint: ["/script/setup.sh"]
My setup.sh to initialize replica set:
#!/bin/bash
echo "Starting replica set initialize"
until mongo --host mongo1 --eval "print(\"waited for connection\")"
do
sleep 2
done
echo "Connection finished"
echo "Creating replica set"
mongo --host mongo1 <<EOF
config = {
"_id" : "shop-mongo-set",
"members" : [
{
"_id" : 0,
"host" : "mongo1:27017"
},
{
"_id" : 1,
"host" : "mongo2:27018"
},
{
"_id" : 2,
"host" : "mongo3:27019"
}
]
}
rs.initiate(config)
EOF
echo "Replica set created"
so as tested with the OP there was a mixup with the network configuration in compose and host network.
when using multiple services in docker-compose each service is a different host in the compose network, so for example if we have multiple services using the same port that won't be an issue because each has a different hostname and IP.
when mapping ports from docker network to host network using bridge adapter usually only one service can use each port and better to use ports >1024 for security and compatibility reasons.
when wanting to connect to different services in the same compose file but no external connection from WAN into the compose network is needed the ports section of the compose file or the -p option in the docker run command isn't needed
after fixing said miss-configuration there shouldn't be any problems around that topic
Related
I am trying to connect to a MongoDB replica set using pymongo, but I keep getting the error: pymongo.errors.ServerSelectionTimeoutError: No replica set members match selector. In the error message it's also specified that my topology type is ReplicaSetNoPrimary, which is odd, as connecting with mongo bash shows a clear primary.
Note that the replica set works fine and is usable via mongo bash on the master node.
Also, I have added firewall rules to allow both inbound and outbound traffic on the specified ports, just to make sure this isn't the issue.
I am using docker-compose for the cluster. The file:
version: "3.9"
services:
mongo-master:
image: mongo:latest
container_name: mongo_master
volumes:
- ./data/master:/data/db
ports:
- 27017:27017
command: mongod --replSet dbrs & mongo --eval rs.initiate(`cat rs_config.json`)
stdin_open: true
tty: true
mongo-slave-1:
image: mongo:latest
container_name: mongo_slave_1
volumes:
- ./data/slave_1:/data/db
ports:
- 27018:27017
command: mongod --replSet dbrs
stdin_open: true
tty: true
mongo-slave-2:
image: mongo:latest
container_name: mongo_slave_2
volumes:
- ./data/slave_2:/data/db
ports:
- 27019:27017
command: mongod --replSet dbrs
stdin_open: true
tty: true
The rs_config.json file used above:
{
"_id" : "dbrs",
"members" : [
{
"_id" : 0,
"host" : "mongo_master:27017",
"priority" : 10
},
{
"_id" : 1,
"host" : "mongo_slave_1:27017"
},
{
"_id" : 2,
"host" : "mongo_slave_2:27017"
}
]
}
The error raises on the last line here:
self.__client = MongoClient(["localhost:27017", "localhost:27018", "localhost:27019"], replicaset="dbrs")
self.__collection = self.__client[self.__db_name][collection.value]
self.__collection.insert_one(dictionary_object)
I ommitted some code for brevity, but you can assume all class attributes and dictionary_object are well defined according to pymongo docs.
Also please note that I have tried many different ways to initialize MongoClient, including a connection string (as in the docs), and the connect=False optional parameter as advised in some blogs. The issue persists...
Edit: I tried adding "mongo_master" to my etc/hosts file pointing at 127.0.0.1 and changing the connection string from localhost to that, and it works with the replica set. This is a bad workaround but maybe can help in figuring out a solution.
Thanks in advance for any help!
To get a connection to a MongoDB replicaset from an external client, you must be able to resolve the hostnames from the local client.
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/#connectivity
Ensure that network traffic can pass securely between all members of the set and all clients in the network.
So, add the following to your /etc/hosts file:
127.0.0.1 mongodb-1
127.0.0.1 mongodb-2
127.0.0.1 mongodb-3
To be able to connect both internally and externally, you will need to run each MongoDB service on different ports.
The following script will initiate a 3-node MongoDB replicaset and run a test client. I recommend using the Bitnami image as it takes care of the replset initiation for you. (Borrowing heavily from this configuration)
#!/bin/bash
PROJECT_NAME=replset_test
MONGODB_VERSION=4.4
PYTHON_VERSION=3.9.6
PYMONGO_VERSION=4.0.1
cd "$(mktemp -d)" || exit
cat << EOF > Dockerfile
FROM python:${PYTHON_VERSION}-slim-buster
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt
COPY ${PROJECT_NAME}.py .
CMD [ "python", "./${PROJECT_NAME}.py" ]
EOF
cat << EOF > requirements.txt
pymongo==${PYMONGO_VERSION}
EOF
cat << EOF > ${PROJECT_NAME}.py
from pymongo import MongoClient
connection_string = 'mongodb://root:password123#mongodb-1:27017,mongodb-2:27018,mongodb-3:27019/mydatabase?authSource=admin&replicaSet=replicaset'
client = MongoClient(connection_string)
db = client.db
db['mycollection'].insert_one({'a': 1})
record = db['mycollection'].find_one()
if record is not None:
print(f'{__file__}: MongoDB connection working using connection string "{connection_string}"')
EOF
cp ${PROJECT_NAME}.py ${PROJECT_NAME}_external.py
cat << EOF > docker-compose.yaml
version: '3.9'
services:
mongodb-1:
image: docker.io/bitnami/mongodb:${MONGODB_VERSION}
ports:
- 27017:27017
environment:
- MONGODB_ADVERTISED_HOSTNAME=mongodb-1
- MONGODB_PORT_NUMBER=27017
- MONGODB_REPLICA_SET_MODE=primary
- MONGODB_ROOT_PASSWORD=password123
- MONGODB_REPLICA_SET_KEY=replicasetkey123
volumes:
- 'mongodb_master_data:/bitnami/mongodb'
mongodb-2:
image: docker.io/bitnami/mongodb:${MONGODB_VERSION}
ports:
- 27018:27018
depends_on:
- mongodb-1
environment:
- MONGODB_ADVERTISED_HOSTNAME=mongodb-2
- MONGODB_PORT_NUMBER=27018
- MONGODB_REPLICA_SET_MODE=secondary
- MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
- MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=password123
- MONGODB_REPLICA_SET_KEY=replicasetkey123
mongodb-3:
image: docker.io/bitnami/mongodb:${MONGODB_VERSION}
ports:
- 27019:27019
depends_on:
- mongodb-1
environment:
- MONGODB_ADVERTISED_HOSTNAME=mongodb-3
- MONGODB_PORT_NUMBER=27019
- MONGODB_REPLICA_SET_MODE=secondary
- MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
- MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=password123
- MONGODB_REPLICA_SET_KEY=replicasetkey123
${PROJECT_NAME}:
container_name: ${PROJECT_NAME}
build: .
depends_on:
- mongodb-1
- mongodb-2
- mongodb-3
volumes:
mongodb_master_data:
driver: local
EOF
docker rm --force $(docker ps -a -q --filter name=mongo) 2>&1 > /dev/null
docker rm --force $(docker ps -a -q --filter name=${PROJECT_NAME}) 2>&1 > /dev/null
docker-compose up --build -d
python ${PROJECT_NAME}.py
docker ps -a -q --filter name=${PROJECT_NAME}
docker logs $(docker ps -a -q --filter name=${PROJECT_NAME})
If all is ok you will get an output confirming both internal and external connectivity:
/tmp/tmp.QM9tQPE8Dj/replset_test.py: MongoDB connection working using connection string "mongodb://root:password123#mongodb-1:27017,mongodb-2:27018,mongodb-3:27019/mydatabase?authSource=admin&replicaSet=replicaset"
d53e8c41ad20
//./replset_test.py: MongoDB connection working using connection string "mongodb://root:password123#mongodb-1:27017,mongodb-2:27018,mongodb-3:27019/mydatabase?authSource=admin&replicaSet=replicaset"
I have a local MongoDB replica set created following this SO answer.
The docker-compose file:
services:
mongo1:
container_name: mongo1
image: mongo:4.2
ports:
- 27017:27017
restart: always
command: ["--bind_ip_all", "--replSet", "rs" ]
mongo2:
container_name: mongo2
image: mongo:4.2
ports:
- 27018:27017
restart: always
command: ["--bind_ip_all", "--replSet", "rs" ]
mongo3:
container_name: mongo3
image: mongo:4.2
ports:
- 27019:27017
restart: always
command: ["--bind_ip_all", "--replSet", "rs" ]
replica_set:
image: mongo:4.2
container_name: replica_set
depends_on:
- mongo1
- mongo2
- mongo3
volume:
- ./initiate_replica_set.sh:/initiate_replica_set.sh
entrypoint:
- /initiate_replica_set.sh
The initiate_replica_set.sh file:
#!/bin/bash
echo "Starting replica set initialize"
until mongo --host mongo1 --eval "print(\"waited for connection\")"
do
sleep 2
done
echo "Connection finished"
echo "Creating replica set"
mongo --host mongo1 <<EOF
rs.initiate(
{
_id : 'rs0',
members: [
{ _id : 0, host : "mongo1:27017" },
{ _id : 1, host : "mongo2:27017" },
{ _id : 2, host : "mongo3:27017" }
]
}
)
EOF
echo "replica set created"
The replica set is brought up successfully and runs fine, but it errors when I try to connect to the replica set:
$ mongo "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs"
MongoDB shell version v5.0.2
connecting to: mongodb://localhost:27017,localhost:27018,localhost:27019/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs
{"t":{"$date":"2021-08-05T21:35:40.667Z"},"s":"I", "c":"NETWORK", "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"rs","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set rs"}}
Error: Could not find host matching read preference { mode: "nearest" } for set rs, rs/localhost:27017,localhost:27018,localhost:27019 :
connect#src/mongo/shell/mongo.js:372:17
#(connect):2:6
exception: connect failed
exiting with code 1
More verbose log:
{
"t": {
"$date": "2021-08-05T21:35:54.531Z"
},
"s": "I",
"c": "-",
"id": 4333222,
"ctx": "ReplicaSetMonitor-TaskExecutor",
"msg": "RSM received error response",
"attr": {
"host": "mongo1:27017",
"error": "HostUnreachable: Error connecting to mongo1:27017 :: caused by :: Could not find address for mongo1:27017: SocketException: Host not found (authoritative)",
"replicaSet": "rs",
"response": "{}"
}
}
What is the cause of the problem and how do I fix it?
There are some partial answers on this issue from various places, here is what I think as a complete answer.
The Cause
Mongo clients use the hostnames listed in the replica set config, not the seed list
Although the connection string is "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs", mongo client does not connect to the members of the replica set with seed addresses localhost:27017 etc, instead the client connects to the members in the replica config set returned from the seed hosts, i.e., the ones in the rs.initiate call. This is why the error message is Error connecting to mongo1:27017 instead of Error connecting to localhost:27017.
Container hostnames are not addressable outside the container network
A mongo client inside the same container network as the mongo server containers can connect to the server via addresses like mongo1:27017; however, a client on the host, which is outside of the container network, can not resolve mongo1 to an IP. The typical solution for this problem is proxy, see Access docker container from host using containers name for details.
The Fix
Because the problem involves docker networking and docker networking varies between Linux and Mac. The fixes are different on the two platforms.
Linux
The proxy fix (via 3rd party software or modifying /etc/hosts file) works fine but sometimes is not viable, e.g., running on remote CI hosts. A simple self-contained portable solution is to update the intiate_replia_set.sh script to initiate the replica set with member IPs instead of hostnames.
intiate_replia_set.sh
echo "Starting replica set initialization"
until mongo --host mongo1 --eval "print(\"waited for connection\")"
do
sleep 2
done
echo "Connection finished"
echo "Creating replica set"
MONGO1IP=$(getent hosts mongo1 | awk '{ print $1 }')
MONGO2IP=$(getent hosts mongo2 | awk '{ print $1 }')
MONGO3IP=$(getent hosts mongo3 | awk '{ print $1 }')
read -r -d '' CMD <<EOF
rs.initiate(
{
_id : 'rs',
members: [
{ _id : 0, host : '${MONGO1IP}:27017' },
{ _id : 1, host : '${MONGO2IP}:27017' },
{ _id : 2, host : '${MONGO3IP}:27017' }
]
}
)
EOF
echo $CMD | mongo --host mongo1
echo "replica set created"
This way the mongo replica set members have container IP instead of hostname in their addresses. And the container IP is reachable from the host.
Alternatively, we can assign static IP to each container explicitly in the docker-compose file, and use static IPs when initiating the replica set. It is a similar fix but with more work.
Mac
The above solution unfortunately does not work for Mac, because docker container IP on Mac is not exposed on the host network interface. https://docs.docker.com/docker-for-mac/networking/#per-container-ip-addressing-is-not-possible
The easiest way to make it work is to add following mapping in /etc/hosts file:
127.0.0.1 mongo1 mongo2 mongo3
I think it is enough to change the name of the replicaSet "rs" to "rs0", this is the name given to your replicaSet in the configuration.
You will have :
$ mongo "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0"
I am using to following docker-compose file to setup my sharded MongoDB which has 3 shards, 3 config servers and 2 router instances:
version: "3.5"
services:
mongorsn1:
container_name: mongors1n1
hostname: mongors1n1
image: mongo
command: mongod --shardsvr --replSet mongors1 --dbpath /data/db --port 27017
ports:
- 27017:27017
expose:
- "27017"
environment:
TERM: xterm
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data1:/data/db
mongors1n2:
container_name: mongors1n2
hostname: mongors1n2
image: mongo
command: mongod --shardsvr --replSet mongors1 --dbpath /data/db --port 27017
ports:
- 27027:27017
expose:
- "27017"
environment:
TERM: xterm
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data2:/data/db
mongors1n3:
container_name: mongors1n3
hostname: mongors1n3
image: mongo
command: mongod --shardsvr --replSet mongors1 --dbpath /data/db --port 27017
ports:
- 27037:27017
expose:
- "27017"
environment:
TERM: xterm
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data3:/data/db
mongocfg1:
container_name: mongocfg1
image: mongo
command: mongod --configsvr --replSet mongors1conf --dbpath /data/db --port 27017
environment:
TERM: xterm
expose:
- "27017"
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config1:/data/db
mongocfg2:
container_name: mongocfg2
image: mongo
command: mongod --configsvr --replSet mongors1conf --dbpath /data/db --port 27017
environment:
TERM: xterm
expose:
- "27017"
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config2:/data/db
mongocfg3:
container_name: mongocfg3
image: mongo
command: mongod --configsvr --replSet mongors1conf --dbpath /data/db --port 27017
environment:
TERM: xterm
expose:
- "27017"
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config3:/data/db
mongos1:
container_name: mongos1
image: mongo
hostname: mongos1
depends_on:
- mongocfg1
- mongocfg2
command: mongos --configdb mongors1conf/mongocfg1:27017,mongocfg2:27017,mongocfg3:27017 --port 27017
ports:
- 27019:27017
expose:
- "27017"
volumes:
- /etc/localtime:/etc/localtime:ro
mongos2:
container_name: mongos2
image: mongo
hostname: mongos2
depends_on:
- mongocfg1
- mongocfg2
command: mongos --configdb mongors1conf/mongocfg1:27017,mongocfg2:27017,mongocfg3:27017 --port 27017
ports:
- 27020:27017
expose:
- "27017"
volumes:
- /etc/localtime:/etc/localtime:ro
As you can see it is up and running correctly.
docker exec -it mongos1 bash -c "echo 'sh.status()' | mongo "
MongoDB shell version v4.4.6
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("f00e3ac7-e114-40b3-b7f8-0e5cf9c56e10") }
MongoDB server version: 4.4.6
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("609e21145937100b3f006e20")
}
shards:
{ "_id" : "mongors1", "host" : "mongors1/mongors1n1:27017,mongors1n2:27017,mongors1n3:27017", "state" : 1 }
active mongoses:
"4.4.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
bye
Now here is the docker-compose file I use to connect Orion and the IoT Agent to my MongoDB shards.
version: "3.5"
services:
orion:
image: atos/smbt/orion:2.5.2
hostname: orion
container_name: orion
restart: unless-stopped
ports:
- "1026:1026"
entrypoint:
- /usr/bin/contextBroker
- -fg
- -multiservice
- -ngsiv1Autocast
- -corsOrigin
- __ALL
command: -dbhost mongors1n1:27017,mongors1n2:27027,mongors1n3:27037 -rplSet mongors1 -logLevel ERROR -noCache
healthcheck:
test: curl --fail -s http://orion:1026/version || exit 1
networks:
- fiware_mongo_sharding_default
iot-agent:
image: atos/smbt/iotagent-ul:1.15.0
hostname: iot-agent
container_name: iot-agent
restart: unless-stopped
expose:
- "4041"
- "7896"
ports:
- "4041:4041"
- "7896:7896"
environment:
- IOTA_CB_HOST=orion
- IOTA_CB_PORT=1026
- IOTA_NORTH_PORT=4041
- IOTA_REGISTRY_TYPE=mongodb
- IOTA_LOG_LEVEL=ERROR
- IOTA_TIMESTAMP=true
- IOTA_CB_NGSI_VERSION=v2
- IOTA_AUTOCAST=true
- IOTA_MONGO_HOST=mongors1n1
- IOTA_MONGO_PORT=27017
- IOTA_MONGO_DB=iotagentul
- IOTA_HTTP_PORT=7896
- IOTA_PROVIDER_URL=http://iot-agent:4041
networks:
- fiware_mongo_sharding_default
networks:
fiware_mongo_sharding_default:
external: true
Orion is connected to MongoDB but with the IoT Agent I get the following error message.
time=2021-05-14T07:51:51.086Z | lvl=ERROR | corr=28202651-30e1-48dc-88a9-53de603e7e6d | trans=28202651-30e1-48dc-88a9-53de603e7e6d | op=IoTAgentNGSI.DbConn | from=n/a | srv=n/a | subsrv=n/a | msg=MONGODB-001: Error trying to connect to MongoDB: MongoNetworkError: failed to connect to server [mongos1:27019] on first connect [MongoNetworkError: connect ECONNREFUSED 172.20.0.9:27019] | comp=IoTAgent
I have tried different things such as:
- IOTA_MONGO_HOST=mongos1
- IOTA_MONGO_PORT=27019
but to no avail unfortunately. Is there something I am doing wrong?
Thanks in advance for your help.
I found the solution myself. What I was doing wrong is that the IoT Agent should be NOT be connected with a Sharded MongoDB as it only expects one Mongo instance. That instance is used to persist and retrieve IoT device created when you provision a sensor via the IoT Agent. Device measurement are sent to the Orion Context Broket which will persist into the Sharded MongoDB. Hope it can help somebody else.
I want to setup MongoDB replica set with docker-compose with only one node. This is my docker-compose.yml file:
version: '3.7'
services:
mongodb1:
image: mongo:latest
command: mongod --replSet rs0
ports:
- 27017:27017
volumes:
- ./mongodb1/data:/data/db
networks:
- mongo-dev-net
setup-rs:
image: mongo:latest
command: mongo mongodb://mongodb1:27017 --eval "rs.initiate();"
depends_on:
- mongodb1
networks:
- mongo-dev-net
networks:
mongo-dev-net:
driver: bridge
It gives me an error while trying to run command in setup-rs service. This is the error:
Error: couldn't connect to server mongodb1:27017, connection attempt failed: SocketException: Error connecting to mongodb1:27017 (MY_IP:27017) :: caused by :: Connection refused :
How should I initiate replica set without using any setup.sh file with only one node? Thanks in advance.
You need to first start your 2 mongodb instances and then call the initiate in a second moment (after both are started)
Something like:
#!/bin/bash
docker-compose up -d
sleep 5
docker exec mongodb1 /scripts/rs-init.sh
And in your mongodb1 you have the rs-init.sh script like:
#!/bin/bash
mongo <<EOF
var config = {
"_id": "dbrs",
"version": 1,
"members": [
{
"_id": 1,
"host": "mongodb1:27017",
"priority": 1
},
]
};
rs.initiate(config, { force: true });
rs.status();
EOF
You can take a look at the detailed step by step here
PS: I didn't test the full solution
In my remote machine, I've set up a docker container machine that I manage using docker-compose.
I created 3 docker containers for each MongoDB instance I want in my replica set
mongodb_01:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
ports:
- 10001:27017
volumes:
- ./mongodb/01:/data/db
mongodb_02:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_01
ports:
- 10002:27017
volumes:
- ./mongodb/02:/data/db
mongodb_03:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_02
ports:
- 10003:27017
volumes:
- ./mongodb/03:/data/db
I also configured the replica set. and this is an excerpt:
"_id" : "rs0",
...
"members" : [
{
"_id" : 0,
"host" : "mongodb_01:27017",
...
},
{
"_id" : 1,
"host" : "mongodb_02:27017",
...
},
{
"_id" : 2,
"host" : "mongodb_03:27017",
...
}
],
...
}
Everything works fine, and intra-communications between other docker images and this replica set works fine using the connection string
mongodb://mongodb_01:27017,mongodb_02:27017,mongodb_03:27017/<database>?replicaSet=rs0
The problem is when I need to connect a remote client to this replica set.
For example, using mongoose via node on my dev machine I get:
MongoNetworkError: failed to connect to server [mongodb_02:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongodb_02 mongodb_02:27017]
Sometimes it fails on mongodb_03.
Edit: as pointed out, here's my connection string from remote machine:
mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
Edit 2: using a client like Mongodb Compass I can successfully connect to the single instances correctly. When I add the replicaset, i got the error.
So I tried to create a dummy container with mongodb (using mongo:latest).
$ docker run -it mongo:latest bash
and running
mongo mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
I get
MongoDB shell version v4.0.6
connecting to: mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?gssapiServiceName=mongodb&replicaSet=rs0
2019-03-04T16:12:54.375+0000 I NETWORK [js] Starting new replica set monitor for rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.377+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to <remote-host>:10003 (1 connections now open to <remote-host>:10003 with a 5 second timeout)
2019-03-04T16:12:54.377+0000 I NETWORK [js] Successfully connected to <remote-host>:10001 (1 connections now open to <remote-host>:10001 with a 5 second timeout)
2019-03-04T16:12:54.378+0000 I NETWORK [js] changing hosts to rs0/mongodb_01:27017,mongodb_02:27017,mongodb_03:27017 from rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.882+0000 W NETWORK [js] Unable to reach primary for set rs0
2019-03-04T16:12:54.882+0000 I NETWORK [js] Cannot reach any nodes for set rs0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
and so on.
Thanks for any help and suggestion !
I faced exactly same problem with you, and I've figured out it.
It is because that your remote client does not know 'mongo1:27017' host. It's is just used inside docker network only.
It is a bit tricky to explain how I've solved this problem. I'll show my docker-compose.yml first.
version: "3.3"
services:
mongo-primary:
container_name: mongo-primary
hostname: mongo-primary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/primary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
extra_hosts:
- "address.whichCanAccess.yourServer:192.168.1.xx"
networks:
- mongo-cluster
ports:
- 27017:27017
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: changeme
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-secondary:
container_name: mongo-secondary
hostname: mongo-secondary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/secondary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27018:27017
restart: always
command: --bind_ip_all -auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-arbiter:
container_name: mongo-arbiter
hostname: mongo-arbiter
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/arbiter/data/arb:/data/arb
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27019:27017
restart: always
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
networks:
mongo-cluster:
!Important part is 'extra_hosts'!! It can make containers access to host computer.
"address.WhichCanAccess.yourServer" <- in my case, my asus router has been set with asus ddns, so it will be XXX.asuscomm.com
"192.168.1.xx" <- IP addres which asus router has assigned the host computer
Maybe some configuration of those compose file are not required.
Run 3 containers with docker-compose.
Next, enter the primary's mongo shell, set replica like below
config = {
"_id": "rs0",
"members": [{
"_id": 0,
"host": "address.whichCanAccess.yourServer:27017"
}, {
"_id": 1,
"host": "address.whichCanAccess.yourServer:27018"
}, {
"_id": 2,
"host": "address.whichCanAccess.yourServer:27019",
arbiterOnly: true
}]
}
rs.initiate(config)
In this way, mongo containers will communicate each other through docker's host network, and it can be accessed from remote IP.
You can use localhost for this purpose. The compose will roughly look like:
version: "3"
services:
mongodb:
image: mongo:4.0.11
ports:
- "27017:27017"
extra_hosts:
- "localhost:0.0.0.0"
volumes:
- "./lambda/docker/mongod.conf:/etc/mongod.conf"
Then when initializing replicaset, make sure to set your host to localhost. Example:
{
// ........
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
// .........
}
]
// ...........
}
Tested on OSX. On linux/windows it may potentially have different behavior.
I ended up using a free version of Atlas for test and integration. Once the my work is complete, I'll use my own replica set on my servers.
That's a bit of a hassle, and I need to triple check everything before production without the support of dev tools and debuggers.