Cannot connect to a working, Dockerized Mongo ReplSet - mongodb

I'm trying to create a Mongo ReplicaSet running locally, using the following docker-compose file:
version: '2.0'
services:
db01:
image: mongo:3.4
mem_limit: 512m
restart: always
ports:
- "27017:27017"
volumes:
- datadb01:/data/db
- ./etc/mongod.conf:/etc/mongod.conf
command: mongod --smallfiles --noIndexBuildRetry --replSet rs0
container_name: db01
networks:
- mongo
db02:
image: mongo:3.4
mem_limit: 512m
restart: always
ports:
- "27018:27017"
volumes:
- datadb02:/data1/db
- ./etc/mongod.conf:/etc/mongod.conf
command: mongod --smallfiles --noIndexBuildRetry --replSet rs0
container_name: db02
networks:
- mongo
So this actually seems to work! I won't post the whole thing but when I run rs.status() I get this:
{
"set" : "rs0",
...
"ok" : 1
}
However, when I try to connect to the replicaSet, rather than just a single node, I get this bunch of error logs:
~/utils/infrastructure/content-repl-set [replay-content*]: mongo --host rs0/127.0.0.1:27017,127.0.0.1:27018
MongoDB shell version v4.0.0
connecting to: mongodb://127.0.0.1:27017,127.0.0.1:27018/?replicaSet=rs0
2018-09-10T17:50:42.173-0400 I NETWORK [js] Starting new replica set monitor for rs0/127.0.0.1:27017,127.0.0.1:27018
2018-09-10T17:50:42.176-0400 I NETWORK [js] Successfully connected to 127.0.0.1:27018 (1 connections now open to 127.0.0.1:27018 with a 5 second timeout)
2018-09-10T17:50:42.177-0400 I NETWORK [js] changing hosts to rs0/db01:27017,db02:27017 from rs0/127.0.0.1:27017,127.0.0.1:27018
2018-09-10T17:50:42.177-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 127.0.0.1:27017 (1 connections now open to 127.0.0.1:27017 with a 5 second timeout)
2018-09-10T17:50:42.684-0400 W NETWORK [js] Unable to reach primary for set rs0
My best guess from that is that the mongo shell (and other mongo drivers, such as pymongo) isn't aware of the container name that I've specified in the docker container. Is there a way to expose this information to my local machine? Or am I off the mark? Thanks for your help

Are you using MacOS? if so, it seems those internal IP addresses in docker are unreachable because the docker (Linux) bridge network is not reachable from the macOS host.
More of it here: https://docs.docker.com/docker-for-mac/networking/

Related

Mongodb: Deploy a minimal mongodb cluster into docker swarm

I'm trying to deploy a minimal mongodb cluster into my development swarm environment like this:
Up to now:
version: "3.3"
networks:
net:
driver: overlay
services:
data1:
image: mongo:3.6
container_name: data1
command: mongod --shardsvr --replSet datars --smallfiles --port 27017
expose:
- 27017
networks:
- net
cfg1:
image: mongo:3.6
container_name: cfg1
command: mongod --configsvr --replSet cfgrs --smallfiles --port 27017
expose:
- 27017
networks:
- net
mongos1:
image: mongo:3.4
container_name: mongos1
command: mongos --configdb cfgrs/cfg1:27017
expose:
- 27017
networks:
- net
Into my config server, I'm getting these message:
2019-09-06T09:22:15.693+0000 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 134 could not get updated shard list from config server due to Read concern majority reads are currently not possible.; will retry after 30s,
Any ideas?

Connecting Docker mongo cluster from my local machine

I have created a mongodb replica in docker on my local machine. After configuration, replica is working if I connect inside the containers. Then when I connect localhost:30001,30002,30003 from my machine but outside the container, it says connection failed, prompting the following error message.
SERVER [node1:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node3
SERVER [node2:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node2
SERVER [node1:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node1
Here is my docker-comopse file
version: '3'
services:
node1:
image: mongo
container_name: "node1"
networks:
- mongocluster
ports:
- 30001:27017
command: mongod --dbpath /data/db --replSet repset
node2:
image: mongo
container_name: "node2"
networks:
- mongocluster
ports:
- 30002:27017
command: mongod --dbpath /data/db --replSet repset
depends_on:
- node1
node3:
image: mongo
container_name: "node3"
networks:
- mongocluster
ports:
- 30003:27017
command: mongod --dbpath /data/db --replSet repset
depends_on:
- node2
networks:
mongocluster:
driver: bridge
And this is how I configured the cluster in mongodb
config = {
"_id": "repset",
"members": [
{
"_id": 0,
"host": "node1:27017"
},
{
"_id": 1,
"host": "node2:27017"
},
{
"_id": 2,
"host": "node3:27017"
}
]
}
I understand my local machine cannot interpret IP for node1, node2 and node3. I am only using it for dev purpose, so what is the quickest (maybe dirty) way to fix it?
Hope the following links help:
mongodb-replica-set-with-docker-and-connecting-net-core-app
mongo-replica-set-docker-localhost
Your guess is correct. Mongo's Replica set self-reports node1,2,3 as an IP address and your client from outside of the container network tries [unsuccessfully] to reach these.
One easy way to fix it would be to put your client inside the container within the same docker network.
Another [dirty] option - adjust hosts file on your local machine to interpret nodeX as the localhost. You'll need to tweak the port numbers to, so host and container port numbers are the same (start mongo on 30001,2,3 in the container)
And perhaps the easiest one - forget about replica set and just run the standalone mongo container.

Connect to a remote dockerized MongoDB replica set

In my remote machine, I've set up a docker container machine that I manage using docker-compose.
I created 3 docker containers for each MongoDB instance I want in my replica set
mongodb_01:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
ports:
- 10001:27017
volumes:
- ./mongodb/01:/data/db
mongodb_02:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_01
ports:
- 10002:27017
volumes:
- ./mongodb/02:/data/db
mongodb_03:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_02
ports:
- 10003:27017
volumes:
- ./mongodb/03:/data/db
I also configured the replica set. and this is an excerpt:
"_id" : "rs0",
...
"members" : [
{
"_id" : 0,
"host" : "mongodb_01:27017",
...
},
{
"_id" : 1,
"host" : "mongodb_02:27017",
...
},
{
"_id" : 2,
"host" : "mongodb_03:27017",
...
}
],
...
}
Everything works fine, and intra-communications between other docker images and this replica set works fine using the connection string
mongodb://mongodb_01:27017,mongodb_02:27017,mongodb_03:27017/<database>?replicaSet=rs0
The problem is when I need to connect a remote client to this replica set.
For example, using mongoose via node on my dev machine I get:
MongoNetworkError: failed to connect to server [mongodb_02:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongodb_02 mongodb_02:27017]
Sometimes it fails on mongodb_03.
Edit: as pointed out, here's my connection string from remote machine:
mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
Edit 2: using a client like Mongodb Compass I can successfully connect to the single instances correctly. When I add the replicaset, i got the error.
So I tried to create a dummy container with mongodb (using mongo:latest).
$ docker run -it mongo:latest bash
and running
mongo mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
I get
MongoDB shell version v4.0.6
connecting to: mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?gssapiServiceName=mongodb&replicaSet=rs0
2019-03-04T16:12:54.375+0000 I NETWORK [js] Starting new replica set monitor for rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.377+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to <remote-host>:10003 (1 connections now open to <remote-host>:10003 with a 5 second timeout)
2019-03-04T16:12:54.377+0000 I NETWORK [js] Successfully connected to <remote-host>:10001 (1 connections now open to <remote-host>:10001 with a 5 second timeout)
2019-03-04T16:12:54.378+0000 I NETWORK [js] changing hosts to rs0/mongodb_01:27017,mongodb_02:27017,mongodb_03:27017 from rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.882+0000 W NETWORK [js] Unable to reach primary for set rs0
2019-03-04T16:12:54.882+0000 I NETWORK [js] Cannot reach any nodes for set rs0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
and so on.
Thanks for any help and suggestion !
I faced exactly same problem with you, and I've figured out it.
It is because that your remote client does not know 'mongo1:27017' host. It's is just used inside docker network only.
It is a bit tricky to explain how I've solved this problem. I'll show my docker-compose.yml first.
version: "3.3"
services:
mongo-primary:
container_name: mongo-primary
hostname: mongo-primary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/primary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
extra_hosts:
- "address.whichCanAccess.yourServer:192.168.1.xx"
networks:
- mongo-cluster
ports:
- 27017:27017
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: changeme
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-secondary:
container_name: mongo-secondary
hostname: mongo-secondary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/secondary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27018:27017
restart: always
command: --bind_ip_all -auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-arbiter:
container_name: mongo-arbiter
hostname: mongo-arbiter
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/arbiter/data/arb:/data/arb
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27019:27017
restart: always
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
networks:
mongo-cluster:
!Important part is 'extra_hosts'!! It can make containers access to host computer.
"address.WhichCanAccess.yourServer" <- in my case, my asus router has been set with asus ddns, so it will be XXX.asuscomm.com
"192.168.1.xx" <- IP addres which asus router has assigned the host computer
Maybe some configuration of those compose file are not required.
Run 3 containers with docker-compose.
Next, enter the primary's mongo shell, set replica like below
config = {
"_id": "rs0",
"members": [{
"_id": 0,
"host": "address.whichCanAccess.yourServer:27017"
}, {
"_id": 1,
"host": "address.whichCanAccess.yourServer:27018"
}, {
"_id": 2,
"host": "address.whichCanAccess.yourServer:27019",
arbiterOnly: true
}]
}
rs.initiate(config)
In this way, mongo containers will communicate each other through docker's host network, and it can be accessed from remote IP.
You can use localhost for this purpose. The compose will roughly look like:
version: "3"
services:
mongodb:
image: mongo:4.0.11
ports:
- "27017:27017"
extra_hosts:
- "localhost:0.0.0.0"
volumes:
- "./lambda/docker/mongod.conf:/etc/mongod.conf"
Then when initializing replicaset, make sure to set your host to localhost. Example:
{
// ........
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
// .........
}
]
// ...........
}
Tested on OSX. On linux/windows it may potentially have different behavior.
I ended up using a free version of Atlas for test and integration. Once the my work is complete, I'll use my own replica set on my servers.
That's a bit of a hassle, and I need to triple check everything before production without the support of dev tools and debuggers.

Cannot reach any nodes for set mongo config server

I want to deploy mongo cluster containing 1 mongo shard (replica-set), 1 configserver (replica-set) and 1 router via docker-compose. I have a problem that it's "Unable to reach primary for set mongoconfigserver".
I found an issue about it:
Unable to launch mongos, but in that case there was no replica-set for config-server, while I have required "--replSet mongoconfigserver" option in commands.
Here is my docker-compose.yml:
version: '3'
services:
# Config servers for metadata
mongoconfigserver-01:
container_name: mongoconfigserver-01
image: mongo
command: mongod --configsvr --replSet mongoconfigserver --dbpath /data/db --port 27017
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config1:/data/db
mongoconfigserver-02:
container_name: mongoconfigserver-02
image: mongo
command: mongod --configsvr --replSet mongoconfigserver --dbpath /data/db --port 27017
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config2:/data/db
mongoconfigserver-03:
container_name: mongoconfigserver-03
image: mongo
command: mongod --configsvr --replSet mongoconfigserver --dbpath /data/db --port 27017
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/config3:/data/db
# First shard
mongo-shard-01a:
image: mongo
command: mongod --port 27018 --replSet mongo-shard-01 --dbpath /data/db
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data1:/data/db
ports:
- 27017:27017
mongo-shard-01b:
image: mongo
command: mongod --port 27018 --replSet mongo-shard-01 --dbpath /data/db
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data2:/data/db
ports:
- 27027:27017
mongo-shard-01c:
image: mongo
command: mongod --port 27018 --replSet mongo-shard-01 --dbpath /data/db
volumes:
- /etc/localtime:/etc/localtime:ro
- /mongo_cluster/data3:/data/db
ports:
- 27037:27017
# Mongo router
mongo-router-01:
container_name: mongo-router-01
image: mongo
depends_on:
- mongoconfigserver-01
- mongoconfigserver-02
- mongoconfigserver-03
- mongo-shard-01a
- mongo-shard-01b
- mongo-shard-01c
command: mongos --configdb mongoconfigserver/mongoconfigserver-01:27017,mongoconfigserver-02:27017,mongoconfigserver-03:27017
volumes:
- /etc/localtime:/etc/localtime:ro
and logs:
mongo-router-01 | 2018-03-14T22:46:40.857+0100 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to mongoconfigserver-02:27017 (1 connections now open to mongoconfigserver-02:27017 with a 5 second timeout)
mongo-router-01 | 2018-03-14T22:46:40.857+0100 I NETWORK [shard registry reload] Successfully connected to mongoconfigserver-03:27017 (1 connections now open to mongoconfigserver-03:27017 with a 5 second timeout)
mongo-router-01 | 2018-03-14T22:46:40.857+0100 I NETWORK [monitoring keys for HMAC] Successfully connected to mongoconfigserver-01:27017 (1 connections now open to mongoconfigserver-01:27017 with a 5 second timeout)
mongo-router-01 | 2018-03-14T22:46:40.858+0100 W NETWORK [monitoring keys for HMAC] Unable to reach primary for set mongoconfigserver
mongo-router-01 | 2018-03-14T22:46:41.359+0100 W NETWORK [mongosMain] Unable to reach primary for set mongoconfigserver
mongo-router-01 | 2018-03-14T22:46:41.359+0100 I NETWORK [mongosMain] Cannot reach any nodes for set mongoconfigserver. Please check network connectivity and the status of the set. This has happened for 2 checks in a row.
mongo-router-01 | 2018-03-14T22:46:41.860+0100 W NETWORK [mongosMain] Unable to reach primary for set mongoconfigserver
Can anybody help with that?
Set the following parameters in the configuration server.
use admin
config = {
_id : "configs",
members : [
{_id : 0, host : "ip:21000" },
{_id : 1, host : "ip:21000" },
{_id : 2, host : "ip:21000" }
]
}
rs.initiate(config);

MongoError: connect ECONNREFUSED in Docker

I have my docker-compose orchestration and I'm getting this error:
connection error: { MongoError: failed to connect to server [172.17.0.2:27018] on first connect [MongoError: connect ECONNREFUSED 172.17.0.2:27018]
This is the code in server.js:
mongoose.connect('mongodb://mongodb:27018');
This happen only when I customize the command of my docker container:
docker-compose.yml:
version: "3"
services:
app:
build: ./my-node-app
depends_on:
- mongodb
mongodb:
image: mongo:3.5
command: mongod --port 27018
If I remove the --port 27018 and point to the default 27017 the error gets fixed.
What could be happening?
You're missing: --bind_ip_all
It seems that newer versions (>3.5) of the mongod daemon listen only to localhost by default.
Your are overriding this:
CMD ["mongod", "--bind_ip_all"]
So, also put "--bind_ip_all" in your docker-compose.yml:
mongodb:
image: mongo:3.5
command: mongod --port 27018 --bind_ip_all