Connecting Docker mongo cluster from my local machine - mongodb

I have created a mongodb replica in docker on my local machine. After configuration, replica is working if I connect inside the containers. Then when I connect localhost:30001,30002,30003 from my machine but outside the container, it says connection failed, prompting the following error message.
SERVER [node1:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node3
SERVER [node2:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node2
SERVER [node1:27017] (Type: UNKNOWN)
|_/ Connection error (MongoSocketOpenException): Exception opening socket
|____/ Unknown host: node1
Here is my docker-comopse file
version: '3'
services:
node1:
image: mongo
container_name: "node1"
networks:
- mongocluster
ports:
- 30001:27017
command: mongod --dbpath /data/db --replSet repset
node2:
image: mongo
container_name: "node2"
networks:
- mongocluster
ports:
- 30002:27017
command: mongod --dbpath /data/db --replSet repset
depends_on:
- node1
node3:
image: mongo
container_name: "node3"
networks:
- mongocluster
ports:
- 30003:27017
command: mongod --dbpath /data/db --replSet repset
depends_on:
- node2
networks:
mongocluster:
driver: bridge
And this is how I configured the cluster in mongodb
config = {
"_id": "repset",
"members": [
{
"_id": 0,
"host": "node1:27017"
},
{
"_id": 1,
"host": "node2:27017"
},
{
"_id": 2,
"host": "node3:27017"
}
]
}
I understand my local machine cannot interpret IP for node1, node2 and node3. I am only using it for dev purpose, so what is the quickest (maybe dirty) way to fix it?

Hope the following links help:
mongodb-replica-set-with-docker-and-connecting-net-core-app
mongo-replica-set-docker-localhost

Your guess is correct. Mongo's Replica set self-reports node1,2,3 as an IP address and your client from outside of the container network tries [unsuccessfully] to reach these.
One easy way to fix it would be to put your client inside the container within the same docker network.
Another [dirty] option - adjust hosts file on your local machine to interpret nodeX as the localhost. You'll need to tweak the port numbers to, so host and container port numbers are the same (start mongo on 30001,2,3 in the container)
And perhaps the easiest one - forget about replica set and just run the standalone mongo container.

Related

Problem while rs.initiate() for setup docker-compose for replica set

I want to setup MongoDB replica set with docker-compose with only one node. This is my docker-compose.yml file:
version: '3.7'
services:
mongodb1:
image: mongo:latest
command: mongod --replSet rs0
ports:
- 27017:27017
volumes:
- ./mongodb1/data:/data/db
networks:
- mongo-dev-net
setup-rs:
image: mongo:latest
command: mongo mongodb://mongodb1:27017 --eval "rs.initiate();"
depends_on:
- mongodb1
networks:
- mongo-dev-net
networks:
mongo-dev-net:
driver: bridge
It gives me an error while trying to run command in setup-rs service. This is the error:
Error: couldn't connect to server mongodb1:27017, connection attempt failed: SocketException: Error connecting to mongodb1:27017 (MY_IP:27017) :: caused by :: Connection refused :
How should I initiate replica set without using any setup.sh file with only one node? Thanks in advance.
You need to first start your 2 mongodb instances and then call the initiate in a second moment (after both are started)
Something like:
#!/bin/bash
docker-compose up -d
sleep 5
docker exec mongodb1 /scripts/rs-init.sh
And in your mongodb1 you have the rs-init.sh script like:
#!/bin/bash
mongo <<EOF
var config = {
"_id": "dbrs",
"version": 1,
"members": [
{
"_id": 1,
"host": "mongodb1:27017",
"priority": 1
},
]
};
rs.initiate(config, { force: true });
rs.status();
EOF
You can take a look at the detailed step by step here
PS: I didn't test the full solution

Connect to a remote dockerized MongoDB replica set

In my remote machine, I've set up a docker container machine that I manage using docker-compose.
I created 3 docker containers for each MongoDB instance I want in my replica set
mongodb_01:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
ports:
- 10001:27017
volumes:
- ./mongodb/01:/data/db
mongodb_02:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_01
ports:
- 10002:27017
volumes:
- ./mongodb/02:/data/db
mongodb_03:
image: mvertes/alpine-mongo
entrypoint: ['/usr/bin/mongod', '--bind_ip_all', '--replSet', 'rs0']
restart: always
depends_on:
- mongodb_02
ports:
- 10003:27017
volumes:
- ./mongodb/03:/data/db
I also configured the replica set. and this is an excerpt:
"_id" : "rs0",
...
"members" : [
{
"_id" : 0,
"host" : "mongodb_01:27017",
...
},
{
"_id" : 1,
"host" : "mongodb_02:27017",
...
},
{
"_id" : 2,
"host" : "mongodb_03:27017",
...
}
],
...
}
Everything works fine, and intra-communications between other docker images and this replica set works fine using the connection string
mongodb://mongodb_01:27017,mongodb_02:27017,mongodb_03:27017/<database>?replicaSet=rs0
The problem is when I need to connect a remote client to this replica set.
For example, using mongoose via node on my dev machine I get:
MongoNetworkError: failed to connect to server [mongodb_02:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongodb_02 mongodb_02:27017]
Sometimes it fails on mongodb_03.
Edit: as pointed out, here's my connection string from remote machine:
mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
Edit 2: using a client like Mongodb Compass I can successfully connect to the single instances correctly. When I add the replicaset, i got the error.
So I tried to create a dummy container with mongodb (using mongo:latest).
$ docker run -it mongo:latest bash
and running
mongo mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?replicaSet=rs0
I get
MongoDB shell version v4.0.6
connecting to: mongodb://<remote-host>:10001,<remote-host>:10002,<remote-host>:10003/<database>?gssapiServiceName=mongodb&replicaSet=rs0
2019-03-04T16:12:54.375+0000 I NETWORK [js] Starting new replica set monitor for rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.377+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to <remote-host>:10003 (1 connections now open to <remote-host>:10003 with a 5 second timeout)
2019-03-04T16:12:54.377+0000 I NETWORK [js] Successfully connected to <remote-host>:10001 (1 connections now open to <remote-host>:10001 with a 5 second timeout)
2019-03-04T16:12:54.378+0000 I NETWORK [js] changing hosts to rs0/mongodb_01:27017,mongodb_02:27017,mongodb_03:27017 from rs0/<remote-host>:10001,<remote-host>:10002,<remote-host>:10003
2019-03-04T16:12:54.882+0000 W NETWORK [js] Unable to reach primary for set rs0
2019-03-04T16:12:54.882+0000 I NETWORK [js] Cannot reach any nodes for set rs0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
and so on.
Thanks for any help and suggestion !
I faced exactly same problem with you, and I've figured out it.
It is because that your remote client does not know 'mongo1:27017' host. It's is just used inside docker network only.
It is a bit tricky to explain how I've solved this problem. I'll show my docker-compose.yml first.
version: "3.3"
services:
mongo-primary:
container_name: mongo-primary
hostname: mongo-primary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/primary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
extra_hosts:
- "address.whichCanAccess.yourServer:192.168.1.xx"
networks:
- mongo-cluster
ports:
- 27017:27017
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: changeme
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-secondary:
container_name: mongo-secondary
hostname: mongo-secondary
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/secondary/data/db:/data/db
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27018:27017
restart: always
command: --bind_ip_all -auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
mongo-arbiter:
container_name: mongo-arbiter
hostname: mongo-arbiter
image: mongo:4.0.11
volumes:
- $HOME/.dockerMongoRepl/arbiter/data/arb:/data/arb
- $HOME/.dockerMongoRepl/keyfile:/data/keyfile
depends_on:
- mongo-primary
extra_hosts:
- ""address.whichCanAccess.yourServer:192.168.1.xx""
networks:
- mongo-cluster
ports:
- 27019:27017
restart: always
command: --bind_ip_all --auth --keyFile /data/keyfile/mongo-cluster-key --replSet rs0 --enableMajorityReadConcern false
networks:
mongo-cluster:
!Important part is 'extra_hosts'!! It can make containers access to host computer.
"address.WhichCanAccess.yourServer" <- in my case, my asus router has been set with asus ddns, so it will be XXX.asuscomm.com
"192.168.1.xx" <- IP addres which asus router has assigned the host computer
Maybe some configuration of those compose file are not required.
Run 3 containers with docker-compose.
Next, enter the primary's mongo shell, set replica like below
config = {
"_id": "rs0",
"members": [{
"_id": 0,
"host": "address.whichCanAccess.yourServer:27017"
}, {
"_id": 1,
"host": "address.whichCanAccess.yourServer:27018"
}, {
"_id": 2,
"host": "address.whichCanAccess.yourServer:27019",
arbiterOnly: true
}]
}
rs.initiate(config)
In this way, mongo containers will communicate each other through docker's host network, and it can be accessed from remote IP.
You can use localhost for this purpose. The compose will roughly look like:
version: "3"
services:
mongodb:
image: mongo:4.0.11
ports:
- "27017:27017"
extra_hosts:
- "localhost:0.0.0.0"
volumes:
- "./lambda/docker/mongod.conf:/etc/mongod.conf"
Then when initializing replicaset, make sure to set your host to localhost. Example:
{
// ........
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
// .........
}
]
// ...........
}
Tested on OSX. On linux/windows it may potentially have different behavior.
I ended up using a free version of Atlas for test and integration. Once the my work is complete, I'll use my own replica set on my servers.
That's a bit of a hassle, and I need to triple check everything before production without the support of dev tools and debuggers.

How to run multiple MongoDBs with different ports within docker-compose?

I am trying to run two MongoDBs within one docker-compose. As I want to store different data in each of the databases, I need to distinguish them by different ports. If I run the following docker-compose, everything works fine for port 27017, but I cannot access port 27018. In the console everything looks the same for both services, only the following log is missing for the service on port 27018:
NETWORK [initandlisten] waiting for connections on port 27017
(shows up for port 27017)
When I try to access both services in the browser, the one on port 27017 works and the one on 27018 doesn't.
I'm a very glad if anybody can help me out with that issue. Working on it since pretty long. Thanks!
Here's my code:
version: "3"
services:
mongo:
image: mvertes/alpine-mongo:4.0.1-0
volumes:
- mongoDBvolume:/data/db
ports:
- "27017:27017"
testmongo:
image: mvertes/alpine-mongo:4.0.1-0
command: mongod --port 27018
volumes:
- mongoDBvolume:/data/testdb
ports:
- "27018:27017"
volumes:
mongoDBvolume:
driver: local
Just remove command: mongod --port 27018, so it will be on port 27017 inside the container.
The flow is like:
Host 27018 <--> docker bridge 27018:27017 <--> mongo container 27017
version: '3.7'
services:
mongo:
image : mongo
container_name: mongo
networks:
- mongo
environment:
- PUID=1000
- PGID=1000
ports:
- "${MONGO_PORT}:27027"
restart: unless-stopped
command: "mongod --bind_ip_all --config /etc/mongo/mongod.conf"
volumes:
- "./data:/data/db"
- "./mongo:/etc/mongo"
networks:
mongo:
driver: bridge
mongod.conf file :
net:
bindIp: 127.0.0.1
port: 27027
setParameter:
enableLocalhostAuthBypass: false

Cannot connect to a working, Dockerized Mongo ReplSet

I'm trying to create a Mongo ReplicaSet running locally, using the following docker-compose file:
version: '2.0'
services:
db01:
image: mongo:3.4
mem_limit: 512m
restart: always
ports:
- "27017:27017"
volumes:
- datadb01:/data/db
- ./etc/mongod.conf:/etc/mongod.conf
command: mongod --smallfiles --noIndexBuildRetry --replSet rs0
container_name: db01
networks:
- mongo
db02:
image: mongo:3.4
mem_limit: 512m
restart: always
ports:
- "27018:27017"
volumes:
- datadb02:/data1/db
- ./etc/mongod.conf:/etc/mongod.conf
command: mongod --smallfiles --noIndexBuildRetry --replSet rs0
container_name: db02
networks:
- mongo
So this actually seems to work! I won't post the whole thing but when I run rs.status() I get this:
{
"set" : "rs0",
...
"ok" : 1
}
However, when I try to connect to the replicaSet, rather than just a single node, I get this bunch of error logs:
~/utils/infrastructure/content-repl-set [replay-content*]: mongo --host rs0/127.0.0.1:27017,127.0.0.1:27018
MongoDB shell version v4.0.0
connecting to: mongodb://127.0.0.1:27017,127.0.0.1:27018/?replicaSet=rs0
2018-09-10T17:50:42.173-0400 I NETWORK [js] Starting new replica set monitor for rs0/127.0.0.1:27017,127.0.0.1:27018
2018-09-10T17:50:42.176-0400 I NETWORK [js] Successfully connected to 127.0.0.1:27018 (1 connections now open to 127.0.0.1:27018 with a 5 second timeout)
2018-09-10T17:50:42.177-0400 I NETWORK [js] changing hosts to rs0/db01:27017,db02:27017 from rs0/127.0.0.1:27017,127.0.0.1:27018
2018-09-10T17:50:42.177-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 127.0.0.1:27017 (1 connections now open to 127.0.0.1:27017 with a 5 second timeout)
2018-09-10T17:50:42.684-0400 W NETWORK [js] Unable to reach primary for set rs0
My best guess from that is that the mongo shell (and other mongo drivers, such as pymongo) isn't aware of the container name that I've specified in the docker container. Is there a way to expose this information to my local machine? Or am I off the mark? Thanks for your help
Are you using MacOS? if so, it seems those internal IP addresses in docker are unreachable because the docker (Linux) bridge network is not reachable from the macOS host.
More of it here: https://docs.docker.com/docker-for-mac/networking/

GOLANG client docker container getting unreachable when attempting to connect to mongodb container

I have a simple program written in golang which connect to a mongodb instance thus:
func main() {
session, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
ensureIndex(session)
}
I use docker compose to spin up two containers one to spin up my GOLANG API and the other to spin up mongo:
version: "3.3"
services:
api:
image: golang:latest
volumes:
- .:/go
working_dir: /go
environment:
- GOPATH=/go
command:
go run /go/src/main.go
ports:
- "8082:8000"
depends_on:
- db
networks:
- api-net
db:
build: .
expose:
- '27017'
container_name: 'mongo'
ports:
- "27017:27017"
networks:
- api-net
networks:
api-net:
driver: bridge
I use a Dockerfile in order to spin up mongo with a conf file that sets the instance to bind to all IPV4 and IPV6 networks:
FROM mongo:latest
COPY mongod.conf /etc/mongod.conf
CMD mongod --config /etc/mongod.conf##
and this is the mongod.conf file:
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
When I run docker-compose.up this is what I get:
Building db
Step 1/3 : FROM mongo:latest
---> a0f922b3f0a1
Step 2/3 : COPY mongod.conf /etc/mongod.conf
---> Using cache
---> f270e718c11e
Step 3/3 : CMD mongod --config /etc/mongod.conf
---> Running in 89ffc2495a2a
Removing intermediate container 89ffc2495a2a
---> fe2677d53122
Successfully built fe2677d53122
Successfully tagged carsupermarket_db:latest
WARNING: Image for service db was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating mongo ... done
Creating carsupermarket_api_1 ... done
Attaching to mongo, carsupermarket_api_1
api_1 | panic: no reachable servers
api_1 |
api_1 | goroutine 1 [running]:
api_1 | main.main()
api_1 | /go/src/main.go:38 +0x291
api_1 | exit status 2
I've trawled Google and stackoverflow and the only thing I could manage to find which is vaguely related to my issue is:
mongod --bind_ip using docker-compose version 2
however, my docker-compose.yml file as it stands 'Should' on paper work.
Can someone please point me in the correct direction as to why my GOLANG code cannot find my mongodb instance.
I think that when you call mgo.Dial("localhost:27017") the localhost part refers to the localhost within the golang container, and not the host the the containers are running on.
Docker will resolved container names as host names, change the dial to mgo.Dial("mongo:27017") and it should work.
From the docker container networking bridge docs
User-defined bridges provide automatic DNS resolution between containers.
Containers on the default bridge network can only access each other by
IP addresses, unless you use the --link option, which is considered
legacy. On a user-defined bridge network, containers can resolve each
other by name or alias.
https://docs.docker.com/network/bridge/
version: "2"
services:
api:
image: golang:latest
volumes:
- .:/go
# Work dir should be defined in dockerfile
#working_dir: /go
environment:
- GOPATH=/go
command:
go run /go/src/main.go
ports:
- "8082:8000"
depends_on:
- db
db:
build: .
container_name: 'mongo'
# You should define some volumes here if you want the data to persist.
You should be able to connect from the golang container to the mongo container using inter container communication using the hostname: 'mongo'