RHEL8 and docker-compose default network error EHOSTUNREACH - docker-compose

We have been using single container Docker images for some time without issues on RHEL8. We need to move toward integrating multiple services using docker-compose but have not been successful in even simple attempts.
We are using Mongo (mongo:4.2.3-bionic) and NodeJS (node:alpine).
We created a simple node application which is trying to add a single document to a MongoDB collection. The code for dbwrite.js is:
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://mongo:27017/", function(err, mongodb) {
if (err) throw err;
var mongodbo = mongodb.db("test");
var doc = {"payload":"test doc"};
mongodbo.collection("test2").insertOne(doc, function(err, res) {
if (err) throw err;
});
mongodb.close();
});
The Dockerfile for dbwrite.js is:
FROM node:alpine
ADD . /
CMD ["node", "dbwrite.js"]
The Mongo container was pulled from DockerHub as was the Node container.
The docker-compose.yaml file:
version: '3.1'
services:
mongo:
image: mongo:4.2.3-bionic
container_name: mongo
restart: always
ports:
- 27017:27017
volumes:
- ./mongo_db:/data/db
app:
image: dbwrite:v0.1
container_name: dbwrite
If we perform "docker-compose up" the dbwrite container throws an error:
dbwrite | /node_modules/mongodb/lib/topologies/server.js:233
dbwrite | throw err;
dbwrite | ^
dbwrite |
dbwrite | MongoNetworkError: failed to connect to server [mongo:27017] on first connect [Error: connect EHOSTUNREACH 172.22.0.2:27017
dbwrite | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1137:16) {
dbwrite | name: 'MongoNetworkError',
dbwrite | [Symbol(mongoErrorContextSymbol)]: {}
dbwrite | }]
dbwrite | at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:438:11)
dbwrite | at Pool.emit (events.js:321:20)
dbwrite | at /node_modules/mongodb/lib/core/connection/pool.js:561:14
dbwrite | at /node_modules/mongodb/lib/core/connection/pool.js:994:11
dbwrite | at /node_modules/mongodb/lib/core/connection/connect.js:31:7
dbwrite | at callback (/node_modules/mongodb/lib/core/connection/connect.js:264:5)
dbwrite | at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:294:7)
dbwrite | at Object.onceWrapper (events.js:428:26)
dbwrite | at Socket.emit (events.js:321:20)
dbwrite | at emitErrorNT (internal/streams/destroy.js:84:8) {
dbwrite | name: 'MongoNetworkError',
dbwrite | [Symbol(mongoErrorContextSymbol)]: {}
dbwrite | }
dbwrite exited with code 1
Rebuilding the container (doing it the hard way -- I know -- but wanting to keep everything as identical as possible), and replacing the Dockerfile CMD line
CMD ["node", "dbwrite.js"]
with
CMD ["ping", "-c", "20", "mongo"]
yields normal ping responses from "mongo" so I believe the default network was created right and the DNS is happening as expected, yet my node application gets EHOSTUNREACH.
dbwrite | 64 bytes from 172.22.0.2: seq=15 ttl=64 time=0.072 ms
dbwrite | 64 bytes from 172.22.0.2: seq=16 ttl=64 time=0.080 ms
dbwrite | 64 bytes from 172.22.0.2: seq=17 ttl=64 time=0.067 ms
dbwrite | 64 bytes from 172.22.0.2: seq=18 ttl=64 time=0.121 ms
dbwrite | 64 bytes from 172.22.0.2: seq=19 ttl=64 time=0.097 ms
dbwrite |
dbwrite | --- mongo ping statistics ---
dbwrite | 20 packets transmitted, 20 packets received, 0% packet loss
dbwrite | round-trip min/avg/max = 0.065/0.086/0.121 ms
dbwrite exited with code 0
If we edit the dbwrite.js code and replace, "mongo" in the connect() method with "localhost" and execute "node dbwrite.js" from the localhost (outside a container), the Document to the Collection. The Mongo container log reports that it is listening on 0.0.0.0.
mongo | 2020-02-10T19:35:26.337+0000 I NETWORK [listener] Listening on 0.0.0.0
mongo | 2020-02-10T19:35:26.337+0000 I NETWORK [listener] waiting for connections on port 27017
While I don't have the output captured, previous executions of "docker network inspect" showed both containers and their assigned IPv4 addresses on 172.22.0.x/16. IPAM showed using the default driver "bridge" on subnet 172.22.0.0/16 and a gateway of 172.22.0.1.
Any suggestions on what could be wrong would be greatly appreciated. We are on the verge of down-grading off RHEL8 to see if that is related to our problem given that Red Hat so vocally claims NOT to support Docker. Seems like it is some network security issue since ICMP ping can traverse the bridge but TCP socket connection cannot.

Related

Default Docker Compose for MongoDB and Mongo-Express times out on Debian

I am trying to setup a docker-compose setup to run MongoDB and Mongo-Express on a Debian virtual machine.
I've read up on problems with volumes, same docker networks, setting ME_CONFIG_MONGODB_SERVER to the Mongodb container name and having problems with UFW.
This is my current docker-compose.yml
version: '3.7'
services:
mongodb:
image: mongo
container_name: mongodb
restart: always
ports:
- '27017:27017'
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
- MONGO_INITDB_DATABASE=default
networks:
- mongo-compose-network
volumes:
- /home/paul/mongodbdocker/data:/data/db
mongo-express:
image: mongo-express
container_name: mongo-express
restart: always
depends_on:
- mongodb
networks:
- mongo-compose-network
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=admin
- ME_CONFIG_MONGODB_ADMINPASSWORD=password
- ME_CONFIG_MONGODB_SERVER=mongodb
- ME_CONFIG_BASICAUTH_USERNAME=admin
- ME_CONFIG_BASICAUTH_PASSWORD=password
ports:
- '8081:8081'
volumes:
- /home/paul/mongodbdocker/data:/data/db
networks:
mongo-compose-network:
driver: bridge
I have disabled the UFW firewall with sudo ufw disable and also edited the ufw configuration as described in here What is the best practice of docker + ufw under Ubuntu from Feng.
The MongoDB starts up, but sadly mongo-express fails on first connection with this error:
mongo-express | Welcome to mongo-express
mongo-express | ------------------------
mongo-express |
mongo-express |
mongo-express | (node:8) [MONGODB DRIVER] Warning: Current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
mongo-express | Could not connect to database using connectionString: mongodb://admin:password#mongodb:27017/"
mongo-express | (node:8) UnhandledPromiseRejectionWarning: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [MongoNetworkTimeoutError: connection timed out
mongo-express | at connectionFailureError (/node_modules/mongodb/lib/core/connection/connect.js:362:14)
mongo-express | at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:330:16)
mongo-express | at Object.onceWrapper (events.js:420:28)
mongo-express | at Socket.emit (events.js:314:20)
mongo-express | at Socket._onTimeout (net.js:483:8)
mongo-express | at listOnTimeout (internal/timers.js:554:17)
mongo-express | at processTimers (internal/timers.js:497:7)]
mongo-express | at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:441:11)
mongo-express | at Pool.emit (events.js:314:20)
mongo-express | at /node_modules/mongodb/lib/core/connection/pool.js:564:14
mongo-express | at /node_modules/mongodb/lib/core/connection/pool.js:1000:11
mongo-express | at /node_modules/mongodb/lib/core/connection/connect.js:32:7
mongo-express | at callback (/node_modules/mongodb/lib/core/connection/connect.js:300:5)
mongo-express | at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:330:7)
mongo-express | at Object.onceWrapper (events.js:420:28)
mongo-express | at Socket.emit (events.js:314:20)
mongo-express | at Socket._onTimeout (net.js:483:8)
mongo-express | at listOnTimeout (internal/timers.js:554:17)
mongo-express | at processTimers (internal/timers.js:497:7)
mongo-express | (node:8) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
mongo-express | (node:8) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Docker-compose service with traefik and internal network

I want to use both internal and an external traefik network in my container.
Problem: when I define an internal network, traefik loses communication with my service.
traefik_webgateway internal network
+----------+ +--------------------------+
| traefik | | +------+ +-----+ |
| +--------------+ | app | | api | |
| | | | | +------+ +-----+ |
| | | proxy | | |
| | | | | +-----+ +------+ |
| | | | | |auth | |worker| |
| +--------------+ +-----+ +------+ |
| | | |
+----------+ +--------------------------+
docker-compose.traefik.yml:
services:
traefik:
image: traefik:v2.4
restart: unless-stopped
ports:
- 80:80
- 8080:8080
- 443:443
networks:
- webgateway
networks:
webgateway:
driver: bridge
docker-compose.yml:
services:
proxy:
networks:
- internal # <=== this causes traefik point the healthcheck to the 172 IP instead of the 192 IP (see Edit below)
- traefik
labels:
- traefik.http....
app:
networks:
- internal
api:
networks:
- internal
auth:
networks:
- internal
worker:
networks:
- internal
networks:
internal:
traefik:
external:
name: traefik_webgateway
I don't want my services to use the external traefik network because I want my services to be namespaced for my green/blue deployment.
I was wondering why this is happening and if there's a solution.
Thanks in advance!
EDIT:
I obtained the proxy container's network:
docker inspect -f '{{range.NetworkSettings.Networks}} {{.IPAddress}}{{end}}' <container id>
And received 2 IPS:
172.18.A.A 192.168.B.B
I have a healthcheck that pings /health. On the traefik dashboard:
http://172.18.A.A:8000
Traefik is unable to communicate with this IP. Sometimes it's able to when it picks the other IP: 192.168.B.B.
From within the proxy container, I'm able to ping proxy (it uses the "B" IP)
PING proxy (192.168.B.B): 56 data bytes
64 bytes from 192.168.16.4: seq=0 ttl=64 time=0.090 ms
64 bytes from 192.168.16.4: seq=1 ttl=64 time=0.080 ms
64 bytes from 192.168.16.4: seq=2 ttl=64 time=0.065 ms
64 bytes from 192.168.16.4: seq=3 ttl=64 time=0.069 ms
I am able to ping: ping 192.168.B.B
I am unable to ping: ping 172.18.A.A

MongoDB Golang driver trying to connect to localhost instead of docker host

ERROR
app | 2020/07/14 13:19:00 server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: localhost:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 127.0.0.1:27017: connect: connection refused }, ] }
I am using docker-compose to manage golang app and mongo db (without replica set) instances. docker-compose file contents:
version: '3.1'
services:
mongodb:
image: mongo
container_name: mongodb
restart: always
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: local
command: mongod --bind_ip mongodb
app:
image: app:1.4
container_name: app
environment:
- MONGO_URL=mongodb
ports:
- 80:8080
depends_on:
- mongodb
restart: always
Running Docker Containers:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d130336be71e app:1.4 "/app/main" 4 minutes ago Up 2 seconds 0.0.0.0:80->8080/tcp app
dc57dd2de645 mongo "docker-entrypoint.s…" 4 minutes ago Up 2 seconds 0.0.0.0:27017->27017/tcp mongodb
For Connecting Golang app with MongoDB
I am using official mongodb go driver (go.mongodb.org/mongo-driver/mongo) and my code is trying to connect to docker host (mongodb):
var dbUser string = "root"
var dbPassword string = "local"
dbURL := os.Getenv("MONGO_URL")
clientOptions := options.Client()
clientOptions.SetDirect(true).ApplyURI("mongodb://" + dbUser + ":" + dbPassword + "#" + dbURL +":27017/?connect=direct")
Troubleshooting
As part of trying to figure out what is wrong I looked at similar issues highlighted in stackoverflow and tried below options:
Tried to use connect=direct in connection URI
Tried to use SetDirect
Added --bindip as highlighted in my docker-compose file
I logged-in into the app container and tried to ping mongodb and it worked:
root#d130336be71e:/app# ping mongodb
PING mongodb (172.18.0.2) 56(84) bytes of data.
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=1 ttl=64 time=0.159 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=2 ttl=64 time=0.200 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=3 ttl=64 time=0.079 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=4 ttl=64 time=0.087 ms
64 bytes from mongodb.app_default (172.18.0.2): icmp_seq=5 ttl=64 time=0.083 ms
^C
--- mongodb ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 60ms
rtt min/avg/max/mdev = 0.079/0.121/0.200/0.050 ms
After trying these options, I am seeking help from Golang community as I am fairly new to Golang.
Thanks in Advance.

Express container unable to connect to Mongo Image with Docker-Compose

I am trying to connect an express container to MongoDB image with docker compose but the connection is being rejected, I can connect to the db with robomongo. I can't get what is happening, this is the express code that connects it:
mongoose.connect('mongodb://localhost:27017/database')
.then(()=>console.log('connection succesfull to url'))
.catch((err)=>console.error(err));
This is the docker-compose file
version: "3"
services:
backend:
build:
context: ../backend
dockerfile: ${PWD}/images/backend/Dockerfile
container_name: backend
ports:
- "${BACKEND_PORT}:${BACKEND_PORT}"
env_file:
- ./deploy.env
environment:
- PORT=3000
- MONGO_CONNECTION=${MONGO_CONNECTION}
command: npm start
links:
- mongodb
depends_on:
- mongodb
front-app:
build:
context: ../front-app
dockerfile: ${PWD}/images/angular/Dockerfile
container_name: front-app
ports:
- "${FRONTEND_PORT}:4200"
env_file:
- ./deploy.env
command: npm start
mongodb:
image: mongo:3.6
container_name: mongo
volumes:
- "${MONGO_DB_DATA}:/data/db"
- "${MONGO_DB_DATA}:/data/configdb"
ports:
- "27017:27017"
This is the error
backend | { MongoNetworkError: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]
backend | at Pool.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/topologies/server.js:564:11)
backend | at Pool.emit (events.js:182:13)
backend | at Pool.EventEmitter.emit (domain.js:442:20)
backend | at Connection.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/connection/pool.js:317:12)
backend | at Object.onceWrapper (events.js:273:13)
backend | at Connection.emit (events.js:182:13)
backend | at Connection.EventEmitter.emit (domain.js:442:20)
backend | at Socket.<anonymous> (/usr/src/app/node_modules/mongodb-core/lib/connection/connection.js:246:50)
backend | at Object.onceWrapper (events.js:273:13)
backend | at Socket.emit (events.js:182:13)
backend | at Socket.EventEmitter.emit (domain.js:442:20)
backend | at emitErrorNT (internal/streams/destroy.js:82:8)
backend | at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
backend | at process._tickCallback (internal/process/next_tick.js:63:19)
backend | name: 'MongoNetworkError',
backend | errorLabels: [ 'TransientTransactionError' ],
backend | [Symbol(mongoErrorContextSymbol)]: {} }
Every process running in docker container thinks that he is "the only one in the world". It means that for this process localhost means: my, container's localhost. And your backend is alone in his container so that's why he cannot find mongodb under localhost.
To fix that problem you should place hostname "mongodb" instead of "localhost" as in docker-compose you can access services using theirs names - it means that mongodb container can also access your backend using "backend" domain.
Please note also that "links" is deprecated in docker and shouldn't be used - it's not required in your configuration as docker-compose gives every service within docker-compose file access to each other using method mentioned above.
Jakub Bujny answer is spot on, I did not understand because it did not include an example. So I am adding one.
mongoose.connect('mongodb://mongodb:27017/database')
.then(()=>console.log('connection succesfull to url'))
.catch((err)=>console.error(err));

Seed mongodb replica set

I want to create replica set of 3 nodes using docker-compose and seed initial data to them. If I remove --replSet and seed data without specifying hosts I have no problems.
docker-compose.yml
master:
image: 'mongo:3.4'
ports:
- '50000:27017'
volumes:
- './restaurants.json:/restaurants.json'
- './setup.js:/docker-entrypoint-initdb.d/00_setup.js'
- './seed.sh:/docker-entrypoint-initdb.d/01_seed.sh'
command: '--replSet rs'
slave1:
image: 'mongo:3.4'
ports:
- '50001:27017'
command: '--replSet rs'
slave2:
image: 'mongo:3.4'
ports:
- '50002:27017'
command: '--replSet rs'
seed.sh
# ...
_wait "slave1"
_wait "slave2"
echo "Starting to import data..."
mongoimport --host="rs/master:27017,slave1:27017,slave2:27017" --db db --collection restaurants --drop --file /restaurants.json > /dev/null
echo "Done."
Log
master_1 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/01_seed.sh
master_1 | Waiting for slave1...
master_1 | .
master_1 | Done.
master_1 | Waiting for slave2...
master_1 | Done.
master_1 | Starting to import data...
master_1 | 2017-11-26T16:06:39.148+0000 [........................] db.restaurants 0B/11.3MB (0.0%)
master_1 | 2017-11-26T16:06:39.653+0000 [........................] db.restaurants 0B/11.3MB (0.0%)
master_1 | 2017-11-26T16:06:39.653+0000 Failed: error connecting to db server: no reachable servers
master_1 | 2017-11-26T16:06:39.653+0000 imported 0 documents
mongoreplication_master_1 exited with code 1
This question is old but i ran into the same issue recently, it's worth noting that the mongo docker-entrypoint.sh script will strip the --replicaSet argument during the initDb phase, see:
https://github.com/docker-library/mongo/blob/master/3.6/docker-entrypoint.sh#L237
So you can't connect to the host that is running the init scripts, you can create another container with the sole purpose of initializing the replicaset however and override the docker-entrypoint.sh