Trying to connect to mongodb service through Consul Connect Sidecar Proxy - mongodb

Question

I have a Minikube set up and a mongo instance running in it. I use Consul + Consul Connect to mesh my services. Only I can not connect to mongo from another service using sidecar upstreams, some weird stuff is happening...
My mongo instance is installed using bitnami helm chart, I just set the service name, set username and change the storage class to match my need, and put consul annotations for service mesh in pod annotation section:
image:
registry: docker.io
repository: bitnami/mongodb
tag: 4.2.5-debian-10-r3
pullPolicy: IfNotPresent
debug: false
serviceAccount:
create: true
name: "svc-identity-data"
usePassword: true
mongodbRootPassword: rootpassword
mongodbUsername: identity
mongodbPassword: identity
mongodbDatabase: company
service:
name: svc-identity-data
annotations: {}
type: ClusterIP
port: 27017
useStatefulSet: true
replicaSet:
enabled: false
useHostnames: true
name: rs0
replicas:
secondary: 1
arbiter: 1
pdb:
enabled: true
minAvailable:
primary: 1
secondary: 1
arbiter: 1
annotations: {}
labels: {}
podAnnotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service": "svc-identity-data"
"consul.hashicorp.com/connect-service-protocol": "tcp"
persistence:
enabled: true
mountPath: /bitnami/mongodb
subPath: ""
storageClass: "standard"
accessModes:
- ReadWriteOnce
size: 8Gi
annotations: {}
configmap:
storage:
dbPath: /bitnami/mongodb/data/db
journal:
enabled: true
directoryPerDB: false
systemLog:
destination: file
quiet: false
logAppend: true
logRotate: reopen
path: /opt/bitnami/mongodb/logs/mongodb.log
verbosity: 0
net:
port: 27017
unixDomainSocket:
enabled: true
pathPrefix: /opt/bitnami/mongodb/tmp
ipv6: false
bindIp: 0.0.0.0
processManagement:
fork: false
pidFilePath: /opt/bitnami/mongodb/tmp/mongodb.pid
setParameter:
enableLocalhostAuthBypass: true
security:
authorization: enabled
Secondly I started a stand-alone mongodb pod to use mongo client, and meshed with consul connect using annotations
apiVersion: v1
kind: Pod
metadata:
name: mongo-client
labels:
name: mongo-client
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service-upstreams": "svc-identity-data:28017"
"consul.hashicorp.com/connect-service-protocol": "tcp"
spec:
containers:
- name: mongo-client
image: mongo:4.2.5
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 27017
I now have a mongodb service and a mongo client pod with an upstream to mongodb service binded on 127.0.0.1:28017
When I try to connect to mongodb service using my upstream I get a behavior I don't understand
> kubectl exec -it mongo-client mongo --host 127.0.0.1 --port 28017 -u root -p rootpassword
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("8c46012d-8083-4029-8495-167bbe8bf063") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
No problem here, everything works perfectly fine to me, but if I use mongo with a connection string instead of separate parameters, I get a connection refused
> kubectl exec -it mongo-client mongo mongodb://root:roopassword#127.0.0.1:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-22T15:04:07.955+0000 I NETWORK [js] DBClientConnection failed to receive message from 127.0.0.1:28017 - HostUnreachable: Connection closed by peer
2020-04-22T15:04:07.968+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host '127.0.0.1:28017' :
connect#src/mongo/shell/mongo.js:341:17
#(connect):2:6
2020-04-22T15:04:07.973+0000 F - [main] exception: connect failed
2020-04-22T15:04:07.973+0000 E - [main] exiting with code 1
I don't understand at all what is the difference between using connection string and separate parameters, if you have any clue or a solution, please let me know.
P.S : I didn't set any secure communication (tls), I'm on a minikube (because I'm a Microservice Architecture and Kubernetes n00b) and it is to experiment service mesh (we need to live in the current era), a solution involving connecting to service without using the sidecar is not the point, by the way connecting directly to service works perfectly using connection string.
> kubectl exec -it mongo-client mongo -mongodb://root:roopassword#svc-identity-data:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://svc-identity-data:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("713febaf-2000-4ca6-8b1f-963c76986e72") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
EDIT : Rebooting minikube make all things work as intended. I will investigate more on the matter to understand why. Maybe someone else will hit the same issue.
EDIT 2 : I discovered one thing : connection error when connecting to mongo through sidecar is random, when I run command until it success, here is what I get
root#mongo-client:/# mongo mongodb://root:rootpassword#localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-24T12:51:15.641+0000 I NETWORK [js] DBClientConnection failed to receive message from localhost:28017 - HostUnreachable: Connection closed by peer
2020-04-24T12:51:15.702+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host 'localhost:28017' :
connect#src/mongo/shell/mongo.js:341:17
#(connect):2:6
2020-04-24T12:51:15.729+0000 F - [main] exception: connect failed
2020-04-24T12:51:15.729+0000 E - [main] exiting with code 1
root#mongo-client:/# mongo mongodb://root:rootpassword#localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("628bfcf9-6d44-4168-ab74-19a717d746f6") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten]
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
And on the side of mongo the log :
2020-04-24T12:51:19.281+0000 I NETWORK [conn6647] end connection 127.0.0.1:54148 (6 connections now open)
2020-04-24T12:51:19.526+0000 I COMMAND [conn6646] command admin.$cmd appName: "MongoDB Shell" command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-256", payload: "xxx", $db: "admin" } numYields:0 reslen:196 locks:{} protocol:op_msg 231ms
2020-04-24T12:51:19.938+0000 I ACCESS [conn6646] Successfully authenticated as principal root on admin from client 127.0.0.1:54142
2020-04-24T12:51:20.024+0000 I NETWORK [listener] connection accepted from 127.0.0.1:54168 #6648 (7 connections now open)
2020-04-24T12:51:20.027+0000 I NETWORK [conn6648] received client metadata from 127.0.0.1:54168 conn6648: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.5" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 10 (buster)"", architecture: "x86_64", version: "Kernel 4.19.94" } }
2020-04-24T12:51:20.215+0000 I NETWORK [conn6648] end connection 127.0.0.1:54168 (6 connections now open)
2020-04-24T12:51:21.328+0000 I NETWORK [conn6646] end connection 127.0.0.1:54142 (5 connections now open)
I am more and more confused, I can not explain that behavior.

Answer 1

I found out the solution, it turns out to be the simpliest issue possible : resources
My minikube wasn't enough to make all pods running swiftly, it was introducing a latency between the sidecar proxy pods even if kubenetes raised no error on any outage.
I'm a kubernetes learner so I didn't think of it right away. Now that I know what happened I can investigate in the right direction to undestand in what extends the latency can be an issue.

Answer 2

The problem may be that CN of the certificate doesn't match the value of hostname in config file of MongoDB. Its is about MongoDB specification and parameters with which you are running it.
CN (common name) or SAN (subject alternative name) of the certificate has to match the value of --hostname that you supply when running mongo.
Your MongoDB URI is:
MONGODB_URI=mongodb://root:roopassword#127.0.0.1:28017/?authSource=admin
the MongoDB is NOT on localhost. Also the MongoDB server needs to allow ANY host to connect to the database. By default it will ONLY allow connections from the SAME runtime. You need to get IP address of service which is assigned to pod with your database container - svc-identity-data has address 10.107.99.51.
Take a look: mongodb-ssl, mongodb-failed-to-connect