MongoDB Kubernetes connection failure with SSL - mongodb

Question

I have three node MongoDB cluster in GCP kubernetes cluster following [1], [2]. I can properly connect with tls=false using mongosh client. Then I enabled tls following [3]. Mongo cluster start properly but I cannot connect from mongosh.
Following is the connection details.
{
"connectionString.standard": "mongodb://mongo-user:stl-m0ng0-dev#mongodb-dev-0.mongodb-dev-svc.dev.svc.cluster.local:27017,mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local:27017,mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local:27017/dev?replicaSet=mongodb-dev&ssl=true",
"connectionString.standardSrv": "mongodb+srv://mongo-user:stl-m0ng0-dev#mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=true",
"password": "xxxxxxx",
"username": "mongo-user"
}
Followings are the certificate details.
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = TLSGenSelfSignedtRootCA, L = $$$$
Validity
Not Before: Jul 27 09:07:50 2022 GMT
Not After : Jul 24 09:07:50 2032 GMT
Subject: CN = *.mongodb-dev-svc.dev.svc.cluster.local, O = client
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c4:44:a6:21:95:85:9a:dc:96:63:8e:76:ed:d9:
3a:59
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-3.mongodb-dev-svc.dev.svc.cluster.local
Signature Algorithm: sha256WithRSAEncryption
7b:78:43:73:ae:2f:ce:97:de:b2:19:56:4c:38:71:8e:3d:ff:
5b:15:79:c1
Will display server certificate info
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = TLSGenSelfSignedtRootCA, L = $$$$
Validity
Not Before: Jul 27 09:07:50 2022 GMT
Not After : Jul 24 09:07:50 2032 GMT
Subject: CN = *.mongodb-dev-svc.dev.svc.cluster.local, O = server
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:bc:1e:4a:a7:4f:c4:01:71:2c:78:eb:ac:c9:53:
24:c1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:mongodb-dev-0.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local, DNS:mongodb-dev-2.mongodb-dev-svc.dev.svc.cluster.local
Signature Algorithm: sha256WithRSAEncryption
16:0f:09:02:66:05:69:7b:91:3b:93:73:86:64:d5:8f:53:2d:
08:19:68:a7
Client side has following error
root#xxxxxxxxxxxxxxxxxx-55955c9fcd-bpp98:/usr/src/app# mongosh "mongodb+srv://mongo-user:stl-m0ng0-dev#mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=false&tlsCAFile=ca.pem&tlsCertificateKeyFile=key.pem"
Current Mongosh Log ID: 62e1029487b960f1bd204b1d
Connecting to: mongodb+srv://<credentials>#mongodb-dev-svc.dev.svc.cluster.local/dev?replicaSet=mongodb-dev&ssl=false&tlsCAFile=ca.pem&tlsCertificateKeyFile=key.pem&appName=mongosh+1.5.1
MongoServerSelectionError: connection <monitor> to 10.120.6.8:27017 closed
Server side has following error
2022-07-27T09:25:44.992+0000 I NETWORK [conn25852] end connection 10.120.6.9:33914 (14 connections now open)
2022-07-27T09:25:44.993+0000 I NETWORK [listener] connection accepted from 10.120.6.9:33918 #25855 (15 connections now open)
2022-07-27T09:25:44.993+0000 E NETWORK [conn25854] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:44.994+0000 I NETWORK [conn25854] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58220 (connection id: 25854)
2022-07-27T09:25:44.994+0000 I NETWORK [conn25854] end connection 10.120.8.127:58220 (14 connections now open)
2022-07-27T09:25:44.995+0000 I NETWORK [listener] connection accepted from 10.120.8.127:58224 #25856 (15 connections now open)
2022-07-27T09:25:44.998+0000 E NETWORK [conn25855] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:44.998+0000 I NETWORK [conn25855] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.6.9:33918 (connection id: 25855)
2022-07-27T09:25:44.998+0000 I NETWORK [conn25855] end connection 10.120.6.9:33918 (14 connections now open)
2022-07-27T09:25:45.000+0000 E NETWORK [conn25856] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:45.000+0000 I NETWORK [conn25856] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58224 (connection id: 25856)
2022-07-27T09:25:45.000+0000 I NETWORK [conn25856] end connection 10.120.8.127:58224 (13 connections now open)
2022-07-27T09:25:45.001+0000 I REPL_HB [replexec-2] Heartbeat to mongodb-dev-1.mongodb-dev-svc.dev.svc.cluster.local:27017 failed after 2 retries, response status: HostUnreachable: stream truncated
2022-07-27T09:25:45.003+0000 I NETWORK [listener] connection accepted from 10.120.8.127:58228 #25858 (14 connections now open)
2022-07-27T09:25:45.007+0000 E NETWORK [conn25858] SSL peer certificate validation failed: unsupported certificate purpose
2022-07-27T09:25:45.007+0000 I NETWORK [conn25858] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unsupported certificate purpose. Ending connection from 10.120.8.127:58228 (connection id: 25858)
2022-07-27T09:25:45.007+0000 I NETWORK [conn25858] end connection 10.120.8.127:58228 (13 connections now open)
Operator log has TLS configuration issue.
2022-07-27T10:06:05.893Z INFO controllers/mongodb_status_options.go:110 TLS config is not yet valid, retrying in 10 seconds
2022-07-27T10:06:15.899Z INFO controllers/replica_set_controller.go:140 Reconciling MongoDB {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG controllers/replica_set_controller.go:142 Validating MongoDB.Spec {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG controllers/replica_set_controller.go:151 Ensuring the service exists {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG agent/agent_readiness.go:101 The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG agent/agent_readiness.go:101 The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG agent/agent_readiness.go:101 The Pod '' doesn't have annotation 'agent.mongodb.com/version' yet {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.900Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.906Z INFO controllers/replica_set_controller.go:462 Create/Update operation succeeded {"ReplicaSet": "dev/mongodb-replica-set","operation": "updated"}
2022-07-27T10:06:15.906Z INFO controllers/mongodb_tls.go:40 Ensuring TLS is correctly configured {"ReplicaSet": "dev/mongodb-replica-set"}
2022-07-27T10:06:15.906Z WARN controllers/mongodb_tls.go:47 CA resource not found: Secret "tls-ca-key-pair" not found {"ReplicaSet": "dev/mongodb-replica-set"}
[1]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/install-upgrade.md
[2]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/deploy-configure.md
[3]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/secure.md

Answer 1

It had two main reasons.
I followed [1] to enable SSL. It create another Statefulset. After that there are two mongo servers. Uninstall operator and re-install and followed last stable release documentation [2]. After it properly detect configmap and secret.
But it gave SSL issue in certificates as Unsupported Certificate in server modules. Following [3] found the issue. We need to remove extended_key_useage from openssl.conf. Otherwise it not work properly.
Important thread [4]
Hope this help.
[1]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/master/docs/secure.md
[2]. https://github.com/mongodb/mongodb-kubernetes-operator/blob/v0.7.4/docs/secure.md
[3]. https://stackoverflow.com/a/61964464/5607943
[4]. https://groups.google.com/g/mongodb-user/c/EmESxx5KK9Q/m/xH6Ul7fTBQAJ