MongoDB connection drops - mongodb

Question

The connection from applications in the Kubernetes cluster to the Mongo DB (Atlas) basically works. However, sometimes the connection drops and I can't figure out why. There are two different errors in my logs that seem to be related:
Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xxxxx-tedsb.gcp.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection timed out}}, {address=xxxxx-shard-00-01-tedsb.gcp.mongodb.net:27017, type=REPLICA_SET_SECONDARY, TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='GCP'}, Tag{name='region', value='EUROPE_NORTH_1'}]}, roundTripTime=34.3 ms, state=CONNECTED}, {address=xxxxx.gcp.mongodb.net:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection timed out}}]
and
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.connection.TlsChannelStreamFactoryFactory$TlsChannelStream.lambda$openAsync$0(TlsChannelStreamFactoryFactory.java:246)
at com.mongodb.connection.TlsChannelStreamFactoryFactory$SelectorMonitor.lambda$start$0(TlsChannelStreamFactoryFactory.java:141)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at com.mongodb.connection.TlsChannelStreamFactoryFactory$TlsChannelStream.lambda$openAsync$0(TlsChannelStreamFactoryFactory.java:218)
... 2 more
I'm not hitting the limit of number of connections and the pod itself is also not under pressure (memory, cpu, disk space). Same goes for the JVM. I'm using the official drivers ("org.mongodb.scala" %% "mongo-scala-driver" % "4.1.0") and Mongo 4.2.8.
The configuration I'm using is pretty basic:
private val uri = configuration.get[String]("mongodb.uri")
private val clientSettings = MongoClientSettings
.builder()
.uuidRepresentation(UuidRepresentation.STANDARD)
.applyConnectionString(new ConnectionString(uri))
.codecRegistry(Registry)
.build()
val client = MongoClient(clientSettings)
I found this (issue)[https://jira.mongodb.org/browse/JAVA-3274] that got resolved in mongo-scala-driver: 4.1.0 and I was hoping that this would fix my issue, but it didn't.
Any idea what I could check next?

Answer 1

Apparently there is a connectivity issue between your pod and two of the three nodes of the replica set. Since the node that is visible is currently a secondary, the driver cannot satisfy the readPreference=primary that is set by default and therefore times out after a while.
I would review your firewall configuration, make sure your pod has access to all three nodes and not just one. Also, ensure that your pod's IP address is whitelisted in Atlas.