Kubernetes how to access application in one namespace from another - mongodb

Question

I have the following components up and running in a kubernetes cluster
A GoLang Application writing data to a mongodb statefulset replicaset in namespace app1
A mongodb replicaset (1 replica) running as a statefulset in the namespace ng-mongo
What I need to do is, I need to access the mongodb database by the golang application for write/read opeations, so what I did was;
Create a headless service for the mongodb in the ng-mongo namespace as below:
# Source: mongo/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: mongo
namespace: ng-mongo
labels:
app: mongo
spec:
ports:
- port: 27017
targetPort: 27017
name: mongo
clusterIP: None
selector:
role: mongo
And then I deployed the mongodb statefulset and initialized the replicaset as below:
kubectl exec -it mongo-0 -n ng-mongo mongosh
rs.initiate({_id: "rs0",members: [{_id: 0, host: "mongo-0"}]})
// gives output
{ ok: 1 }
Then I created an ExternalName service in the app1 namespace linking the above mongo service in step 1, look below:
# Source: app/templates/svc.yaml
kind: Service
apiVersion: v1
metadata:
name: app1
namespace: app1
spec:
type: ExternalName
externalName: mongo.ng-mongo.svc.cluster.local
ports:
- port: 27017
And at last, I instrumented my golang application as follows;
// Connection URI
const mongo_uri = "mongodb://app1" <-- Here I used the app1, as the ExternalName service's name is `app1`
<RETRACTED-CODE>
And then I ran the application, and checked the logs. Here is what I found:
2022/11/22 12:49:47 server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongo-0:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp: lookup mongo-0 on 10.96.0.10:53: no such host }, ] }
Update: I haven't set any usernames or passwords for the mongodb
Can someone help me why this is happening?

Answer 1

After some digging, I was able to find the issue.
When specifying the host entry for the rs.initiate({}), I should provide the FQDN of the relevant mongodb instance (in my case it is the mongo-0 pod). Therefore, my initialisation command should look like this;
rs.initiate({_id: "rs0",members: [{_id: 0, host: "mongo-0.mongo.ng-mongo.svc.cluster.local:27017"}]})

Answer 2

From my understanding of what you are trying to do,
Your Pod(golang application) and app1 Service are already in the same namespace.
However, looking at the log,
2022/11/22 12:49:47 server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mongo-0:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp: lookup mongo-0 on 10.96.0.10:53: no such host }, ] }
The log means that the domain named 'mongo-0' could not be found in DNS. (Note that 10.96.0.10 IP is probably kube-dns)
Your application tries to connect to the domain mongo-0, but the domain mongo-0 does not exist in DNS (It means there is no service named mongo-0 on app1 namespace).
What is the 'mongo-0' that your Application trying to access?
(Obviously the log shows an attempt to access the domain mongo-0 and your golang applications mongo_uri indicates mongodb://app1)
Finding out why your application are trying to connect to the mongo-0 domain will help solve the problem.
Hope this helps you.