Running replicated MongoDB 4.2 in Kubernetes: replset config not being received - mongodb

Question

I'm running a Kubernetes cluster on AWS and need to configure a replicated MongoDB 4.2 Database.
I'm using StatefulSets in order for other Pods (e.g., REST API NodeJS Pod) to easily connect to the mongo instances (example dsn: "mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/app").
mongo-configmap.yaml (provides a shell script to perform the replication initialization upon mongo container creation):
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-init
data:
init.sh: |
#!/bin/bash
# wait for the readiness health check to pass
until ping -c 1 ${HOSTNAME}.mongo; do
echo "waiting for DNS (${HOSTNAME}.mongo)..."
sleep 2
done
until /usr/bin/mongo --eval 'printjson(db.serverStatus())'; do
echo "connecting to local mongo..."
sleep 2
done
echo "connected to local."
HOST=mongo-0.mongo:27017
until /usr/bin/mongo --host=${HOST} --eval 'printjson(db.serverStatus())'; do
echo "connecting to remote mongo..."
sleep 2
done
echo "connected to remote."
if [[ "${HOSTNAME}" != 'mongo-0' ]]; then
until /usr/bin/mongo --host=${HOST} --eval="printjson(rs.status())" \
| grep -v "no replset config has been received"; do
echo "waiting for replication set initialization"
sleep 2
done
echo "adding self to mongo-0"
/usr/bin/mongo --host=${HOST} --eval="printjson(rs.add('${HOSTNAME}.mongo'))"
fi
if [[ "${HOSTNAME}" == 'mongo-0' ]]; then
echo "initializing replica set"
/usr/bin/mongo --eval="printjson(rs.initiate(\
{'_id': 'rs0', 'members': [{'_id': 0, \
'host': 'mongo-0.mongo:27017'}]}))"
fi
echo "initialized"
while true; do
sleep 3600
done
mongo-service.yaml:
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
app: mongo
spec:
clusterIP: None
ports:
- port: 27017
selector:
app: mongo
mongo-statefulset.yaml (2 containers inside one Pod, 1 for the actual DB, the other for initialization of the replication):
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
labels:
app: mongo
spec:
selector:
matchLabels:
app: mongo
serviceName: "mongo"
replicas: 3
template:
metadata:
labels:
app: mongo
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongodb
image: mongo:4.2
command:
- mongod
args:
- --replSet
- rs0
- "--bind_ip_all"
ports:
- containerPort: 27017
name: web
volumeMounts:
- name: database
mountPath: /data/db
livenessProbe:
exec:
command:
- /usr/bin/mongo
- --eval
- db.serverStatus()
initialDelaySeconds: 10
timeoutSeconds: 10
- name: init-mongo
image: mongo:4.2
command:
- bash
- /config/init.sh
volumeMounts:
- name: config
mountPath: /config
volumes:
- name: config
configMap:
name: "mongo-init"
volumeClaimTemplates:
- metadata:
name: database
annotations:
volume.beta.kubernetes.io/storage-class: mongodb-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 5Gi
After applying these configurations, the 3 mongo pods start running (mongo-0, mongo-1, mongo-2).
However, other pods can't connect to these mongo pods.
Further look into the mongo-0 pods (should be primary instance) reveals that the replication did not work.
kubectl exec -it mongo-0 -- /bin/bash
Then running 'mongo' to start the mongo shell, and entering 'rs.status()' into the mongo shell results in the following output:
{
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "no replset config has been received",
"code" : 94,
"codeName" : "NotYetInitialized"
}

Answer 1

Apparently, mongo-4.x images do not come with 'ping' installed, therefore, the rest of the script was not executed.
Adding these two lines to the script in mongo-configmap.yaml fixes the problem:
apt-get update
apt-get install iputils-ping --yes

Answer 2

After start running all pods, then hit this command
kubectl exec -it mongo-0 -- /bin/bash
(here mongo-0 is pod name)
now start mongo shell,
mongo
now check pod is initiate or not
rs.status()
If not then, initiate and make it primary by hitting these commands one by one
rs.initiate()
var cfg = rs.conf()
cfg.members[0].host=”mongo-0.mongo:27017”
(here is mongo-0 is pod name and mongo is serive name.)
now reconfig primary node
rs.reconfig(cfg)
Add all slaves to primary node
rs.add(“mongo-1.mongo:27017”)
rs.add(“mongo-2.mongo:27017”)
(here is mongo-1 and mongo-2 is pod name and mongo is serive name.)
now check status
rs.status()
now exit from primary shell and go to secondary (slave) node
exit
exit
kubectl exec -it mongo-1 -- /bin/bash
mongo
rs.secondaryOk()
check status and exit
rs.status()
exit
exit
now do same for other secondary (slave) nodes