tldr: What will I need to do in order to use an elastic IP in my MongoDB replicaset configuration?
We have a three-node MongoDB replicaset running on EC2. One of the instances in the set was retired by AWS yesterday, and so we were forced to stop and restart the EC2 instance.
Unfortunately, when we first configured the replicaset, we were fairly new to AWS and not aware that the public DNS address of the instances was subject to change. We used the public DNS of each instance in the replicaset configuration, and in all of the application connection strings in our code. After reading up on the subject yesterday, I tried to get the node back online by assigning an elastic IP to the instance and changing the replicaset configuration to use that IP. After some pain, I was able to get the other two nodes back up and running with that configuration, but the instance with the elastic IP refused to re-join the replicaset, and the error in mongod.log says:
[rsStart] replSet info self not present in the repl set configuration
After yet more reading, I found that I should not have used the actual elastic IP in the config, but rather the public DNS name of the elastic IP. My question is, before I take everything offline again to try this change, what exactly will I need to do in order to use the elastic IP in the replicaset configuration? I found some information on this 10Gen page: http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/#communication-across-regions that made me think I might need to mess with the hostname of the instance and/or the hosts file, but I haven't been able to find anybody describing my exact scenario.
Any thoughts?
It turned out to be a pretty simple fix; once I changed the replicaset configuration to use the public DNS of the elastic IP, the mongo node came back online. I didn't have to touch the hostname or the hosts file.
You should never use an Elastic IP for internal traffic like replication. You will be charged $0.01/GB for this traffic, whereas using the internal IP would be free.
If you're using something like replica sets, you really should be running in a VPC. Unlike normal EC2 instances, instances in an VPC keep the same private IP addresses and Elastic IP addresses even when stopped.
Related
Just trying to make an app on k8s to connect to MongoDB atlass,
So far tried the following:
Changed the DNSpolicy to Default and many others - no luck
Created nginx-ingress link- so have the main IP address of the cluster
Added that IP to IP access list - but still no luck
The cluster tier is M2 - so no private peering or private endpoints.
The Deployment/Pod that is trying to connect will not have an a DNS assigned to it, it is simply a service running inside of the k8s and processing rabbitmq messages.
So not sure on what I should whitelist if the service is never exposed.
I assume it would have to be something with Nodes or K8s egress or something, but not sure where to even look
Tried pretty much everything I could and still cannot find clear documentation on how to achieve the desired result apart from whitelisting All IP addresses
UPDATE: Managed to find this article https://www.digitalocean.com/community/questions/urgent-how-to-connect-to-mongodb-atlas-cluster-from-a-kubernetes-pod
So now im trying to find a way to persist Node IP addresses, as I understand during the scale up or down or upgrade of nodes it will create new IP addresses.
So is there a way to persist them?
I’m little bit unsure what would the correct connection URI for my applications for the Mongodb statefulset. I have three replicas running in my cluster, each one in separate node.
Should I configure the pods OR the headless service (load balancer for the pods)?
The documentation is directing to use pods, like this (Running MongoDB on Kubernetes with StatefulSets | Kubernetes):
mongodb://user:pwd#mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/dbname_?
But I have got it working with the service as well:
mongodb://user:pwd#mongodb-headless.svc.cluster.local:27017/dbname_?authSource=admin&replicaSet=rs0
But, I don’t know what is the correct URI? The problem I’m having is that when some of the replicas goes down, for some reason, the application crashes as the database connection is lost. I think this is where the headless-service comes in picture, but no, the documentation says to configure the pods. And if I scale the replicas I need to reconfigure the URI. This does not sound so dynamic.
I’m also facing some issues with the headless service, as if it is in different namespace I cannot get the connection work with namespace defined, like:
mongodb-headless.namespace.svc.cluster.local:27017
Have I missed something?
Thank you in any advance!
EDIT: added replicaset for service/lb URI example (I had this configured...)
I think your way of referencing the headless service will result in mongodb only using the first in the set.
Another way is using the MongoDB's DNS seed list connection format together with Kubernetes' support for DNS SRV records. If you named your Service's port mongodb, then the following connection string ought to work:
mongodb+srv://user:pwd#mongodb-headless.namespace.svc.cluster.local:27017/dbname_?
MongoDB clients will use DNS to get a seed list on connection, which stays up to date with the actual Pods running.
Note that this enables tls by default, which you probably do not want.
What's the best/easier way to connect an AWS RDS instance to my kubernetes cluster running on gcloud?
Likely the only way there is, is using their public availability option upon creating the instance. That really is just fine. You will get a DNS hostname that also powers their loadbalancing feature.
Note, their are similar question for connection within AWS, which report DNS problems, that do not apply in your case Ref: Accessing Postgres RDS from Kubernetes cluster in AWS
One thing you can do is: Add a label for some of your nodes with kubectl label nodes in order to have control over their public IP addresses, they are ephemeral so I think you will want to reserve a static IP for those nodes you choose.
Each virtual machine is named with its corresponding node's name, so you can use the same name you used in the previous kubectl label nodes command.
Now all you have to do is, add each IP to the security group that corresponds to your RDS instance.
I have followed tutorials and set up working mongodb replica sets however when it comes to exposing them as a service I am stuck with using a LoadBalancer which directs to any pod. In most cases this ends up being a secondary database and not terrible helpful. I have also managed to get separate mongodb replicas set up and then tried to connect to those externally however connections fail because internal replicaset ips are all through local google cloud dns.
What I am hoping for is something like this.
Then (potentially) there is a single connection uri that could connect you to your mongodb replicaset without needing individual mongodb connection details.
I'm not sure if this is possible but any help is greatly appreciated!
The loadbalancer type service will route traffic to any one pod matches its selector, which is not how mongodb replica set works. The connection string should contain all instances in the set. You probably need to expose each replica instance with type=loadbalancer. Then you may connect via "mongodb://mongo-0_IP,mongo-1_IP,mongo-2_IP:27017/dbname_?"
If you configure a mongodb replica set with stateful sets, you should
also create a headless service. Then you can connect to the replica set with a url like:
“mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/dbname_?”
Here mongo-0,mongo-1,mongo-2 are pod names and "mongo" is the headless service name.
If you still want to able to connect to a specific mongo instance, you can create separate service (of type=NodePort) for each of the deployment/replica and then you should be able to connect to a specific mongo instance using <any-node-ip>:<nodeport>
But you'll not be able to leverage the advantages of having a mongo replica set in this case.
I have a mongoDB replica set in azure
I have:
server1 Primary
server2 secondary
server3 Arbiter
I have a dev environment on my local machine that I want to point to this mongoDB instance
What do I open on my Azure Firewall to make sure this configuration is setup with best practices.
Do I create a load balanced endpoint to the Primary and Secondary or do I create a single endpoint to the arbiter, or perhaps even something else?
thanks!
MongoDB will not play well with a load-balanced endpoint (as you might end up sending traffic to a secondary, and you'd have no control over this unless you implemented a custom probe for each VM, and then you'd need to update the probe's status based on the replicaset node's health, for each node). The MongoDB client-side driver is designed to work with a replicaset's topology to make the correct decision on which node to communicate with. Each replicaset node should have a discrete addressable ip:port. If you have all your instances in a single cloud service (e.g. myservice.cloudapp.net) then you'll need one port per instance (since they'd all share a single ip address). If each instance is in a different cloud service, then you can have the same port for each, with different dns name / ip address for each.
The best solution with an iptables is to open the third with an ip rule. It's open in the twice configuration and secure. This solution is the best architecture for your code.