Just trying to make an app on k8s to connect to MongoDB atlass,
So far tried the following:
Changed the DNSpolicy to Default and many others - no luck
Created nginx-ingress link- so have the main IP address of the cluster
Added that IP to IP access list - but still no luck
The cluster tier is M2 - so no private peering or private endpoints.
The Deployment/Pod that is trying to connect will not have an a DNS assigned to it, it is simply a service running inside of the k8s and processing rabbitmq messages.
So not sure on what I should whitelist if the service is never exposed.
I assume it would have to be something with Nodes or K8s egress or something, but not sure where to even look
Tried pretty much everything I could and still cannot find clear documentation on how to achieve the desired result apart from whitelisting All IP addresses
UPDATE: Managed to find this article https://www.digitalocean.com/community/questions/urgent-how-to-connect-to-mongodb-atlas-cluster-from-a-kubernetes-pod
So now im trying to find a way to persist Node IP addresses, as I understand during the scale up or down or upgrade of nodes it will create new IP addresses.
So is there a way to persist them?
Related
I'm hosting a Kubernetes cluster on VMs/VPS from a random cloud provider not providing any Kubernetes things at all, meaning with a dedicated public IP address and to allow the trafic coming to the worker nodes, I'm defining my Service with the spec.externalIPs with the fixed list of IP addresses.
I'm looking for a way to get that list updated when a node is drained/down automatically.
I had a look at the existing operators from https://operatorhub.io/ but I haven't found any that seem to cover my use case.
The idea would that when the event of a node passing to NotReady is emitted, the Service is updated with the Nodes being Ready.
Is there any operator that could allow doing that?
After some time working on this, I finally figured out that this is not possible, at least today, there's no known operator or what so ever that could update the field with the IP addresses.
And even if it was the case, there would be delays to update the DNS records.
What I've done instead is to buy another VPS, installing HAproxy in order to proxy the Kubernetes API trafic to the master nodes, and the web trafic (both 80 and 443) to the Kubernetes worker nodes.
HAproxy monitors the nodes, and add/remove nodes automagically and in a very quick way.
With this, you just need one DNS record, pointing to the Load Balancer (or VIP of the Load Balancers in order to avoid SPOF), and HAproxy will do the rest!
Current set-up
Cluster specs: Managed Kubernetes on Digital Ocean
Goal
My pods are accessing some websites but I want to use a proxy first.
Problem
The proxy I need to use is only taking 1 IP address in an "allow-list".
My cluster is using different nodes, with node-autoscaler so I have multiple and changing IP addresses.
Solutions I am thinking about
Setting-up a proxy (squid? nginx?) outside of the cluster (Currently not working when I access an HTTPS website)
Istio could let me set-up a gateway? (No knowledge of Istio)
Use GCP managed K8s, and follow the answers on Kubernetes cluster outgoing traffic IP. But all our stack is on Digital Ocean and the pricing is better there.
I am curious to know what is the best practice, easiest solution or if anyone experienced such use-case before :)
Best
You could set up all your traffic to go through istio-egressgateway.
Then you could manipulate the istio-egressgateway to always be deployed on the same node of the cluster, and whitelist that IP address.
Pros: super easy. BUT. If you are not using Istio already, to set up Istio just for this is may be killing a mosquito with a bazooka.
Cons: Need to make sure the node doesn't change the IP address. Otherwise the istio-egressgateway itself might not get deployed (if you do not have the labels added to the new node), and you will need to reconfigure everything for the new node (new IP address). Another con might be the fact that if the traffic goes up, there is an HPA, which will deploy more replicas of the gateway, and all of them will be deployed on the same node. So, if you are going to have lots of traffic, may be it would be a good idea to isolate one node, just for this purpose.
Another option would be as you are suggesting; a proxy. I would recommend an Envoy proxy directly. I mean, Istio is going to be using Envoy anyways right? So, just get the proxy directly, put it in a pod, do the same thing as I mentioned before; node affinity, so it will always run on the same node, so it will go out with the same IP.
Pros: You are not installing entire service mesh control plane for one tiny thing.
Cons: Same as before, as you still have the issue of the node IP change if something goes wrong, plus you will need to manage your own Deployment object, HPA, configure the Envoy proxy, etc. instead of using Istio objects (like Gateway and a VirtualService).
Finally, I see a third option; to set up a NAT gateway outside the cluster, and configure your traffic to go through it.
Pros: You won't have to configure any kubernetes object, therefor there will be no need to set up any node affinity, therefor there will be no node overwhelming or IP change. Plus you can remove the external IP addresses from your cluster, so it will be more secure (unless you have other workloads that need to reach internet directly). Also , probably having a single node configured as NAT will be more resilient then a kubernetes pod, running in a node.
Cons: May be a little bit more complicate to set up?
And there is this general Con, that you can whitelist only 1 IP address, so you will always have a single point of failure. Even NAT gateway; it still can fail.
The GCP static IP won't help you. What is suggesting the other post is to reserve an IP address, so you can re-use it always. But it's not that you will have that IP address automatically added to a random node that goes down. Human intervention is needed. I don't think you can have one specific node to have a static IP address, and if it goes down, the new created node will pick the same IP. That service, to my knowledge, doesn't exist.
Now, GCP does offer a very resilient NAT gateway. It is managed by Google, so shouldn't fail. Not cheap though.
Is there option to connect to external database cluster from POD? I need to connect to elastic search, zookeeeper, Kafka and couchbase, each of them has its own cluster. Per my understanding the documentation, I can define multi external IPs, but I cannot find how will k8s behave if one of them is down. I am working with pure k8s 1.6 now, and we will migrate to 1.7 soon. Information about OpenShift 3.7 will be also welcome because I cannot find anything specific in its documentation.
The k8s doc on your link has more info on exposing services running on k8s but not externally
You generally want to expose your service using a DNS entry and manage the HA for that service separately.
For example you can a single DNS entry mykafka.mydomain.com and then assign IP addresses to that entry:
kafka1 ip
kafka2 ip
kafka3 ip
You can see that approach on the Openshift docs in the USING AN EXTERNAL DOMAIN NAME section. Yes, its not clear from the docs whether k8s/openshift does a round robin on the multiple IPs for an external service and if automatically fails over.
Hope it helps.
I have a Kubernetes cluster (1.3.2) in the the GKE and I'd like to connect VMs and services from my google project which shares the same network as the cluster.
Is there a way for a VM that's internal to the subnet but not internal to the cluster itself to connect to the service without hitting the external IP?
I know there's a ton of things you can do to unambiguously determine the IP and port of services, such as the ENVs and DNS...but the clusterIP is not reachable outside of the cluster (obviously).
Is there something I'm missing? An important component to this is that this is meant to be a service "public" to the project, such that I don't know which VMs on the project will want to connect to the service (this could rule out loadBalancerSourceRanges). I understand the endpoint which the services actually wraps is the internal IP I can hit, but the only good way to get to that IP is though the Kube API or kubectl, both of which are not prod-ideal ways of hitting my service.
Check out my more thorough answer here, but the most common solution to this is to create bastion routes in your GCP project.
In the simplest form, you can create a single GCE Route to direct all traffic w/ dest_ip in your cluster's service IP range to land on one of your GKE nodes. If that SPOF scares you, you can create several routes pointing to different nodes, and traffic will round-robin between them.
If that management overhead isn't something you want to do going forward, you could write a simple controller in your GKE cluster to watch the Nodes API endpoint, and make sure that you have a live bastion route to at least N nodes at any given time.
GCP internal load balancing was just released as alpha, so in the future, kube-proxy on GCP could be implemented using that, which would eliminate the need for bastion routes to handle internal services.
tldr: What will I need to do in order to use an elastic IP in my MongoDB replicaset configuration?
We have a three-node MongoDB replicaset running on EC2. One of the instances in the set was retired by AWS yesterday, and so we were forced to stop and restart the EC2 instance.
Unfortunately, when we first configured the replicaset, we were fairly new to AWS and not aware that the public DNS address of the instances was subject to change. We used the public DNS of each instance in the replicaset configuration, and in all of the application connection strings in our code. After reading up on the subject yesterday, I tried to get the node back online by assigning an elastic IP to the instance and changing the replicaset configuration to use that IP. After some pain, I was able to get the other two nodes back up and running with that configuration, but the instance with the elastic IP refused to re-join the replicaset, and the error in mongod.log says:
[rsStart] replSet info self not present in the repl set configuration
After yet more reading, I found that I should not have used the actual elastic IP in the config, but rather the public DNS name of the elastic IP. My question is, before I take everything offline again to try this change, what exactly will I need to do in order to use the elastic IP in the replicaset configuration? I found some information on this 10Gen page: http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/#communication-across-regions that made me think I might need to mess with the hostname of the instance and/or the hosts file, but I haven't been able to find anybody describing my exact scenario.
Any thoughts?
It turned out to be a pretty simple fix; once I changed the replicaset configuration to use the public DNS of the elastic IP, the mongo node came back online. I didn't have to touch the hostname or the hosts file.
You should never use an Elastic IP for internal traffic like replication. You will be charged $0.01/GB for this traffic, whereas using the internal IP would be free.
If you're using something like replica sets, you really should be running in a VPC. Unlike normal EC2 instances, instances in an VPC keep the same private IP addresses and Elastic IP addresses even when stopped.