Is it possible to grant blanket access to my CloudSQL instance from ALL (current and future) GCE instances? I've tried adding the /16 internal network block address for my project's instances (copied from the "networks" tab under "Compute Engine": 10.240.0.0/16) but that won't save - it appears that I can only add single-machine (/32) IP addresses.
You need to use the external IP of your machine, although they are both (GCE and Cloud SQL) in Google's datacenters, you cannot communicate between the two using internal IPs.
I do not think there is a native way to allow access from any instance in your project. The only way would be to make your own app to run on one of your instances and use the GCE api to periodically query running instances, get their external ip's, and then use the CloudSQL API to modify the security configuration on the CloudSQL instance.
You could improve this slightly creating a pool of static IP's that you assign to your GCE machines that are going to access your CloudSQL instance, that way the IP's would not change, the side affect is that you would be charged for IP's that you have reserved but do not have allocated to instances.
Apart from that you would have to put a rule to allow any IP access (e.g. 1.0.0.0/0), which would not be a good idea.
Related
We are currently providing our software as a software-as-a-service on Amazon EC2 machines. Our software is a microservice-based application with around 20 different services.
For bigger customers we use dedicated installations on a dedicated set of VMs, the number of VMs (and number of instances of our microservices) depending on the customer's requirements. A common requirement of any larger customer is that our software needs access to the customer's datacenter (e.g., for LDAP access). So far, we solved this using Amazon's virtual private gateway feature.
Now we want to move our SaaS deployments to Kubernetes. Of course we could just create a Kubernetes cluster across an individual customer's VMs (e.g., using kops), but that would offer little benefit.
Instead, perspectively, we would like to run a single large Kubernetes cluster on which we deploy the individual customer installations into dedicated namespaces, that way increasing resource utilization and lowering cost compared to the fixed allocation of machines to customers that we have today.
From the Kubernetes side of things, our software works fine already, we can deploy multiple installations to one cluster just fine. An open topic is however the VPN access. What we would need is a way to allow all pods in a customer's namespace access to the customer's VPN, but not to any other customers' VPNs.
When googleing for the topic, I found approaches that add a VPN client to the individual container (e.g., https://caveofcode.com/2017/06/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/) which is obviously not an option).
Other approaches seem to describe running a VPN server inside K8s (which is also not what we need).
Again others (like the "Strongswan IPSec VPN service", https://www.ibm.com/blogs/bluemix/2017/12/connecting-kubernetes-cluster-premises-resources/ ) use DaemonSets to "configure routing on each of the worker nodes". This also does not seem like a solution that is acceptable to us, since that would allow all pods (irrespective of the namespace they are in) on a worker node access to the respective VPN... and would also not work well if we have dozens of customer installations each requiring its own VPN setup on the cluster.
Is there any approach or solution that provides what we need, .i.e., VPN access for the pods in a specific namespace only?
Or are there any other approaches that could still satisfy our requirement (lower cost due to Kubernetes worker nodes being shared between customers)?
For LDAP access, one option might be to setup a kind of LDAP proxy, so that only this proxy would need to have VPN access to the customer network (by running this proxy on a small dedicated VM for each customer, and then configuring the proxy as LDAP endpoint for the application). However, LDAP access is only one out of many aspects of connectivity that our application needs depending on the use case.
If your IPSec concentrator support VTI, it's possible route the traffic using firewall rules. For example, PFSense suports it: https://www.netgate.com/docs/pfsense/vpn/ipsec/ipsec-routed.html.
Using VTI, you can direct traffic using some kind of policy routing: https://www.netgate.com/docs/pfsense/routing/directing-traffic-with-policy-routing.html
However, i can see two big problems here:
You cannot have two IPSEC tunnels with the conflicted networks. For example, your kube network is 192.168.0.0/24 and you have two customers: A (172.12.0.0/24) and B (172.12.0.0/12). Unfortunelly, this can happen (unless your customer be able to NAT those networks).
Find the ideals criteria for rule match (to allow the routing), since your source network are always the same. Use mark packages (using iptables mangle or even through application) can be a option, but you will still get stucked on the first problem.
A similar scenario is founded on WSO2 (API gateway provider) architecture. They solved it using reverse-proxy in each network (sad but true) https://docs.wso2.com/display/APICloud/Expose+your+On-Premises+Backend+Services+to+the+API+Cloud#ExposeyourOn-PremisesBackendServicestotheAPICloud-ExposeyourservicesusingaVPN
Regards,
UPDATE:
I don't know if you use GKE. If yes, maybe use Alias-IP can be an option: https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips. The PODs IPs will be routable from VPC. So, you can apply some kind of routing policy based on their CIDR.
I'm porting an application to k8s. The application currently consists of pairs of dockers, Trusted and Untrusted, where Trusted is connected to the regular bridge network and talks to internal services, while Untrusted is connected to a separate network allowing access only to external IPs. Untrusted accesses the network according to user-generated data, hence must have internet access and must not be able to access internal IPs.
Trusted and Untrusted communicate using a pair of FIFOs since they run on the same machine (Using a Unix Domain Socket was ~20% slower, I didn't yet test local TCP/IP but I suspect a bigger performance hit - the service is not entirely horizontally scalable due to ordering constraints so single-machine performance matters).
I've hit a wall porting this setup to k8s: the original idea was to use a Pod for each pair of dockers and using emptyDir to share the FIFOs, but there seems to be no way to create separate network limitations for a single container in a Pod since they all share a networking container. Is there a way to do this?
What's the alternative if this isn't possible? Setting up the untrusted containers in a separate namespace and applying a limited network access policy allowing only tightly-controlled access to the rest of the cluster?
There is an option for google cloud dataproc to stop(Not delete) the cluster (Master + Worker nodes) and start as well but when we do so, external IP address of master and worker nodes are changing which causes problem for using Hue and other IP based Web UI on it.
Is there any option to persist the same IP after restart?
Though Dataproc doesn't currently provide a direct option for using static IP addresses, you can use the underlying Compute Engine interfaces to add a static IP address to your master node, possibly removing the previous "ephemeral IP address".
That said, if you're accessing your UIs through external IP addresses, that presumably means you also had to manage your firewall rules to carefully limit the inbound IP ranges. Depending on what UIs you're using, if they're not using HTTPS/SSL then that's still not ideal even if you have firewall rules limiting access from other external sources.
The recommended way to access your Dataproc UIs is through SSH tunnels; you can even add the gcloud compute ssh and browser-launching commands to a shell script for convenience if you don't want to re-type all the SSH flags each time. This approach would also ensure that links work in pages like the YARN ResourceManager, since those will be using GCE internal hostnames which your external IP address would not work for.
The Setup
ECS (Containerized) Application (Node.js, API Only)
Auto Scaling Group for ECS Container Instances
Load Balancer in front of auto scaling group
VPC covering all instances and ELB
Database hosted in another VPC, not managed explicitly (MongoDB Atlas), likely not the same region.
The Problem
I want my database to use good security policies, therefore I opt for whitelisting IPs as Atlas recommends - rather than opening up my database to the world with 0.0.0.0/0.
Each server has its own IP address, and in an autoscaling event it would need to be added by automation to the Atlas security rules (which is possible, not ideal).
How can I (using NAT Gateways? Elastic IPs?) get one IP for all of my load balanced instances.
Failed Solutions?
I tried using a NAT Gateway, essentially scenario 2 where all my of instances were in a private subnet, the NAT was in a public subnet with internet access, and the instances went through it to get to the database. This worked! Elastic IP on the NAT and I was able to authorize it on Atlas however it had weird issues where the instance wouldn't respond for 65 - 75 seconds, intermittently when pinged. I suspect this is due to the fact that it's not technically available on the internet and there's some routing that I don't fully understand happening. Once you got a 200 though everything would work fine, for a bit, then another 70 second latency and back to good again...
Really appreciate the input, have been searching for a while with no luck!
Have you tried a VPC peering connection? As long as the VPC CIDR blocks do not overlap, this is a good option because you can use security groups and private IPs between the peered VPCs.
http://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/Welcome.html
I'm attempting to transition existing applications to Kubernetes that work as follows:
An outside service calls our application through a load balancer with a new session.
Our application returns the ip of the server that processed the request.
All subsequent calls from the outside service for that session are made directly to the same server (bypassing the load balancer)
Is there any way to do this in kubernetes? I understand that pod ip's are not exposed externally, is there some way to expose them directly?
Also, I don't think I can use sessionAffinity="ClientIP" because the requests will all come in from the same place. Is there a way to write custom sessionAffinity type?
It kind of depends on how your network is set up and what you mean by an "outside service", but the answer is most likely "no".
If you're running using one of the default cluster creation scripts in a cloud environment, pod IP addresses are not routable from the Internet, so any service not in the same private network as your cluster won't be able to talk directly to pods.
However, depending on what cloud provider you're on, you'll likely get the behavior that you want anyways by just continuing to make all calls through to the external IP of a service of type LoadBalancer. For instance, on the Google Cloud Platform, the cloud load balancer that gets created for such services by default maintains connection affinity by 5-tuple (src ip and port, dst ip and port, L4 protocol), which sounds like it's what you want, since you want balancing per session rather than per IP.
As for creating a new sessionAffinity type, that's not an easy thing to extend, since it requires changing Kubernetes source code. If that's really a path you want to take, it's likely that you'd want to run your own load balancer within your cluster rather than relying on the built-in load balancing.