I have an ECS-Fargate cluster created inside VPC.
If I want to access above mentioned AWS services from fargate task, what needs to be done?
I see following options from different documentations I read:
Create private link to each AWS service
Create NAT gateway
Not sure which one is correct and recommended option?
To be clear, an ECS cluster is an abstracted entity and does not dictate where you connect the workloads you are running within it. If we stick to the Fargate launch type this means that tasks could be launched either on a private subnet or on a public subnet:
If you launch them in a public subnet (and you assign a public IP to the tasks) then these tasks can reach the public endpoints of the services you mentioned and nothing else (from a networking routing perspective) is required.
If you launch them in a private subnet you have two options that are those you called out in your question.
I don't think there is a golden rule for what's best. The decision is multi-dimensional (cost, ease of setup, features, observability and control, etc). I'd argue the NAT GW route is easier to setup regardless of the number of services you need to add but you may lose a bit of visibility and all your traffic will go outside of the VPC (for some customers this is ok, for others it's not). Private Links will give you tighter control but they may be more work to setup (especially if you need to reach many services).
Related
This is going to be more of a conceptual question.
I'm fairly new to Kubernetes and VPCs, and I'm currently studying in order to take part in designing a Kubernetes Cluster on GCP (Google Cloud Platform), and my role in that would be to address our security concerns.
Recently, I've been introduced to the concept of a "Private Kubernetes Cluster", which runs on a VPC and only allows traffic of allowed agents and from inside the VPC, with the Control Plane being accessible by a Bastion, for instance.
The thing is, I'm not sure if doing this would mean completely air-gapping the Cluster, blocking any access from the internet outside of the VPC or if I'm still able to use this to serve public web services, such as websites and APIs, whilst using the VPC to secure the control plane.
Any insights on that? I would also appreciate some documentation and related articles.
I still haven't got to the implementation part, since I'm trying to make sure I know what I'm doing beforehand.
Edit: According to the documentation, I am able to expose some of my cluster's nodes by using Cloud NAT. But would this defeat the purpose of even having a private cluster?
The thing is, I'm not sure if doing this would mean completely
air-gapping the Cluster, blocking any access from the internet outside
of the VPC or if I'm still able to use this to serve public web
services, such as websites and APIs, whilst using the VPC to secure
the control plane.
Yes, you will be able to Host your web application and you can expose those with the LoadBalancer even if you Cluster is private.
With a public cluster, your Worker node will be having the External/Public IPs while in private cluster worker nodes won't be having public IP.
You can create the service type LoadBalancer or use the Ingress to expose the application.
If public API access is required you can use the NAT gateway. you can configure your firewall rules to allow egress traffic to the specific public API endpoint you want to access.
Edit: According to the documentation, I am able to expose some of my
cluster's nodes by using Cloud NAT. But would this defeat the purpose
of even having a private cluster?
Yes right, The main advantage of Private GKE cluster I am seeing it does not have any Public/External IP address so can't be accessed from outside only accessed from within the VPC network. It can help protect clusters from un-auth access and reduce the surface of attacks on apps also.
Refer the Github for terraform and other details.
We're integrating with a new partner that requires us to use VPN when communicating with them (over HTTPS). We're running all of our services in a (non-private) Google Kubernetes Engine (GKE) cluster and it's only a single pod that needs to communicate with the partner's API.
The problem we face is that our partner's VPN provider won't allow us to use the private IP-range provided by GKE, 10.244.0.0/14, because the subnet is too large.
Preferably, we don't want to deploy something outside our GKE cluster, like a Compute Engine instance, that is somehow used to proxy our traffic (we will of course do it if this is the only/best way to proceed). We're hoping that, perhaps, it'll be possible to create a new node pool in the same cluster with a different (smaller) subnet, but so far we haven't found a way to do this. We've also looked briefly at CloudVPN, but if we understand it correctly, it only works with private GKE clusters.
Question:
What's the recommended way to obtain a smaller subnet/IP-range for a pod in an existing (public) GKE cluster to allow it to communicate with a third-party API over VPN?
The problem I see is that you have to maintain your VPN connection within your pod, it is possible but looks like an antipattern.
I would recommend using CloudVPN in a separate GCP project (due to cost separation and security) to establish the connection with a specific and limited VPC and then route that traffic to the pod, that might be in a specific ip range as you mentioned.
Take a look at the docs on how to create the vpn:
https://cloud.google.com/network-connectivity/docs/vpn/concepts/overview
Redirect traffic between VPCs:
https://cloud.google.com/vpc/docs/vpc-peering
Create the nodepool with an IP range: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create
Assign your deployment to that nodepool
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
I'm trying to set up a Kubernetes cluster for production (with kubeadm) on a private cloud (IONOS Cloud). I've been trying to figure it all out and configure it as best as possible for a few weeks now. Still there is one thing I don't quite understand or see if it is possible.
The initial cluster would be 5 nodes, 3 masters and 2 workers. The master servers with a load balancer and the workers with another load balancer. Each server has 2 interfaces, one for the public network and one for the private network (192.168.3.0/24). Each server has a firewall that by default blocks all packets.
In order not to have to create firewall rules on each server, so that they can see each other over the public network, I would like any kind of communication between Kubernetes nodes to be over the private network.
I can think of other reasons to use the private network for inter-node communication, such as: speed, latency, security...
However, I have not been able to configure it and I don't know if it is really possible to create this scenario. The problems I encounter are the following:
In order to access the API (e.g. kubectl) of the cluster from outside I need to expose the control-plane endpoint with the public IP of the balancer. If I do this, then the database endpoints etcd are exposed on the public network. Then, the other nodes, in the process of joining the cluster (kubeadm join) need to get some information from the databases etcd, therefore, they necessarily need visibility over the public network between them.
Once all the nodes are joined, the "kubernetes" service has the endpoints of all the control-plane endpoints (public). When I try to deploy Calico (or another CNI) they never finish deploying because they make queries to the kubernetes service, and if they don't have visibility between them, it fails.
It seems that whatever I do, if I publish the API on the public network, I need all the nodes to see each other over the public network.
Maybe I'm making my life too complicated, and the simplest thing to do is to open a firewall rule for each node, but I don't know if this is a good practice.
Architecture diagram (I still cannot embed images)
We are currently providing our software as a software-as-a-service on Amazon EC2 machines. Our software is a microservice-based application with around 20 different services.
For bigger customers we use dedicated installations on a dedicated set of VMs, the number of VMs (and number of instances of our microservices) depending on the customer's requirements. A common requirement of any larger customer is that our software needs access to the customer's datacenter (e.g., for LDAP access). So far, we solved this using Amazon's virtual private gateway feature.
Now we want to move our SaaS deployments to Kubernetes. Of course we could just create a Kubernetes cluster across an individual customer's VMs (e.g., using kops), but that would offer little benefit.
Instead, perspectively, we would like to run a single large Kubernetes cluster on which we deploy the individual customer installations into dedicated namespaces, that way increasing resource utilization and lowering cost compared to the fixed allocation of machines to customers that we have today.
From the Kubernetes side of things, our software works fine already, we can deploy multiple installations to one cluster just fine. An open topic is however the VPN access. What we would need is a way to allow all pods in a customer's namespace access to the customer's VPN, but not to any other customers' VPNs.
When googleing for the topic, I found approaches that add a VPN client to the individual container (e.g., https://caveofcode.com/2017/06/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/) which is obviously not an option).
Other approaches seem to describe running a VPN server inside K8s (which is also not what we need).
Again others (like the "Strongswan IPSec VPN service", https://www.ibm.com/blogs/bluemix/2017/12/connecting-kubernetes-cluster-premises-resources/ ) use DaemonSets to "configure routing on each of the worker nodes". This also does not seem like a solution that is acceptable to us, since that would allow all pods (irrespective of the namespace they are in) on a worker node access to the respective VPN... and would also not work well if we have dozens of customer installations each requiring its own VPN setup on the cluster.
Is there any approach or solution that provides what we need, .i.e., VPN access for the pods in a specific namespace only?
Or are there any other approaches that could still satisfy our requirement (lower cost due to Kubernetes worker nodes being shared between customers)?
For LDAP access, one option might be to setup a kind of LDAP proxy, so that only this proxy would need to have VPN access to the customer network (by running this proxy on a small dedicated VM for each customer, and then configuring the proxy as LDAP endpoint for the application). However, LDAP access is only one out of many aspects of connectivity that our application needs depending on the use case.
If your IPSec concentrator support VTI, it's possible route the traffic using firewall rules. For example, PFSense suports it: https://www.netgate.com/docs/pfsense/vpn/ipsec/ipsec-routed.html.
Using VTI, you can direct traffic using some kind of policy routing: https://www.netgate.com/docs/pfsense/routing/directing-traffic-with-policy-routing.html
However, i can see two big problems here:
You cannot have two IPSEC tunnels with the conflicted networks. For example, your kube network is 192.168.0.0/24 and you have two customers: A (172.12.0.0/24) and B (172.12.0.0/12). Unfortunelly, this can happen (unless your customer be able to NAT those networks).
Find the ideals criteria for rule match (to allow the routing), since your source network are always the same. Use mark packages (using iptables mangle or even through application) can be a option, but you will still get stucked on the first problem.
A similar scenario is founded on WSO2 (API gateway provider) architecture. They solved it using reverse-proxy in each network (sad but true) https://docs.wso2.com/display/APICloud/Expose+your+On-Premises+Backend+Services+to+the+API+Cloud#ExposeyourOn-PremisesBackendServicestotheAPICloud-ExposeyourservicesusingaVPN
Regards,
UPDATE:
I don't know if you use GKE. If yes, maybe use Alias-IP can be an option: https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips. The PODs IPs will be routable from VPC. So, you can apply some kind of routing policy based on their CIDR.
For improved performance and availability we'd like to distribute certain services from out stack across different Kubernetes clusters in different parts of the world (GCP regions).
The majority of our stack will continue to run in one cluster / region but some user facing services will be deployed all over the world.
Some of these services need to access other services in our main cluster.
Q: How can we reliably access services in a different Kubernetes cluster?
Using internal load balancers seems to be out of the question as those are per region only.
We'd like to keep the communication between our services inside the private GCP network and avoid going over the public internet. So an public ingress also wouldn't work.
VPC networks are global resources, not restricted by regional boundaries, and so with the correct firewall rules set up, you should be able to access any internal resource from any other resource "right out of the box", assuming they are in the same VPC network and same project.
Take a look at VPN Peering: https://cloud.google.com/vpc/docs/vpc-peering
It allows you to connect two vpcs (in different regions) so that they can communicate privately.
You may have to recreate/reconfigure your Kubernetes in order to support this vpc architecture.