For improved performance and availability we'd like to distribute certain services from out stack across different Kubernetes clusters in different parts of the world (GCP regions).
The majority of our stack will continue to run in one cluster / region but some user facing services will be deployed all over the world.
Some of these services need to access other services in our main cluster.
Q: How can we reliably access services in a different Kubernetes cluster?
Using internal load balancers seems to be out of the question as those are per region only.
We'd like to keep the communication between our services inside the private GCP network and avoid going over the public internet. So an public ingress also wouldn't work.
VPC networks are global resources, not restricted by regional boundaries, and so with the correct firewall rules set up, you should be able to access any internal resource from any other resource "right out of the box", assuming they are in the same VPC network and same project.
Take a look at VPN Peering: https://cloud.google.com/vpc/docs/vpc-peering
It allows you to connect two vpcs (in different regions) so that they can communicate privately.
You may have to recreate/reconfigure your Kubernetes in order to support this vpc architecture.
Related
This is going to be more of a conceptual question.
I'm fairly new to Kubernetes and VPCs, and I'm currently studying in order to take part in designing a Kubernetes Cluster on GCP (Google Cloud Platform), and my role in that would be to address our security concerns.
Recently, I've been introduced to the concept of a "Private Kubernetes Cluster", which runs on a VPC and only allows traffic of allowed agents and from inside the VPC, with the Control Plane being accessible by a Bastion, for instance.
The thing is, I'm not sure if doing this would mean completely air-gapping the Cluster, blocking any access from the internet outside of the VPC or if I'm still able to use this to serve public web services, such as websites and APIs, whilst using the VPC to secure the control plane.
Any insights on that? I would also appreciate some documentation and related articles.
I still haven't got to the implementation part, since I'm trying to make sure I know what I'm doing beforehand.
Edit: According to the documentation, I am able to expose some of my cluster's nodes by using Cloud NAT. But would this defeat the purpose of even having a private cluster?
The thing is, I'm not sure if doing this would mean completely
air-gapping the Cluster, blocking any access from the internet outside
of the VPC or if I'm still able to use this to serve public web
services, such as websites and APIs, whilst using the VPC to secure
the control plane.
Yes, you will be able to Host your web application and you can expose those with the LoadBalancer even if you Cluster is private.
With a public cluster, your Worker node will be having the External/Public IPs while in private cluster worker nodes won't be having public IP.
You can create the service type LoadBalancer or use the Ingress to expose the application.
If public API access is required you can use the NAT gateway. you can configure your firewall rules to allow egress traffic to the specific public API endpoint you want to access.
Edit: According to the documentation, I am able to expose some of my
cluster's nodes by using Cloud NAT. But would this defeat the purpose
of even having a private cluster?
Yes right, The main advantage of Private GKE cluster I am seeing it does not have any Public/External IP address so can't be accessed from outside only accessed from within the VPC network. It can help protect clusters from un-auth access and reduce the surface of attacks on apps also.
Refer the Github for terraform and other details.
We're integrating with a new partner that requires us to use VPN when communicating with them (over HTTPS). We're running all of our services in a (non-private) Google Kubernetes Engine (GKE) cluster and it's only a single pod that needs to communicate with the partner's API.
The problem we face is that our partner's VPN provider won't allow us to use the private IP-range provided by GKE, 10.244.0.0/14, because the subnet is too large.
Preferably, we don't want to deploy something outside our GKE cluster, like a Compute Engine instance, that is somehow used to proxy our traffic (we will of course do it if this is the only/best way to proceed). We're hoping that, perhaps, it'll be possible to create a new node pool in the same cluster with a different (smaller) subnet, but so far we haven't found a way to do this. We've also looked briefly at CloudVPN, but if we understand it correctly, it only works with private GKE clusters.
Question:
What's the recommended way to obtain a smaller subnet/IP-range for a pod in an existing (public) GKE cluster to allow it to communicate with a third-party API over VPN?
The problem I see is that you have to maintain your VPN connection within your pod, it is possible but looks like an antipattern.
I would recommend using CloudVPN in a separate GCP project (due to cost separation and security) to establish the connection with a specific and limited VPC and then route that traffic to the pod, that might be in a specific ip range as you mentioned.
Take a look at the docs on how to create the vpn:
https://cloud.google.com/network-connectivity/docs/vpn/concepts/overview
Redirect traffic between VPCs:
https://cloud.google.com/vpc/docs/vpc-peering
Create the nodepool with an IP range: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create
Assign your deployment to that nodepool
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
I am facing one scenario where I have to access one Kubernetes service of GCP PROJECT X from a pod running in another GCP Project Y.
I know we can access service from one namespace in another namespace in the same project by using
servicename.namespacename.svc.cluster.local
how can I do if I have to do similar across different GCP projects?
Agree with #cperez08, but adding my 5 cents.
I think you can try Set up clusters with Shared VPC
With Shared VPC, you designate one project as the host project, and
you can attach other projects, called service projects, to the host
project. You create networks, subnets, secondary address ranges,
firewall rules, and other network resources in the host project. Then
you share selected subnets, including secondary ranges, with the
service projects. Components running in a service project can use the
Shared VPC to communicate with components running in the other service
projects.
You can use Shared VPC with both zonal and regional clusters. Clusters
that use Shared VPC cannot use legacy networks and must have Alias IPs
enabled.
You can configure Shared VPC when you create a new cluster. Google
Kubernetes Engine does not support converting existing clusters to the
Shared VPC model.
If I understood well project X and Y are completely different clusters, thus, I am not sure if that's possible, take a look to this https://kubernetes.io/blog/2016/07/cross-cluster-services/ maybe you can have re-architect your services by federating in case High Availability is needed.
On the other hand, you can always access to the resources through a public endpoint/domain if they are not in someway connected.
I have two Kubernetes clusters cluster-a, cluster-b in Google Cloud GCP.
Can i call a service exposed with ambasador in cluster (cluster-a) from a different cluster (cluster-b) in the same GCP project but different VPC's ?
Right now i can call the service by the ambasador service name (when I do it in the same cluster).
I have read about Internal TCP/UDP Load Balancing, but it only works when cluster-a and cluster-b are in the same VPC network and my clusters are in different VPC's.
There is a different approach to accomplish it ?
VPCs on GCP aren't routed to each other by default, so your requests won't be reaching the remote CIDRs. For that, you want to use VPC Network Peering to make each VPC reachable to each other.
Note that firewall rules still apply for both VPCs, so you have to create them in order to establish full communication.
Finally, this will only allow network communication between your VPCs. If you rule out this as the issue and you're still experiencing lack of connectivity, it might be related to your Ambassador configuration, in which case, I'd recommend posting either information about that or create another question for that specifically.
We want to run a multi-tenant scenario that requires tenant separation on a network level.
The idea is that every tenant receives a dedicated node and a dedicated network that other tenants nodes can join. Tenant nodes should be able to interact with each other in that network.
Networks should not be able to talk with each other (true network isolation).
Are there any architectural patterns to achieve this?
One Kubernetes cluster per tenant?
One Kubernetes cluster for all tenants, with one subnet per tenant?
One Kubernetes cluster across VPCs (speaking in AWS terms)?
The regular way to deal with multi-tenancy inside kubernetes is to use namespaces. But this is within a kube cluster, meaning you still have the same underlying networking solution shared by all tenants. That is actualy fine, as you have Network Policies to restrict networking in the cluster.
You can obviously run autonomous clusters per tenant, yet this is not exactly multi-tenancy then, just multiple clusters. Networking can be configured on node level to route as expected, but you'd still be left with an issue of cross-cluster service discovery etc. Federation can help a bit with that, but I would still advise to chase Namespaces+Policies approach.
I see four ways to run multi-tenant k8s clusters at network-level:
Namespaces
Ingress rules
allow/deny and ingress/egress Network Policies
Network-aware Zones