I have a subnet mask for my subnet set to 10.0.0.0/9. When setting up kubernetes, google asks for a master ip range for kubernets. I set this to 10.0.0.0/28 but I have no idea if this is correct or how these two things are related? Is there any info on that?
Also, did I do that right? I assume the kubernetes has to be using the ips of the subnet.
thanks,
Dean
"Master IP Range" is only relevant in GKE when you enable Private Network.
When creating a private cluster, the Master IP Range has the following information message:
Master IP range is a private RFC 1918 range for the master's VPC. The master range must not overlap with any subnet in your cluster's VPC. The master and your cluster use VPC peering to communicate privately.
This setting is permanent.
Since 10.0.0.0/28 is a range inside 10.0.0.0/9, it will not effectively isolate the cluster.
I Created a vpc subnet with 10.0.0.0/9 and tried to create the cluster with Master IP Range 10.0.0.0/28, look at the message I get while creating it:
If you look at Creating a Private GKE Cluster you can find many configuration examples for different access types.
Example:
If your subnet is 10.0.0.0/9 you must use a Master IP Range outside of that range.
Since the first half of /9 ends in 10.127.255.255 you can set master network to be anything inside 10.128.0.0/9, 172.16.0.0/12 or 192.168.0.0/16 as long it does not overlaps any other vpc or subnet in your project.
Here you can learn more about GKE Networking.
If you have any doubts let me know in the comments.
Related
I want to connect GKE (Google Kubernetes Engine) cluster to MongoDB Atlas. But I need to green the IP of my nodes (allow them). But sometimes I have 3 nodes, sometimes I have 10 and sometimes nodes are falling down and re-creating - constant changing means a no single IP.
I have tried to create NAT on the GCP followed this guide: https://medium.com/google-cloud/using-cloud-nat-with-gke-cluster-c82364546d9e
Also I want to green my cluster's IP in the Google Maps APIs so I can use the Directions API, for example.
This is a common situation, since there may be many other third party APIs that I want to enable that require incoming requests from certain IPs only, besides Atlas or Google Maps..
How can I achieve this?
Private GKE cluster means the nodes do not have public IP addresses but you mentioned
the actual outbound transfer goes from the node's IP instead of
the NAT's
Looks like you have a public cluster of GKE, you have to use the same NAT option to get outbound egress single IP.
If you are using the ingress which means there is a single point for incoming request to cluster but if your Nodes have public IP PODs will use Node's IP when there is an outgoing request unless you use NAT or so.
Your single outbound IP will be there, so all requests going out of PODs won't have node's IP instead they will use the NAT IP.
how to set up the NAT gateway
https://registry.terraform.io/modules/GoogleCloudPlatform/nat-gateway/google/latest/examples/gke-nat-gateway
Here is terraform ready for the GKE clusters, you just have to run this terraform example bypassing project ID and others vars.
The above terraform example will create the NAT for you and verify the PODs IP as soon as NAT is set. You mostly won't require any changes in NAT terraform script.
GitHub link: https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway/tree/v1.2.3/examples/gke-nat-gateway
if you dont have idea of terraform you can follow this article to setup the NAT which will stop the SNAT for PODs : https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a
Private GKE cluster means the nodes do not have public IP addresses. If the service on the other end is receiving packets from node's own IP then you have a public cluster.
You can find further explanation in this document.
If you want a static, public IP for the entire GKE cluster, you should consider Ingress for External Load Balancing. You can find instructions on how to configure it here.
I'm wondering if anyone can help with my issue, here's the setup:
We have 2 separate kubernetes clusters in GKE, running on v1.17, and they each sit in a separate project
We have set up VPC peering between the two projects
On cluster 1, we have 'service1' which is exposed by an internal HTTPS load balancer, we don't want this to be public
On cluster 2, we intend on being able to access 'service1' via the internal load balancer, and it should do this over the VPC peering connection between the two projects
Here's the issue:
When I'm connected via SSH on a GKE node on cluster 2, I can successfully run a curl request to access https://service1.domain.com running on cluster 1, and get the expected response, so traffic is definitely routing from cluster 2 > cluster 1. However, when I'm running the same curl command from a POD, running on a GKE node, the same curl request times out.
I have run as much troubleshooting as I can including telnet, traceroute etc and I'm really stuck why this might be. If anyone can shed light on the difference here that would be great.
I did wonder whether pod networking is somehow forwarding traffic over the clusters public IP rather than over the VPC peering connection.
So it seems you're not using a "VPC-native" cluster and what you need is "IP masquerading".
From this document:
"A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses."
You can use ip-masq-agent or k8s-custom-iptables. After this, it will work since it will be like you're making a call from node, not inside of pod.
As mentioned in one of the answers IP aliases (VPC-native) should work out of the box. If using a route based GKE cluster rather than VPC-native you would need to use custom routes.
As per this document
By default, VPC Network Peering with GKE is supported when used with
IP aliases. If you don't use IP aliases, you can export custom routes
so that GKE containers are reachable from peered networks.
This is also explained in this document
If you have GKE clusters without VPC native addressing, you might have
multiple static routes to direct traffic to VM instances that are
hosting your containers. You can export these static routes so that
the containers are reachable from peered networks.
The problem your facing seems similar to the one mentioned in this SO question, perhaps your pods are using IPs outside of the VPC range and for that reason cannot access the peered VPC?
UPDATE: In Google cloud, I tried to access the service from another cluster which had VPC native networking enabled, which I believe allows pods to use the VPC routing and possibly the internal IPs.
Problem solved :-)
I have my Redshift cluster deployed in a VPC inside private subnets . I need to allow an IP address to access the cluster from outside the VPC . To add that IP as a whitelist and access the cluster I tried the below .
Created an inbound rule in the security group which is attached to the redshift cluster . Added the ip-address/32 as source , port 5439 , protocol tcp , type redshift.
Added the redshift cluster in the public subnet .
I did check in https://forums.aws.amazon.com/thread.jspa?threadID=134301 . He faced the same issue too .
The steps I tried didn't work . Appreciate any suggestion which can make that IP address to access the cluster.
Thanks in advance.
As the second step you did, I assume you've already put the Redshift cluster to public subnet in your VPC, then make sure your networkACL allows ingress port 5439 and egress ephemeral ports.
I think you need to make your redshift cluster "publicly accessible".
After that, just modify your associated VPC security group to allow access from specific IP addresses, and you should be able to connect to the cluster from outside the VP.
AWS forum
AWS documentation
If the IP address which is outside the VPC of Redshift is in your AWS account, or in an other account; the VPC peering between two VPC can be an option.
If you peer two VPCs; one with Redshift and the other is the VPC of the other IP address, then it is possible two enable network traffic between two
You should enable traffic by routing tables entries for new IP ranges too.
And the security group entries should be added into Redshift's Inbound rules
I am trying to further understand what exactly is happening when I provision a private cluster in Google's Kubernetes Engine.
Google provides this example here of provisioning a private cluster where the control plane services (e.g. Kubernetes API) live on the 172.16.0.16/28 subnet.
https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters
gcloud beta container clusters create pr-clust-1 \
--private-cluster \
--master-ipv4-cidr 172.16.0.16/28 \
--enable-ip-alias \
--create-subnetwork ""
When I run this command, I see that:
I now have a few gke subnets in my VPC belong to the cluster subnets for nodes and services. These are in the 10.x.x.x/8 range.
I don't have any subnets in the 172.16/16 address space.
I do have some new pairing rules and routes that seem to be related. For example, there is a new route peering-route-a08d11779e9a3276 with a destination address range of 172.16.0.16/28 and next hop gke-62d565a060f347e0fba7-3094-3230-peer. This peering role then points to gke-62d565a060f347e0fba7-3094-bb01-net
gcloud compute networks subnets list | grep us-west1
#=>
default us-west1 default 10.138.0.0/20
gke-insti3-subnet-62d565a0 us-west1 default 10.2.56.0/22
gcloud compute networks peerings list
#=>
NAME NETWORK PEER_PROJECT PEER_NETWORK AUTO_CREATE_ROUTES STATE STATE_DETAILS
gke-62d565a060f347e0fba7-3094-3230-peer default gke-prod-us-west1-a-4180 gke-62d565a060f347e0fba7-3094-bb01-net True ACTIVE [2018-08-23T16:42:31.351-07:00]: Connected.
Is gke-62d565a060f347e0fba7-3094-bb01-net a peered VPC in which the Kubernetes management endpoints live (the control plane stuff in the 172.16/16 range) that Google is managing for the GKE service?
Further - how are my requests making it to the Kubernetes API server?
The Private Cluster feature of GKE depends on the Alias IP Ranges feature of VPC networking, so there are multiple things happening when you create a private cluster:
The --enable-ip-alias flag tells GKE to use a subnetwork that has two secondary IP ranges: one for pods and one for services. This allows the VPC network to understand all the IP addresses in your cluster and route traffic appropriately.
The --create-subnetwork flag tells GKE to create a new subnetwork (gke-insti3-subnet-62d565a0 in your case) and choose its primary and secondary ranges automatically. Note that you could instead choose the secondary ranges yourself with --cluster-ipv4-cidr and --services-ipv4-cidr. Or you could even create the subnetwork yourself and tell GKE to use it with the flags --subnetwork, --cluster-secondary-range-name, and --services-secondary-range-name.
The --private-cluster flag tells GKE to create a new VPC network (gke-62d565a060f347e0fba7-3094-bb01-net in your case) in a Google-owned project and connect it to your VPC network using VPC Network Peering. The Kubernetes management endpoints live in the range you specify with --master-ipv4-cidr (172.16.0.16/28 in your case). An Internal Load Balancer is also created in the Google-owned project and this is what your worker nodes communicate with. This ILB allows traffic to be load-balanced across multiple VMs in the case of a Regional Cluster. You can find this internal IP address as the privateEndpoint field in the output of gcloud beta container clusters describe. The important thing to understand is that all communication between master VMs and worker node VMs happens over internal IP addresses, thanks to the VPC peering between the two networks.
Your private cluster also has an external IP address, which you can find as the endpoint field in the output of gcloud beta container clusters describe. This is not used by the worker nodes, but is typically used by customers to manage their cluster remotely, e.g., using kubectl.
You can use the Master Authorized Networks feature to restrict which IP ranges (both internal and external) have access to the management endpoints. This feature is strongly recommended for private clusters, and is enabled by default when you create the cluster using the gcloud CLI.
Hope this helps!
I have deployed a Kubernetes cluster to GCP. For this cluster, I added some deployments. Those deployments are using external resources that protected with security policy to reject connection from unallow IP address.
So, in order to pod to connect the external resource, I need manually allow the node (who hosting the pod) IP address.
It's also possible to me to allow range of IP address, where one of my nodes are expected to be running.
Untill now, I just find their internal IP addresses range. It looks like this:
Pod address range 10.16.0.0/14
The question is how to find the range of external IP addresses for my nodes?
Let's begin with the IPs that are assigned to Nodes:
When we create a Kubernetes cluster, GCP in the backend creates compute engines machines with a specific internal and external IP address.
In your case, just go to the compute engine section of the Google Cloud Console and capture all the external IPs of the VM whose initials starts with gke-(*) and whitelist it.
Talking about the range, as such in GCP only the internal IP ranges are known and external IP address are randomly assigned from a pool of IPs hence you need to whitelist it one at a time.
To get the pod description and IPs run kubectl describe pods.
If you go to the compute engine instance page it shows the instances which make the cluster. it shows the external ips on the right side. For the the ip of the actual pods use the Kubectl command.