We are attempting to make several private Kubernetes clusters. We can find limited documentation on specific settings for the private cluster, therefore we are running into issues related to the subnetwork IP ranges.
Say we have 3 clusters: We set the Master Address Range to 172.16.0.0/28, 172.16.0.16/28 and 172.16.0.32/28 respectively.
We leave Network and Subnet set to "default". We are able to create 2 clusters that way, however, upon spin-up of the 3rd cluster, we receive the error of "Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5." We suspect that we are setting up the subnetwork IP ranges incorrectly, but we are not sure what we are doing wrong, or why there is more than 1 secondary range per subnetwork, to begin with.
Here is a screenshot of the configuration for one of the clusters:
We are setting these clusters up through the UI.
This cluster has VPC-native (alias IP) enabled, which use 2 secondary ranges per cluster.
See https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#secondary_ranges
According to
Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.
the max is 5. That's why the 3rd one failed to create.
The best approach is to create a new subnetwork for each cluster. This way, each subnetwork only requires 2 secondary ranges, and you won't hit the limit of 5.
For anyone who lands here from Google and is wondering how to list / see the subnet names that have been created using GKE as described in OP's question:
To list subnets for a region (and potentially modify or delete a Subnet, since you won't know the name) use the beta gcloud command:
gcloud beta container subnets list-usable
I landed here while looking for the answer and figured others trying to determine the best way to structure their subnets / ranges might be able to use the above command (which took me forever to track down).
(Can't comment on Alan's answer due to low reputation)
You can create a new subnetwork:
go to the "VPC network"
click on "default" (under name)
click on "Add subnet"
define the subnet range / zone
Then on GKE when you create a new cluster, select your new subnetwork.
This should allow you to create more clusters without running into the error Google Compute Engine: Exceeded maximum supported number of secondary ranges per subnetwork: 5.
You can create clusters via gcloud and add --create-subnetwork "".
See: https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--create-subnetwork
This will create a new subnet with each new cluster so the "5 Secondary IP ranges per subnet" quota won't be reached.
I run into the same issue with a error message below
ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=This operation will exceed max secondary ranges per subnetwork (30) for subnet "default", consider reusing existing secondary ranges or use a different subnetwork.
I think the problem is we create GEK clusters that shares the same subnet "default" and eventually they exceed the max secondary ranges.
I think the best practice would be to create dedicated subnet for each cluster.
You Creating a cluster and subnet simultaneously
gcloud container clusters create cluster-name \
--region=region \
--enable-ip-alias \
--create-subnetwork name=subnet-name
see below to find more subnet configuration
https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#creating_cluster_and_subnet
NOTE: GKE tries to clean up the created subnetwork when the cluster is deleted. But if the subnetwork is being used by other resources, GKE does not delete the subnetwork, and you must manage the life of the subnetwork yourself.
Related
I am having some issue with GKE(autopilot).
I am deploying statefulsets and for each statefulset I deploy a service with public IP.
But after deploying like 10-12 statefulsets, if I try deploying any new it remains red(Unschedulable) with message "Insufficient cpu".
When I go to cluster section is show a different message saying:
Can’t scale up because instances in managed instance groups hosting node pools ran out of IPs
Image of error: https://i.imgur.com/t8I4Yij.png
I am new to GKE and tried doing what suggested in links of those image but it seems most of steps give error saying its not supported in autopilot mode.
Any help/suggestion is appreciated.
Thanks.
If you are on GKE autopilot ideally it will create the new nodes in cluster if out of CPU or no space left to schedule the PODs.
However if it's issue of IP you can read more : https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips#not_enough_space
Cluster autoscaler might not have enough unallocated IP address space
to use to add new nodes or Pods, resulting in scale-up failures, which
are indicated by eventResult events with the reason
scale.up.error.ip.space.exhausted. You can add more IP addresses for
nodes by expanding the primary subnet, or add new IP addresses for
Pods using discontiguous multi-Pod CIDR. For more information, see Not
enough free IP space for Pods.
but you are on autopilot so wont be able to access underlaying subnet and node pools of cluster maybe.
Unfortunately, the only option at this point is to create a new cluster and make sure that the CIDR ranges you assign to the cluster have enough available IPs for the number of nodes you believe you'll need. The default setting for Autopilot should be enough.
I have av VPC with three private subnets (10.176.128.0/20, 10.176.144.0/20, 10.176.160.0/20) and three public subnets (10.176.0.0/20, 10.176.16.0/20, 10.176.32.0/20). All private subnets have the tag kubernetes.io/role/internal-elb=1 and public have the tag kubernetes.io/role/elb=1.
I run all my worker nodes in managed node groups and AWS eks has been responsible for creating a default security group for the cluster. That security group is what I'm referring to later.
I have two namespaces in my kubernetes cluster, test and stage and in each namespace I have 3 services with loadbalancer and they expose 8 ports in each namespace. The loadbalancer is of type nlb.
Now to the problem, each service with a loadbalancer creates 4 rules per port in my security group for my nodes, one for each subnet it's located in and one for all trafic (0.0.0.0/0). 8 * 4 * 2 = 64 and max number of rules per security group is 60 according to AWS, so when I'm about to create the last LB I get the error about RulesPerSecurityGroupLimitExceeded.
Two ways to solve this as I see it, either have more security groups attached to my nodes or somehow config so there are less rules created per port. Thing is that actually one rule with be enough of them since 0.0.0.0/0 would allow all my subnets as well. Another option might be that I'm doing something wrong in the design. The first option to add more security groups I have tried and failed with, still tries to add the rules to the one that is already full.
We are hitting this issue as well. One thing you can do is request a quota increase on rules per security group in the AWS console. Feels to me like that is only going to postpone the issue slightly though.
On AWS EKS
I'm adding deployment with 17 replicas (requesting and limiting 64Mi memory) to a small cluster with 2 nodes type t3.small.
Counting with kube-system pods, total running pods per node is 11 and 1 is left pending, i.e.:
Node #1:
aws-node-1
coredns-5-1as3
coredns-5-2das
kube-proxy-1
+7 app pod replicas
Node #2:
aws-node-1
kube-proxy-1
+9 app pod replicas
I understand that t3.small is a very small instance. I'm only trying to understand what is limiting me here. Memory request is not it, I'm way below the available resources.
I found that there is IP addresses limit per node depending on instance type.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html?shortFooter=true#AvailableIpPerENI .
I didn't find any other documentation saying explicitly that this is limiting pod creation, but I'm assuming it does.
Based on the table, t3.small can have 12 IPv4 addresses. If this is the case and this is limiting factor, since I have 11 pods, where did 1 missing IPv4 address go?
The real maximum number of pods per EKS instance are actually listed in this document.
For t3.small instances, it is 11 pods per instance. That is, you can have a maximum number of 22 pods in your cluster. 6 of these pods are system pods, so there remains a maximum of 16 workload pods.
You're trying to run 17 workload pods, so it's one too much. I guess 16 of these pods have been scheduled and 1 is left pending.
The formula for defining the maximum number of pods per instance is as follows:
N * (M-1) + 2
Where:
N is the number of Elastic Network Interfaces (ENI) of the instance type
M is the number of IP addresses of a single ENI
So, for t3.small, this calculation is 3 * (4-1) + 2 = 11.
Values for N and M for each instance type in this document.
For anyone who runs across this when searching google. Be advised that as of August 2021 its now possible to increase the max pods on a node using the latest AWS CNI plugin as described here.
Using the basic configuration explained there a t3.medium node went from a max of 17 pods to a max of 110 which is more then adequate for what I was trying to do.
This is why we stopped using EKS in favor of a KOPS deployed self-managed cluster.
IMO EKS which employs the aws-cni causes too many constraints, it actually goes against one of the major benefits of using Kubernetes, efficient use of available resources.
EKS moves the system constraint away from CPU / memory usage into the realm of network IP limitations.
Kubernetes was designed to provide high density, manage resources efficiently. Not quite so with EKS’s version, since a node could be idle, with almost its entire memory available and yet the cluster will be unable to schedule pods on an otherwise low utilized node if pods > (N * (M-1) + 2).
One could be tempted to employ another CNI such as Calico, however would be limited to worker nodes since access to master nodes is forbidden.
This causes the cluster to have two networks and problems will arise when trying to access K8s API, or working with Admissions Controllers.
It really does depend on workflow requirements, for us, high pod density, efficient use of resources, and having complete control of the cluster is paramount.
connect to you EKS node
run this
/etc/eks/bootstrap.sh clusterName --use-max-pods false --kubelet-extra-args '--max-pods=50'
ignore nvidia-smi not found the output
whole script location https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh
EKS allows to increase max number of pods per node but this can be done only with Nitro instances. check the list here
Make sure you have VPC CNI 1.9+
Enable Prefix delegation for VPC_CNI plugin
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
If you are using self managed node group, make sure to pass the following in BootstrapArguments
--use-max-pods false --kubelet-extra-args '--max-pods=110'
or you could create the node group using eksctl using
eksctl create nodegroup --cluster my-cluster --managed=false --max-pods-per-node 110
If you are using managed node group with a specified AMI, it has bootstrap.sh so you could modify user_data to do something like this
/etc/eks/bootstrap.sh my-cluster \ --use-max-pods false \ --kubelet-extra-args '--max-pods=110'
Or simply using eksctl by running
eksctl create nodegroup --cluster my-cluster --max-pods-per-node 110
For more details, check AWS documentation https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
Does GKE supports vertical node auto-scaling?
For example:
I have a GKE cluster with only one node-pool and two nodes that node-pool, in case there is a requirement for more memory or CPU by any pod, I do not want any other nodes / compute instances to be created, is there a way in which the configuration of existing nodes change and extra memory / CPU get added?
Basically, existing instances / nodes get upgraded to become instances with higher configuration.
I'm afraid that Google Cloud does not provide feature you want.
There is no way to add CPU/RAM in current node pool VMs as nodes used in Google Kubernetes Engine are based on Google Compute Engine Virtual Machines and resources are specified before creation. If you would like to change CPU/RAM resources in node pool, you need to do it manually as #Emre Odabaş pointed. However, all nodes must be recreated with higher resources availability.
To fulfill your needs you can choose which Machine Type you want to use, however you can create Custom Machine Type.
As workaround you could consider to have 2 node pools:
one with regular node-pool-1 with e2-medium (2 vCPU, 4GB Memory)
$ gcloud container node-pools create node-pool-1 --machine-type=e2-medium --cluster=cluster-1 --zone=europe-north1-b ---num-nodes=2
and node-pool-2 with e2-standard-4 (4 vCPU, 16 GB Memory).
$ gcloud container node-pools create node-pool-2 --machine-type=e2-standard-4 --cluster=cluster-1 --zone=europe-north1-b --num-nodes=0
However, this would work only when you would know that you do not have enough resources and change it manually:
Regular load: 2 node from node-pool-1
Higher load: 1 node from node-pool-1 and 1 node from node-pool-2
Full load: 2 nodes form node-pool-2
You could manually change node pools with different node types. AFAIK there is no vertical node auto scaler in GKE.
I have a GKE cluster running in us-central1 with a preemptable node pool. I have nodes in each zone (us-central1-b,us-central1-c,us-central1-f). For the last 10 hours, I get the following error for the underlying node vm:
Instance '[instance-name]' creation failed: The zone
'[instance-zone]'
does not have enough resources available to fulfill
the request. Try a different zone, or try again
later.
I tried creating new clusters in different regions with different machine types, using HA (multi-zone) settings and I get the same error for every cluster.
I saw an issue on Google Cloud Status Dashboard and tried with the console, as recommended, and it errors out with a timeout error.
Is anyone else having this problem? Any idea what I may be dong wrong?
UPDATES
Nov 11
I stood up a cluster in us-west2, this was the only one which would work. I used gcloud command line, it seems the UI was not effective. There was a note similar to this situation, use gcloud not ui, on the Google Cloud Status Dashboard.
I tried creating node pools in us-central1 with the gcloud command line, and ui, to no avail.
I'm now federating deployments across regions and standing up multi-region ingress.
Nov. 12
Cannot create HA clusters in us-central1; same message as listed above.
Reached out via twitter and received a response.
Working with the K8s guide to federation to see if I can get multi-cluster running. Most likely going to use Kelsey Hightowers approach
Only problem, can't spin up clusters to federate.
Findings
Talked with google support, need a $150/mo. package to get a tech person to answer my questions.
Preemptible instances are not a good option for a primary node pool. I did this because I'm cheap, it bit me hard.
The new architecture is a primary node pool with committed use VMs that do not autoscale, and a secondary node pool with preemptible instances for autoscale needs. The secondary pool will have minimum nodes = 0 and max nodes = 5 (for right now); this cluster is regional so instances are across all zones.
Cost for an n1-standard-1 sustained use (assuming 24/7) a 30% discount off list.
Cost for a 1-year n1-standard-1 committed use is about ~37% discount off list.
Preemptible instances are re-provisioned every 24hrs., if they are not taken from you when resource needs spike in the region.
I believe I fell prey to a resource spike in the us-central1.
A must-watch for people looking to federate K8s: Kelsey Hightower - CNCF Keynote | Kubernetes Federation
Issue appears to be resolved as of Nov 13th.