So the question is in the title.
I'm wondering what is the purpose of cluster-name parameter from kubernetes controller_manager?
Reading the code of the cluster manager you'll find that the cluster-name is passed down to the service controller and the persistent volume controller which then pass them down to their related objects (Load balancers, persistent volumes, ...).
In both cases these pass down the cluster name to the related cloud provider (see interface) which could use it within the naming of their provider specific objects. This makes sense in case you run more than one Kubernetes cluster side by side on the same provider.
The GCE and AWS cloud providers do this for example, some others don't.
So having two clusters with the same cluster name configuration for the controller manager could therefore cause issues due to name collisions within the objects created by the cloud provider.
lets you create more than one cluster and helps you distinguish between them. Most folks just use kubernetes (which is the default) When you setup your kubectl you provide it as well.
this is from the k8s site # https://kubernetes.io/docs/getting-started-guides/scratch/
You should pick a name for your cluster. Pick a short name for each cluster which is unique from future cluster names. This will be used in several ways:
by kubectl to distinguish between various clusters you have access to. You will probably want a second one sometime later, such as for testing new Kubernetes releases, running in a different region of the world, etc.
Kubernetes clusters can create cloud provider resources (e.g. AWS ELBs) and different clusters need to distinguish which resources each created. Call this CLUSTER_NAME
Related
I am now running two kubernetes clusters.
First Cluster is running on bare metal, and Second Cluster is running on EKS.
but since maintaining EKS costs a lot, so I am finding ways to change this service as Single Cluster that autoscales on AWS.
I did tried to consider several solutions such as RHACM, Rancher and Anthos.
But those solutions are for controlling multi cluster.
I just want to change this cluster as "onpremise based cluster that autoscales (on AWS) when lack of resources"
I could find "EKS anywhere" solution but since price is too high, I want to build similar architecture.
need advice for any use cases for ingress controller, or (physical) loadbalancer, or other architecture that could satisfies those conditions
Cluster API is probably what you need. It is a concept of creating Clusters with Machine objects. These Machine objects are then provisioned using a Provider. This provider can be Bare Metal Operator provider for your bare metal nodes and Cluster API Provider AWS for your AWS nodes. All resting in a single cluster (see the docs below for many other provider types).
You will run a local Kubernetes cluster which will have the Cluster API running in it. This will include components that will allow you to be able to create different Machine objects and tell Kubernetes also how to provision those machines.
Here is some more reading:
Cluster API Book: Excellent reading on the topic.
Documentation for CAPI Provider - AWS.
Documentation for the Bare Metal Operator I worked on this project for a couple of years and the community is pretty amazing. This GitHub repository hosts the CAPI Provider for bare metal nodes.
This should definitely get you going. You can start by running different providers individually to get a taste of how they work and then work with Cluster API and see it in function.
I am a beginner and learning about Kubernetes.
As per my understanding, namespace is a virtual cluster backed by the same physical cluster.
In which usecases do we go for separate physical Kubernetes cluster?
What are the main resources that can be saved by opting for namespace instead of physical Kubernetes cluster? (Kubernetes objects present in one namespace of the physical cluster can be shared by all other namespaces, like the ones in kube-system? And are the nodes in the physical Kubernetes cluster shared by all the namespaces but it is not possible to share nodes between multiple physical Kubernetes clusters?)
A namespace isn't a "virtual cluster" in any meaningful way; it's just a way to group together resources. For instance, these Services are different because they're in different namespaces:
kubectl describe service --namespace n1 foo
kubectl describe service --namespace n2 foo
But a service in n1 can make a call to foo.n2.svc.cluster.local without doing any special setup.
A namespace is a natural boundary for Kubernetes RBAC settings. If one object directly references another (e.g., a pod mounts a persistent volume claim or gets environment variables from a config map) they generally must be in the same namespace.
In an actual cluster, the nodes are shared. A given node can run any pod from any namespace (unless that's specifically configured at the pod level); kubectl describe node will show this. If a pod makes very heavy use of some resource (CPU, memory, disk I/O) this can impact other pods running on the same node. That completely ignores namespace boundaries.
You want different clusters when you want things to actually be separated: when a service in one environment shouldn't be able to call a service in a different environment, when cluster-level resources like NodePort services need to be separated, if you have different policies around things like PersistentVolume allocation.
Sharing a cluster means that you need fewer copies of some cluster-global processes (the Kubernetes core, service meshes like Istio) and you can share nodes. That could result in better utilization of large nodes.
You might, for example, separate your test and production environments into separate clusters. These would have different external-DNS settings, separate ingress controllers, and separate node pools. You couldn't accidentally send requests into the test environment from outside, and a load test on the test environment wouldn't impact the production environment.
Generally a separate physical cluster is necessary
To meet compliance and security standards such as PCI DSS, HIPPA etc.
To provide dedicated physical resources to critical workloads.
To separate different environments such as DEV, TEST, PROD
A multi tenant cluster shared by many tenants using their own namespace is useful for saving cost. Namespace separation is logical where the resources of all namespaces still reside in same ETCD storage but with different keys. This is not a problem in separate dedicated physical cluster because in that case the cluster will have separate ETCD as well.
Access to resources across namespaces is controlled by RBAC via kubernetes API Server. But you can access everything from all namespaces if you get access to ETCD directly bypassing the API Server.
You need to put lot of best practices and protection in a multi tenant cluster so that tenants from different namespaces do not step on each other toes. This not that much necessary in a separate dedicated physical cluster.
We have a requirement to setup a geo redundant cluster. I am looking at sharing an external etcd cluster to run two kubernetes clusters. It may sound absurd at first, but the requirements have come down to it..I am seeking some direction to whether it is possible, and if not, what are the challenges.
Yes it is possible, you can have a single etcd cluster and multiple k8s clusters attached to it. The key to achieve it, is to use -etcd-prefix string flag from kubernetes apiserver. This way each cluster will use different root path for storing its resources and avoid possible conflict with second cluster in the etcd. In addition to it, you should also setup the appropriate rbac rules and certificates for each k8s cluster. You can find more detailed information about it in the following article: Multi-tenant external etcd for Kubernetes clusters.
EDIT: Ooh wait, just noticed that you want to have those two clusters to behave as master-slave. In that case you could achieve it by assign to the slave cluster a read-only role in the etcd and change it to read-write when it has to become master. Theoretically it should work, but I have never tried it and I think the best option is to use builtin k8s mechanism for high-availability like leader-election.
We are working on provisioning our service using Kubernetes and the service needs to register/unregister some data for scaling purposes. Let's say the service handles long-held transactions so when it starts/scales out, it needs to store the starting and ending transaction ids somewhere. When it scales out further, it will need to find the next transaction id and save it with the ending transaction id that is covered. When it scales in, it needs to delete the transaction ids, etc. ETCD seems to make the cut as it is used (by Kubernetes) to store deployment data and not only that it is close to Kubernetes, it is actually inside and maintained by Kubernetes; thus we'd like to find out if that is open for our use. I'd like to ask the question for both EKS, AKS, and self-installed. Any advice welcome. Thanks.
Do not use the kubernetes etcd directly for an application.
Access to read/write data to the kubernetes etcd store is root access to every node in your cluster. Even if you are well versed in etcd v3's role based security model avoid sharing that specific etcd instance so you don't increase your clusters attack surface.
For EKS and GKE, the etcd cluster is hidden in the provided cluster service so you can't break things. I would assume AKS takes a similar approach unless they expose the instances to you that run the management nodes.
If the data is small and not heavily updated, you might be able to reuse the kubernetes etcd store via the kubernetes API. Create a ConfigMap or a custom resource definition for your data and edit it via the easily securable and namespaced functionality in the kubernetes API.
For most application uses run your own etcd cluster (or whatever service) to keep Kubernetes free to do it's workload scheduling. The coreos etcd operator will let you define and create new etcd clusters easily.
I wish to know how to create a GCP Kubernetes Engine cluster spanning two regions. For instance, a cluster has some instances at "us-west1" region, and others at "us-central1" region.
My use case is to verify "failure-domain.beta.kubernetes.io/region" topology key is working as expected. I am aware of:
1. cluster federation: not supported yet for Kubernetes Engine
2. multi-cluster ingress: in development, but may not something I am looking for
3. regional cluster: not applicable as it focuses on replication in only one region
I am aware that my use case is not atypical.
It is possible, but I cannot say that will be a stable and fully functional configuration.
There are no standard tools to do what you want, but you can connect external nodes to your cluster from a different region manually. It will not work with kubeadm, but if you will setup kubelet manually - it will work, but with many limitations:
No auto-updates.
You should manage the connection between regions manually (you should have a private network with direct routing within all your nodes).
You can have problems with logs, monitoring, load balancing, etc.
You will pay for the traffic between internal and external nodes as for the external traffic.
Finally, although it is possible, I cannot recommend you to use it. If you really want to get a multi-region cluster - setup it yourself by kubeadm and use kubefed to create a federation.