Novice to provisioning resources & configuration for kubernetes setup.
Registered a domain & subdomain with GoDaddy.
Installed kops & kubectl in EC2 instance.
Created a bucket for this cluster to store configuration
I ran below command on AWS EC2 instance:
kops create cluster --cloud=aws --zones=us-east-1b \
—dns-zone=subdomain.domain-aws.com \
—name=cluster1.subdomain.domain-aws.com --yes
My understanding is:
1) kops performs provisioning of AWS services(resources) for a two node cluster. VPC, subnets, auto-scaling, EC2(hosting master & nodes) etc...
2) kops creates the configuration file of this provisioning & stores it in KV cluster store of master
3) kops configure cluster store, apiserver, controller, scheduler installations in master
4) kops configures kubelet, container engine & kube-proxy on each node.
kubectl is a client for apiserver, that talks to API server and performs tasks to create PODS, get information about cluster etc..
Questions
1) How does kops remember the state of provisioning & cluster configuration, if I need to later upgrade & configure the cluster? state maintenance...
2) How to view the current cluster provisioning & configuration, done by kops? as a configuration file..
1) kops performs provisioning of AWS services(resources) for a two node cluster. VPC, subnets, auto-scaling, EC2(hosting master & nodes) etc...
This is generally correct. A couple points: it doesn't have to be a two node cluster, for instance you can specify the desired number of nodes with the --node-count; also, it doesn't directly create any EC2 VMs for you, it creates autoscaling groups (ASGs) and those ASGs are "responsible" for creating and maintaining the desired number of master and worker VMs.
2) kops creates the configuration file of this provisioning & stores it in KV cluster store of master
Unless this behaviour has changed recently, no, the configuration settings are stored in bucket you created to store configuration. See these docs.
3) kops configure cluster store, apiserver, controller, scheduler installations in master
4) kops configures kubelet, container engine & kube-proxy on each node.
As above, unless this changed recently, kops does not install the cluster store on the master. It does configure apiserver and scheduler installations on the master nodes. You can see more of what gets installed on masters and workers, and how these are installed, in these docs
Based on those docs, looks like the masters have the following:
kube-apiserver
kube-controller-manager (which runs miscellaneous controllers)
kube-scheduler (which assigns pods to nodes)
etcd
dns-controller
and on workers:
kube-proxy (which configures iptables so that the k8s-network will work)
kubelet
Docker
and perhaps Docker can be swapped out for other container engines?
kubectl is a client for apiserver, that talks to API server and performs tasks to create PODS, get information about cluster etc..
Yes.
1) How does kops remember the state of provisioning & cluster configuration, if I need to later upgrade & configure the cluster? state maintenance...
That is stored in the "state store" which is typically an S3 bucket, like the one you said you provisioned to store configuration. You can look at the kops docs for how to do things like perform upgrades, change configuration, scale up the Kubernetes cluster, etc. You don't need to edit the files in the buckets directly, kops has commands for most of those operations.
2) How to view the current cluster provisioning & configuration, done by kops? as a configuration file..
These docs have some info. For example, you could run:
kops get cluster1.subdomain.domain-aws.com
In general, I think searching and reading the docs would be very helpful for you in answering these questions and future questions you may have in understanding and using kops.
Related
To read secret from etcd in AKS Cluster, Used below command
ETCDCTL_API=3 etcdctl --endpoints=<endpoint> --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key get / --prefix --keys-only
Error: open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory.
Where the certificates will be stored by default?
refered the doc https://docs.starlingx.io/security/kubernetes/etcd-certificates-c1fc943e4a9c.html for certification path.
It seems to me that you're having the wrong image about AKS (and Managed Kubernetes solutions in general).
Basically:
Managed Kubernetes solutions (like AKS, GKE EKS) are having some of the cluster components abstracted from the user (meaning you won't be able to access them).
Kubernetes clusters that are not managed by a cloud provider (like on-premise) are giving the user access to pretty much everything.
Above bullet points were only to narrow down the issue. There are a lot of differences between cloud-managed and self-managed solutions and I encourage you to check them out.
Example reference:
Serverfault.com: What is the point of running self managed Kubernetes cluster
In short terms:
You will not get the access to etcd on AKS.
You won't find the etcd certificates on your VM or Azure Cloud Shell
Citing official Microsoft documentation:
Control plane
When you create an AKS cluster, a control plane is automatically created and configured. This control plane is provided at no cost as a managed Azure resource abstracted from the user. You only pay for the nodes attached to the AKS cluster. The control plane and its resources reside only on the region where you created the cluster.
The control plane includes the following core Kubernetes components:
Component
Description >
kube-apiserver
The API server is how the underlying Kubernetes APIs are exposed. This component provides the interaction for management tools, such as kubectl or the Kubernetes dashboard.
etcd
To maintain the state of your Kubernetes cluster and configuration, the highly available etcd is a key value store within Kubernetes.
kube-scheduler
When you create or scale applications, the Scheduler determines what nodes can run the workload and starts them.
kube-controller-manager
The Controller Manager oversees a number of smaller Controllers that perform actions such as replicating pods and handling node operations.
AKS provides a single-tenant control plane, with a dedicated API server, scheduler, etc. You define the number and size of the nodes, and the Azure platform configures the secure communication between the control plane and nodes. Interaction with the control plane occurs through Kubernetes APIs, such as kubectl or the Kubernetes dashboard.
While you don't need to configure components (like a highly available etcd store) with this managed control plane, you can't access the control plane directly. Kubernetes control plane and node upgrades are orchestrated through the Azure CLI or Azure portal. To troubleshoot possible issues, you can review the control plane logs through Azure Monitor logs.
To configure or directly access a control plane, deploy a self-managed Kubernetes cluster using Cluster API Provider Azure.
-- Docs.microsoft.com: Azure: AKS: Concepts clusters workloads: Control plane
Well, I read the user guide of AWS EKS service. I created a managed node group for the EKS cluster successfully.
I don't know how to add the nodes running on my machine to the EKS cluster. I don't know whether EKS support. I didn't find any clue in its document. I read the 'self-managed node group' chapter, which supports add a self-managed EC2 instances and auto-scaling group to the EKS cluster rather than a private node running on other cloud instance like azure, google cloud or my machine.
Does EKS support? How to do that if supports?
This is not possible. It is (implicitly) called out in this page. All worker nodes need to be deployed in the same VPC where you deployed the control plane (not necessarily the same subnets though). EKS Anywhere (to be launched later this year) will allow you to deploy a complete EKS cluster (control plane + workers) outside of an AWS region (but it won't allow running the control plane in AWS and workers locally).
As far as I know, EKS service doesn't support adding self nodes to the cluster. But the 'EKS Anywhere' service does, which has not been online yet, but soon.
I am trying to create Kubernetes cluster having 1 master and 2 worker nodes by using the tool kubeadm in my on-premise machines. I am following the Kubernetes official documentation for forming the cluster from the following url:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
After installing all the runtime and completing before begin pre-requistics steps, I found in the document as the first step of forming the cluster is Create load balancer for kube-apiserver.
My Doubt
When I created the single master 3 worker nodes cluster using kubespray tool, I did not created any separate load balancer for that. So here when I am following the kubeadm tool, Do I need to create the load balancer actually for forming ?
Why are both tools showing different way, Since I did not created load balancer by using kubespray tool. Now I am trying to create cluster with kubeadm tool.
Speaking of load balancers creation during Kubernetes deployment using Kubeadm it depends on your setup. It is not mandatory to setup load balancer. Your cluster will still work, but without load balancing, it's going to be hard to qualify this cluster as HA.
In a single master setup as it is in your case, the master node manages the etcd database, API server, controller manager and scheduler, along with the worker nodes. However, if that single master node fails, all the worker node fail as well and entire cluster will be lost.
Learn more here: kubernetes-ha-kubeadm.
Kubeadm covers the needs of a life-cycle management for Kubernetes clusters, including self-hosted layouts, dynamic discovery services, etc. Kubespray is more about generic configuration, initial clustering, and bootstrapping.
Kubespray is a good choice when you either are familiar with Ansible or seek a possibility to switch between multiple platforms. If your priority is tight integration with unique features offered by the supported clouds, and you plan to stick with your provider, kops may be a better option.
Deploying a loadbalancer is up to a user and is not covered by ansible roles in Kubespray. By default, it only configures a non-HA endpoint, which points to the access_ip or IP address of the first server node in the kube-master group. It can also configure clients to use endpoints for a given loadbalancer type. More information you can find here: kubespray-lb.
Here you have comparision of Kubernetes deployment tools: Kubernetes Deployment Tools.
I am using kubernete to manage docker cluster. Right now, I can set up POD autoscale using Horizontal Pod Scaler, that is fine.
And now I think the next step is to autoscale nodes. I think for HPA, the auto-created pod is only started in the already created nodes, but if all the available nodes are utilized and no available resource for any more pods, I think the next step is to automatically create node and have node join the k8s master.
I googled a lot and there are very limited resources to introduce this topic.
Can anyone please point me to any resource how to implement this requirement.
Thanks
One way to do using AWS and setting up your own Kubernetes cluster is by following these steps :
Create an Instance greater than t2.micro (will be master node).
Initialize the Kubernetes cluster using some tools like Kubeadm. After the initialisation would be completed you would get a join command, which needs to e run on all the nodes who want to join the cluster. (Here is the link)
Now create an Autoscaling Group on AWS with start/boot script containing that join command.
Now whenever the utilisation specified by you in autoscaling group is breached the scaling would happen and the node(s) would automatically join the Kubernetes cluster. This would allow the Kubernetes to schedule pods on the newly joined nodes based on the HPA.
(I would suggest to use Flannel as pod network as it automatically removes the node from Kubernetes cluster when it is not available)
kubernetes operations (kops) helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line.
Features:
Automates the provisioning of Kubernetes clusters in AWS and GCE
Deploys Highly Available (HA) Kubernetes Masters
Most of the managed kubernetes service providers provide auto scaling feature of the nodes
Elastic Kubernetes Service EKS- configure cluster auto scalar
Google Kubernetes Engine
GKE Auto Scalar
Auto scaling feature needs to be supported by the underlying cloud provider. Google cloud supports auto scaling during cluster creation or update by passing flags --enable-autoscaling --min-nodes and --max-nodes to the corresponding gcloud commands.
Examples:
gcloud container clusters create mytestcluster --zone=us-central1-b --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
gcloud container clusters update mytestcluster --enable-autoscaling --min-nodes=1 --max-nodes=15
below link would be helpful
https://medium.com/kubecost/understanding-kubernetes-cluster-autoscaling-675099a1db92
I set up a Kubernetes cluster with a single master node and two worker nodes using kubeadm, and I am trying to figure out how to recover from node failure.
When a worker node fails, recovery is straightforward: I create a new worker node from scratch, run kubeadm join, and everything's fine.
However, I cannot figure out how to recover from master node failure (without interrupting the deployments running on the worker nodes). Do I need to backup and restore the original certificates or can I just run kubeadm init to create a new master from scratch? How do I join the existing worker nodes?
I ended up writing a Kubernetes CronJob backing up the etcd data. If you are interested: I wrote a blog post about it: https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html
In addition to that you may want to backup all of /etc/kubernetes/pki to avoid issues with secrets (tokens) having to be renewed.
For example, kube-proxy uses a secret to store a token and this token becomes invalid if only the etcd certificate is backed up.
As per your mention about Master's backup , actually if you mean backup procedures (like traditional/legacy backups tools/techs) isn't mentioned directly in the official documentation (as i know), but you can take your precautions by some Options/Workarounds :
Setup HA Masters (only for GCE)
Set up High-Availability Kubernetes Masters
Setup HA etcd cluster / Master Load Balancer
Setting-up-an-ha-etcd-cluster
Set up master Load Balancer
Operating etcd clusters for Kubernetes
OS file Systems Snapshot/backup
kubeadm init will definitely not work out of the box, as that will create a new cluster altogether, credentials, ip space, etc.
At a minimum, restoring the master node will require a backup of your etcd data. This typically lives in /var/lib/etcd directory.
You will also need the kubeadm config from the cluster
kubeadm config view should output this. (upward of v1.8)
The step-by-step to restore a master node really isn't so clean cut, which is why they introduce HA - High Availability. This is a much safer way of maintaining redundancy and uptime. Particularly because restoring anything from etcd can be a real pain (in my humble opinion and experience).
If I may go a bit off topic from your question, if you are still getting started with Kubernetes and not deeply invested in kubeadm, i would suggest you consider creating your cluster with kops instead. It supports HA already and I found kops to be more robust and easier to use to either kubeadm and kube-aws (the coreos cluster builder).
https://kubernetes.io/docs/getting-started-guides/kops/