I have followed this blog to set up open5GS on AWS: https://aws.amazon.com/blogs/opensource/open-source-mobile-core-network-implementation-on-amazon-elastic-kubernetes-service/
I've set up the infrastructure using open5gs-infra.yaml
I've configured the bastion host and run step 5 properly (by providing the correct ARN value)
I've initialised the DocumentDB
I updated the CoreDNS configmap and restarted coredns pods
I then ran the cloudformation yaml file for the creation of the worker node group. However, the workernode group doesn't join the cluster. I've double-checked the parameters that I feed to the cloudformation template. I've even tried to edit the authConfig manually after the worker node group has been created so that the worker nodes can join the cluster. But that doesn't work.
Since there are no worker nodes, the pods can't be scheduled and the cluster is non-usable. What can I do so that the worker node group joins the cluster?
Below debugging steps helped me resolve it.
Verify if SecurityGroups allow connectivity bw API server and worker nodes using "Run Reachability Analyzer" (here
Add required policies to IAM role eksAdminRoleForLambda-v5G-Core (here
AmazonEKSWorkerNodePolicy
AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly
Run "TroubleshootEKSWorkerNode" (here
Related
I would like to run an OpenDJ cluster in my kubernetes cluster. In order to do so I used this procedure
I've created a StetefulSet, service and storageclass.
In order to initialize the replication, I need to run several commands dsreplication enable and dsreplication initialize-all inside one of the StatefulSet replicas because when I try running those commands through a kubernetes job as a separated pod it sends the following error - The local instance is not configured or you do not have permissions to access it as the server has not initialized as one of the opendj cluster's replicas (Which has the initial command that launch the opendj server).
How would you run those commands from the statefulsets pods? (When I keep thinking about scaling - how will I join new pods to the cluster when they'll launch with the hpa?)
Or maybe the better question is how can I run commands on my cluster from a remote pod?
Thanks.
When using eksctl to create Kubernetes cluster using AWS EKS, the process get stuck waiting for the nodes to join the cluster:
nodegroup "my-cluster" has 0 node(s)
waiting for at least 3 node(s) to become ready in “my-cluster”
timed out (after 25m0s) waiting for at least 3 nodes to join the cluster and become ready in "my-cluster"
The message is displayed, without any additional logs, until the process eventually times out. It looks like behind the scenes, the newly created nodes are unable to communicate with the Kubernetes cluster
When using an existing VPC network, you have to make sure that the VPC conforms with all EKS-specific requirements [1, 2]. The blog post by logz.io provides detailed guidance on setting up a VPC network, as well as an example AWS Cloud Formation template that you can use as the basis [3].
Missing IAM Policies
The AmazonEKSWorkerNodePolicy and AmazonEKS_CNI_Policy policies [4] are required by the EKS worker nodes to be able to communicate with the cluster.
By default, eksctl automatically generates a role containing these policies. However, when you use “attachPolicyARNs” property to attach specific policies by ARN, you have to include these policies explicitly [5]:
nodeGroups:
- name: my-special-nodegroup
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
[1] https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html
[2] https://eksctl.io/usage/vpc-networking
[3] https://logz.io/blog/amazon-eks
[4] https://docs.aws.amazon.com/eks/latest/userguide/worker_node_IAM_role.html
5] https://eksctl.io/usage/iam-policies/
I face the same issue and found the real cause of this issue.
The issue is with the created VPC. I have also created the VPC and subnets but I have not created the route table and internet gate which was causing the issue.
Issue got resolved,once mapped the route table and internet gateway.
Novice to provisioning resources & configuration for kubernetes setup.
Registered a domain & subdomain with GoDaddy.
Installed kops & kubectl in EC2 instance.
Created a bucket for this cluster to store configuration
I ran below command on AWS EC2 instance:
kops create cluster --cloud=aws --zones=us-east-1b \
—dns-zone=subdomain.domain-aws.com \
—name=cluster1.subdomain.domain-aws.com --yes
My understanding is:
1) kops performs provisioning of AWS services(resources) for a two node cluster. VPC, subnets, auto-scaling, EC2(hosting master & nodes) etc...
2) kops creates the configuration file of this provisioning & stores it in KV cluster store of master
3) kops configure cluster store, apiserver, controller, scheduler installations in master
4) kops configures kubelet, container engine & kube-proxy on each node.
kubectl is a client for apiserver, that talks to API server and performs tasks to create PODS, get information about cluster etc..
Questions
1) How does kops remember the state of provisioning & cluster configuration, if I need to later upgrade & configure the cluster? state maintenance...
2) How to view the current cluster provisioning & configuration, done by kops? as a configuration file..
1) kops performs provisioning of AWS services(resources) for a two node cluster. VPC, subnets, auto-scaling, EC2(hosting master & nodes) etc...
This is generally correct. A couple points: it doesn't have to be a two node cluster, for instance you can specify the desired number of nodes with the --node-count; also, it doesn't directly create any EC2 VMs for you, it creates autoscaling groups (ASGs) and those ASGs are "responsible" for creating and maintaining the desired number of master and worker VMs.
2) kops creates the configuration file of this provisioning & stores it in KV cluster store of master
Unless this behaviour has changed recently, no, the configuration settings are stored in bucket you created to store configuration. See these docs.
3) kops configure cluster store, apiserver, controller, scheduler installations in master
4) kops configures kubelet, container engine & kube-proxy on each node.
As above, unless this changed recently, kops does not install the cluster store on the master. It does configure apiserver and scheduler installations on the master nodes. You can see more of what gets installed on masters and workers, and how these are installed, in these docs
Based on those docs, looks like the masters have the following:
kube-apiserver
kube-controller-manager (which runs miscellaneous controllers)
kube-scheduler (which assigns pods to nodes)
etcd
dns-controller
and on workers:
kube-proxy (which configures iptables so that the k8s-network will work)
kubelet
Docker
and perhaps Docker can be swapped out for other container engines?
kubectl is a client for apiserver, that talks to API server and performs tasks to create PODS, get information about cluster etc..
Yes.
1) How does kops remember the state of provisioning & cluster configuration, if I need to later upgrade & configure the cluster? state maintenance...
That is stored in the "state store" which is typically an S3 bucket, like the one you said you provisioned to store configuration. You can look at the kops docs for how to do things like perform upgrades, change configuration, scale up the Kubernetes cluster, etc. You don't need to edit the files in the buckets directly, kops has commands for most of those operations.
2) How to view the current cluster provisioning & configuration, done by kops? as a configuration file..
These docs have some info. For example, you could run:
kops get cluster1.subdomain.domain-aws.com
In general, I think searching and reading the docs would be very helpful for you in answering these questions and future questions you may have in understanding and using kops.
I am using kubernete to manage docker cluster. Right now, I can set up POD autoscale using Horizontal Pod Scaler, that is fine.
And now I think the next step is to autoscale nodes. I think for HPA, the auto-created pod is only started in the already created nodes, but if all the available nodes are utilized and no available resource for any more pods, I think the next step is to automatically create node and have node join the k8s master.
I googled a lot and there are very limited resources to introduce this topic.
Can anyone please point me to any resource how to implement this requirement.
Thanks
One way to do using AWS and setting up your own Kubernetes cluster is by following these steps :
Create an Instance greater than t2.micro (will be master node).
Initialize the Kubernetes cluster using some tools like Kubeadm. After the initialisation would be completed you would get a join command, which needs to e run on all the nodes who want to join the cluster. (Here is the link)
Now create an Autoscaling Group on AWS with start/boot script containing that join command.
Now whenever the utilisation specified by you in autoscaling group is breached the scaling would happen and the node(s) would automatically join the Kubernetes cluster. This would allow the Kubernetes to schedule pods on the newly joined nodes based on the HPA.
(I would suggest to use Flannel as pod network as it automatically removes the node from Kubernetes cluster when it is not available)
kubernetes operations (kops) helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line.
Features:
Automates the provisioning of Kubernetes clusters in AWS and GCE
Deploys Highly Available (HA) Kubernetes Masters
Most of the managed kubernetes service providers provide auto scaling feature of the nodes
Elastic Kubernetes Service EKS- configure cluster auto scalar
Google Kubernetes Engine
GKE Auto Scalar
Auto scaling feature needs to be supported by the underlying cloud provider. Google cloud supports auto scaling during cluster creation or update by passing flags --enable-autoscaling --min-nodes and --max-nodes to the corresponding gcloud commands.
Examples:
gcloud container clusters create mytestcluster --zone=us-central1-b --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
gcloud container clusters update mytestcluster --enable-autoscaling --min-nodes=1 --max-nodes=15
below link would be helpful
https://medium.com/kubecost/understanding-kubernetes-cluster-autoscaling-675099a1db92
From this article(https://cloudplatform.googleblog.com/2018/06/Kubernetes-best-practices-upgrading-your-clusters-with-zero-downtime.html) I learnt that it is possible to create a new node pool, and cordon and drain old nodes one by one, so that workloads get re-scheduled to new nodes in the new pool.
To me, a new node pool seems to indicate a new cluster. The reason: we have two node pools in GKE, and they're listed as two separate clusters.
My question is: after the pods under a service get moved to a new node, if that service is being called from other pods in the old node, will this inter-cluster service call fail?
You don't create a new cluster per se. You upgrade the master(s) and then you create a new node pool with nodes that have a newer version. Make sure the new node pool shares the same network as the original node pool.
If you have a service with one replica (one pod) if that pod is living in one of the nodes you are upgrading you need to allow time for Kubernetes to create a new replica on a different node that is not being upgraded. For that time, your service will be unavailable.
If you have a service with multiple replicas chances are that you won't see any downtime unless for some odd reason all your replicas are scheduled on the same node.
Recommendation: scale your resources which serve your services (Deployments, DaemonSets, StatefulSets, etc) by one or two replicas before doing node upgrades.
StatefulSet tip: You will have some write downtime if you are running something like mysql in a master-slave config when you reschedule your mysql master.
Note that creating a new node Pool does not create a new cluster. You can have multiple node pools within the same cluster. Workloads within the different node pools will still interact with each other since they are in the same cluster.
gcloud container node-pools create (the command to create node pools) requires that you specify the --cluster flag so that the new node pool is created within an existing cluster.
So to answer the question directly, following the steps from that Google link will not cause any service interruption nor will there be any issues with pods from the same cluster communicating with each other during your migration.