How to set node allocatable computation on kubernetes? - kubernetes

I'm reading the Reserve Compute Resources for System Daemons task in Kubernetes docs and it briefly explains how to allocate a compute resource to a node using kubelet command and flags --kube-reserved, --system-reserved and --eviction-hard.
I'm learning on Minikube for masOS and as far as I got, minikube is configured to use command kubectl along with minikube command.
For local learning purposes on minikube I don't need to have it set (maybe it can't be done on minikube) but
How this could be done let's say in K8's development environment on a node?

This could be be done by:
1. Passing config file during cluster initialization or initilize kubelet with additional parameters via config file,
For cluster initialization using config file it should contains at least:
kind: InitConfiguration
kind: ClusterConfiguration
additional configuratuion types like:
kind: KubeletConfiguration
In order to get basic config file you can use kubeadm config print init-defaults
2. For the live cluster please consider reconfigure current cluster using steps "Generate the configuration file" and "Push the configuration file to the control plane" like described in "Reconfigure a Node's Kubelet in a Live Cluster"
3. I didn't test it but for minikube - please take a look here:
Note:
Minikube has a “configurator” feature that allows users to configure the Kubernetes components with arbitrary values. To use this feature, you can use the --extra-config flag on the minikube start command.
This flag is repeated, so you can pass it several times with several different values to set multiple options.
This flag takes a string of the form component.key=value, where component is one of the strings from the below list, key is a value on the configuration struct and value is the value to set.
Valid keys can be found by examining the documentation for the Kubernetes componentconfigs for each component. Here is the documentation for each supported configuration:
kubelet
apiserver
proxy
controller-manager
etcd
scheduler
Hope this helped:
Additional community resources:
Memory usage in kubernetes cluster

Related

minikube start - howto modify KubeletConfiguration passed to kubeadm?

I would like to set the value KubeletConfiguration.cpuCFSQuota = false in the config.yaml passed to kubeadm when launching minikube to turn off CPU resource checking, but I have not managed to find the options to do this through the documentation here https://minikube.sigs.k8s.io/docs/handbook/config/ . The closest solution I have found is to use the option --extra-config=kubelet.cpu-cfs-quota=false but the --cpu-cfs-quota option for the kubelet has been deprecated and no longer has an effect.
Any ideas appreciated.
Environment:
Ubuntu 20.04
Minikube 1.17.1
Kubernetes 1.20.2
Driver docker (20.10.2)
Thanks,
Piers.
Using the --extra-config=kubelet. flag alongside minikube start is a good approach but you would also need to Set Kubelet parameters via a config file.
As you already noticed the --cpu-cfs-quota flag:
Enable CPU CFS quota enforcement for containers that specify CPU
limits (DEPRECATED: This parameter should be set via the config file
specified by the Kubelet's --config flag.
So you need to set that parameter by creating a kubelet config file:
The configuration file must be a JSON or YAML representation of the
parameters in this struct. Make sure the Kubelet has read permissions
on the file.
Here is an example of what this file might look like:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
memory.available: "200Mi"
Now you can use that config file to set cpuCFSQuota = false:
// cpuCFSQuota enables CPU CFS quota enforcement for containers that
// specify CPU limits.
// Dynamic Kubelet Config (beta): If dynamically updating this field, consider that
// disabling it may reduce node stability.
// Default: true
// +optional`
CPUCFSQuota *bool `json:"cpuCFSQuota,omitempty"
and than call minikube with --extra-config=kubelet.config=/path/to/config.yaml
Alternately, you can start your minikube without the --extra-config flag and than start the Kubelet with the --config flag set to the path of the Kubelet's config file. The Kubelet will then load its config from this file.
I know these are a few steps more than you expected but setting the kubelet parameters via a config file is the recommended approach because it simplifies node deployment and configuration management.

How to change kubelet configuration via kubeadm

I'm fairly new to Kubernetes and trying to wrap my head around how to manage ComponentConfigs in already running clusters.
For example:
Recently I initialized a kubeadm cluster in a test environment running Ubuntu. When I did that, I found CoreDNS to be in a CrashLoopBackoff which turned out to be the case because Ubuntu was configured to use systemd-resolved and so the resolv.conf had a loopback resolver configured. After reading the docs for coredns, I found out that a solution for that would be to change the resolvConf parameter for kubelet - either via commandline arguments or in the config.
So how would one do this properly in a kubeadm-managed cluster?
Reading [this page in the documentation][1] I didn't really get a clue, because it seems to be tailored to the case of initializing a new cluster or joining new nodes.
Of course, in this particular situation I could just use "Kubeadm reset" and initialize it again with a --config parameter but that doesn't seem to be the right solution for a running cluster.
So after digging a bit deeper I found several infos:
I could change the /var/lib/kubelet/kubeadm-flags.env on the node directly, but AFAICT this only makes sense for node-specific changes.
There is a ConfigMap in the kube-system namespace named kubelet-config-1.14. This seems promising for upcoming nodes joining the cluster to get the right configuration - but would changing that CM affect the already running Kubelet?
There is a marshalled version of the running config in /var/lib/config/kubelet.yaml that I could change, but AFAIU this would be overriden by kubelet itself periodically (?) or at least during a kubeadm upgrade.
There seems to be an option to specify a configmap in the node object, to let kubelet dynamically load the configuration from there, but given that there is already an existing configmap it seems more sensible to change that one.
I seemingly had success by some combination of changing aforementioned CM, running kubeadm upgrade something afterwards and rebooting the machine (since restarting the kubelet did not fix the CoreDNS issue ... but maybe I was to impatient).
So I am now asking:
What is the recommended way to carry out changes to the kubelet configuration (or any other configuration I could affect via kubeadm-config.yaml) that works and is upgrade-safe for cases where the configuration is not node-specific?
And if this involves running kubeadm ... config --config - how do I extract the existing Kubeadm-config in a way that I can feed it back to to kubeadm?
I am entirely happy with pointers to the right documentation, I just didn't find the right clues myself.
TIA
What you are looking for is well described in official documentation.
The basic workflow for configuring a Kubelet is as follows:
Write a YAML or JSON configuration file containing the Kubelet’s configuration.
Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
Update the Kubelet’s corresponding Node object to use this ConfigMap.
In addition there is DynamicKubeletConfig Feature Gate is enabled by default starting from Kubernetes v1.11, but you need some additional steps to activate it. You need to remember about, that Kubelet’s --dynamic-config-dir flag must be set to a writable directory on the Node.

How to change --horizontal-pod-autoscaler-sync-period field in kube-controller-manager to 5sec in gke

I am trying to set up an horizontal pod auto scaling in GKE. No proper documentation found to reduce the --horizontal-pod-autoscaler-sync-period to 5 sec using kube-controller-manager.
In the below link it says there is a possibility of changing the flags:
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
Is there any proper implementation steps to this?
You are not able do this on GKE, EKS and other managed clusters.
In order to change/add flags in kube-controller-manager - you should have access to your /etc/kubernetes/manifests/ directory on master node and be able to modify parameters in /etc/kubernetes/manifests/kube-controller-manager.yaml.
GKE, EKS and other clusters manages only by their providers without getting you permissions to have access to master nodes.
But you can create cluster with kubeadm init and configure/change as you like.
you can stop your minikube cluster and start it with your extra configs ...
minikube start --extra-config 'controller-manager.horizontal-pod-autoscaler-sync-period=5s'
for more details, you can go through https://minikube.sigs.k8s.io/docs/handbook/config/#modifying-kubernetes-defaults

How to add flag to Kubernetes controller manager

I'm new to K8s. In process to config Openstack Cinder as K8s StorageClass, i have to add some flags to my kube controller manager, and I found that it's my big problem.
I'm using K8s 1.11 in VMs, and my K8s cluster has a kube-controller-manager pod, but I don't know how to add these flags to my kube-controller-manager.
After hours search, i found that there's a lot of task require add flag to kube-controller-manager, but no exactly document guide me how to do that. Please share me the way to go over it.
Thank you.
You can check /etc/kubernetes/manifests dir on your master nodes.
This dir would contain yaml files for master components.
These are also known as static pods.
More Info : https://kubernetes.io/docs/tasks/administer-cluster/static-pod/
Update these files and you would be able to see your changes as kubelet should restart the pod on file change.
As a more long term solution, you will need to incorporate the flags to the tooling that you use to generate your k8s cluster.

How to change fluentd config for GKE-managed logging agent?

I have a container cluster in Google Container Engine with Stackdriver logging agent enabled. It is correctly pulling stdout logs from my containers. Now I would like to change the fluentd config to specify a log parser so that the logs shown in the GCP Logging view will have the correct severity and component.
Following this Stackdriver logging guide from kubernetes.io, I have attempted to:
Get the fluentd ConfigMap as a yml file
Added a new <filter> according to my log4js log format
Created a new ConfigMap named fluentd-cm-2 in kube-system namespace
Edited the DaemonSet for fluentd and set its ConfigMap to fluentd-cm-2. I did this using kubectl edit ds instead of kubectl replace -f because the latter failed with an error message: "the object has been modified", even after getting a fresh copy of the DaemonSet yaml.
Unexpected result: The DaemonSet is restarted, but its configuration is reverted back to the original ConfigMap, so my changes did not take effect.
I have also tried editing the ConfigMap directly (kubectl edit cm fluentd-gcp-config-v1.1 --namespace kube-system) and saved it, but it was also reverted.
I noticed that the DaemonSet and ConfigMap for fluentd are tagged with addonmanager.kubernetes.io/mode: Reconcile. I would conclude that GKE has overwritten my settings because of this "reconcile" mode.
So, my question is: how can I change the fluentd configuration in a Google Container Engine cluster, when the logging agent was installed by GKE on cluster provisioning?
Please take a look at the Prerequisites section on the documentation page you mentioned. It's mentioned there, that on GKE you cannot change the default Stackdriver Logging integration. The reason is that GKE maintains this configuration: updates the agent, watches its health and so on. It's not possible to provide the same level of support for all possible configurations.
However, you can always disable the default integration and deploy your own, patched version of DaemonSet. You can find out how to disable the default integration in the GKE documentation:
gcloud beta container clusters update [CLUSTER-NAME] \
--logging-service=none
Note, that after you disabled the default integration, you have to maintain the new deployment yourself: update the agent, set the resources, watch its health.
Here is a solution for using your own fluentd daemonset that is very much like the one included with GKE.
https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd