I am debugging certain behavior from my application pods; i am launching on K8s cluster. In order to do that I am increasing logging by increasing verbosity of deployment by adding --v=N flag to Kubectl create deployment command.
my question is : how can i configure increased verbosity globally so all pods start reporting increased verbosity; including pods in kube-system name space.
i would prefer if it can be done without re-starting k8s cluster; but if there is no other way I can re-start.
thanks
Ankit
For your applications, there is nothing global as that is not something that has global meaning. You would have to add the appropriate config file settings, env vars, or cli options for whatever you are using.
For kubernetes itself, you can turn up the logging on the kubelet command line, but the defaults are already pretty verbose so I’m not sure you really want to do that unless you’re developing changes for kubernetes.
Related
I'm using the openshift playground. I deploy a sample application, and export the yaml for the pod.
While trying to edit some of the fields I ran across this message
Forbidden: unsafe sysctl "kernel.msgmax" is not allowed
Searching around the link https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#listing-all-sysctl-parameters describes how some parameters are labelled unsafe and cannot be changed but the safe ones can
But even the safe sysctls throw error,
spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations
Is it the playground environment that is limiting changes to kernel parameters? Would I need to have my own minikube installation to enable changing the unsafe sysctl parameters?
Apart from the minikube/kubelet alternatives given to edit/enable unsafe sysctls, is there a different way? What would be a good way to customize kernel parameters for a pod?
The safe sysctls throwing that error is expected behavior. What you need to do is delete the pod before applying the edited yaml to the cluster.You can also avoid this error if you use a deployment instead of a pod directly.
Please read thoroughly this documentation section. Everything is clearly explained there.
As to setting Unsafe Sysctls, you need to additionally enable them on node-level:
All safe sysctls are enabled by default.
All unsafe sysctls are disabled by default and must be allowed
manually by the cluster admin on a per-node basis. Pods with disabled
unsafe sysctls will be scheduled, but will fail to launch.
With the warning above in mind, the cluster admin can allow certain
unsafe sysctls for very special situations such as high-performance or real-time application tuning. Unsafe sysctls are enabled on a
node-by-node basis with a flag of the kubelet; for example:
kubelet --allowed-unsafe-sysctls \
'kernel.msg*,net.core.somaxconn' ...
For Minikube, this can be done via the extra-config flag:
minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"...
Only namespaced sysctls can be enabled this way.
As to...
But even the safe sysctls throw error,
spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations
This is completely different error message and it has nothing to do with restrictions related to changing sysctls in your Pod definition. Note that you cannot change majority of your Pod specification via kubectl edit apart from just a few exceptions listed in the message above. Specifically you cannot change them without recreating your Pod so in this case instead of editing it you can simply run:
kubectl get pod pod-name -o yaml > my-pod.yaml
Then you can edit your required Pod spec fields, and redeploy it:
kubectl apply -f my-pod.yaml
Alternatively you may edit your Deployment as #Arghya Sadhu already suggested in his answer. Deployment controller will recreate those Pods for you with updated specification.
Is it the playground environment that is limiting changes to kernel
parameters? Would I need to have my own minikube installation to
enable changing the unsafe sysctl parameters?
Not really. You can enable them on every node which is part of your cluster by re-configuring your kubelets. As to changing kubelet configuration, it might be done differently depending on your kubernetes installation. In case it was created with kubeadm you just need to edit the following file:
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
then run:
sudo systemctl daemon-reload
and restart your kubelet by running:
sudo systemctl restart kubelet.service
Apart from the minikube/kubelet alternatives given to edit/enable
unsafe sysctls, is there a different way? What would be a good way to
customize kernel parameters for a pod?
Answered above.
I hope it clarified your doubts about setting both safe and unsafe sysctls in a Kubernetes Cluster.
I'm fairly new to Kubernetes and trying to wrap my head around how to manage ComponentConfigs in already running clusters.
For example:
Recently I initialized a kubeadm cluster in a test environment running Ubuntu. When I did that, I found CoreDNS to be in a CrashLoopBackoff which turned out to be the case because Ubuntu was configured to use systemd-resolved and so the resolv.conf had a loopback resolver configured. After reading the docs for coredns, I found out that a solution for that would be to change the resolvConf parameter for kubelet - either via commandline arguments or in the config.
So how would one do this properly in a kubeadm-managed cluster?
Reading [this page in the documentation][1] I didn't really get a clue, because it seems to be tailored to the case of initializing a new cluster or joining new nodes.
Of course, in this particular situation I could just use "Kubeadm reset" and initialize it again with a --config parameter but that doesn't seem to be the right solution for a running cluster.
So after digging a bit deeper I found several infos:
I could change the /var/lib/kubelet/kubeadm-flags.env on the node directly, but AFAICT this only makes sense for node-specific changes.
There is a ConfigMap in the kube-system namespace named kubelet-config-1.14. This seems promising for upcoming nodes joining the cluster to get the right configuration - but would changing that CM affect the already running Kubelet?
There is a marshalled version of the running config in /var/lib/config/kubelet.yaml that I could change, but AFAIU this would be overriden by kubelet itself periodically (?) or at least during a kubeadm upgrade.
There seems to be an option to specify a configmap in the node object, to let kubelet dynamically load the configuration from there, but given that there is already an existing configmap it seems more sensible to change that one.
I seemingly had success by some combination of changing aforementioned CM, running kubeadm upgrade something afterwards and rebooting the machine (since restarting the kubelet did not fix the CoreDNS issue ... but maybe I was to impatient).
So I am now asking:
What is the recommended way to carry out changes to the kubelet configuration (or any other configuration I could affect via kubeadm-config.yaml) that works and is upgrade-safe for cases where the configuration is not node-specific?
And if this involves running kubeadm ... config --config - how do I extract the existing Kubeadm-config in a way that I can feed it back to to kubeadm?
I am entirely happy with pointers to the right documentation, I just didn't find the right clues myself.
TIA
What you are looking for is well described in official documentation.
The basic workflow for configuring a Kubelet is as follows:
Write a YAML or JSON configuration file containing the Kubelet’s configuration.
Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
Update the Kubelet’s corresponding Node object to use this ConfigMap.
In addition there is DynamicKubeletConfig Feature Gate is enabled by default starting from Kubernetes v1.11, but you need some additional steps to activate it. You need to remember about, that Kubelet’s --dynamic-config-dir flag must be set to a writable directory on the Node.
I have a kubernetes-based application that uses an operator to build and deploy containers in pods. Sometimes I'd like to run containers in privileged mode to enable performance tracing, but since I'm not deploying the pod/containers directly from a manifest, I cannot simply add privileged mode and the debugfs filesystem mount.
That leaves me to fork the operator code, change where it builds the container spec, and redeploy with the modified operator. Doable, but awkward.
So my question is, is it possible to impose additional attributes to be added to container specs based on some clusterwide setting, either before pods are deployed by the operator? Or to modify the container spec after deployment? I tried that with kubectl edit pod mypod, but that didn't work.
This is on a physical cluster installed with kubespray.
There are three things to consider:
Your operator can create a controller (e.g. Deployment) instead of Pod, which allows modifications in the Pod Spec area, thus triggering Deployment's rollout (see rolling update strategy).
Use MutatingAdmissionWebhook
so before creating the Pod, its manifest would be modified/overwritten on the fly.
More info regarding MutatingAdmissionWebhook can be found here and here.
A workaround solution in a form of modifying the supply spec -> swapping the pod-a.
More about this was discussed here.
Please let me know if any of the above helped.
I have a running GKE cluster with cockroachDB active. It's been running for quite a while and I don't want to reinitialize it from scratch - it uses the (almost) standard cockroachDB supplied yaml file to start. I need to change a switch in the exec line to modify the logging level -- currently it's set to the below (but that is logging all information messages as well as errors)
exec /cockroach/cockroach start --logtostderr --insecure --advertise-host $(hostname -f) --http-host 0.0.0.0 --join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroa
chdb-2.cockroachdb --cache 25% --max-sql-memory 25%"
How do I do this without completely stopping the DB?
Kubernetes allows you to update StatefulSets in a rolling manner, such that only one pod is brought down at a time.
The simplest way to make changes is to run kubectl edit statefulset cockroachdb. This will open up a text editor in which you can make the desired change to the command, then save and exit. After that, Kubernetes should handle replacing the pods one-by-one with new pods that use the new command.
For more information:
https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html#step-10-upgrade-the-cluster
https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets
https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#in-place-updates-of-resources
I need add argument --authentication-token-webhook in Kubelet. I can change file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf at all nodes step by step with my hands. But it is not funny )). How can I change Kubelet arguments from single point?
You can either
configure your Kubernetes workers via tools like Puppet or Ansible. Write your service drop-in once and deploy it via the tool to all nodes. Make sure you don't restart all kubelets at once (keyword serial for Ansible). Also, don't change 10-kubeadm.conf, drop in another file like 20-kubeadm-extra-args.conf and set the environment variable KUBELET_EXTRA_ARGS.
or use a Kubernetes feature called DynamicKubeletConfig. Beware that this is an alpha feature (as of Kubernetes 1.10) and has to be enabled by hand. I wouldn't recommend this method (yet, as long as it's an alpha feature), but it might become a feasible option in the future.