I'm using the openshift playground. I deploy a sample application, and export the yaml for the pod.
While trying to edit some of the fields I ran across this message
Forbidden: unsafe sysctl "kernel.msgmax" is not allowed
Searching around the link https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#listing-all-sysctl-parameters describes how some parameters are labelled unsafe and cannot be changed but the safe ones can
But even the safe sysctls throw error,
spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations
Is it the playground environment that is limiting changes to kernel parameters? Would I need to have my own minikube installation to enable changing the unsafe sysctl parameters?
Apart from the minikube/kubelet alternatives given to edit/enable unsafe sysctls, is there a different way? What would be a good way to customize kernel parameters for a pod?
The safe sysctls throwing that error is expected behavior. What you need to do is delete the pod before applying the edited yaml to the cluster.You can also avoid this error if you use a deployment instead of a pod directly.
Please read thoroughly this documentation section. Everything is clearly explained there.
As to setting Unsafe Sysctls, you need to additionally enable them on node-level:
All safe sysctls are enabled by default.
All unsafe sysctls are disabled by default and must be allowed
manually by the cluster admin on a per-node basis. Pods with disabled
unsafe sysctls will be scheduled, but will fail to launch.
With the warning above in mind, the cluster admin can allow certain
unsafe sysctls for very special situations such as high-performance or real-time application tuning. Unsafe sysctls are enabled on a
node-by-node basis with a flag of the kubelet; for example:
kubelet --allowed-unsafe-sysctls \
'kernel.msg*,net.core.somaxconn' ...
For Minikube, this can be done via the extra-config flag:
minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"...
Only namespaced sysctls can be enabled this way.
As to...
But even the safe sysctls throw error,
spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations
This is completely different error message and it has nothing to do with restrictions related to changing sysctls in your Pod definition. Note that you cannot change majority of your Pod specification via kubectl edit apart from just a few exceptions listed in the message above. Specifically you cannot change them without recreating your Pod so in this case instead of editing it you can simply run:
kubectl get pod pod-name -o yaml > my-pod.yaml
Then you can edit your required Pod spec fields, and redeploy it:
kubectl apply -f my-pod.yaml
Alternatively you may edit your Deployment as #Arghya Sadhu already suggested in his answer. Deployment controller will recreate those Pods for you with updated specification.
Is it the playground environment that is limiting changes to kernel
parameters? Would I need to have my own minikube installation to
enable changing the unsafe sysctl parameters?
Not really. You can enable them on every node which is part of your cluster by re-configuring your kubelets. As to changing kubelet configuration, it might be done differently depending on your kubernetes installation. In case it was created with kubeadm you just need to edit the following file:
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
then run:
sudo systemctl daemon-reload
and restart your kubelet by running:
sudo systemctl restart kubelet.service
Apart from the minikube/kubelet alternatives given to edit/enable
unsafe sysctls, is there a different way? What would be a good way to
customize kernel parameters for a pod?
Answered above.
I hope it clarified your doubts about setting both safe and unsafe sysctls in a Kubernetes Cluster.
Related
I am still bit of kubernetes newbe. But I am looking for a way to give developers controlled access to kubectl exec command . I want to give them run most of the read-only command but prevent some high risk command and also prevent interactive download/install etc. Also want to log all their action during sessions for audit purpose.
I am not seeing any straight forward way to do it using rbac. Also not seeing those options in rancher either. I am looking for some guidance and direction to achieve such capability.
I am sure some of you have achieved it some way.
Kubernetes RBAC can only validate whenever you can or cannot exec into pods, (by checking create verb on pods/exec resource), after that it switches to SPDY protocol and passes your input and returning back output from analog of docker exec on your container runtime, without actually caring about what's going in and out
With rbac you also have to specify pod name, which might be problematic if you are using Deployments, where each new revision will generate a different pod name. Since pattern matching is not implemented in rbac - you would have to change your role every time new pod name is generated.
So the answer is "No, you can' do it with rbac"
An alternative solution would be to use some kind of CI/CD (jenkins,gitlab-ci etc.) or orchestration tool (rundeck, ansible-tower etc) where you will create some kind of script, where your developers would pass arguments to a job, controlled by you, i.e.
kubectl exec deploy/foo -- /bin/bar baz "$DEV_ARGUMENT"
Which, essentially, means, that you would be responsible for managing access to that job/script, creating and maintaining serviceAccount for that script, etc.
If you are afraid of image mutability, i.e. you don't want your developers to install something in running container, but otherwise are okay with giving them shell on it (remember, they can still read any secrets/env vars/configMaps and even serviceAccount tokens that pod uses of you mount them by default), you should consider the following:
Don't run your containers as root. Try to use images, that support rootles operation, and then either specify correct non-root UID in runAsUser field in securityContext, or configure runAsNonRoot: true flag to deny containers running as root.
Better general solution would be to utilize PodSecurityPolicy (deprecated, removed in 1.25), Pod Security Admission or some 3rd party admission contoller like OPA Gatekeeper to deny containers running as root in your namespace
You can also make your pods immutable by using readOnlyRootFilesystem in security context, which will deny write operation to pod ephemeral storage (but if you mounted any volume as RW - they still will be accessible to write operations). Feasibility of this approach depends on whenever your apps use some kind of temporary files of not
Relevant links:
kubernetes RBAC role verbs to exec to pod
https://github.com/kubernetes/kubernetes/issues/44703#issuecomment-324826356 - issue, discussing current rbac limitations
https://itnext.io/how-it-works-kubectl-exec-e31325daa910
https://erkanerol.github.io/post/how-kubectl-exec-works/ - bot links explaining how exec actually works
I'm fairly new to Kubernetes and trying to wrap my head around how to manage ComponentConfigs in already running clusters.
For example:
Recently I initialized a kubeadm cluster in a test environment running Ubuntu. When I did that, I found CoreDNS to be in a CrashLoopBackoff which turned out to be the case because Ubuntu was configured to use systemd-resolved and so the resolv.conf had a loopback resolver configured. After reading the docs for coredns, I found out that a solution for that would be to change the resolvConf parameter for kubelet - either via commandline arguments or in the config.
So how would one do this properly in a kubeadm-managed cluster?
Reading [this page in the documentation][1] I didn't really get a clue, because it seems to be tailored to the case of initializing a new cluster or joining new nodes.
Of course, in this particular situation I could just use "Kubeadm reset" and initialize it again with a --config parameter but that doesn't seem to be the right solution for a running cluster.
So after digging a bit deeper I found several infos:
I could change the /var/lib/kubelet/kubeadm-flags.env on the node directly, but AFAICT this only makes sense for node-specific changes.
There is a ConfigMap in the kube-system namespace named kubelet-config-1.14. This seems promising for upcoming nodes joining the cluster to get the right configuration - but would changing that CM affect the already running Kubelet?
There is a marshalled version of the running config in /var/lib/config/kubelet.yaml that I could change, but AFAIU this would be overriden by kubelet itself periodically (?) or at least during a kubeadm upgrade.
There seems to be an option to specify a configmap in the node object, to let kubelet dynamically load the configuration from there, but given that there is already an existing configmap it seems more sensible to change that one.
I seemingly had success by some combination of changing aforementioned CM, running kubeadm upgrade something afterwards and rebooting the machine (since restarting the kubelet did not fix the CoreDNS issue ... but maybe I was to impatient).
So I am now asking:
What is the recommended way to carry out changes to the kubelet configuration (or any other configuration I could affect via kubeadm-config.yaml) that works and is upgrade-safe for cases where the configuration is not node-specific?
And if this involves running kubeadm ... config --config - how do I extract the existing Kubeadm-config in a way that I can feed it back to to kubeadm?
I am entirely happy with pointers to the right documentation, I just didn't find the right clues myself.
TIA
What you are looking for is well described in official documentation.
The basic workflow for configuring a Kubelet is as follows:
Write a YAML or JSON configuration file containing the Kubelet’s configuration.
Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
Update the Kubelet’s corresponding Node object to use this ConfigMap.
In addition there is DynamicKubeletConfig Feature Gate is enabled by default starting from Kubernetes v1.11, but you need some additional steps to activate it. You need to remember about, that Kubelet’s --dynamic-config-dir flag must be set to a writable directory on the Node.
I have a kubernetes-based application that uses an operator to build and deploy containers in pods. Sometimes I'd like to run containers in privileged mode to enable performance tracing, but since I'm not deploying the pod/containers directly from a manifest, I cannot simply add privileged mode and the debugfs filesystem mount.
That leaves me to fork the operator code, change where it builds the container spec, and redeploy with the modified operator. Doable, but awkward.
So my question is, is it possible to impose additional attributes to be added to container specs based on some clusterwide setting, either before pods are deployed by the operator? Or to modify the container spec after deployment? I tried that with kubectl edit pod mypod, but that didn't work.
This is on a physical cluster installed with kubespray.
There are three things to consider:
Your operator can create a controller (e.g. Deployment) instead of Pod, which allows modifications in the Pod Spec area, thus triggering Deployment's rollout (see rolling update strategy).
Use MutatingAdmissionWebhook
so before creating the Pod, its manifest would be modified/overwritten on the fly.
More info regarding MutatingAdmissionWebhook can be found here and here.
A workaround solution in a form of modifying the supply spec -> swapping the pod-a.
More about this was discussed here.
Please let me know if any of the above helped.
I am debugging certain behavior from my application pods; i am launching on K8s cluster. In order to do that I am increasing logging by increasing verbosity of deployment by adding --v=N flag to Kubectl create deployment command.
my question is : how can i configure increased verbosity globally so all pods start reporting increased verbosity; including pods in kube-system name space.
i would prefer if it can be done without re-starting k8s cluster; but if there is no other way I can re-start.
thanks
Ankit
For your applications, there is nothing global as that is not something that has global meaning. You would have to add the appropriate config file settings, env vars, or cli options for whatever you are using.
For kubernetes itself, you can turn up the logging on the kubelet command line, but the defaults are already pretty verbose so I’m not sure you really want to do that unless you’re developing changes for kubernetes.
I tried (unsuccessfully) to set up an initializer admission controller on k8s 1.10, running in minikube. kubectl does not show 'initializerconfiguration' as a valid object type and attempting 'kubectl create -f init.yaml' with a file containing an initializerConfiguration object (similar to the exmaple found here: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#configure-initializers-on-the-fly) returns this:
no matches for kind "InitializerConfiguration" in version "admissionregistration.k8s.io/v1alpha1"
(I tried with /v1beta1 as well, because kubectl api-versions doesn't show admissionregistration.k8s.io/v1alpha1 but does have .../v1beta1; no luck with that, either).
"Initializers" is enabled in the --admission-control option for kube-apiserver and all possible APIs are also turned on by default in minikube - so it should have worked, according to the k8s documentation.
According to the document mentioned in question:
Enable initializers alpha feature
Initializers is an alpha feature, so it is disabled by default. To
turn it on, you need to:
Include “Initializers” in the --enable-admission-plugins flag when starting kube-apiserver. If you have multiple kube-apiserver
replicas, all should have the same flag setting.
Enable the dynamic admission controller registration API by adding admissionregistration.k8s.io/v1alpha1 to the --runtime-config flag
passed to kube-apiserver, e.g.
--runtime-config=admissionregistration.k8s.io/v1alpha1. Again, all
replicas should have the same flag setting.
NOTE: For those looking to use this on minikube, use this to pass runtime-config to the apiserver:
minikube start --vm-driver=none --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1=true