When an application needs to call events API to get all events of its cluster, as a programmer I may define a role like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["events"]
verbs: ["list"]
What makes me confusing is the apiGroups part, I can use "events.k8s.io", or simply "", or "events.k8s.io" and "" both...
What is this apiGroups thing? I tried to read official documentation but all I found is this:
API groups make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the apiVersion field of a serialized object.
There are several API groups in Kubernetes:
The core (also called legacy) group is found at REST path /api/v1. The core group is not specified as part of the apiVersion field, for example, apiVersion: v1.
The named groups are at REST path /apis/$GROUP_NAME/$VERSION and use apiVersion: $GROUP_NAME/$VERSION (for example, apiVersion: batch/v1). You can find the full list of supported API groups in the Kubernetes API reference.
This doesn't help me understand it... Why there are named groups and core groups, why I can use "" and "events.k8s.io" together?
If my resource is events, why do I need to explicitly tell K8s that there is an API group named "events.k8s.io" as if the events in "events.k8s.io" and events in resources are two different things...
This question had been haunting me for daysðŸ˜
Why there are named groups and core groups
The reason is historical. Kubernetes first considered some resources part of a core group, meaning the core of Kubernetes is constituted by these resources (resources in the core group includes Pods, Events).
The purposes of the named API groups (or namespaces) (such as events.k8s.io) is to group resources by thematic. You'll have networking.k8s.io for resources related to networking.
The "core" resources were considered such an integral part of Kubernetes that they did not need a group (I'm simplifying).
why I can use "" and "events.k8s.io" together?
If my resource is events, why do I need to explicitly tell K8s that there is an API group named "events.k8s.io" as if the events in "events.k8s.io" and events in resources are two different things...
The API is not exactly the same. In the API reference you can see that in the events.k8s.io you can see some field staring with deprecated which are deprecated field(s) assuring backward compatibility with core.v1 Event type. So depending on which endpoint you use, the API semantic is not exactly the same (but Kubernetes has mechanisms to translate from one version to the other). Since some tools might rely on the core v1 version and not yet have migrated to events.k8s.io/v1, both need to exists.
There has been other "group migration" of that sort, notably from the extensions group to other group (Ingress migrated to networking.k8s.io, Deployment to apps)
It has to do with Kubernetes controllers & CustomResourceDefinition.
When you write an operator for Kubernetes, you would define custom objects. For example, Elastic (www.elastic.co) has an operator deploying ElasticSearch/Kibana. They decided to call one of their object "elasticsearch".
Those resource names are arbitrary, and nothing guarantee there won't be someone else, working on its own operator, using the same names. Thus, when you define your custom resources, you would also affect them with an API group.
The API group is also arbitrary, though would usually identify a unique controller/operator/set of functionality. In the case of the ElasticSearch operator, they went with an API group "k8s.elastic.co". When installing this operator CRD on your cluster, a kubectl get crd would list objects such as "elasticsearchs.k8s.elastic.co", "kibanas.k8s.elastic.co".
You could very well deploy another operator, which implements its own "elasticsearch" or "kibana" objects, within its own API group.
Thus, when you write your Roles / ClusterRoles, that apiGroup field is crucial configuring your RBAC. The resources array lists short names, the apiGroup is then used resolving the fully qualified CRD name.
While kubernetes legacy objects use some empty string, other native objects may use networking.k8s.io, batch, extensions, ... In the meantime, custom controllers would all use their own api group, whenever they rely on their own CRDs.
Now, I'm not sure I know the answer to 'why I can use "" and "events.k8s.io" together?'. I would say it's historic?
I did not see any cases where setting multiple api groups in a rule was benefic. In my opinion, it's misleading, a source of mistake, ... when you find a rule with two api groups, you can be pretty sure the person who wrote it did not understand what they were doing.
At best, one combination of apigroup / resource grants what you meant to. Worst case, one combination of apigroup / resource grants something you did not want to allow in the first place.
Erratum: events are an exception, though. As pointed out in some kubernetes GitHub Issue ( https://github.com/kubernetes/kubernetes/issues/94857 ), some member would argue this is not a bug: we really need both API groups. Which is kind of surprising.
Not sure why would that be. Sidenote, the schema for Event in core/v1 and events.k8s.io isn't the same:
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#event-v1-events-k8s-io
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#event-v1-core
So they're not the same object strictly speaking, although from what I can see, they return the same data / in terms of etcd, it's the same data, with two JSON representations depending on which api group you query:
$ k get event.events.k8s.io -n kube-system
LAST SEEN TYPE REASON OBJECT MESSAGE
58m Warning Unhealthy pod/metrics-server-7df46996d8-c5wf5 Readiness probe failed: Get "https://10.233.108.212:8443/healthz": context deadline exceeded
...
$ k get event -n kube-system
LAST SEEN TYPE REASON OBJECT MESSAGE
58m Warning Unhealthy pod/metrics-server-7df46996d8-c5wf5 Readiness probe failed: Get "https://10.233.108.212:8443/healthz": context deadline exceeded
...
Not sure what's the rational here. events.k8s.io isn't anything new, I haven't read about some transition from one to the other ... That's a good question you have.
Related
If I have a microservice app within a namespace, I can easily get all of my namespaced resources within that namespace using the k8s api. I cannot, however, view what non-namespaced resources are being used by the microservice app. If I want to see my non-namespaced resources, I can only see them all at once, with no indication of which ones are dependancies in the microservice app.
How can I find my dependancies related to my application? I'd like to be able to get reference to things like PersistentVolumes, StorageClasses, ClusterRoles, etc. that are being used by the app's namespaced resources.
Your code, running in a pod container inside a namespace, runs using a serviceaccount set using pod.spec.serviceAccountName.
If not set, it'll run using the default serviceaccount.
You need to create a clusterRole in order to grant access to cluster-wide resources specific verbs, then in the pod namespace assign this clusterRole to the serviceaccount, via a roleBinding targetting the clusterRole create before.
Then your pod, using a kubernetes client, and using the "in-cluster config" auth method, will be able to query the apiserver to get/list/watch/delete/patch... the said cluster-wide resources.
This is a definitely a non-trivial task because of the many ways such dependency can come into play: whenever an object "uses" another one, there we could identify a dependency. The issue is that this "use" relation can take many forms: e.g., a Pod can reference a Volume in its definition (which would be a sort of direct dependency), but can also use a PersistentVolumeClaim which would then instantiate a PV through use of a StorageClass -- and these relations are only known to Kubernetes at run time, when the YAML definitions are applied.
In other words:
To chase dependencies, you would have to inspect the YAML description of resources in-use, knowing the semantics of each: there's no single depends: value in each but one would need to follow e.g., the spec.storageClass of a PVC, the spec.volumes: of a Pod, etc.
In some cases, this would not even be enough: e.g., for matching Services and Pods this would not even be enough, as one would have to match ports on each side.
All of this would need to be done by extracting YAML from a running K8s cluster, since some relations between resources would not be known until they are instantiated.
You could check How do you visualise dependencies in your Kubernetes YAML files? article by Daniele Polencic shows a few tools that can be used to visualize dependencies:
There isn't any static tool that analyses YAML files. But you can visualise your dependencies in the cluster with Weave Scope, KubeView or tracing the traffic with Istio.
I want to use OPA(Open Policy Agent) in kubernetes but have some questions which are not still clear for me:
Let’s take a look at a specific case together: for instance, there is a pod creation in a namespace and we can know the namespace from the pod object at OPA. But, can we get the namespace object separately to learn the authority which this namespace belongs to?
More explicitly, I mean can we do requests through the Kubernetes cluster by OPA?
for instance, there is a pod creation with the name of Test. I just want to allow this creation for only an authority called TestAuthority. When the pod is created, we know the namespace data but not the authority. To figure out the authority which this pod belongs to, I need to have the namespace object and look out its labels. Can we do so by OPA?
Additionally, can we say allow pod creation with the names of Test1, Test2, and Test3? So, any pod creation with the name of Test4 should be denied.
Thank you in advance for your help
(1) Yes, see https://github.com/open-policy-agent/kube-mgmt#caching or https://github.com/open-policy-agent/gatekeeper#replicating-data depending on which integration you are wanting to use. Both allow replicating objects from kubernetes into OPA to reference in the policies.
(2) Yes, you can write policies like that.
credit:Patrick East answered from OPA Slack
We are going to provide customers a function by deploying and running a container in customers kubernetes environment. After the job is done, we will clean up the container. Currently, the plan is to use k8s default namespace, but I'm not sure whether it can be a concern for customers. I don't have much experience in k8s related field. Should we give customers' an option to specify a namespace to run container, or just use the default namespace? I appreciate your suggestions!
I would recommend you not use (!?) the default namespace for anything ever.
The following is more visceral than objective but it's drawn from many years' experience of Kubernetes. In 2016, a now former colleague and I blogged about the use of namespaces:
https://kubernetes.io/blog/2016/08/kubernetes-namespaces-use-cases-insights/
NB since then, RBAC was added and it permits enforcing separation, securely.
Although it exists as a named (default) namespace, it behaves as if there is (the cluster has) no namespace. It may be (!?) that it was retcon'd into Kubernetes after namespaces were added
Unless your context is defined to be a specific other namespace, kubectl ... behaves as kubectl ... --namespace=default. So, by accident it's easy to pollute and be impacted by pollution in this namespace. I'm sure your team will use code for your infrastructure but mistakes happen and "I forgot to specify the namespace" is easily done (and rarely wanted).
Using non-default namespaces becomes very intentional, explicit and, I think, precise. You must, for example (per #david-maze answer) be more intentional about RBAC for the namespace's resources.
Using namespaces is a mechanism that promotes multi-tenancy which is desired for separation of customers (business units, versions etc.)
You can't delete the default namespace but you can delete (and by consequence delete all the resources constrained by) any non-default namespace.
I'll think of more, I'm sure!
Update
Corollary: generally don't constrain resources to namespace in specs but use e.g. kubectl apply --filename=x.yaml --namespace=${NAMESPACE}
I'd consider the namespace name pretty much a required option. I would default to the namespace name specified in the .kube/config file, if that's at all a choice for you. (That may not be default.)
RBAC rules or organizational policies also might mean the default namespace can't or shouldn't be used. One of the clusters I work with is a shared cluster where each user has their own namespace, enforced by RBAC policies; except for cluster admins, nobody gets to use default, and everybody needs to be able to configure the namespace to run in their own.
I am trying to follow something similar to the Istio injection model where Istio is able to run a mutating admission webhook in order to auto inject their sidecar.
We would like to do something similar, but with some config maps. We have a need to mount config maps to all new pods in a given namespace, always mounted at the same path. Is it possible to create a mutating admission webhook that will allow me to mount this config map at the known path while admitting new pods?
docs to mutating webhooks: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
This should be entirely possible and is, in fact, an aligned use-case for a custom mutating admission webhook. Unfortunately, the official documentation on actually implementing them is somewhat sparse.
This is the most useful reference material I found when I was working on this mutating admission webhook.
The general process is as follows:
Generate and deploy certificates as demonstrated here
Deploy MutatingWebhookConfiguration to your cluster. This configuration allows us to tell the API server where to send the AdmissionReview objects. This is also where you should specify which operations on which API resources in which namespaces you want to target.
Write the actual webhook server, which should accept AdmissionReview objects at the specified endpoint (/mutate is the convention) and return AdmissionResponse objects with the mutated object, as is shown here (note: in the linked example, I added an annotation to incoming pods that fit a certain criteria, while your application would add a field for the ConfigMap)
Deploy the webhook server and expose it using normal methods (Deployment and Service, or whatever fits your use case). Make sure it's accessible at the location you specified in the configuration for the MutatingWebhookConfiguration
Hope this was enough information! Let me know if I left anything too vague / was unclear.
We have several namespaces in a kubernetes (openshift) cluster and we want to provide some default objects in every namespace in an automated way. If for some reason one of these default objects is deleted (or changed), it should be recreated automatically.
In openshift this already happens with some serviceaccounts and secrets for these accounts, but it is not easily extendable if i'm not mistaking. E.g. the the 'builder' and 'deployer' serviceaccount are created and recreated automatically by openshift. We are looking for a similar solution that supports all types of objects.
Does anyone has an example of how to accomplish this? It is possible to watch the etcd for changes and react to these changes by creating/modifying objects, but i can't seem to find a good example to start with.