We're trying to create different kuberentes secrets and offer access to specific secrets through specific service accounts that are assigned to pods. For example:
Secrets
- User-Service-Secret
- Transaction-Service-Secret
Service Account
- User-Service
- Transaction-Service
Pods
- User-Service-Pod
- Transaction-Service-Pod
The idea is to restrict access to User-Service-Secretsecret to User-Service service account that is assign to User-Service-Pod. So we can set this all up with the relevant kuberentes resources (i.e. ServiceAccount, Role, RoleBinding), but we realize that this may not be actually enforced, because Transaction-Service-Pod can just as easily read the User-Service-Secret secret when Pod starts up, even though the service account its assign to doesn't have get permission to the User-Service-Secret.
How do we actually enforce the RBAC system?
FYI we are using EKS
First it is important to distinguish between API access to the secret and consuming the secret as an environment variable or a mounted volume.
TLDR:
RBAC controls who can access a secret (or any other resource) using K8s API requests.
Namespaces or the service account's secrets attribute control if a pod can consume a secret as an environment variable or through a volume mount.
API access
RBAC is used to control if an identity (in your example the service account) is allowed to access a resource via the K8s API. You control this by creating a RoleBinding (namespaced) or a ClusterRoleBinding (cluster-wide) that binds an identity to a Role (namespaced) or a ClusterRole (not-namespaced) to your identity (service account). Then, when you assign the service account to a pod by setting the serviceAccountName attribute, running kubectl get secret in that pod or the equivalent method from one of the client libraries would mean you have credentials available to make the API request.
Consuming Secrets
This however is independent of configuring the pod to consume the secret as an environment variable or a volume mount. If the container spec in a pod spec references the secret it is made available inside that container. Note, per container, not per pod. You can limit what secret a pod can mount by having the pods in different namespaces, because a pod can only refer to a secret in the same namespace. Additionally, you can use the service account's secrets attribute, to limit what secrets a pod with thet service account can refer to.
$ kubectl explain sa.secrets
KIND: ServiceAccount
VERSION: v1
RESOURCE: secrets <[]Object>
DESCRIPTION:
Secrets is the list of secrets allowed to be used by pods running using
this ServiceAccount. More info:
https://kubernetes.io/docs/concepts/configuration/secret
ObjectReference contains enough information to let you inspect or modify
the referred object.
You can learn more about the security implications of Kubernetes secrets in the secret documentation.
The idea is to restrict access to User-Service-Secret secret to User-Service service account that is assign to User-Service-Pod. So we can set this all up with the relevant Kubernetes resources (i.e. ServiceAccount, Role, RoleBinding), but we realize that this may not be actually enforced, because Transaction-Service-Pod can just as easily read the User-Service-Secret secret when Pod starts up, even though the service account its assign to doesn't have get permission to the User-Service-Secret.
Yes, this is correct.
This is documented for Kubernetes on privilege escalation via pod creation - within a namespace.
Users who have the ability to create pods in a namespace can potentially escalate their privileges within that namespace. They can create pods that access their privileges within that namespace. They can create pods that access secrets the user cannot themselves read, or that run under a service account with different/greater permissions.
To actually enforce this kind of Security Policies, you probably have to add an extra layer of policies via the admission controller. The Open Policy Agent in the form of OPA Gatekeeper is most likely a good fit for this kind of policy enforcement.
Related
I am trying to create a K8s cluster in Azure AKS and when cluster is ready I can see couple of resources are created within the default namespace. Example secret, configmap:
As a security recommendation NO k8s resources should be created under the default namespace so how to avoid it? It's created by default during cluster creation.
I have found the same question asked here:
User srbose-msft (Microsoft employee) explained the principle of operation very well:
In Kubernetes, a ServiceAccount controller manages the ServiceAccounts inside namespaces, and ensures a ServiceAccount named "default" exists in every active namespace. [Reference]
TokenController runs as part of kube-controller-manager. It acts asynchronously. It watches ServiceAccount creation and creates a corresponding ServiceAccount token Secret to allow API access. [Reference] Thus, the secret for the default ServiceAccount token is also created.
Trusting the custom CA from an application running as a pod usually requires some extra application configuration. You will need to add the CA certificate bundle to the list of CA certificates that the TLS client or server trusts. For example, you would do this with a golang TLS config by parsing the certificate chain and adding the parsed certificates to the RootCAs field in the tls.Config struct.
You can distribute the CA certificate as a ConfigMap that your pods have access to use. [Reference] AKS implements this in all active namespaces through ConfigMaps named kube-root-ca.crt in these namespaces.
You shall also find a Service named kubernetes in the default namespace. It has a ServiceType of ClusterIP and exposes the API Server Endpoint also named kubernetes internally to the cluster in the default namespace.
All the resources mentioned above will be created by design at the time of cluster creation and their creation cannot be prevented. If you try to remove these resources manually, they will be recreated to ensure desired goal state by the kube-controller-manager.
Additionally:
The Kubernetes clusters should not use the default namespace Policy is still in Preview. Currently the schema does not explicitly allow for Kubernetes resources in the default namespace to be excluded during policy evaluation. However, at the time of writing, the schema allows for labelSelector.matchExpressions[].operator which can be set to NotIn with appropriate labelSelector.matchExpressions[].values for the Service default/kubernetes with label:
component=apiserver
The default ServiceAccount, the default ServiceAccount token Secret and the RootCA ConfigMap themselves are not created with any labels and hence cannot to added to this list. If this is impeding your use-case I would urge you to share your feedback at https://techcommunity.microsoft.com/t5/azure/ct-p/Azure
kubectl explain serviceaccount.secrets describes ServiceAccount Secrets as the secrets allowed to be used by Pods running using this ServiceAccount, but what effect does adding a Secret name to this list have?
The ServiceAccount token Secret (which is automatically added to this list) gets automatically mounted as a volume into all containers in a Pod running using this ServiceAccount (as long as the ServiceAccount admission controller is enabled), but what happens for other secrets?
It holds the name of all secrets containing tokens for that SA so when the controller goes to rotate things, it knows where to find them.
We need to disable the automount of service account from our existing deployments in AKS cluster. There are 2 ways to do by adding the property "automountserviceaccount : false" in either in the service account manifest or pod template.
We are using separate service account specified in our application deployments, however when we looked in the namespace, there are default service account also created.
So inorder to secure our cluster, do we need to disable the automount property for both default and application specific service accounts?.
Since our app already live, will there be any impact by adding this to the service account s.
How to know the used service accounts of a pod and it's dependencies ?
So inorder to secure our cluster, do we need to disable the automount property for both default and application specific service accounts?.
The design behind the default ServiceAccount is that it does not have any rights unless you give them some. So from a security point of view there is not much need to disable the mount unless you granted them access for some reason. Instead, whenever an application truly needs some access, go ahead and create a ServiceAccount for that particular application and grant it the permissions it needs via RBAC.
Since our app already live, will there be any impact by adding this to the service account s.
In case you truly want to disable the mount there won't be an impact on your application if it didn't use the ServiceAccount beforehand. What is going to happen though, is that a new Pod will be created and the existing one is being delete. However, if you properly configured readinessProbes and a rolling update strategy, then Kubernetes will ensure that there will be no downtime.
How to know the used service accounts of a pod and it's dependencies ?
You can check what ServiceAccount a Pod is mounting by executing kubectl get pods <pod-name> -o yaml. The output is going to show you the entirety of the Pod's manifest and the field spec.serviceAccountName contains information on which ServiceAccount the Pod is mounting.
Disable auto-mount of default service account:
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
automountServiceAccountToken: false
https://gist.github.com/pjbgf/0a8c8a1459e5a2eb20e9d0852ba8c4be
Is there a way to programmatically update a kubernetes secret from a pod? that is, not using kubectl.
I have a secret mounted on a pod and also exposed via an environment variable. I would like to modify it from my service but it looks that it's read only by default.
You can use the Kubernetes REST API with the pod's serviceaccount's token as credentials (found at /var/run/secrets/kubernetes.io/serviceaccount/token inside the pod), you just need to allow the service account to edit secrets in the namespace via a role.
See Secret for the API docs
The API server is internally reachable via https://kubernetes.default
I am trying to understand security implication of granting kubernetes serviceaccount permission to perform deployments, service creation, etc. The role is clusterrole with namespaced rolebinding.
Use case are using service account to automate/orchestrate some tasks inside cluster. Version is 1.16
Service account can used to grant different access level to different purpose. for developer ClusterRole need to list, get and watch resources but not delete. but Admin can delete and create resources.
If you are developing K8 operator, operator need to communicate with cluster for create, update and delete resources then operator need service with ClusterRoleBinding of all verbs. so that operator can perform all operation on resources. but not good practice to assign full permission to regular deployment.
Let me discuss about this.
According to the docs.
A Role always _sets permissions within a particular namespace__; when you create a Role, you have to specify the namespace it belongs in
ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced; it can't be both.
A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.
If you want to define a role within a namespace, use a Role; if you want to define a role cluster-wide, use a ClusterRole.
A RoleBinding may reference any Role in the same namespace. Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding.
So answering your question - handling ClusterRoles it could be definitively security risk.
The best solution is to grant as minimal permissions as possible.
Grant a role to an application-specific service account (best practice)
Grant a role to the "default" service account in a namespace
Grant a role to all service accounts in a namespace
Grant a limited role to all service accounts cluster-wide (discouraged)
Grant super-user access to all service accounts cluster-wide (strongly discouraged)
There is no one answer for this question because you need to plan and test action/permissions between your application/deployment and kubernetes API. F.e at namesapaced or non namesapced resources, inside one namespace or across the entire cluster.
In you example you can simply use Role/Rolebinding if you are working inside the same namespace. You can use ClusterRole/Rolebinding and extend permissions by additional RoleBinding allowing the ServiceAccount to create new k8s objects into another namespace.
Assuming we are talking about ServiceAccount for deployment, here you can find good advice for "RBAC in Deployments: A use case"
If you create ServiceAccount for your deployment and create appropriate Role/ClusterRole and Rolebinding/ClusterRoleBinding:
you can perform:
kubectl can-i get secrets --as=system:serviceaccount:[namespace]:[service_account_name] -n [target_namespace]
For testing please take a look also at Access Clusters Using the Kubernetes API.
This command will show you if particular ServiceAccount (defined by subject in Rolebinding/ClusterRolebinding) has permissions (defined by verbs in Role/ClusteRole) to get secrets (defined by resources Role/ClusterRole) in specified namespace.
Following this approach you can verify if your deployment has enough permissions to perform all required operation against Kubernetes API.
While working with RBAC in Kubernetes you should consider mentioned below topics:
Have multiple users with different properties, establishing a proper authentication mechanism.
Have full control over which operations each user or group of users can execute.
Have full control over which operations each process inside a pod can execute.
Limit the visibility of certain resources of namespaces.