I have the following pod definition, (notice the explicitly set service account and secret):
apiVersion: v1
kind: Pod
metadata:
name: pod-service-account-example
labels:
name: pod-service-account-example
spec:
serviceAccountName: example-sa
containers:
- name: busybox
image: busybox:latest
command: ["sleep", "10000000"]
env:
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: example-secret
key: secret-key-123
It successfully runs. However if I use the the same service account example-sa, and try to retrieve the example-secret it fails:
kubectl get secret example-secret
Error from server (Forbidden): secrets "example-secret" is forbidden: User "system:serviceaccount:default:example-sa" cannot get resource "secrets" in API group "" in the namespace "default"
Does RBAC not apply for pods? Why is the pod able to retrieve the secret if not?
RBAC applies to service accounts, groups, users and not to pods.When you refer a secret in the env of a pod , service account is not being used to get the secret.Kubelet is getting the secret by using its own kubernetes client credential. Since kubelet is using its own credential to get the secret it does not matter whether the service account has RBAC to get secret or not because its not used.
Service account is used when you want to invoke Kubernetes API from a pod using kubernetes standard client library or Kubectl.
Code snippet of Kubelet for reference.
Related
I used the following guide to set up my chaostoolkit cluster: https://chaostoolkit.org/deployment/k8s/operator/
I am attempting to kill a pod using kubernetes, however the following error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:chaostoolkit-run:chaostoolkit-b3af262edb\" cannot list resource \"pods\" in API group \"\" in the namespace \"task-dispatcher\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
I set my serviceAccountName to an RBAC that I created but for some reason my kubernetes defaults to "system:serviceaccount:chaostoolkit-run:chaostoolkit-b3af262edb".
apiVersion: v1
kind: ConfigMap
metadata:
name: my-chaos-exp
namespace: chaostoolkit-run
data:
experiment.yaml: |
---
version: 1.0.0
title: Terminate Pod Experiment
description: If a pod gets terminated, a new one should be created in its place in a reasonable amount of time.
tags: ["kubernetes"]
secrets:
k8s:
KUBERNETES_CONTEXT: "docker-desktop"
method:
- type: action
name: terminate-k8s-pod
provider:
type: python
module: chaosk8s.pod.actions
func: terminate_pods
arguments:
label_selector: ''
name_pattern: my-release-rabbitmq-[0-9]$
rand: true
ns: default
---
apiVersion: chaostoolkit.org/v1
kind: ChaosToolkitExperiment
metadata:
name: my-chaos-exp
namespace: chaostoolkit-crd
spec:
serviceAccountName: test-user
automountServiceAccountToken: false
pod:
image: chaostoolkit/chaostoolkit:full
imagePullPolicy: IfNotPresent
experiment:
configMapName: my-chaos-exp
configMapExperimentFileName: experiment.yaml
restartPolicy: Never
Error which is shared is using default service account "choastoolkit". Look like the role associated might not proper permissions.
The service account "test-user" which is been used in ChaosToolkitExperiment defintion should have proper role access to delete pod.
Please specify proper service account having proper role access.
I am using Vault CSI Driver on Charmed Kubernetes v1.19 where I'm trying to retrieve secrets from Vault for a pod running in a separate namespace (webapp) with its own service account (webapp-sa) following the steps in the blog.
As I have been able to understand so far, the Pod is trying authenticate to the Kubernetes API, so that later it can generate a Vault token to access the secret from Vault.
$ kubectl get po webapp
NAME READY STATUS RESTARTS AGE
webapp 0/1 ContainerCreating 0 22m
It appears to me there's some issue authenticating with the Kubernetes API.
The pod remains stuck in the Container Creating state with the message - failed to create a service account token for requesting pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned webapp/webapp to host-03
Warning FailedMount 4m38s (x23 over 35m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod webapp/webapp, err: rpc error: code = Unknown desc = error making mount request: **failed to create a service account token for requesting pod** {webapp xxxx webapp webapp-sa}: the server could not find the requested resource
I can get the vault token using cli in the pod namespace:
$ vault write auth/kubernetes/login role=database jwt=$SA_JWT_TOKEN
Key Value
--- -----
token <snipped>
I do get the vault token using the API as well:
$ curl --request POST --data #payload.json https://127.0.0.1:8200/v1/auth/kubernetes/login
{
"request_id":"1234",
<snipped>
"auth":{
"client_token":"XyZ",
"accessor":"abc",
"policies":[
"default",
"webapp-policy"
],
"token_policies":[
"default",
"webapp-policy"
],
"metadata":{
"role":"database",
"service_account_name":"webapp-sa",
"service_account_namespace":"webapp",
"service_account_secret_name":"webapp-sa-token-abcd",
"service_account_uid":"123456"
},
<snipped>
}
}
Reference: https://www.vaultproject.io/docs/auth/kubernetes
As per the vault documentation, I've configured Vault with the Token Reviewer SA as follows:
$ cat vault-auth-service-account.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-token-review-binding
namespace: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: vault
Vault is configured with JWT from the Token Reviewer SA as follows:
$ vault write auth/kubernetes/config \
token_reviewer_jwt="< TOKEN Reviewer service account JWT>" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
kubernetes_ca_cert=#ca.crt
I have defined a Vault Role to allow the webapp-sa access to the secret:
$ vault write auth/kubernetes/role/database \
bound_service_account_names=webapp-sa \
bound_service_account_namespaces=webapp \
policies=webapp-policy \
ttl=72h
Success! Data written to: auth/kubernetes/role/database
The webapp-sa is allowed access to the secret as per the Vault Policy defined as follows:
$ vault policy write webapp-policy - <<EOF
> path "secret/data/db-pass" {
> capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: webapp-policy
Pod and it's SA is defined as follows:
$ cat webapp-sa-and-pod.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: webapp-sa
---
kind: Pod
apiVersion: v1
metadata:
name: webapp
spec:
serviceAccountName: webapp-sa
containers:
- image: registry/jweissig/app:0.0.1
name: webapp
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
providerName: vault
secretProviderClass: "vault-database"
Does anyone have any clue as to why the Pod won't authenticate with
the Kubernetes API?
Do I have to enable flags on the kube-apiserver for Token Review API
to work?
Is it enabled by default on Charmed Kubernetes v1.19?
Would be grateful for any help.
Regards,
Sana
I am running Airflow in Kubernetes
One pod, 2 containers - webserver and scheduler.
KubernetesExecutor in configs
But due to organizational settings, the scheduler with the default service account can't work, not enough roles. I can't change this setting
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:<account_name>:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"<namespace_name>\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
So I created Service Account with the needed roles, roleBinging and etc. How can I set Airflow to run the scheduler with that SA?
You can specify the desired SA to use in your pod spec as discussed in the link below:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
serviceAccountName: build-robot
automountServiceAccountToken: false
...
I'm going through this post, where we bind a Role to a Service Account and then query the API Server using said Service Account. The role only has list permission to the pods resource.
I did an experiment where I mounted a random Secret into a Pod that is using the above Service Account and my expectation was that the Pod would attempt to query the Secret and fail the creation process, but the pod is actually running successfully with the secret mounted in place.
So I'm left wondering when does a pod actually needs to query the API Server for resources or if the pod creation process is special and gets the resources through other means.
Here is the actual list of resources I used for my test:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: example-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: example-role
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: example-rb
subjects:
- kind: ServiceAccount
name: example-sa
roleRef:
kind: Role
name: example-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: example-secret
data:
password: c3RhY2tvdmVyZmxvdw==
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
serviceAccountName: example-sa
containers:
- name: webserver
image: nginx
volumeMounts:
- name: secret-volume
mountPath: /mysecrets
volumes:
- name: secret-volume
secret:
secretName: example-secret
...
I must admit that at first I didn't quite get your point, but when I read your question again I think now I can see what it's all about. First of all I must say that your initial interpretation is wrong. Let me explain it.
You wrote:
I did an experiment where I mounted a random Secret into a Pod that
is using the above Service Account
Actually the key word here is "I". The question is: who creates a Pod and who **mounts a random Secret into this Pod ? And the answer to that question from your perspective is simple: me. When you create a Pod you don't use the above mentioned ServiceAccount but you authorize your access to kubernetes API through entries in your .kube/config file. During the whole Pod creation process the ServiceAccount you created is not used a single time.
and my expectation was that the
Pod would attempt to query the Secret and fail the creation process,
but the pod is actually running successfully with the secret mounted
in place.
Why would it query the Secret if it doesn't use it ?
You can test it in a very simple way. You just need to kubectl exec into your running Pod and try to run kubectl, query kubernetes API directly or use one of the officially supported kubernetes cliet libraries. Then you will see that you're allowed to perform only specific operations, listed in your Role i.e. list Pods. If you attempt to run kubectl get secrets from within your Pod, it will fail.
The result you get is totally expected and there is nothig surprising in the fact that a random Secret is successfully mounted and a Pod is being created successfully every time. It's you who query kubernetes API and request creation of a Pod with a Secret mounted. **It's not Pod's
ServiceAccount.
So I'm left wondering when does a pod actually needs to query the API
Server for resources or if the pod creation process is special and
gets the resources through other means.
If you don't have specific queries e.g. written in python that use Kubernetes Python Client library that are run by your Pod or you don't use kubectl command from within such Pod, you won't see it making any queries to kubernetes API as all the queries needed for its creation process are performed by you, with permissions given to your user.
I'd like to deploy pods into my GKE Kubernetes cluster that use images from a private, third-party Docker registry (not GCP's private Docker registry).
How do I provide my GKE Kubernetes cluster with credentials to that private repository so that the images can be pulled when required?
You need to create a secret that holds the credentials needed to download images from the private registry. This process is explained on Kubernetes documentation, but it looks like
kubectl create secret docker-registry regsecret --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
Then, once your secret has been created, you need to specify that you want to use this secret to pull images from the registry when creating the pod's containers with the imagePullSecrets key containing the name of the secret created above, like
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regsecret