I am running Airflow in Kubernetes
One pod, 2 containers - webserver and scheduler.
KubernetesExecutor in configs
But due to organizational settings, the scheduler with the default service account can't work, not enough roles. I can't change this setting
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:<account_name>:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"<namespace_name>\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
So I created Service Account with the needed roles, roleBinging and etc. How can I set Airflow to run the scheduler with that SA?
You can specify the desired SA to use in your pod spec as discussed in the link below:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
serviceAccountName: build-robot
automountServiceAccountToken: false
...
Related
I'm going through this post, where we bind a Role to a Service Account and then query the API Server using said Service Account. The role only has list permission to the pods resource.
I did an experiment where I mounted a random Secret into a Pod that is using the above Service Account and my expectation was that the Pod would attempt to query the Secret and fail the creation process, but the pod is actually running successfully with the secret mounted in place.
So I'm left wondering when does a pod actually needs to query the API Server for resources or if the pod creation process is special and gets the resources through other means.
Here is the actual list of resources I used for my test:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: example-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: example-role
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: example-rb
subjects:
- kind: ServiceAccount
name: example-sa
roleRef:
kind: Role
name: example-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
name: example-secret
data:
password: c3RhY2tvdmVyZmxvdw==
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
serviceAccountName: example-sa
containers:
- name: webserver
image: nginx
volumeMounts:
- name: secret-volume
mountPath: /mysecrets
volumes:
- name: secret-volume
secret:
secretName: example-secret
...
I must admit that at first I didn't quite get your point, but when I read your question again I think now I can see what it's all about. First of all I must say that your initial interpretation is wrong. Let me explain it.
You wrote:
I did an experiment where I mounted a random Secret into a Pod that
is using the above Service Account
Actually the key word here is "I". The question is: who creates a Pod and who **mounts a random Secret into this Pod ? And the answer to that question from your perspective is simple: me. When you create a Pod you don't use the above mentioned ServiceAccount but you authorize your access to kubernetes API through entries in your .kube/config file. During the whole Pod creation process the ServiceAccount you created is not used a single time.
and my expectation was that the
Pod would attempt to query the Secret and fail the creation process,
but the pod is actually running successfully with the secret mounted
in place.
Why would it query the Secret if it doesn't use it ?
You can test it in a very simple way. You just need to kubectl exec into your running Pod and try to run kubectl, query kubernetes API directly or use one of the officially supported kubernetes cliet libraries. Then you will see that you're allowed to perform only specific operations, listed in your Role i.e. list Pods. If you attempt to run kubectl get secrets from within your Pod, it will fail.
The result you get is totally expected and there is nothig surprising in the fact that a random Secret is successfully mounted and a Pod is being created successfully every time. It's you who query kubernetes API and request creation of a Pod with a Secret mounted. **It's not Pod's
ServiceAccount.
So I'm left wondering when does a pod actually needs to query the API
Server for resources or if the pod creation process is special and
gets the resources through other means.
If you don't have specific queries e.g. written in python that use Kubernetes Python Client library that are run by your Pod or you don't use kubectl command from within such Pod, you won't see it making any queries to kubernetes API as all the queries needed for its creation process are performed by you, with permissions given to your user.
I have the following pod definition, (notice the explicitly set service account and secret):
apiVersion: v1
kind: Pod
metadata:
name: pod-service-account-example
labels:
name: pod-service-account-example
spec:
serviceAccountName: example-sa
containers:
- name: busybox
image: busybox:latest
command: ["sleep", "10000000"]
env:
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: example-secret
key: secret-key-123
It successfully runs. However if I use the the same service account example-sa, and try to retrieve the example-secret it fails:
kubectl get secret example-secret
Error from server (Forbidden): secrets "example-secret" is forbidden: User "system:serviceaccount:default:example-sa" cannot get resource "secrets" in API group "" in the namespace "default"
Does RBAC not apply for pods? Why is the pod able to retrieve the secret if not?
RBAC applies to service accounts, groups, users and not to pods.When you refer a secret in the env of a pod , service account is not being used to get the secret.Kubelet is getting the secret by using its own kubernetes client credential. Since kubelet is using its own credential to get the secret it does not matter whether the service account has RBAC to get secret or not because its not used.
Service account is used when you want to invoke Kubernetes API from a pod using kubernetes standard client library or Kubectl.
Code snippet of Kubelet for reference.
According to Spring Cloud Kubernetes docs, in order to discover services/pods in RBAC enabled Kubernetes distros:
you need to make sure a pod that runs with spring-cloud-kubernetes has access to the Kubernetes API. For any service accounts you assign to a deployment/pod, you need to make sure it has the correct roles. For example, you can add cluster-reader permissions to your default service account depending on the project you’re in.
What are cluster-reader permissions in order to discover services/pods?
Error I receiving is:
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://x.x.x.x/api/v1/namespaces/jx-staging/services.
Message: Forbidden!Configured service account doesn't have access.
Service account may have been revoked. services is forbidden:
User "system:serviceaccount:jx-staging:default" cannot list services in the namespace "jx-staging"
Read endpoints and services seems to be a bare minimum for Spring Cloud Kubernetes to discover pods and services.
Example adds permissions to default service account in default namespace.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-read-role
rules:
- apiGroups:
- ""
resources:
- endpoints
- pods
- services
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-read-rolebinding
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-read-role
apiGroup: rbac.authorization.k8s.io
Kubernetes generally categorizes roles into two types:
Role: This are specific to the namespace to which they are granted
ClusterRole: Applies to the whole cluster, meaning that it applies to all namespaces
So what the Spring Cloud Kubernetes docs mean there is that in order to be able to read properly discover services/pods across all namespaces, the ServiceAccount which will be associated with the application should have a ClusterRole that allows it to read Pods, Services etc.
This part of the Kubernetes docs (which also contains great examples) is a must-read for a general understanding of Kubernetes RBAC.
I am setting up a namespace for my application that has statefulsets, deployments, and secrets into that namespace. Using RBAC, I am defining specific roles and binding them to a service account that is used by the deployment/statefulset. This works as expected.
Now when I try to test if the secrets are secure by not assigning any service account to the deployment, it still pulls down the secrets. The default service account in the namespace is bound with the view clusterrole which should not have access to secrets.
Any clue what is happening here?
Thanks in advance.
I believe you need to assign a RoleBinding to the default service account on your namespace. For example:
kubectl create rolebinding myapp-view-binding --clusterrole=view --serviceaccount=default:default --namespace=default
The view role should prevent you from reading secrets.
Now when I try to test if the secrets are secure by not assigning any service account to the deployment...
If you don't assign a service account to your deployment, the default service account in the deployment's namespace will be used.
... it still pulls down the secrets
Try set the automountServiceAccountToken: false on the pod. That will ensure the service account token is not automatically mounted. So something like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-pod
spec:
...
template:
...
spec:
serviceAccountName: default
automountServiceAccountToken: false
I use the Kubernetes ServiceAccount plugin to automatically inject a ca.crt and token in to my pods. This is useful for applications such as kube2sky which need to access the API Server.
However, I run many hundreds of other pods that don't need this token. Is there a way to stop the ServiceAccount plugin from injecting the default-token in to these pods (or, even better, have it off by default and turn it on explicitly for a pod)?
As of Kubernetes 1.6+ you can now disable automounting API credentials for a particular pod as stated in the Kubernetes Service Accounts documentation
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
serviceAccountName: build-robot
automountServiceAccountToken: false
...
Right now there isn't a way to enable a service account for some pods but not others, although you can use ABAC to for some service accounts to restrict access to the apiserver.
This issue is being discussed in https://github.com/kubernetes/kubernetes/issues/16779 and I'd encourage you to add your use can to that issue and see when it will be implemented.