I am trying out the capability where 2 pods deployed to the same worker node in EKS are associated to different service accounts. Below are the steps
Each service account is associated to a different role one with access to SQS and other without access.
Used eksutil to associate OIDC provider with cluster and also created iamserviceaccount with service account in kubernetes and role with policy for accessing SQS attached (implicit annotation of service account with IAM role provided by eksctl create iamserviceaccount).
But when I try to start the pod which has service account tied to role with SQS access, I am getting access denied for SQS, however if I add SQS permissions to worker node instance role, its working fine.
Am I missing any steps and is my understanding correct?
So, there are a few things required to get IRSA to work:
There has to be an OIDC provider associated with the cluster, following the directions here.
The IAM role has to have a trust relationship with the OIDC provider, as defined in the AWS CLI example here.
The service account must be annotated with a matching eks.amazonaws.com/role-arn.
The pod must have the appropriate service account specified with a serviceAccountName in its spec, as per the API docs.
The SDK for the app needs to support the AssumeRoleWithWebIdentity API call. Weirdly, the aws-sdk-go-v2 SDK doesn't currently support it at all (the "old" aws-sdk-go does).
It's working with the node role because one of the requirements above isn't met, meaning the credential chain "falls through" to the underlying node role.
Related
I'm a little confused about the default Service Account in new created Namespace in my Minikube.
Does it have any permissions? It seems not because I can't find any rolebinding or clusterrolebindung which references this SA
Then why is it created when it does not have a permission, or is there a use case around that?
and lastly, why are service accounts by default mount to pods?
Regards
ralph
The default service account doesn’t have enough permissions to retrieve the services running in the same namespace.
Kubernetes follows the convention of closed-to-open which means that by default no user or service account has any permissions.
To fulfill this request, we need to create a role binding associating the default service account with an appropriate role.This is similar to how we assign a viewer role to the service account that can give permission to list pods.
Pods have the default service account assigned even when you don’t ask for it. This is because every pod in the cluster needs to have one (and only one) service account assigned to it.
Refer Kubernetes namespace default service account for more information.
I have a very specific scenario I'm trying to solve for:
Using Kubernetes (single cluster)
Installing Vault on that cluster
sending GitLab containers to the same cluster.
I need to install Vault in such a way that:
Vault lives in it's own namespace (easy/solved)
Vault's service account (vault-auth) is available to all other namespaces (unsolved)
GitLab's default behavior is to put all apps/services into their own namespaces with the Project ID; EG: repo_name+project_id. It's predictable but the two options are:
When the app is in its own namespace it cannot access the Vault service account in the 'vault' Namespace. It requires you to create a vault service account in each application namespace; hot garbage, or...
Put ALL apps + Vault in the default namespace and applications can easily find the 'vault-auth' service account. Messy but totally works.
To use GitLab in the way it is intended (and I don't disagree) is to leave each app in it's own namespace. The question then becomes:
How would one create the Kubernetes Service Account for Vault (vault-auth) so that Vault the application is in it's own namespace but the service account itself is available to ALL namespaces?
Then, no matter the namespace that GitLab creates, the containers have equal access to the 'vault-auth' service account.
I've recently been making use of the GKE Workload Identity feature. I'd be interested to know in more detail how the gke-metadata-server component works.
GCP client code (gcloud or other language SDKs) falls through to the GCE metadata method
Request made to http://metadata.google.internal/path
(guess) Setting GKE_METADATA_SERVER on my node pool configures this to resolve to the gke-metadata-server pod on that node.
(guess) the gke-metadata-server pod with --privileged and host networking has a means of determining the source (pod IP?) then looking up the pod and its service account to check for the iam.gke.io/gcp-service-account annotation.
(guess) the proxy calls the metadata server with the pods 'pseudo' identity set (e.g. [PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]) to get a token for the service account annotated on its Kubernetes service account.
If this account has token creator / workload ID user rights to the service account presumably the response from GCP is a success and contains a token, which is then packaged and set back to the calling pod for authenticated calls to other Google APIs.
I guess the main puzzle for me right now is the verification of the calling pods identity. Originally I thought this would use the TokenReview API but now I'm not sure how the Google client tools would know to use the service account token mounted into the pod...
Edit follow-up questions:
Q1: In between step 2 and 3, is the request to metadata.google.internal routed to the GKE metadata proxy by the setting GKE_METADATA_SERVER on the node pool?
Q2: Why does the metadata server pod need host networking?
Q3: In the video here: https://youtu.be/s4NYEJDFc0M?t=2243 it's taken as a given that the pod makes a GCP call. How does the GKE metadata server identify the pod making the call to start the process?
Before going into details, please familiarize yourself with these components:
OIDC provider: Runs on Google’s infrastructure, provides cluster specific metadata and signs authorized JWTs.
GKE metadata server: It runs as a DaemonSet meaning one instance on every node, exposes pod specific metadata server (it will provide backwards compatibility with old client libraries), emulates existing node metadata server.
Google IAM: issues access token, validates bindings, validates OIDC signatures.
Google cloud: accepts access tokens, does pretty much anything.
JWT: JSON Web token
mTLS: Mutual Transport Layer Security
The steps below explain how GKE metadata server components work:
Step 1: An authorized user binds the cluster to the namespace.
Step 2: Workload tries to access Google Cloud service using client libraries.
Step 3: GKE metadata server is going to request an OIDC signed JWT from the control plane. That connection is authenticated using mutual TLS (mTLS) connection with node credential.
Step 4: Then the GKE metadata server is going use that OIDC signed JWT to request an access token for the [identity namespace]/[Kubernetes service account] from IAM. IAM is going to validate that the appropriate bindings exist on identity namespace and in the OIDC provider.
Step 5: And then IAM validates that it was signed by the cluster’s correct OIDC provider. It will then return an access token for the [identity namespace]/[kubernetes service account].
Step 6: Then the metadata server sends the access token it just got back to IAM. IAM will then exchange that for a short lived GCP service account token after validating the appropriate bindings.
Step 7: Then GKE metadata server returns the GCP service account token to the workload.
Step 8: The workload can then use that token to make calls to any Google Cloud Service.
I also found a video regarding Workload Identity which you will find useful.
EDIT Follow-up questions' answers:
Below are answers to your follow-up questions:
Q1: In between step 2 and 3, is the request to metadata.google.internal routed to the gke metadata proxy by the setting GKE_METADATA_SERVER on the node pool?
You are right, GKE_METADATA_SERVER is set on the node pool. This exposes a metadata API to the workloads that is compatible with the V1 Compute Metadata APIs. Once workload tries to access Google Cloud service, the GKE metadata server performs a lookup (the metadata server checks to see if a pod exists in the list whose IP matches the incoming IP of the request) before it goes on to request the OIDC token from the control plane.
Keep in mind that GKE_METADATA_SERVER enumeration feature can only be enabled if Workload Identity is enabled at the cluster level.
Q2: Why does the metadata server pod need host networking?
The gke-metadata-server intercepts all GCE metadata server requests from pods, however pods using the host network are not intercepted.
Q3: How does the GKE metadata server identify the pod making the call to start the process?
The pods are identified using iptables rules.
I'm developing a service running in Google Kubernetes Engine and I would like to use Google Cloud functionality from that service.
I have created a service account in Google Cloud with all the necessary roles and I would like to use these roles from the pod running my service.
I have read this: https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform
and I was wondering if there is an easier way to "connect" the two kinds of service accounts ( defined in Kubernetes - defined in Google Cloud IAM ) ?
Thanks
I don't think there is any direct link. K8s service accounts are purely internal. You could try granting GIAM permissions to serviceaccount:name but that seems unlikely to work. More likely you would put the Google SA credentials in a secret and then write an RBAC policy giving your K8s SA read access to it.
Read the topic which I have shared. You need to enable Workload Identity on your cluster and then you can annotate Kubernetes service account with IAM on google.
gke-document
I created a kubernetes cluster under my user account on IBM Bluemix, and added another into my organization. But he can't see my cluster. Is there any other configure?
To manage cluster access, see this link from the IBM Bluemix Container Service documentation. Summarised here:
Managing cluster access
You can grant access to your cluster to other users, so that they can
access the cluster, manage the cluster, and deploy apps to the
cluster.
Every user that works with IBM Bluemix Container Service must be
assigned a service-specific user role in Identity and Access
Management that determines what actions this user can perform.
Identity and Access Management differentiates between the following
access permissions.
IBM Bluemix Container Service access policies
Access policies determine the cluster management actions that you can
perform on a cluster, such as creating or removing clusters, and
adding or removing extra worker nodes.
Cloud Foundry roles
Every user must be assigned a Cloud Foundry user role. This role
determines the actions that the user can perform on the Bluemix
account, such as inviting other users, or viewing the quota usage. To
review the permissions of each role, see Cloud Foundry roles.
RBAC roles
Every user who is assigned an IBM Bluemix Container Service access
policy is automatically assigned an RBAC role. RBAC roles determine
the actions that you can perform on Kubernetes resources inside the
cluster. RBAC roles are set up for the default namespace only. The
cluster administrator can add RBAC roles for other namespaces in the
cluster. See Using RBAC Authorization in the
Kubernetes documentation for more information.