Unable to run 'kubectl' commands after using impersonation to fetch GKE cluster credentials - kubernetes

Question

My Objective
I want to use GCP impersonation to fetch my GKE cluster credentials. And then I want to run kubectl commands.
Initial Context
I have a GCP project called rakib-example-project
I have 2 ServiceAccounts in the project called:
owner#rakib-example-project.iam.gserviceaccount.com
it has the project-wide roles/owner role - so it can do anything and everything inside the GCP project
executor#rakib-example-project.iam.gserviceaccount.com
it only has the project-wide roles/iam.serviceAccountTokenCreator role - so it can impersonate the owner ServiceAccount in the GCP project
I have 1 GKE cluster in the project called my-gke-cluster
The Problem
✅ I have authenticated as the executor ServiceAccount:
$ gcloud auth activate-service-account --key-file=my_executor_sa_key.json
Activated service account credentials for: [executor#rakib-example-project.iam.gserviceaccount.com]
✅ I have fetched GKE cluster credentials by impersonating the owner:
$ gcloud container clusters get-credentials my-gke-cluster \
--zone asia-southeast1-a \
--project rakib-example-project \
--impersonate-service-account=owner#rakib-example-project.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [owner#rakib-example-project.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [owner#rakib-example-project.iam.gserviceaccount.com].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for my-gke-cluster.
❌ I am failing to list cluster nodes due to missing container.nodes.list permission:
$ kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "executor#rakib-example-project.iam.gserviceaccount.com" cannot list resource "nodes" in API group "" at the cluster scope: requires one of ["container.nodes.list"] permission(s).
But I have already impersonated the Owner ServiceAccount. Why would it still have missing permissions? 😧😧😧
My Limitations
It works well if i grant my executor ServiceAccount the roles/container.admin role. However, I am not allowed to grant such roles to my executor ServiceAccount due to compliance requirements. I can only impersonate the owner ServiceAccount and THEN do whatever I want through it - not directly.

Answer 1

If you have a look to your kubeconfig file at this location ~/.kube/config, you can see the list of authorization and the secrets, such as
- name: gke_gdglyon-cloudrun_us-central1-c_knative
user:
auth-provider:
config:
access-token: ya29.<secret>-9XQmaTQodj4kS39w
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: "2020-08-25T17:48:39Z"
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
You see external references (expiry-key and token-key) and a cmd-path. The command path is interesting because when a new token need to be generated, it will be called.
However, you see any mention of the impersonation. You have to add it in the command path, to be used by default. For this, add it in your config like this:
gcloud config set auth/impersonate_service_account owner#rakib-example-project.iam.gserviceaccount.com
Now, every use of the gcloud CLI will use the impersonate service account, and it's what you want to generate a valid access_token to reach your GKE cluster