Service Account Unable to Interact with GKE Cluster in Another Project - kubernetes

Question

A service account ($SERVICE_ACCOUNT_A) from one Google Cloud Platform (GCP) project ($PROJECT_A) is unable to interact with a Google Kubernetes Engine (GKE) cluster ($GKE_CLUSTER_B) within another GCP project ($PROJECT_B); where:
$PROJECT_A is the name of the project $SERVICE_ACCOUNT_A lives within
$SERVICE_ACCOUNT_A is of the form some-name#some-project-name#.iam.gserviceaccount.com
$PROJECT_B is the name of the project the $GKE_CLUSTER_B cluster lives within
$GKE_CLUSTER_B is a GKE cluster name, not context, of the form: some_cluster
$SERVICE_ACCOUNT_A is unable to interact with a $GKE_CLUSTER_B despite possessing roles from $PROJECT_B containing permissions that should allow it to do so.
I.e., first I created a custom role $ROLE:
gcloud iam roles create $ROLE \
--description="$ROLE_DESCRIPTION" \
--permissions=container.clusters.get,container.clusters.list \
--project=$PROJECT_B \
--title='$ROLE_TITLE'
#=>
Created role [$ROLE].
description: $ROLE_DESCRIPTION
etag: . . .
includedPermissions:
- container.clusters.get
- container.clusters.list
name: projects/$PROJECT_B/roles/$ROLE
stage: . . .
title: $ROLE_TITLE
then I associated $ROLE, from $PROJECT_B, with $SERVICE_ACCOUNT_A:
gcloud projects add-iam-policy-binding $PROJECT_B \
--member=serviceAccount:$SERVICE_ACCOUNT_A \
--role=projects/$PROJECT_B/roles/$ROLE
#=>
Updated IAM policy for project [$PROJECT_B].
auditConfigs:
. . .
and I am able to see $ROLE under $SERVICE_ACCOUNT_A:
gcloud projects get-iam-policy $PROJECT_B \
--flatten='bindings[].members' \
--format='value(bindings.role)' \
--filter="bindings.members:${SERVICE_ACCOUNT_A}"
#=>
projects/$PROJECT_B/roles/$ROLE
with the proper permissions:
gcloud iam roles describe $ROLE \
--flatten='includedPermissions' \
--format='value(includedPermissions)' \
--project=$PROJECT_B
#=>
container.clusters.get
container.clusters.list
but still unable to get $SERVICE_ACCOUNT_A to interact with $GKE_CLUSTER_B.
Why?

Answer 1

You need to enable the Kubernetes Engine API (found here) for $PROJECT_A, even if $PROJECT_A doesn't have or need a GKE cluster.
You can confirm this by creating a new JSON key for $SERVICE_ACCOUNT_A:
gcloud iam service-accounts keys create \
./some-key.json \
--iam-account="${SERVICE_ACCOUNT_A}" \
--key-file-type="json"
#=>
created key [$KEY_ID] of type [json] as [./some-key.json] for [$SERVICE_ACCOUNT_A]
activate the service account:
gcloud auth activate-service-account \
"${SERVICE_ACCOUNT_A}" \
--key-file=./some-key.json
#=>
Activated service account credentials for: [$SERVICE_ACCOUNT_A]
confirm it's active:
cloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
. . .
* $SERVICE_ACCOUNT_A
your#account.user
. . .
To set the active account, run:
$ gcloud config set account `ACCOUNT`
and attempt to interact with $GKE_CLUSTER_B:
gcloud container clusters list --project=$PROJECT_B
#=>
ERROR: (gcloud.container.clusters.list) ResponseError: code=403, message=Kubernetes Engine API has not
been used in project $PROJECT_A_ID before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=$PROJECT_A_ID
then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our
systems and retry.
where $PROJECT_A_ID is a numeric id of the form: xxxxxxxxxxxxx.
Visit the address returned along with the 403 above (here) and enable the Kubernetes Engine API. $SERVICE_ACCOUNT_A should now be able to interact with GKE clusters within $PROJECT_B:
gcloud container clusters list \
--project=$PROJECT_B \
--format='value(name)
#=>
. . .
some_cluster
. . .
including $GKE_CLUSTER_B.