I’m trying to create a cluster in GKE project-1 with shared network of project-2.
Roles given to Service account:
project-1: Kubernetes Engine Cluster Admin, Compute Network Admin, Kubernetes Engine Host Service Agent User
project-2: Kubernetes Engine Service Agent, Compute Network User, Kubernetes Engine Host Service Agent User
Service Account is created under project-1.
API & Services are enabled in both Projects.
But I am getting this error persistently.
Error: googleapi: Error 403: Kubernetes Engine Service Agent is missing required permissions on this project. See Troubleshooting | Kubernetes Engine Documentation | Google Cloud for more info: required “container.hostServiceAgent.use” permission(s) for “projects/project-2”., forbidden
data "google_compute_network" "shared_vpc" {
name = "network-name-in-project-2"
project = "project-2"
}
data "google_compute_subnetwork" "shared_subnet" {
name = "subnet-name-in-project-2"
project = "project-2"
region = "us-east1"
}
# cluster creation under project 1
# project 1 specified in Provider
resource "google_container_cluster" "mowx_cluster" {
name = var.cluster_name
location = "us-east1"
initial_node_count = 1
master_auth {
username = ""
password = ""
client_certificate_config {
issue_client_certificate = false
}
}
remove_default_node_pool = true
cluster_autoscaling {
enabled = false
}
# cluster_ipv4_cidr = var.cluster_pod_cidr
ip_allocation_policy {
cluster_secondary_range_name = "pods"
services_secondary_range_name = "svc"
}
network = data.google_compute_network.shared_vpc.id
subnetwork = data.google_compute_subnetwork.shared_subnet.id
}
This is a community wiki answer based on the discussion in the comments and posted for better visibility. Feel free to expand it.
The error you encountered:
Error: googleapi: Error 403: Kubernetes Engine Service Agent is missing required permissions on this project. See Troubleshooting | Kubernetes Engine Documentation | Google Cloud for more info: required “container.hostServiceAgent.use” permission(s) for “projects/project-2”., forbidden
means that the necessary service agent was not created:
roles/container.serviceAgent - Kubernetes Engine Service Agent:
Gives Kubernetes Engine account access to manage cluster resources.
Includes access to service accounts.
The official troubleshooting docs describe a solution for such problems:
To resolve the issue, if you have removed the Kubernetes Engine Service Agent role from your Google Kubernetes Engine service account,
add it back. Otherwise, you must re-enable the Kubernetes Engine API,
which will correctly restore your service accounts and permissions.
You can do this in the gcloud tool or the Cloud Console.
The solution above works as in your use case the account was missing so it had to be (re)created.
For me, even though the gke serice account existed and had the roles Kubernetes Engine Host Service Agent and Kubernetes Engine Service Agent in the both the service and host projects, I still got the 443 error.
The problem was that the service account needed to have roles/compute.networkUser and roles/compute.instanceAdmin applied to the VPC's subnetwork binding of the VPC.
See: resource google_compute_subnetwork_iam_binding
See also module "shared_vpc_access"
Related
I am trying to create an end point for an API to be deployed into existing GKE cluster by following the instructions in Getting started with Cloud Endpoints for GKE with ESPv2
I clone the sample code in the repo and modified the content of openapi.yaml:
# [START swagger]
swagger: "2.0"
info:
description: "A simple Google Cloud Endpoints API example."
title: "Endpoints Example"
version: "1.0.0"
host: "my-api.endpoints.my-project.cloud.goog"
I then deployed it via the command:
endpoints/getting-started (master) $ gcloud endpoints services deploy openapi.yaml
Now I can see that it has been created:
$ gcloud endpoints services list
NAME TITLE
my-api.endpoints.my-project.cloud.goog
I also have postgreSQL service account:
$ gcloud iam service-accounts list
DISPLAY NAME EMAIL DISABLED
my-postgresql-service-account my-postgresql-service-acco#my-project.iam.gserviceaccount.com False
In the section Endpoint Service Configuration of documentation it says to add the role to the attached service account for the endpoint service as follows, but I get this error:
$ gcloud endpoints services add-iam-policy-binding my-api.endpoints.my-project.cloud.goog
--member serviceAccount:my-postgresql-service-acco#my-project.iam.gserviceaccount.com
--role roles/servicemanagement.serviceController
ERROR: (gcloud.endpoints.services.add-iam-policy-binding) User [myusername#mycompany.com] does not have permission to access services instance [my-api.endpoints.my-project.cloud.goog:getIamPolicy] (or it may not exist): No access to resource: services/my-api.my-project.cloud.goog
The previous lines show the service exits, I guess? Now I am not sure how to resolve this? What permissions do I need? who can give me permission and what permissions he should have? how can I check? Is there any other solution?
The issue got resolved after I was assigned the role of "Project_Admin". It was not ideal as it was giving too much permission to me. The role "roles/endpoints.portalAdmin" was also tried but did not help.
I am trying to use Google's preferred "Workload Identity" method to enable my GKE app to securely access secrets from Google Secrets.
I've completed the setup and even checked all steps in the Troubleshooting section (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?hl=sr-ba#troubleshooting) but I'm still getting the following error in my logs:
Unhandled exception. Grpc.Core.RpcException:
Status(StatusCode=PermissionDenied, Detail="Permission
'secretmanager.secrets.list' denied for resource
'projects/my-project' (or it may not exist).")
I figured the problem was due to the node pool not using the correct service account, so I recreated it, this time specifying the correct service account.
The service account has the following roles added:
Cloud Build Service
Account Kubernetes Engine Developer
Container Registry Service Agent
Secret Manager Secret Accessor
Secret Manager Viewer
The relevant source code for the package I am using to authenticate is as follows:
var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
var request = new ListSecretsRequest
{
ParentAsProjectName = ProjectName.FromProject(projectName),
};
var secrets = secretManagerServiceClient.ListSecrets(request);
foreach(var secret in secrets)
{
var value = secretManagerServiceClient.AccessSecretVersion($"{secret.Name}/versions/latest");
string secretVal = this.manager.Load(value.Payload);
string configKey = this.manager.GetKey(secret.SecretName);
data.Add(configKey, secretVal);
}
Data = data;
Ref. https://github.com/jsukhabut/googledotnet
Am I missing a step in the process?
Any idea why Google is still saying "Permission 'secretmanager.secrets.list' denied for resource 'projects/my-project' (or it may not exist)?"
Like #sethvargo mentioned in the comments, you need to map the service account to your pod because Workload Identity doesn’t use the underlying node identity and instead maps a Kubernetes service account to a GCP service account. Everything happens at the per-pod level in Workload identity.
Assign a Kubernetes service account to the application and configure it to act as a Google service account.
1.Create a GCP service account with the required permissions.
2.Create a Kubernetes service account.
3.Assign the Kubernetes service account permission to impersonate the GCP
service account.
4.Run your workload as the Kubernetes service account.
Hope you are using project ID instead of project name in the project or secret.
You cannot update the service account of an already created pod.
Refer the link to add service account to the pods.
This my from terraform.tf
terraform {
backend "azurerm" {
resource_group_name = "tstate"
storage_account_name = "strorageaccount1"
container_name = "terraform.tfstate"
access_key = "asdfg45454..."
}
}
This fails when my storage account is not in "all networks". My settings of storage account network is given below. Blob storage private or public it works so no problem there. But "all networks" must be enabled for it to work. How can I make it work with "all networks" disabled? The error I get is as follows:
Error: Failed to get existing workspaces: storage: service returned
error: StatusCode=403, ErrorCode=AuthorizationFailure,
ErrorMessage=This request is not authorized to perform this operation.
There is no IP or Vnet needed as Azure default agent is running the devops pipeline. And the SPN has owner access on subscription. What am I missing?
Well, you explicitly forbid almost any service (or server) to access your storage account. with the exception of "trusted Microsoft services". However, your Azure DevOps Build Agent does not fall under that category.
So, you need to whitelist your build agent first. There are two ways you can do this:
Use a self-hosted agent that you run inside a VNET. Then allow access from that VNET in your firewall rules of your storage account
If you want to stick with managed build agents: Run a AZ CLI or Azure Powershell script first, that does fetch the public IP of your build agent (https://api.ipify.org) and add that to your firewall. After terraform finished, have another script that removes that IP exception again.
I need to authenticate to a Kubernetes cluster provisioned in GKE using the Kubernetes Python client and the Google Cloud python client. I would prefer not to shell out to gcloud for several reasons:
relying on the system shell gcloud in a Python script when I have a native Google Cloud library is inelegant
it requires the system to have gcloud
I would have to switch users to the relevant ServiceAccount and switch back
It incurs the cost of starting/joining another process
As such, the workflow of gcloud container clusters get-credentials (which delegates to gcloud config config-helper) will not suffice to get me the API key I need. How do I get the equivalent output with the Google Cloud Python API?
Here is what I have so far:
import kubernetes.client
import googleapiclient.discovery
import base64
# get the cluster object from GKE
gke = googleapiclient.discovery.build('container', 'v1', credentials=config['credentials'])
name = f'projects/{config["project_id"]}/locations/{config["location"]}/{parent}/clusters/{config["name"]}'
gke_clusters = gke.projects().locations().clusters()
gke_cluster = gke_clusters.get(name=name).execute()
# set up Kubernetes Config
kube_config = kubernetes.client.Configuration()
kube_config.host = 'https://{0}/'.format(gke_cluster['endpoint'])
kube_config.verify_ssl = True
#kube_config.api_key['authenticate'] = "don't know what goes here"
# regretably, the Kubernetes client requires `ssl_ca_cert` to be a path, not the literal cert, so I will write it here.
kube_config.ssl_ca_cert = 'ssl_ca_cert'
with open(kube_config.ssl_ca_cert, 'wb') as f:
f.write(base64.decodestring(gke_cluster['masterAuth']['clusterCaCertificate'].encode()))
# use Kubernetes client to do something
kube_client = kubernetes.client.ApiClient(configuration=kube_config)
kube_v1 = kubernetes.client.CoreV1Api(kube_client)
kube_v1.list_pod_for_all_namespaces(watch=False)
Below is a solution that pulls the access token out of the googleapiclient, rather than copy-pasting things manually.
import googleapiclient.discovery
from tempfile import NamedTemporaryFile
import kubernetes
import base64
def token(*scopes):
credentials = googleapiclient._auth.default_credentials()
scopes = [f'https://www.googleapis.com/auth/{s}' for s in scopes]
scoped = googleapiclient._auth.with_scopes(credentials, scopes)
googleapiclient._auth.refresh_credentials(scoped)
return scoped.token
def kubernetes_api(cluster):
config = kubernetes.client.Configuration()
config.host = f'https://{cluster["endpoint"]}'
config.api_key_prefix['authorization'] = 'Bearer'
config.api_key['authorization'] = token('cloud-platform')
with NamedTemporaryFile(delete=False) as cert:
cert.write(base64.decodebytes(cluster['masterAuth']['clusterCaCertificate'].encode()))
config.ssl_ca_cert = cert.name
client = kubernetes.client.ApiClient(configuration=config)
api = kubernetes.client.CoreV1Api(client)
return api
def run(cluster):
"""You'll need to give whichever account `googleapiclient` is using the
'Kubernetes Engine Developer' role so that it can access the Kubernetes API.
`cluster` should be the dict you get back from `projects.zones.clusters.get`
and the like"""
api = kubernetes_api(cluster)
print(api.list_pod_for_all_namespaces())
Figuring this out took longer than I care to admit. #Ivan's post helped a lot.
In order to authenticate to a GKE cluster, you can use a service account to connect to a project and then a generated secret key from GKE to authenticate to a cluster. Here are the steps:
Create a service account in GCP. Go to IAM > Service Accounts > create a service account. Give it a Project Owner role. Once SA is created, create a key and download it as json.
Upload key.json to a folder where you have .py script
Get API_TOKEN. This is your main question, you can get it by reading a token file:
First run kubectl get secrets
You will get ‘default-token-xxxxx’
run kubectl describe secrets default-token-xxxxx (replace xxxxx with your token name).
The token parameter displayed is your “API-KEY”. Copy it inside your script.
Creating a script. It is a bit different then yours for few reasons: you need to authenticate to a project first with a service account, then you need to pass the api_token, but also you need to get SSL certificate when authenticating to GKE master.
import base64, pprint
import kubernetes.client
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file("key.json")
gke = googleapiclient.discovery.build('container', 'v1', credentials=credentials)
name = 'projects/your_project/locations/your_zone/clusters/your_gke_cluster'
gke_clusters = gke.projects().locations().clusters()
gke_cluster = gke_clusters.get(name=name).execute()
kube_config = kubernetes.client.Configuration()
kube_config.host = 'https://{}'.format(gke_cluster['endpoint'])
kube_config.verify_ssl = True
kube_config.api_key['authorization'] = 'your_api_token'
kube_config.api_key_prefix['authorization'] = 'Bearer'
kube_config.ssl_ca_cert = 'ssl_ca_cert'
with open(kube_config.ssl_ca_cert, 'wb') as f:
f.write(base64.decodestring(gke_cluster['masterAuth']['clusterCaCertificate'].encode()))
kube_client = kubernetes.client.ApiClient(configuration=kube_config)
kube_v1 = kubernetes.client.CoreV1Api(kube_client)
pprint.pprint(kube_v1.list_pod_for_all_namespaces())
Specific fields:
your_project - from GCP
your _zone - where gke cluster is created
your _gke_cluster - GKE cluster name
your_api_key - what you get in step 3.
This should be enough to authenticate you to a GKE cluster.
I have a gcp service account with kubernetes engine admin role, I get an error trying to create new cluster. Any ideas?
gcloud auth activate-service-account --key-file service_account.json
gcloud config set project my_proj_name
gcloud container clusters create dummy --num-nodes=3 -m n1-standard-8 --zone europe-west1-b
ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=The user does not have access to service account "default".
.
Issue solved after setting the service account role to the 'Editor' since kubernetes Engine Admin was not sufficient to create new cluster.
Seems your user does not have access to the service account or service account does not have the right role. You can check service account user role to your user and give Editor, compute admin and service account actor roles.