I am learning GCP and wanted to create a Kubernetes cluster with instance, here is what I did and what I followed with no success:
First set the region to my default us-east1-b:
xenonxie#cloudshell:~ (rock-perception-263016)$ gcloud config set compute/region us-east1-b
Updated property [compute/region].
Now proceed to create it:
xenonxie#cloudshell:~ (rock-perception-263016)$ gcloud container clusters create my-first-cluster --num-nodes 1
ERROR: (gcloud.container.clusters.create) One of [--zone, --region]
must be supplied: Please specify location.
So it seems default region/zone us-east1-b is NOT picked up
I then run the same command again with region specified explicitly:
xenonxie#cloudshell:~ (rock-perception-263016)$ gcloud container clusters create my-first-cluster --num-nodes 1 --zone us-east1-b
WARNING: Currently VPC-native is not the default mode during cluster
creation. In the future, this will become the default mode and can be
disabled using --no-enable-ip-alias flag. Use
--[no-]enable-ip-alias flag to suppress this warning. WARNING: Newly
created clusters and node-pools will have node auto-upgrade enabled by
default. This can be disabled using the --no-enable-autoupgrade
flag. WARNING: Starting in 1.12, default node pools in new clusters
will have their legacy Compute Engine instance metadata endpoints
disabled by default. To create a cluster with legacy instance metadata
endpoints disabled in the default node pool,run clusters create with
the flag --metadata disable-legacy-endpoints=true. WARNING: Your Pod
address range (--cluster-ipv4-cidr) can accommodate at most 1008
node(s). This will enable the autorepair feature for nodes. Please see
https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for
more information on node autorepairs. ERROR:
(gcloud.container.clusters.create) ResponseError: code=403,
message=Kubernetes Engine API is not enabled for this project. Please
ensure it is enabled in Google Cloud Console and try again: visit
https://console.cloud.google.com/apis/api/container.googleapis.com/overview?project=rock-perception-263016
to do so.
From the warning/error it seems I need to enable Kubernetes API, and a link is provided to me already, wonderful, I then clicked the link and it took me to enable it, which I did, right after I enabled it, I was prompt to create credential before I can use the API.
Clicking into it and choosing the right API, as you can see from the screenshot, it doesn't give me a button to create the credential:
What is missing here?
Thank you very much.
Once the API is created, you can go ahead and create the cluster. The credentials are not used when you use gcloud since the SDK will wrap the API call and use your logged-in user credentials.
As long as the Kubernetes Engine API shows as enabled, you should be able to run the same command you used and the cluster will be created. Most of those are just warnings letting you know about default settings that you did not specify
Related
I want to create a Google Cloud Kubernetes cluster programmatically in a GitLab CI script .gitlab-ci.yml in order to do e2e testing against it. Therefore I
created a separate project project-e2e (in order to separate the billing)
enabled Kubernetes Engine API
created a service account gitlab-ci#project-e2e.iam.gserviceaccount.com with a key in JSON format which I'm providing through CI variables and using as shown below
made the service account App Engine Admin, Compute Admin, Kubernetes Engine Admin, Kubernetes Engine Cluster Admin, Editor, Service Account User and Owner following permission-role mappings described at https://cloud.google.com/kubernetes-engine/docs/reference/api-permissions and https://cloud.google.com/compute/docs/access/iam
The script however fails due to missing permissions of the created service account which should be covered by the assigned roles by as far as I understand (the output contains the commands the stage in the CI script consists of):
$ echo "$GOOGLE_KEY" > key.json
$ gcloud config set project project-e2e
Updated property [core/project].
$ gcloud auth activate-service-account --key-file key.json --project project-e2e
Activated service account credentials for: [gitlab-ci#project-e2e.iam.gserviceaccount.com]
$ gcloud config set compute/zone us-central1-a
Updated property [compute/zone].
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
$ gcloud container clusters create project-e2e-$CI_COMMIT_SHORT_SHA --project project-e2e --service-account=gitlab-ci#project-e2e.iam.gserviceaccount.com
WARNING: In June 2019, node auto-upgrade will be enabled by default for newly created clusters and node pools. To disable it, use the `--no-enable-autoupgrade` flag.
WARNING: Starting in 1.12, new clusters will have basic authentication disabled by default. Basic authentication can be enabled (or disabled) manually using the `--[no-]enable-basic-auth` flag.
WARNING: Starting in 1.12, new clusters will not have a client certificate issued. You can manually enable (or disable) the issuance of the client certificate using the `--[no-]issue-client-certificate` flag.
WARNING: Currently VPC-native is not the default mode during cluster creation. In the future, this will become the default mode and can be disabled using `--no-enable-ip-alias` flag. Use `--[no-]enable-ip-alias` flag to suppress this warning.
WARNING: Starting in 1.12, default node pools in new clusters will have their legacy Compute Engine instance metadata endpoints disabled by default. To create a cluster with legacy instance metadata endpoints disabled in the default node pool, run `clusters create` with the flag `--metadata disable-legacy-endpoints=true`.
WARNING: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Required "container.clusters.create" permission(s) for "projects/project-e2e". See https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_deleted for more info.
I tried
omitting --service-account=gitlab-ci#project-e2e.iam.gserviceaccount.com which has no effect
finding answers in https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_deleted which seems unrelated
adding and omitting --no-enable-legacy-authorization which has no effect
I have a cluster on GKE whose node pool I create when I want to use the cluster, and delete when I'm done with it.
It's a two node cluster with the master in europe-west2-a and with and whose node zones are europe-west2-a and europe-west2-b.
The most recent creation resulted in the node in zone B failing with NetworkUnavailable because RouteController failed to create a route. The reason was because Could not create route xxx 10.244.1.0/24 for node xxx after 342.263706ms: instance not found.
Why would this be happening all of a sudden, and what can I do to fix it?!
You didn't mention which version of GKE you are using so just for clarification:
Changes in access scopes
Beginning with Kubernetes version 1.10, gcloud and GCP Console no longer grants the compute-rw access scope on new clusters and new node pools by default. Furthermore, if --scopes is specified in gcloud container create, gcloud no longer silently adds compute-rw or storage-ro.
In any case you can still revert to legacy access scopes but this is not recommended approach.
Hope this help.
With gke 1.13.6-gke.13, some of the default scopes were changed, including the compute-rw scope being removed. I think that due to the age of the cluster, this scope was necessary for a route to be correctly created between nodes in a node pool.
In the end, my gcloud creation command had these scopes:
--scopes https://www.googleapis.com/auth/projecthosting,storage-rw,monitoring,trace,compute-rw
I made a small setup of Kubernetes cluster. After few hours I tried to delete it. But, it impossible to delete the cloud resource.
I tried from the GCP UI:
and from with gcloud:
$ gcloud container clusters delete nat-test-cluster
The following clusters will be deleted.
- [nat-test-cluster] in [europe-west3-c]
Do you want to continue (Y/n)? Y
ERROR: (gcloud.container.clusters.delete) Some requests did not succeed:
- args: [u'ResponseError: code=404, message=Not found: projects/proj/zones/europe-west3-c/clusters/nat-test-cluster.\nCould not find [nat-test-cluster] in [europe-west3-c].\nDid you mean [nat-test-cluster] in [us-central1-a]?']
exit_code: 1
message: ResponseError: code=404, message=Not found: projects/dotnet-core-cluster/zones/europe-west3-c/clusters/nat-test-cluster.
Could not find [nat-test-cluster] in [europe-west3-c].
Did you mean [nat-test-cluster] in [us-central1-a]?
Those machines are looks like still working, but in accessible. I dont know what else to try. Contact gcp billing support to stop the billing but they said I dont have technial support plan and they can't help me. So annoying that I need to pay for support for problems not in my control.
How to delete this cluster? What to do?
If we look at the documentation for deleting a cluster found here:
https://cloud.google.com/sdk/gcloud/reference/container/clusters/delete
We find that it has an optional zone parameter. When you create (or delete) a cluster, you MUST supply a zone. If you do NOT supply a zone, your default zone (as believed by the gcloud command) will be used. In your output, we seem to see that your gcloud command thinks its DEFAULT zone is europe-west3-c while it appears that the zone in which the cluster lives is us-central1-a. I believe the solution will be to add the --zone us-central1-a parameter to your gcloud command.
I was in the same situation as you.
But I just managed to delete the cluster.
Delete all Services.
Delete all Workloads.
Delete all Applications.
Go to the Clusters page.
Push the ... icon next to the cluster you want to delete.
Select "Delete"
It would say again, there is a repairing undergoing. But you can select "Delete" again in the modal.
Voila!
I'm running kubernetes on GCE. I used kube-up.sh to create the cluster and the nodes and masters are all running the image gci-stable-56-9000-84-2. I deleted a few nodes today which triggered the autoscaler to create new ones. But they failed with the following error.
Instance 'kubernetes-minion-30gb-20180131-9jwn' creation failed: The
resource
'projects/google-containers/global/images/gci-stable-56-9000-84-2' was
not found (when acting as 'REDACTED')
Is it possible this image was deleted somehow? I don't think I changed any access controls or permissions for any service accounts.
The image is present on this page:
https://cloud.google.com/container-optimized-os/docs/release-notes#cos-stable-56-9000-84-2
This error could be due to authentication issues. Re-authenticate to the gcloud command-line tool with command ‘gcloud auth login’
It could be as well that the Kubernetes Engine service account has been deleted or edited. Check this: https://cloud.google.com/kubernetes-engine/docs/troubleshooting#error_404
When I run a simple command on my local shell with gcloud sdk.
$ kubectl get pod
I get such error:
Error from server (Forbidden): pods is forbidden: User "client" cannot list pods at the cluster scope: Unknown user "client"
The same command runs fine on GCP cloud shell, and the output of
$ gcloud auth list
is as expected:
Credentialed Accounts
ACTIVE ACCOUNT
* foo#bar.com
I also tried to create clusterrolebinding, but get similar error.
This happens when you disable Legacy Authorisation in the cluster settings, because the client certificate that you are using is a legacy authentication method. So it looks like what is happening is the client authentication succeeds but the authorisation fails, as expected. ("Unknown user" in the error message, confusingly, seems to mean the user is unknown to the authorisation system, not to the authentication system.)
You can either disable the use of the client certificate with
gcloud config unset container/use_client_certificate
and then regenerate your kubectl config with
gcloud container clusters get-credentials my-cluster
Or you can simply re-enable Legacy Authorisation in the cluster settings in the Google Cloud Console, or using the command:
gcloud container clusters update [CLUSTER_NAME] --enable-legacy-authorization
I understand this issue has now been resolved, but I would like to add some information about why this issue can occur, as it may be useful to anyone who comes across a similar issue.
Kubernetes Engine users can authenticate to the Kubernetes API using Google OAuth2 access tokens, which means that when users create a new cluster, Kubernetes Engine configures kubectl to authenticate the user to the cluster.
It's also possible to authenticate to the cluster using legacy methods which include using the cluster certificate and/or username and passwords. This is defined in the gcloud config.
The configuration of gcloud in, for example the Cloud Shell may be different from an installation of gcloud elsewhere, for example on a home workstation.
The:
Error from server (Forbidden): pods is forbidden: User "client" cannot
list pods at the cluster scope: Unknown user "client"
error suggests that gcloud config set container/use_client_certificate is set to True i.e. that gcloud is expecting a client cluster certificate to authenticate to the cluster (this is what the 'client' in the error message refers to).
As #Yanwei has discovered, unsetting container/use_client_certificate by issuing the following command in the glcoud config ends the need for a legacy certificate or credentials and prevents the error message:
gcloud config unset container/use_client_certificate
Issues such as this may be more likely if you are using an older version of gcloud on your home workstation or elsewhere.
There is some information on this here.
Found out there is some issue with gcloud config. This command solved it:
gcloud config unset container/use_client_certificate
In addition to setting
gcloud config unset container/use_client_certificate
Also make sure you do not have this env variable set to True
CLOUDSDK_CONTAINER_USE_CLIENT_CERTIFICATE