I cannot find a way to remove GPU (accelerator resource) from Google Kubernetes Engine (GKE) cluster. There is no official documentation on how to make change to it. Can you suggest a proper way to do so? The UI is gray out and it cannot allow me to make change from the console.
Here is the screenshot when I click to edit cluster.
Thank you
You cannot edit settings of a Node Pool once it is created.
You should create a new node pool with the settings you want (GPU, machine type etc) and delete the old node pool.
There's a tutorial on how to migrate to a new node pool smoothly here: https://cloud.google.com/kubernetes-engine/docs/tutorials/migrating-node-pool If you don't care about pods terminating gracefully, you can create a new pool and just delete the old one.
You can find more content about this at https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-upgrading-your-clusters-with-zero-downtime.
Related
Is there a way to allow access to all APIs for an existing node pool in GKE? When creating a node pool, you can choose it, but I can’t find a way to change it.
To change the API Access scope on a running GKE Cluster, you can create a new node pool with your desired scope, migrate the workload, and then delete the old node pool. In that way, the cluster will be available all the time.
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes
We have been running a cluster on GKE for around three years. As such, legacy authorization is enabled.
The control plane has been getting updated automatically, and our node pools are running a mixture of 1.12 and 1.14.
We have an increasing number of services, and are planning on incrementally adopting istio.
We want to enable a minimal RBAC setup without causing errors and downtime of our services.
I haven't been able to find any guides for how to accomplish this. Some people say just to enable RBAC authorization on the GKE cluster, but I assume that would take down all of our services.
It has also been implied that k8s can run in a hybrid ABAC/RBAC mode, but we can't tell if it is or not!
Is there a good guide for migrating to RBAC for GKE?
If you cluster is Regional you won't have downtime in your application when upgrade, but if your cluster is single-zonal or multi-zonal the best approach here is:
Add a new node pool
Cordon the old node pool to migrate the applications to the new node pool
Delete the old node pool after all pods are migrated.
It is the safesty way to update your node pool (zonal) without downtimes. Please read the references below to understand in details every step.
References:
https://kubernetes.io/docs/concepts/architecture/nodes/#reliability
https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-nodes-and-cluster
I have a cluster with several workloads and different configurations on GCP's Kubernetes Engine.
I want to create a clone of this existing cluster along with cloning all the workloads in it. It turns out, you can clone a cluster but not the workloads.
So, at this point, I'm copying the deployment yaml's of the workloads from the cluster which is working fine, and using them for the newly created workload's in the newly created cluster.
When I'm deploying the pods of this newly created workload, the pods are stuck in the pending state.
In the logs of the container, I can see that the error has something to do with Redis.
The error it shows is, Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete].
Also, when I'm connected with the first cluster and run the command,
kubectl get secrets -n=development, it shows me a bunch of secrets which are supposed to be used by my workload.
However, when I'm connected with the second cluster and run the above kubectl command, I just see one service related secret.
My question is how do I make my workload of the newly created cluster to use the configurations of the already existing cluster.
I think there are few things that can be done here:
Try to use kubectl config command and set the same context for both of your clusters.
You can find more info here and here
You may also try to use Kubernetes Cluster Federation. But bear in mind that it is still in alpha.
Remember that keeping your config in a version control system is generally a very good idea. You want to store it before the cluster applies defaults while exporting.
Please let me know if that helped.
One of our Google Kubernetes Engine clusters has lost access to Google Cloud Platform via it's main service account. It was not using the service account 'default', but a custom one, but it's now gone. Is there a way to restore or change the service account for a GKE cluster after it has been created? Or are we just out of luck and do we have to re-create the cluster?
Good news! We found a way to solve the issue without having to re-create the entire cluster.
Create a new node-pool and make sure it has the default permissions to Google Cloud Platform (this is the case if you create the pool via the Console UI).
'Force' all workloads on the new node pool (e.g. by using node labels).
Re-deploy the workloads.
Remove the old (broken ) node pool.
Hope this helps anyone with the same issue in the future.
Looks like you are out of luck. According to the documentation, gcloud container clusters update command does not let you update service account.
It's not possible to do it, either restore a service account or update the cluster for a new one, you can edit Compute Engine instances but since the cluster is managed as a group, you can't edit them, even if you could, if you had the autoscaler or the auto repair node feature, new nodes wouldn't have the new service account.
So, it seems you're out of luck, you will have to recreate the cluster.
How can I specify a specific vm type for the cluster master (I don't want to use an high memory instance for relative an inactive node).
Also, is there any way to add nodes to a cluster and choosing the type of vm? (this can solve the first problem)
Update November 2015:
Now that Google Container Engine is no longer in alpha, you don't need to worry about the size of your cluster master, as it is part of the managed service.
You can now easily add/remove nodes from your cluster through the cloud console UI but they will all be the same machine type that you originally choose for your cluster.
If you are running OSS Kubernetes on GCE, then you can set the MASTER_SIZE environment variable in cluster/gce/config-default.sh before creating your cluster.
If you are running on GKE, we unfortunately don't yet offer the option to customize the size of your master differently than the size of your nodes. We hope to offer more flexibility in cluster provisioning soon.
There is currently not a way to resize your cluster after you create it. I'm actually working on this for OSS Kubernetes in Issue #3168.