Currently, we run kops based cluster of the version 15. We are planning to upgrade it to the version 16 first and then further. However, api versions for various kubernetes services in yaml's will also need to change. How would you address this issue before the cluster upgrade? Is there any way to enumerate all objects in the cluster with incompatible api versions or what would be the best approach for it? I suspect the objects created by kops, e.g. kube-system objects will be upgraded automatically.
When you upgrade the cluster, the API server will take care to upgrade all existing resources in the cluster. The problem arise when you want to deploy more resources and after the upgrade these are still using the old API versions. In this case your deployment (say kubectl apply) will fail.
I.e nothing already running in the cluster will break. But future deployments will if they still use old versions.
The resources managed by kOps already use new API versions.
Related
I want to roll back to specific version of kubernetes. My current version is 1.21.
Is there any system specifications for kubernetes?
If you are using a managed service, you probably won't be able to roll back, and I would strongly recommend AGAINST rolling back even if you can.
Managed services like GKE, AKS and EKS will only allow you to pick from the latest couple of versions (normally between 3-4 minor versions), but will not allow you to downgrade a minor version (e.g. you can't downgrade from 1.21 to 1.20 for example see here for GKE example)
Rolling back a version will re-introduce any bugs and security issues that were fixed by the upgrade. So essentially, you are making your cluster less secure by downgrading.
Clients such as kubectl will also flag up skew warnings such as in this question, and the rolled back clusters will start rejecting deployments if you've already updated them for new apiVersions
For example if the version you migrated from had an API version something/v1beta and the new version required you to use something/v1 then if you tried to deploy a deployment on the rolled back cluster that used something/v1 (to meet the new cluster version), the rolled back cluster would reject that.
I have production stage hosted in Google Kubernetes Engine with Kubernetes version 1.12.9-gke.15.
My team is planning to upgrade it to Kubernetes version 1.13.11-gke.5.
A capture of list of Kubernetes version
I have read some articles to upgrade Kubernetes. However, they use kubeadm not GKE.
How to update api versions list in Kubernetes here's a example that use GKE.
If you guys have experience in upgrading kubernetes cluster in GKE or even kubeadm. Please share what should i do before upgrading the version ?
Should i upgrade the version to 1.13.7-gke.24 and then to 1.13.9-gke.3 and so on ?
You first should check if you are not using any depreciated features. For example check the Changelogs for version 1.12 and 1.13 to make sure you won't loose any functionality after the upgrade.
You will have to remember that if you have just one master node you will loose access to if for few minutes while control plane is being updated. After master node is set then worker nodes will follow.
There is a great post about Kubernetes best practices: upgrading your clusters with zero downtime, which talks about location for nodes and a beta option being Regional
When creating your cluster, be sure to select the “regional” option:
And that’s it! Kubernetes Engine automatically creates your nodes and masters in three zones, with the masters behind a load-balanced IP address, so the Kubernetes API will continue to work during an upgrade.
And they explain how does Rolling update works and how to do them.
Also you might consider familiarizing yourself with documentation for Cluster upgrades, as it discusses how automatic and manual upgrades work on GKE.
As you can see from your current version 1.12.9-gke.15 you cannot upgrade to 1.14.6-gke.1. You will need to upgrade to 1.13.11-gke.5 and once this is done you will be able to upgrade to latest GKE version.
GCP Kubernetes is upgraded manually and generally does not require you to do much. But if you are you looking for manual upgrade options maybe this will help.
https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster
A point worth mentioning is too, make sure you have persistence volumes for services that require to do so viz. like DB, etc And for these, you will have to back them up manually.
I have a problem to migrate kubernetes cluster to other google project, not so familiar with GKE. Assuming my cluster is k8s-prod-xyz in xyz-proj project.
Now, i have a new project called xyz-new-proj and the Kubernetes cluster is still empty. I want to move or migrate the k8s-prod-xyz from xyz-proj to xyz-new-proj.
Node, PVC, Services, etc should be transfered or migrated. Have you guys experienced this case ? Or should i create new Kubernetes cluster in new project and then run the deployment from zero ?
You can use GKE feature Clone an existing cluster (however this works only within the same project) along with Heptio Velero tool. I guess the solution described in this article is currently the fastest and most convenient way of performing such migration.
When the storage version of a Kubernetes API resource changes, is it still necessary to manually read and write back resources as describe here or does the apiserver now deal with this automatically?
For example, if I wanted to remove the deprecated extensions/v1beta1 version of deployments from my cluster and migrate to apps/v1 would it be enough to specify --storage-versions=extensions=apps/v1 on the apiserver and then ‘wait for a bit’ before setting something like ---runtime-config=api/all=true,extensions/v1beta1/deployments=false? Or would I have to use the update-storage-objects.sh script after setting --storage-versions=extensions=apps/v1?
Additionally, would specifying --storage-versions=extensions=apps/v1 cause any issues for ingress resources that still use API version extensions/v1beta1 but have no conversion to apps/v1?
does the apiserver now deal with this automatically?
No, the api-server does not do it automatically, you need to do it manually.
Regarding the upgrade between API versions, all necessary steps are described in the official documentation:
This is an infrequent event, but it requires careful management. There
is a sequence of steps to upgrade to a new API version.
Turn on the new API version.
Upgrade the cluster’s storage to use the new version.
Upgrade all config files. Identify users of the old API version endpoints.
Update existing objects in the storage to new version by running cluster/update-storage-objects.sh.
Turn off the old API version.
Step 4 is not only about storage but also about all resources related to the old version which you have in the cluster.
Additionally, would specifying --storage-versions=extensions=apps/v1 cause any issues for ingress resources that still use API version extensions/v1beta1 but have no conversion to apps/v1?
Versioning of each type of resource is independent. Storage and Ingress are different resources so there are no relations between their versions and different versions should not affect them in any way.
The recommended method for doing this is still in flux. Removing API versions is currently prohibited: https://github.com/kubernetes/kubernetes/issues/52185
Im usually upgrading the cluster with a new API version then upgrade the config files but not removing the old API. Only once I had to remove a old API version due to a bug. You can do this by running kubectl get apiservice to list all available versions then kubectl delete apiservice some_api and you don't have to set any other flag.
I began to try Google Container Engine recently. I would you like to upgrade the Kubernetes Cluster to the latest version available, if possible without downtime. Is there any way to do this?
Unfortunately, the best answer we currently have is to create a new cluster and move your resources over, then delete the old one.
We are very actively working on making cluster upgrades reliable (both nodes and the master), but upgrades are unlikely to work for the majority of currently existing clusters.
We now have a checked-in upgrade tool for master and nodes: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/gce/upgrade.sh