We are developing k8s CSI driver
Currently in order to upgrade driver we delete the installed operator pods, cdrs and roles and recreate them from new version images.
What is suggested way to do upgrade? Or is uninstall/install is the suggested method?
I couldn't find any relevant information
We also have support of installing from OpenShift. Is there any difference regarding upgrade from OpenShift?
You should start from this documentation:
This page describes to CSI driver developers how to deploy their
driver onto a Kubernetes cluster.
Especially:
Deploying a CSI driver onto Kubernetes is highlighted in detail in
Recommended Mechanism for Deploying CSI Drivers on Kubernetes.
Also, you will find there all the necessary info with an example.
Your question lacks some details regarding your use case but I strongly recommend starting from the guide I have presented you.
Please, let me know if that helps.
CSI drivers can differ, but I believe the best approach is to do rolling update of your plugin's DaemonSet. It will happen automatically once you apply the new DaemonSet configuration, e.g. newer docker image.
For more details, see https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
For example:
kubectl get -n YOUR-NAMESPACE daemonset YOUR-DAEMONSET --export -o yaml > plugin.yaml
vi plugin.yaml # Update your image tag(s)
kubectl apply -n YOUR-NAMESPACE -f plugin.yaml
A shorted way to update just the image:
kubectl set image ds/YOUR-DAEMONSET-NAME YOUR-CONTAINER-NAME=YOUR-IMAGE-URL:YOUR-TAG -n YOUR-NAMESPACE
Note: I found that I also needed to restart (kill) the pod with the external provisioner. There's probably a more elegant way to handle this, but it works in a pinch.
kubectl delete pod -n YOUR-NAMESPACE YOUR-EXTERNAL-PROVISIONER-POD
Related
As the question shows, I have very low knowledge about kubernetes. Following a tutorial, I made a Kubernetes cluster to run a web app on a local server using Minikube. I have applied the kubernetes components and they are running but the Web-Server does not respond to HTTP requests. My problem is that all the system that I have created is like a black box for me and I have literally no idea how to open it and see where the problem is. Can you explain how I can debug such implementaions in a wise way. Thanks.
use a tool like https://github.com/kubernetes/kubernetes-dashboard
You can install kubectl and kubernetes-dashboard in a k8s cluster (https://kubernetes.io/docs/tasks/tools/install-kubectl/), and then use the kubectl command to query information about a pod or container, or use the kubernetes-dashboard web UI to query information about the cluster.
For more information, please refer to https://kubernetes.io/
kubectl get pods
will show you all your pods and their status. A quick check to make sure that all is at least running.
If there are pods that are unhealthy, then
kubectl describe pod <pod name>
will give some more information.. eg image not found etc
kubectl log <pod name> --all
is often the next step , use -f to follow the logs as you exercise your api.
It is possible to link up images running in a pod with most ide debuggers, but instructions will differ depending on language and ide used...
I'm running GKE cluster and there is a deployment that uses image which I push to Container Registry on GCP, issue is - even though I build the image and push it with latest tag, the deployment keeps on creating new pods with the old one cached - is there a way to update it without re-deploying (aka without destroying it first)?
There is a known issue with the kubernetes that even if you change configmaps the old config remains and you can either redeploy or workaround with
kubectl patch deployment $deployment -n $ns -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
is there something similar with cached images?
I think you're looking for kubectl set or patch which I found there in kubernetes documentation.
To update image of deployment you can use kubectl set
kubectl set image deployment/name_of_deployment name_of_deployment=image:name_of_image
To update image of your pod you can use kubectl patch
kubectl patch pod name_of_pod -p '{"spec":{"containers":[{"name":"name_of_pod_from_yaml","image":"name_of_image"}]}}'
You can always use kubectl edit to edit which allows you to directly edit any API resource you can retrieve via the command line tool.
kubectl edit deployment name_of_deployment
Let me know if you have any more questions.
1) You should change the way of your thinking. Destroying pod is not bad. Application downtime is what is bad. You should always plan your deployments in such a way that it can tolerate one pod death. Use multiple replicas for stateless apps and use clusters for stateful apps. Use Kubernetes rolling update for any changes to your deployments. Rolling updates have many extremely important settings which directly influence the uptime of your apps. Read it carefully.
2) The reason why Kubernetes launches old image is that by default it uses
imagePullPolicy: IfNotPresent. Use imagePullPolicy: Always and it will always try to pull latest version on redeploy.
One of the points in the kubectl best practices section in Kubernetes Docs state below:
Pin to a specific generator version, such as kubectl run
--generator=deployment/v1beta1
But then a little down in the doc, we get to learn that except for Pod, the use of --generator option is deprecated and that it would be removed in future versions.
Why is this being done? Doesn't generator make life easier in creating a template file for resource definition of deployment, service, and other resources? What alternative is the kubernetes team suggesting? This isn't there in the docs :(
kubectl create is the recommended alternative if you want to use more than just a pod (like deployment).
https://kubernetes.io/docs/reference/kubectl/conventions/#generators says:
Note: kubectl run --generator except for run-pod/v1 is deprecated in v1.12.
This pull request has the reason why generators (except run-pod/v1) were deprecated:
The direction is that we want to move away from kubectl run because it's over bloated and complicated for both users and developers. We want to mimic docker run with kubectl run so that it only creates a pod, and if you're interested in other resources kubectl create is the intended replacement.
For deployment you can try
kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
and
Note: kubectl run --generator except for run-pod/v1 is deprecated in v1.12.
I am using a POD directly to manage our C* cluster in a K8s cluster, not using any high-level controller. When I want to upgrade C*, I want to do an image update. Is this a good pattern to update image for an upgrade?
I saw the high-level deployment controller support image update too, but that causes the POD to delete and recreate, which in turn causes the IP to change. I don't want to change the IP and I found if I directly update the POD image, it can cause a restart and also keep the IP. This is the exact behavior I want, is this pattern right?
Is it safe to use in production?
I believe you can follow the K8s documentation for a more 'production ready' upgrade strategy. Basically, use the updateStrategy=RollingUpdate:
$ kubectl patch statefulset cassandra -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}
and then update the image:
$ kubectl patch statefulset web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cassandra:next-version"}]'
and watch your updates:
$ kubectl get pod -l app=cassandra -w
There's also Staging the Update in case you'd like to update each C* node individually, for example, why if the new version turns out to be incompatible, then you can revert that C* back to the original version.
Also, familiarize with all the Cassandra release notes before doing the upgrade.
enter image description hereI tried to used the instructions from this link https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md but I was not able to install it. specifically I dont know what this instruction means "Ensure that kubecfg.sh is exported." I dont even know where I can find this I did this sudo find / -name "kubecfg.sh" and I found no results.
moving on to the next step "kubectl create -f deploy/kube-config/influxdb/" when I did this it says kube-system not found I am using latest version of kubernetes version 1.0.1
These instructions are broken can any one provide some instructions on how to install this? I have kubernetes cluster up and running I was able to create and delete pods and so on and default is the only namespace I have when i do kubectl get pods,svc,rc --all-namespaces
Changing kube-system to default in the yaml files is just getting me one step further but I am unable to access the UI and so on. so installing kube-system makes more sense however I dont know how to do it and any instructions on installing influxdb and grafana to get it up and running will be very helpful
I am using latest version of kubernetes version 1.0.1
FYI, the latest version is v1.2.3.
... it says kube-system not found
You can create the kube-system namespace by running
kubectl create namespace kube-system.
Hopefully once you've created the kube-system namespace the rest of the instructions will work.
We had the same issue deploying grafana/influxdb. So we dug into it:
Per https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md since we don’t have an external load balancer, we changed the port type on the grafana service to NodePort which made it accessible at port 30397.
Then looked at the controller configuration here: https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/influxdb-grafana-controller.yaml and noticed the comment about using the api-server proxy which we wouldn’t be doing by exposing the NodePort, so we deleted the GF_SERVER_ROOT_URL environment variable from the config. At that point Grafana at least seemed to be running, but it looked like it was having trouble reaching influxdb.
We then changed the datasource to use localhost instead of monitoring-influxd and was able to connect. We're getting data on the cluster usage now, though individual pod data doesn’t seem to be working.