We have a requirement that no requests receive 404's when doing a rolling deploy. Currently we achieve this by deploying the new assets container to all servers before continuing with a rolling deploy. With nginx's "try_files" this ensures that as the new code is being rolled out we can serve both the old and new versions of assets. Does Kubernetes have any features to support this type of workflow?
You can either use Deployment API (for Kubernetes >= v1.2) or kubectl rolling-update (for < v1.2) to manage the rolling deploy of your Kubernetes Pods (each is a co-located group of containers and volumes). You'll also need to create Services for accessing those Pods (Services redirect traffic to Pods). During the rolling deploy, a user will be redirected to either the Pod with old or new versions of assets container.
One possibility would be using Deployments. You can run a Deployment with several Replica's.
Now when you update to a new version, the Deployment takes care that one old version is removed and a new version is added one after another until your new version is fully rolled out. The rollout strategy can be
When running nginx in your Pod's for serving static files, I would suggest adding the following to your container specification:
lifecycle:
preStop:
exec:
command: ["/usr/sbin/nginx", "-s", "quit"]
This will send a QUIT Signal to your nginx before the Pod is getting destroyed. This way you can be sure that no new connections get accepted by nginx, just before the server is going down.
Related
Let's say I have 10 pods running a stable version, and I wish to replace the image of one of them to run a newer version before a full rollout.
Is there a way to do that?
Not as such: every pod managed by a Deployment is expected to be identical, including running the same image. You can't change a pod's image once it's been created, and if you change the Deployment's image, it will try to recreate all of its managed pods.
If the only thing you're worried about is the pod starting up, the default behavior of a deployment is to start 25% of its specified replicas with the new image. The old pods will continue running uninterrupted until the new replicas successfully start and pass their readiness checks. If the new pods immediately go into CrashLoopBackOff state, the old pods will still be running.
If you want to start a pod specifically as a canary deployment, you can create a second Deployment to handle that. You'll need to include some label on the pods (for instance, canary: 'true') where you can distinguish the canary from main pods. This would be present in the pod spec, and in the deployment selector, but it would not be present in the corresponding Service selector: the Service matches both canary and non-canary pods. If this runs successfully then you can remove the canary Deployment and update the image on the main Deployment.
Like the other answer mentioned, It sounds like you are talking about a canary deployment. You can do this with Kubernetes and also with Istio. I prefer Istio as it gives you some great control over traffic weighting. I.e you could send 1% of traffic to the canary and 99% to the control. Great for testing in production. It also lets you route using HTTP headers.
https://istio.io/latest/blog/2017/0.1-canary/
If you want to do it with k8s just create two deployments with unique deployment names (myappv1 & myappv2 for example) with the same app= label. Then you can just create a service with the selector = whatever your app label is. The svc will round robin between the two v1 and v2 deployments.
I am trying to deploy updates to pods. However I want the current pods to terminate only when all the containers inside the pod have terminated and their process is complete.
The new pods can keep waiting to start untill all container in the old pods have completed. We have a mechanism to stop old pods from picking up new tasks and therefore they should eventually terminate.
It's okay if twice the pods exist at some instance of time. I tried finding solution for this in kubernetes docs but wan't successful. Pointers on how / if this is possible would be helpful.
well I guess then you may have to create a duplicate kind of deployment with new image as required and change the selector in service to new deployment, which will prevent external traffic from entering pre-existing pods and new calls can go to new pods. Then later you can check for something like -
Kubectl top pods -c containers
and if the load appears to be static and low, then preferrably you can delete the old pods related deployment later.
But for this thing everytime the service selectors have to be updated and likely for keeping track of things you can append the git commit hash to the service selector to keep it unique everytime.
But rollback to previous versions if required from inside Kubernetes cluster will be difficult, so preferably you can trigger the wanted build again.
I hope this makes some sense !!
I'm running GKE cluster and there is a deployment that uses image which I push to Container Registry on GCP, issue is - even though I build the image and push it with latest tag, the deployment keeps on creating new pods with the old one cached - is there a way to update it without re-deploying (aka without destroying it first)?
There is a known issue with the kubernetes that even if you change configmaps the old config remains and you can either redeploy or workaround with
kubectl patch deployment $deployment -n $ns -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
is there something similar with cached images?
I think you're looking for kubectl set or patch which I found there in kubernetes documentation.
To update image of deployment you can use kubectl set
kubectl set image deployment/name_of_deployment name_of_deployment=image:name_of_image
To update image of your pod you can use kubectl patch
kubectl patch pod name_of_pod -p '{"spec":{"containers":[{"name":"name_of_pod_from_yaml","image":"name_of_image"}]}}'
You can always use kubectl edit to edit which allows you to directly edit any API resource you can retrieve via the command line tool.
kubectl edit deployment name_of_deployment
Let me know if you have any more questions.
1) You should change the way of your thinking. Destroying pod is not bad. Application downtime is what is bad. You should always plan your deployments in such a way that it can tolerate one pod death. Use multiple replicas for stateless apps and use clusters for stateful apps. Use Kubernetes rolling update for any changes to your deployments. Rolling updates have many extremely important settings which directly influence the uptime of your apps. Read it carefully.
2) The reason why Kubernetes launches old image is that by default it uses
imagePullPolicy: IfNotPresent. Use imagePullPolicy: Always and it will always try to pull latest version on redeploy.
I got one service with auto scaling which means it can have 2 pods or 4 pods running. My issue is that this service is a reactjs application with service-worker.
With my current deployment strategy it will create a new pod and kill one at a time, which causes issues when the clients gets alarmed that there is a new update & tries to fetch new assets from server & the loadbalancer forwards it to the old pods.
So basicly I am wondering if it's possible to change to a strategy that creates x pods & replaces them all at the same time?
Use the Recreate deployment strategy to first kill all old pods and then create new ones.
Alternatively, if you're looking to first create a parallel set of new pods, reroute traffic to these new pods and then kill the old pods (i.e., a blue/green deployment), check this guide.
add the spec.strategy.type in your deployment.yaml manfest and set it to "Recreate"
this will kill all the existing pods before new ones are created.
spec:
strategy:
type: Recreate
The strategy you are using is the - RollingUpdate , which is the default if you dont specify any.
Follow this approach though it is manual and meets your requirement.
Say, you are running version 1.0 ( with label version:1.0 ) in the cluster and you want to upgrade to version 2.0
Deploy version 2.0 with label version:2.0
Verify that pods are running and your version 2.0 app is running fine.
edit version 1.0 service selector to use label version:2.0
Delete version 1.0 deployment
I am using minikube and building my projects by tearing down the previous project and rebuilding it with
kubectl delete -f myprojectfiles
kubectl apply -f myprojectfiles
The files are a deployment and a service.
When I access my website I get a 503 error as I'm waiting for kubernetes to bring up the deployment. Is there anyway to speed this up? I see that my application is already built because the logs show it is ready. However it stays showing 503 for what feels like a few minute before everything in kubernetes triggers and starts serving me the application.
What are some things I can do to speed up the uptime?
Configure what is called readinessProbe, it won't fasten your boot up time, but it will help you by not giving false sense that application is up and running. With this your traffic will only be sent to your application pod when it is ready to accept the connection. Please read about it here.
FWIW your application might be waiting on some dependency to be up and running, also add these kinda health checks to that dependency pod.
You should not delete your Kubernetes resources. Use either kubectl apply or kubectl replace to update your project.
If you delete it, the nginx ingress controller won't find any upstream for a short period of time and puts on a blacklist for some seconds.
Also you should make sure, that you use Deployment which is able to do a rolling update without any downtime.