I got one service with auto scaling which means it can have 2 pods or 4 pods running. My issue is that this service is a reactjs application with service-worker.
With my current deployment strategy it will create a new pod and kill one at a time, which causes issues when the clients gets alarmed that there is a new update & tries to fetch new assets from server & the loadbalancer forwards it to the old pods.
So basicly I am wondering if it's possible to change to a strategy that creates x pods & replaces them all at the same time?
Use the Recreate deployment strategy to first kill all old pods and then create new ones.
Alternatively, if you're looking to first create a parallel set of new pods, reroute traffic to these new pods and then kill the old pods (i.e., a blue/green deployment), check this guide.
add the spec.strategy.type in your deployment.yaml manfest and set it to "Recreate"
this will kill all the existing pods before new ones are created.
spec:
strategy:
type: Recreate
The strategy you are using is the - RollingUpdate , which is the default if you dont specify any.
Follow this approach though it is manual and meets your requirement.
Say, you are running version 1.0 ( with label version:1.0 ) in the cluster and you want to upgrade to version 2.0
Deploy version 2.0 with label version:2.0
Verify that pods are running and your version 2.0 app is running fine.
edit version 1.0 service selector to use label version:2.0
Delete version 1.0 deployment
Related
Let's say I have 10 pods running a stable version, and I wish to replace the image of one of them to run a newer version before a full rollout.
Is there a way to do that?
Not as such: every pod managed by a Deployment is expected to be identical, including running the same image. You can't change a pod's image once it's been created, and if you change the Deployment's image, it will try to recreate all of its managed pods.
If the only thing you're worried about is the pod starting up, the default behavior of a deployment is to start 25% of its specified replicas with the new image. The old pods will continue running uninterrupted until the new replicas successfully start and pass their readiness checks. If the new pods immediately go into CrashLoopBackOff state, the old pods will still be running.
If you want to start a pod specifically as a canary deployment, you can create a second Deployment to handle that. You'll need to include some label on the pods (for instance, canary: 'true') where you can distinguish the canary from main pods. This would be present in the pod spec, and in the deployment selector, but it would not be present in the corresponding Service selector: the Service matches both canary and non-canary pods. If this runs successfully then you can remove the canary Deployment and update the image on the main Deployment.
Like the other answer mentioned, It sounds like you are talking about a canary deployment. You can do this with Kubernetes and also with Istio. I prefer Istio as it gives you some great control over traffic weighting. I.e you could send 1% of traffic to the canary and 99% to the control. Great for testing in production. It also lets you route using HTTP headers.
https://istio.io/latest/blog/2017/0.1-canary/
If you want to do it with k8s just create two deployments with unique deployment names (myappv1 & myappv2 for example) with the same app= label. Then you can just create a service with the selector = whatever your app label is. The svc will round robin between the two v1 and v2 deployments.
I see there are two ways to move back to the older deployment version. One is using rollout undo command and another option is to deploy again to the older version. Is there any difference between the two or they both are interchangeable?
As I understood, you're asking for a difference between doing undo and manually changing pod definitions to the exact previous state. If that's the case - read below.
When you do a new deployment and if in that deployment your pod definitions' hash has been modified - the Deployment Controller will create a new ReplicaSet (let's call it A) in order to roll out new version, but at the same time it will decrease replica count in the existing ReplicaSet (let's call it B) - so you have 2 ReplicaSets (A, B). How it does this - depends on rollout strategy you choose (for example: rolling updates, blue-green deployment and so on).
When you do kubectl rollout undo deploy <your deployment> - the Deployment Controller will basically decrease the number of replicas in your newly created ReplicaSet (A) and increase the number of replicas in the old ReplicaSet (B).
But, when you do, as you said: deploy again to the older version - you basically do a new deployment, so new ReplicaSet (C) will be created in order to roll out your new version (even if it's not a new version), and your existing ReplicaSets(A) replica count will be decreased.
So, basically the difference is the ReplicaSets which gets created.
Read: Deployments for more info
The whole flow is as following:
Deployment Controller manages ReplicaSets
ReplicaSet changes desired pod count in etcd
Scheduler schedules the pod
Kubelet creates/terminates actual pods
And all of them talk to API Server and watch for changes in resource definitions via watch mechanism, again via API Server
when you undo the roll out, you are updating in a way that is not reflected in source control. The preferred way is to revert your YAML and apply previous versions- then your revisions match with the tracked configuration.
kubectl rollout history deployment xyz
REVISION --> these do not reflect correctly, get a new number with undo roll-out
Well first of all there is no direct way of option is to deploy again in kubernetes. Yes the undo way is to going back the previous version of your deployment. and command to go back is
kubectl undo deployment ABC
In my team, we sometimes scale down to just one pod in Openshift to make testing easier. If we then do a rolling update with the desired replica count set to 2, Openshift scales up to two pods before performing a rolling deploy. It is a nuisance, because the new "old" pod can start things that we don't expect to be started before the new deployment starts, and so we have to remember to take down the one pod before the new deploy.
Is there a way to stop the old deployment from scaling up to the desired replica count while the new deployment is scaled up to the desired replica count? Also, why does it work this way?
OpenShift Master:
v3.11.200
Kubernetes Master:
v1.11.0+d4cacc0
OpenShift Web Console:
3.11.200-1-8a53b1d
From our Openshift template:
- apiVersion: v1
kind: DeploymentConfig
spec:
replicas: 2
strategy:
type: Rolling
This is expected behavior when using RollingUpdate strategy. It removes old pods one by one, while adding new ones at the same time, keeping the application available throughout the whole process, and ensuring there’s no drop in its capacity to handle requests. Since you have only one pod, Kubernetes scales the deployment to keep the strategy and zero-downtime as requested in the manifest.
It scales up to 2, because if not specified maxSurge defaults to 25%. It means that there can be at most 25% more pod instances than the desired count during an update.
If you want to ensure that this won't be scaled you might change the strategy to Recreate. This will cause all old pods to be deleted before the new ones are created. Use this strategy when your application doesn’t support running multiple versions in parallel and requires the old version to be stopped completely before the new one is started. However please note that, this strategy does involve a short period of time when your app becomes completely unavailable.
Here`s a good document that describes rolling update strategy. It is worth also checking official kubernetes documentation about deployments.
We are using Google kubernetes to deploy our microservices, circlci for deployment integration, we define our k8s files inside the githup repos.
The problem we are facing is that some services are taking time on startup by loading database schema and other pre data, but google kubernetes shutdown old pods before the new pods fully started.
can we tell kubernetes somehow to wait till the new pods are fully loaded or at least to wait 10 seconds before shutdown the old pods.
Yes, this is possible. Based on the description it sounds like you are using a single deployment that gets updated. The new pods are created and the old ones are removed before the new ones become ready.
To address this, you need to have a proper readinessProbe configured or readinessGates on the pods so that the pod status only becomes ready once it actually is ready. If you are not sure what to put as the probe, you can also define initialDelaySeconds with a guess at how much time you think the pod needs to start up.
You should also look into using the deployment spec field for minReadySeconds as well as defining a proper deployment strategy. You can ensure that the rolling update creates new pods (by defining the maxSurge field) and ensure that the old pod is not removed until the new one is ready and receiving traffic (using the maxUnavailable field = 0).
an example would be:
spec:
replicas: 3
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
This will maintain 3 working replicas at any given time. When a new version is pushed, 1 new pod will be created with the new image. No pods will be taken offline until the new one is in ready state. Once it is, one of the old pods will be terminated and the cycle goes again. Feel free to change the maxSurge value to a higher number if you want the rollout to happen in one go.
We have a requirement that no requests receive 404's when doing a rolling deploy. Currently we achieve this by deploying the new assets container to all servers before continuing with a rolling deploy. With nginx's "try_files" this ensures that as the new code is being rolled out we can serve both the old and new versions of assets. Does Kubernetes have any features to support this type of workflow?
You can either use Deployment API (for Kubernetes >= v1.2) or kubectl rolling-update (for < v1.2) to manage the rolling deploy of your Kubernetes Pods (each is a co-located group of containers and volumes). You'll also need to create Services for accessing those Pods (Services redirect traffic to Pods). During the rolling deploy, a user will be redirected to either the Pod with old or new versions of assets container.
One possibility would be using Deployments. You can run a Deployment with several Replica's.
Now when you update to a new version, the Deployment takes care that one old version is removed and a new version is added one after another until your new version is fully rolled out. The rollout strategy can be
When running nginx in your Pod's for serving static files, I would suggest adding the following to your container specification:
lifecycle:
preStop:
exec:
command: ["/usr/sbin/nginx", "-s", "quit"]
This will send a QUIT Signal to your nginx before the Pod is getting destroyed. This way you can be sure that no new connections get accepted by nginx, just before the server is going down.