Currently I am using this in my pipeline
kubectl apply -f deployment.yaml && kubectl rollout status -f deployment.yaml
With this in yaml
readinessProbe:
tcpSocket:
port: 90
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 1
successThreshold: 1
livenessProbe:
tcpSocket:
port: 90
initialDelaySeconds: 120
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 1
successThreshold: 1
For me, kubectl rollout is running for a very long time, blocking the deployment pipeline. From the documentation
By default 'rollout status' will watch the status of the latest rollout until it's done
My question:
1/ Which actions are the parts that contribute to the deployment "until it's done" (resource creation, resource teardown?... )
2/ Does readinessProbe and livenessProbe contribute to the deployment time
The criteria for this are in the kubectl source. A deployment is "complete" if:
It hasn't timed out
Its updated-replica count is at least its desired-replica count (every new pod has been created)
Its current-replica count is at most its updated-replica count (every old pod has been destroyed)
Its available-replica count is at least its updated-replica count (every new pod is running)
You can use kubectl get deployment -w or kubectl get pod -w to watch a deployment actually happen in real time; the kubectl get -w option watches the given resources and prints out a new line whenever they change. You'll see the following sequence occur (with default Deployment settings, one at a time for "small" deployments):
A new pod is created
The new pod passes its probes and become ready
An old pod is terminated
The old pod actually exits and is deleted
So for kubectl rollout status deployment/... to finish, all of these steps must happen – new pods are created, new pods all pass their health checks, old pods are destroyed – for every replica in the deployment.
Related
I have a web app on Kubernetes that gets "spun-up" via user action. That user action triggers a python script that applies a deployment, service, and ingress. The frontend shows a spinner until the app is ready to make connections. I'm currently checking the status of the deployment and checking that the "Available" status is "True", at which point I hide the spinner and load the app.
The problem is, every once in a while, users will experience a 503: Temporarily Unavailable error. A quick browser refresh fixes the problem, so this appears to be some sort of race condition.
My question is, why am I getting a 503 error if the deployment is marked as "Available"? Does this mean that the ingress or the service is sometimes taking longer to initialize?
I currently have the following probes on my deployment's app container:
readinessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 5
initialDelaySeconds: 5
livenessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 5
initialDelaySeconds: 5
I'm using Azure AKS and ingress-nginx.
For newly created deployments check these. Both should be true, order does not matter.
kubectl get deployment <name> -ojson | jq ".status.availableReplicas" equal to desired, or >= 1, on your preference.
kubectl get ingress <name> -ojson | jq ".status.loadBalancer" is not empty. It means that ingress controller initialized for your host.
For updated deployments (anything that required pods to be recreated). Both should be true, order does not matter.
kubectl get deployment <name> -ojson | jq ".status.availableReplicas" equal to desired.
kubectl get deployment <name> -ojson | jq ".status.updatedReplicas" equal to desired.
Ingress will already be initialized here.
Here is my Readiness Probe Configuration:
readinessProbe:
httpGet:
path: /devops/versioninfo/api
port: 9001
initialDelaySeconds: 300
timeoutSeconds: 3
periodSeconds: 10
failureThreshold: 60
Here is my rolling update strategy:
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 0
Because it will take a long time for my pods to be ready, but when the deployment is rolling update, old pods will be deleted when the new one's status is running whose ready health is not ok.
How to let the rolling update strategy be that the new one is ready and then delete the old one.
You can try increasing the minReadySeconds option in the Deployment spec. Basically, tell the deployment that you need to at least wait X number of seconds before you can say one particular pod is ready.
✌️
Running a Spring Boot application inside a OpenShift Pod. To execute the readiness and liveness probe, I created an appropriate YAML file. However the Pod fails and responds that he was not able to pass the readiness check (after approximately 5 minutes).
My goal is to execute the readiness probe every 20 minutes. But I assume that it is failing because it adds up the initalDelaySeconds together with the periodSeconds. So I guess that the first check after the pod has been started will be executed after 22 minutes.
Following the related configuration of the readiness probe.
readinessProbe:
failureThreshold: 3
httpGet:
path: /actuator/health
port: 8080
scheme: HTTP
initialDelaySeconds: 120
periodSeconds: 1200
successThreshold: 1
timeoutSeconds: 60
Is my assumption right? How to avoid it (Maybe increase the timeout regarding the kubelet)?
Your configuration is correct and the initialDelaySeconds and periodSeconds do not sum up. So, the first readinessProbe HTTP call will exactly in 2 min after you start your POD.
I would look for an issue in your app itself, first thing that comes to my mind is that your path is /actuator/health, shouldn't it be just /health? That is the default in case of Spring Boot Actuator.
If that doesn't help, then the best would be to debug it: exec into your container and use curl to check if your health endpoint works correctly (it should return HTTP Code 200).
Let's say I have a deployment template like this
spec:
minReadySeconds: 15
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 5
How will this affect the newly versions of my app? Will the minReadySeconds and initialDelaySeconds count at the same time? Will the initialDelaySeconds come first then minReadySeconds?
From Kubernetes Deployment documentation:
.spec.minReadySeconds is an optional field that specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. This defaults to 0 (the Pod will be considered available as soon as it is ready). To learn more about when a Pod is considered ready, see Container Probes
So your newly created app pod have to be ready for .spec.minReadySeconds seconds to be considered as available.
initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated.
So initialDelaySeconds comes before minReadySeconds.
Lets say, container in the pod has started at t seconds. Readiness probe will be initiated at t+initialDelaySeconds seconds. Assume Pod become ready at t1 seconds(t1 > t+initialDelaySeconds). So this pod will be available after t1+minReadySeconds seconds.
I am creating an helm chart that should install 2 services.
It has a dependency that first postgresql service will be installed.
Then the other service should use the database user,password,hostname and port for the postgresql service installed.
Since I need to get these details run time I.e soon installed postgresql service of course user details I will use as env variables, hostname and port to be used once postgresql is deployed.
I tried using some template functions and subchart concepts that I got from different sites.. but nothing is solving the requirement.
Is there any examples that I can get to match the above requirement ?
There are a couple of ways you could do this, for ex. using a InitContainer to check if DB is up, but I will show you with a sample example in the charts. I am using Wordpress Chart as an example
livenessProbe:
httpGet:
path: /wp-login.php
port: http
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
httpGet:
path: /wp-login.php
port: http
initialDelaySeconds: 30
timeoutSeconds: 3
periodSeconds: 5
I have removed some lines for brevity.
The readiness probe will start acting after a initialDelaySeconds of 30 seconds, will check every periodSeconds i.e. 5 seconds to see if the page responds. Unless the readiness probe succeeds, the traffic won't be sent to this pod. If the probe succeeds then we are good.
The second check - liveness probe does something more. It is starting 120 seconds after the pod is deployed. But if the check fails, it will restart the pod and it will restart failureThreshold times i.e. 6 times.
Coming back to your question and how to solve this:
Use liveness and readiness probes in the applications which are dependent on the database
Use some defaults based on your experience and optimize them as you go.
More information about the readiness and liveness probes can be found here