I have a web app on Kubernetes that gets "spun-up" via user action. That user action triggers a python script that applies a deployment, service, and ingress. The frontend shows a spinner until the app is ready to make connections. I'm currently checking the status of the deployment and checking that the "Available" status is "True", at which point I hide the spinner and load the app.
The problem is, every once in a while, users will experience a 503: Temporarily Unavailable error. A quick browser refresh fixes the problem, so this appears to be some sort of race condition.
My question is, why am I getting a 503 error if the deployment is marked as "Available"? Does this mean that the ingress or the service is sometimes taking longer to initialize?
I currently have the following probes on my deployment's app container:
readinessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 5
initialDelaySeconds: 5
livenessProbe:
httpGet:
path: /
port: 3000
periodSeconds: 5
initialDelaySeconds: 5
I'm using Azure AKS and ingress-nginx.
For newly created deployments check these. Both should be true, order does not matter.
kubectl get deployment <name> -ojson | jq ".status.availableReplicas" equal to desired, or >= 1, on your preference.
kubectl get ingress <name> -ojson | jq ".status.loadBalancer" is not empty. It means that ingress controller initialized for your host.
For updated deployments (anything that required pods to be recreated). Both should be true, order does not matter.
kubectl get deployment <name> -ojson | jq ".status.availableReplicas" equal to desired.
kubectl get deployment <name> -ojson | jq ".status.updatedReplicas" equal to desired.
Ingress will already be initialized here.
Related
Currently I am using this in my pipeline
kubectl apply -f deployment.yaml && kubectl rollout status -f deployment.yaml
With this in yaml
readinessProbe:
tcpSocket:
port: 90
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 10
failureThreshold: 1
successThreshold: 1
livenessProbe:
tcpSocket:
port: 90
initialDelaySeconds: 120
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 1
successThreshold: 1
For me, kubectl rollout is running for a very long time, blocking the deployment pipeline. From the documentation
By default 'rollout status' will watch the status of the latest rollout until it's done
My question:
1/ Which actions are the parts that contribute to the deployment "until it's done" (resource creation, resource teardown?... )
2/ Does readinessProbe and livenessProbe contribute to the deployment time
The criteria for this are in the kubectl source. A deployment is "complete" if:
It hasn't timed out
Its updated-replica count is at least its desired-replica count (every new pod has been created)
Its current-replica count is at most its updated-replica count (every old pod has been destroyed)
Its available-replica count is at least its updated-replica count (every new pod is running)
You can use kubectl get deployment -w or kubectl get pod -w to watch a deployment actually happen in real time; the kubectl get -w option watches the given resources and prints out a new line whenever they change. You'll see the following sequence occur (with default Deployment settings, one at a time for "small" deployments):
A new pod is created
The new pod passes its probes and become ready
An old pod is terminated
The old pod actually exits and is deleted
So for kubectl rollout status deployment/... to finish, all of these steps must happen – new pods are created, new pods all pass their health checks, old pods are destroyed – for every replica in the deployment.
I would like to ask on the mechanism for stopping the pods in kubernetes.
I read https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods before ask the question.
Supposably we have a application with gracefully shutdown support
(for example we use simple http server on Go https://play.golang.org/p/5tmkPPMiSSt).
Server has two endpoints:
/fast, always send 200 http status code.
/slow, wait 10 seconds and send 200 http status code.
There is deployment/service resource with that configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app/name: test
template:
metadata:
labels:
app/name: test
spec:
terminationGracePeriodSeconds: 120
containers:
- name: service
image: host.org/images/grace:v0.1
livenessProbe:
httpGet:
path: /health
port: 10002
failureThreshold: 1
initialDelaySeconds: 1
readinessProbe:
httpGet:
path: /health
port: 10002
failureThreshold: 1
initialDelaySeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: test
spec:
type: NodePort
ports:
- name: http
port: 10002
targetPort: 10002
selector:
app/name: test
To make sure the pods deleted gracefully I conducted two test options.
First option (slow endpoint) flow:
Create deployment with replicas value equal 1.
Wait for pod readness.
Send request on /slow endpoint (curl http://ip-of-some-node:nodePort/slow) and delete pod (simultaneously, with 1 second out of sync).
Expected:
Pod must not end before http server completed my request.
Got:
Yes, http server process in 10 seconds and return response for me.
(if we pass --grace-period=1 option to kubectl, then curl will write - curl: (52) Empty reply from server)
Everything works as expected.
Second option (fast endpoint) flow:
Create deployment with replicas value equal 10.
Wait for pods readness.
Start wrk with "Connection: close" header.
Randomly delete one or two pods (kubectl delete pod/xxx).
Expected:
No socket errors.
Got:
$ wrk -d 2m --header "Connection: Close" http://ip-of-some-node:nodePort/fast
Running 2m test # http://ip-of-some-node:nodePort/fast
Thread Stats Avg Stdev Max +/- Stdev
Latency 122.35ms 177.30ms 1.98s 91.33%
Req/Sec 66.98 33.93 160.00 65.83%
15890 requests in 2.00m, 1.83MB read
Socket errors: connect 0, read 15, write 0, timeout 0
Requests/sec: 132.34
Transfer/sec: 15.64KB
15 socket errors on read, that is, some pods were disconnected from the service before all requests were processed (maybe).
The problem appears when a new deployment version is applied, scale down and rollout undo.
Questions:
What's reason of that behavior?
How to fix it?
Kubernetes version: v1.16.2
Edit 1.
The number of errors changes each time, but remains in the range of 10-20, when removing 2-5 pods in two minutes.
P.S. If we will not delete a pod, we don't got errors.
Does Kubernetes support green-blue deployment?
Yes, it does. You can read about it on Zero-downtime Deployment in Kubernetes with Jenkins,
A blue/green deployment is a change management strategy for releasing software code. Blue/green deployments, which may also be referred to as A/B deployments require two identical hardware environments that are configured exactly the same way. While one environment is active and serving end users, the other environment remains idle.
Container technology offers a stand-alone environment to run the desired service, which makes it super easy to create identical environments as required in the blue/green deployment. The loosely coupled Services - ReplicaSets, and the label/selector-based service routing in Kubernetes make it easy to switch between different backend environments.
I would also recommend reading Kubernetes Infrastructure Blue/Green deployments.
Here is a repository with examples from codefresh.io about blue green deployment.
This repository holds a bash script that allows you to perform blue/green deployments on a Kubernetes cluster. See also the respective blog post
Prerequisites
As a convention the script expects
The name of your deployment to be $APP_NAME-$VERSION
Your deployment should have a label that shows it version
Your service should point to the deployment by using a version selector, pointing to the corresponding label in the deployment
Notice that the new color deployment created by the script will follow the same conventions. This way each subsequent pipeline you run will work in the same manner.
You can see examples of the tags with the sample application:
service
deployment
You might be also interested in Canary deployment:
Another deployment strategy is using Canaries (a.k.a. incremental rollouts). With canaries, the new version of the application is gradually deployed to the Kubernetes cluster while getting a very small amount of live traffic (i.e. a subset of live users are connecting to the new version while the rest are still using the previous version).
...
The small subset of live traffic to the new version acts as an early warning for potential problems that might be present in the new code. As our confidence increases, more canaries are created and more users are now connecting to the updated version. In the end, all live traffic goes to canaries, and thus the canary version becomes the new “production version”.
EDIT
Questions:
What's reason of that behavior?
When new deployment is being applied old pods are being removed and new ones are being scheduled.
This is being done by Control Plan
For example, when you use the Kubernetes API to create a Deployment, you provide a new desired state for the system. The Kubernetes Control Plane records that object creation, and carries out your instructions by starting the required applications and scheduling them to cluster nodes–thus making the cluster’s actual state match the desired state.
You have only setup a readinessProbe, which tells your service if it should send traffic to the pod or not. This is not a good solution as like you can see in your example if you have 10 pods and remove one or two there is a gap and you receive socket error.
How to fix it?
You have to understand this is not broken so it doesn't need a fix.
This might be mitigated by implementing a check in your application to make sure it's sending request to working address or utilize other features like load balancing like ingress.
Also when you are updating deployment you can do checks before deleting the pod to check if it does have any traffic incoming/outgoing and roll the update to only not used pods.
I'm trying to use exec probes for readiness and liveness in GKE. This is because it is part of Kubernetes' recommended way to do health checks on gRPC back ends. However when I put the exec probe config into my deployment yaml and apply it, it doesn't take effect in GCP. This is my container yaml:
- name: rev79-uac-sandbox
image: gcr.io/rev79-232812/uac:latest
imagePullPolicy: Always
ports:
- containerPort: 3011
readinessProbe:
exec:
command: ["bin/grpc_health_probe", "-addr=:3011"]
initialDelaySeconds: 5
livenessProbe:
exec:
command: ["bin/grpc_health_probe", "-addr=:3011"]
initialDelaySeconds: 10
But still the health checks fail and when I look at the health check configuration in the GCP console I see a plain HTTP health check directed at '/'
When I edit a health check in GCP console there doesn't seem to be any way to choose an exec type. Also I can't see any mention of liveness checks as contrasted to readiness checks even though these are separate Kubernetes things.
Does Google cloud support using exec for health checks?
If so, how do I do it?
If not, how can I health check a gRPC server?
TCP probes are useful when we are using gRPC Services rather than using HTTP probes.
- containerPort: 3011
readinessProbe:
tcpSocket:
port: 3011
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 3011
initialDelaySeconds: 15
periodSeconds: 20
the kubelet will attempt to open a socket to your container on the specified port. If it can establish a connection, the container is considered healthy, if it can’t it is considered a failure
define-a-tcp-liveness-probe
Exec probes work in GKE just the same way they work everywhere. You can view liveness probe result in "kubectl describe pod". Or you can simply log in into pod, execute command and see its return code.
The server has to implement the grpc probe protocol as indicated here as indicated in this article
Both answers from Vasily Angapov and Suresh Vishnoi should in theory work, however in practice they don't (at least in my practice).
So my solution was to start another server on my backend container - an HTTP server that simply has the job of executing the health check whenever it gets a request and returning a 200 status if it passes and a 503 if it fails.
I also had to open a second port on my container for that server to listen on.
Hi I try to use custom health check with GCP LoadBalancer.
I have added readinessProbe & livenessProbe like this:
readinessProbe:
httpGet:
path: /health
port: dash
initialDelaySeconds: 5
periodSeconds: 1
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 10
livenessProbe:
httpGet:
path: /health
port: dash
initialDelaySeconds: 5
periodSeconds: 1
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 10
But when I create my ingress I haven't got my custom health check
Path LB
I FINALIZED an answer. What I was trying to do was impossible. My GCE Ingress used a backend on port 80 . But in my ReadinessProbe I told him to check on port 8080 and on the /health path. This is impossible!
The port of the service declared in the Ingress backend must be the same as that declared in the readinessProbe. Only the path can be different. If we do not respect this pattern, it is / that is associated with the Health Check GCP path.
From a network point of view this is logical, the Health Check GCP is "out" of the Kube cluster, if we tell it to route on port 80 but our ReadinessProbe is on another port, how it can ensure that even if the port associated with the ReadinessProbe meets port 80 (which is the one on which it must route traffic) also respond.
In summary, the port of the backend declared in Ingress must have a readinessProbe on the same port. The only thing we can customize is the path.
I think you are confused between resources in GCP.
The code you posted is at no moment in relation to a Load balancer resource, as it's a kubernetes health check for pod states. If you want to know if the probes are working, check your pod state, if it's not running describe your pod and look at the logs, should indicate an issue with the probes.
I'm going to guess that you have an ingress resource somewhere in your kubernetes conf wich creates the lb and all the resources around it like the health check (still guessing that the image you posted is in relation to that).
If you are using GKE you should leave the google automated resource conf from k8s config you deployed as it is, cause you may brake some things that google is already maintaining for you.
I am creating an helm chart that should install 2 services.
It has a dependency that first postgresql service will be installed.
Then the other service should use the database user,password,hostname and port for the postgresql service installed.
Since I need to get these details run time I.e soon installed postgresql service of course user details I will use as env variables, hostname and port to be used once postgresql is deployed.
I tried using some template functions and subchart concepts that I got from different sites.. but nothing is solving the requirement.
Is there any examples that I can get to match the above requirement ?
There are a couple of ways you could do this, for ex. using a InitContainer to check if DB is up, but I will show you with a sample example in the charts. I am using Wordpress Chart as an example
livenessProbe:
httpGet:
path: /wp-login.php
port: http
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
httpGet:
path: /wp-login.php
port: http
initialDelaySeconds: 30
timeoutSeconds: 3
periodSeconds: 5
I have removed some lines for brevity.
The readiness probe will start acting after a initialDelaySeconds of 30 seconds, will check every periodSeconds i.e. 5 seconds to see if the page responds. Unless the readiness probe succeeds, the traffic won't be sent to this pod. If the probe succeeds then we are good.
The second check - liveness probe does something more. It is starting 120 seconds after the pod is deployed. But if the check fails, it will restart the pod and it will restart failureThreshold times i.e. 6 times.
Coming back to your question and how to solve this:
Use liveness and readiness probes in the applications which are dependent on the database
Use some defaults based on your experience and optimize them as you go.
More information about the readiness and liveness probes can be found here