I have specified a deployment in kubernetes :
apiVersion: v1
kind: List
items:
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: quote-data
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
run: quote-data
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
run: quote-data
spec:
containers:
- image: quote-data:2
imagePullPolicy: Always
name: quote-data
readinessProbe:
httpGet:
path: /api/data/health
port: 5000
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 1
successThreshold: 1
ports:
- containerPort: 5000
protocol: TCP
Can we use the readinessprobe to disable the traffic for this pod (few minutes after the start phase) ?
In my test, the service (type: LoadBalancer) continue to send the traffic on the bad pod even if the readinessprobe is failed. In the service, the Ready status of the pod = false but the service still continues to send on the wrong pod
UPDATE :
it seems work with a curl but not with Postman.
I have a pods different between two request with a simple curl command.
I have always the same pod with Postman
Yes, the traffic will be disabled if you put initialDelaySeconds: xxx on readinessProbe. For this is not working on all Kubernetes version, for sure not on 1.15 and bellow.
Related
My GKE cluster is displaying "Scale down blocked by pod" note, and clicking it then going to the Logs Explorer it shows a filtered view with log entries for the pods that had the incident: no.scale.down.node.pod.not.enough.pdb . But that's really strange since the pods on the log entries having that message do have PDB defined for them. So it seems to me that GKE is wrongly reporting the cause of the blocking of the node scale down. These are the manifests for one of the pods with this issue:
apiVersion: v1
kind: Service
metadata:
labels:
app: ms-new-api-beta
name: ms-new-api-beta
namespace: beta
spec:
ports:
- port: 8000
protocol: TCP
targetPort: 8000
selector:
app: ms-new-api-beta
type: NodePort
The Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ms-new-api-beta
name: ms-new-api-beta
namespace: beta
spec:
selector:
matchLabels:
app: ms-new-api-beta
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
labels:
app: ms-new-api-beta
spec:
containers:
- command:
- /deploy/venv/bin/gunicorn
- '--bind'
- '0.0.0.0:8000'
- 'newapi.app:app'
- '--chdir'
- /deploy/app
- '--timeout'
- '7200'
- '--workers'
- '1'
- '--worker-class'
- uvicorn.workers.UvicornWorker
- '--log-level'
- DEBUG
env:
- name: ENV
value: BETA
image: >-
gcr.io/.../api:${trigger['tag']}
imagePullPolicy: Always
livenessProbe:
failureThreshold: 5
httpGet:
path: /rest
port: 8000
scheme: HTTP
initialDelaySeconds: 120
periodSeconds: 20
timeoutSeconds: 30
name: ms-new-api-beta
ports:
- containerPort: 8000
name: http
protocol: TCP
readinessProbe:
httpGet:
path: /rest
port: 8000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 2
resources:
limits:
cpu: 150m
requests:
cpu: 100m
startupProbe:
failureThreshold: 30
httpGet:
path: /rest
port: 8000
periodSeconds: 120
imagePullSecrets:
- name: gcp-docker-registry
The Horizontal Pod Autoscaler:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: ms-new-api-beta
namespace: beta
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ms-new-api-beta
targetCPUUtilizationPercentage: 100
And finally, the Pod Disruption Budget:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: ms-new-api-beta
namespace: beta
spec:
minAvailable: 0
selector:
matchLabels:
app: ms-new-api-beta
no.scale.down.node.pod.not.enough.pdb is not complaining about the lack of a PDB. It is complaining that, if the pod is scaled down, it will be in violation of the existing PDB(s).
The "budget" is how much disruption the Pod can permit. The platform will not take any intentional action which violates that budget.
There may be another PDB in place that would be violated. To check, make sure to review pdbs in the pod's namespace:
kubectl get pdb
I am looking for how to have a spare/cold replica/pod in my Kubernetes configuration. I assume it would go in my Kuberentes deployment or HPA configuration. Any idea how I would make it so I have 2 spare/cold instances of my app always ready, but only get put into the active pods once HPA requests another instance? My goal is to have basically zero startup time on a new pod when HPA says it needs another instance.
apiVersion: apps/v1
kind: Deployment
metadata:
name: someName
namespace: someNamespace
labels:
app: someName
version: "someVersion"
spec:
replicas: $REPLICAS
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: someMaxSurge
maxUnavailable: someMaxUnavailable
selector:
matchLabels:
app: someName
version: someVersion
template:
metadata:
labels:
app: someName
version: "someVersion"
spec:
containers:
- name: someName
image: someImage:someVersion
imagePullPolicy: Always
resources:
limits:
memory: someMemory
cpu: someCPU
requests:
memory: someMemory
cpu: someCPU
readinessProbe:
failureThreshold: someFailureThreshold
initialDelaySeconds: someInitialDelaySeconds
periodSeconds: somePeriodSeconds
timeoutSeconds: someTimeoutSeconds
livenessProbe:
httpGet:
path: somePath
port: somePort
failureThreshold: someFailureThreshold
initialDelaySeconds: someInitialDelay
periodSeconds: somePeriodSeconds
timeoutSeconds: someTimeoutSeocnds
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: someName
namespace: someNamespace
spec:
minAvailable: someMinAvailable
selector:
matchLabels:
app: someName
version: "someVersion"
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: someName-hpa
namespace: someNamespace
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: someName
minReplicas: someMinReplicas
maxReplicas: someMaxReplicas
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: someAverageUtilization
I am just wanting to always have 2 spare for scaling, or if one becomes unavailable or any reason
It is a good practice to have at least two replicas for services on Kubernetes. This helps if e.g. a node goes down or you need to do maintenance of the node. Also set Pod Topology Spread Constraints so that those pods are scheduled to run on different nodes.
Set the number of replicas that you minimum want as desired state. In Kubernetes, traffic will be load balanced to the replicas. Also use Horizontal Pod Autoscaler to define when you want to autoscale to more replicas. You can set the requirements low for autoscaling, if you want to scale up early.
I've set up 2 replicas of a deployment.
when I use
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
It only updates 1 pod when I update it via set Image. The second pod doesnot get updated with new code. This means 1 have 2 pods running different images.
When I set maxSurge 25% and maxUnavailable 25%, the pods don't get replaced at all.
Here's the complete yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "89"
creationTimestamp: 2018-11-26T09:40:48Z
generation: 94
labels:
io.kompose.service: servicing
name: servicing
namespace: default
resourceVersion: "6858872"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/servicing
uid: 5adb98c8-f15f-11e8-8752-42010a800188
spec:
replicas: 2
selector:
matchLabels:
io.kompose.service: servicing
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: servicing
spec:
containers:
- env:
- name: JWT_KEY
value: ABCD
- name: PORT
value: "3001"
image: gcr.io/something/something
imagePullPolicy: Always
name: servicing-container
ports:
- containerPort: 3001
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 3001
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 25m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 2
conditions:
- lastTransitionTime: 2018-12-13T11:55:00Z
lastUpdateTime: 2018-12-13T11:55:00Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 94
readyReplicas: 2
replicas: 2
updatedReplicas: 2
You have setup initialDelaySeconds to 5, periodSeconds to 5 and failureThreshold to 3, it means that kubernetes will wait initial 5 seconds to do first probe to your application is ready or not and then periodically probe your app every 5 seconds if it is ready or not and will do it 3 times. So your application will be checked at 10 second, 15 second and 20 second and if pod doesn't come up in that time, it bails out without upgrading.
You might need to increase this failureThreshold so that your app does have enough time to come up.
Also, I would suggest you to make maxUnavailable to 0 so that pod will be deleted only when the new pod comes up in its replacement.
Check my answer here for better understanding:
Kubernetes 0 Downtime using Readiness Probe and RollBack strategy not working
I am trying to implement the Rolling update of deployments in Kubernetes. I have followed a lot of articles that say that there would be zero downtime but when I run curl continuously. A couple of my requests failed before getting a response back. Below is the deployment file.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-deployment
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: my-image
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
The next thing I did was added
MinReadySeconds: 120
This takes care of this issue but it is not an optimum solution since we want to switch to the next pod as soon as it starts servicing requests and kill the old pod. I have two questions -
Can there be a condition when both the pods - the new and the old are
running and both start servicing the traffic? That would also not be
ideal as well. Since we want only one pod to service the request at a
time.
Is there any other out of the box solution that Kubernetes provides
to do a rolling deployment?
Try this. this should work for you . try doing a update of your image.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-deployment
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: nginx
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
For your better understanding check this link
I have a situation where I have zero endpoints available for one service. To test this, I specially crafted a yaml descriptor that uses a simple node server to set and retrieve the ready/live status for a pod:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nodejs-deployment
labels:
app: nodejs
spec:
replicas: 3
selector:
matchLabels:
app: nodejs
template:
metadata:
labels:
app: nodejs
spec:
containers:
- name: nodejs
image: nodejs_server
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /is_alive
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 10
readinessProbe:
httpGet:
path: /is_ready
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
name: nodejs-service
labels:
app: nodejs
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
app: nodejs
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nodejs-ingress
spec:
backend:
serviceName: nodejs-service
servicePort: 80
The node server has methods to set and retrieve the liveness and readiness.
When the app start I can see that 3 replicas are created and the status of them is ready. OK then now I trigger manually the status of their readiness to set to false [from outside the ingress]. One pod is correctly removed from the endpoint so no traffic is routed to it[that's OK as this is the expected behavior]. When I set all the ready-statuses to false for all pods the endpoints list is empty [still the expected behavior].
At that point I cannot set ready=true from outside the ingress as the traffic is not routed to any pod. Is there a way here for example of triggering a restart of the pod when the ready is not achieved after n-timer or n-seconds? Or when the endpoints list is empty?
Well, that is perfectly normal and expected behaviour. What you can do, on the side, is to forward traffic from localhost to a particular pod with kubectl port-forward. That way you can access the pod directly, without ingresses etc. and set it's readiness back to ok. If you want to restart when host it not ready for to long, just use the same endpoint for liveness probe, but trigger it after more tries.