I have progressDeadlineSeconds set to 120 seconds
I deploy and run kubectl rollout status deployment mydeployment
The deployment failed with 0 of 1 updated replicas available - CrashLoopBackOff
But the kubectl is still hanging forever with message: Waiting for deployment "mydeployment" rollout to finish: 0 of 1 updated replicas are available...
Why is this happening, progressDeadlineSeconds is suppose to force it to fail and cause kubectl rollout status deployment to exit with a non-zero return code right?
You are correct, kubectl rollout status returns a non-zero exit code if the Deployment has exceeded the progression deadline. Progress Deadline Seconds:
.spec.progressDeadlineSeconds is an optional field that specifies
the number of seconds you want to wait for your Deployment to progress
before the system reports back that the Deployment has failed
progressing - surfaced as a condition with Type=Progressing,
Status=False. and Reason=ProgressDeadlineExceeded in the status of
the resource. The Deployment controller will keep retrying the
Deployment. This defaults to 600. In the future, once automatic
rollback will be implemented, the Deployment controller will roll back
a Deployment as soon as it observes such a condition.
If specified, this field needs to be greater than
.spec.minReadySeconds.
Which brings us to Min Ready Seconds
.spec.minReadySeconds is an optional field that specifies the
minimum number of seconds for which a newly created Pod should be
ready without any of its containers crashing, for it to be considered
available. This defaults to 0 (the Pod will be considered available as
soon as it is ready).
Without your exact deployment configs it is hard to tell where the problem is but there are few things to check out:
Try setting both the progressDeadlineSeconds and minReadySeconds remembering that the later must have a smaller value.
progressDeadlineSeconds might not be respected when used with Replicas: 1. Check your maxUnavailable param in order to see if it allows deployment full unavailability.
As a workaround you can specify a timeout period for your kubectl rollout status command. For example: kubectl rollout status deployment mydeployment --timeout=120s
If you don't want to wait for the rollout to finish then you can use --watch=false. You will than have to check the status manually by running kubectl describe deployment and kubectl get deployment commands which might not be ideal.
Please let me know if that helps.
Related
As per official documentation, kubectl rollout status returns a non-zero exit code if the Deployment has exceeded the progression deadline.
Progress Deadline Seconds
Now let's say I would like to create 2 pods as a part of my deployment.
Once the first pod is successfully up, will the progressDeadlineSeconds reset to 0 again and check for second pod to come up until progressDeadlineSeconds or is it the waiting time for complete deployment?
example:
progressDeadlineSeconds is 120, deployment needs to create 2 pods.
pod1 gets takes 80 seconds to come up.
for pod2, will it wait for 40 seconds or will it reset to 120 seconds before reporting success/failure ?
I am trying to use Kubernetes Deployment , i would like to know whether this is same as kubectl apply -f deployment.yaml or does this wait for the deployments to be up and running . because when i used kubernetes deployment to create a basic pod which i know will not work, i got this error
Error: Waiting for rollout to finish: 0 of 1 updated replicas are available...
Is this just giving me the error from kubernetes or the entire terraform script fails because of this?
According to the documentation
A Deployment ensures that a specified number of pod “replicas” are running at any one time. In other words, a Deployment makes sure that a pod or homogeneous set of pods are always up and available. If there are too many pods, it will kill some. If there are too few, the Deployment will start more.
So, It will wait to ensure number of expected replicas are up
I have Deployment that runs 5 pods.
I want to restart all pods each 5min.
Currently I'm doing it with python scritpt that is running kubectl get po and checks AGE, if the AGE bigger than 5 min it deletes the pod.
Is there another way to achieve that?
You could do a liveness check to achieve this, but why would you do it? Deployment is for LongRunning Tasks.
A liveness check will reschedule a pod if its not true (gives a other exit code than 0)
For more Info here:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
When I am in local, with minikube, I checked that each time I rebuild my app, and roll it out with :
kubectl rollout restart deployment/api-local -n influx
The container lasts 30 sec to be terminated. I have no problem for production matters, but in local dev, I end losing a lot of time waiting, as I can rebuild my app 100 times maybe ( = losing 50 minutes a day ).
Is there some trick to shorten this terminating time ? I understand this would not be part of k8s best practices, but it would make sense to me.
Set the terminationGracePeriodSeconds in the deployment/pod configuration. Per the reference docs, it is:
...Optional duration in seconds the pod needs to terminate gracefully.
May be decreased in delete request. Value must be non-negative
integer. The value zero indicates delete immediately. If this value is
nil, the default grace period will be used instead. The grace period
is the duration in seconds after the processes running in the pod are
sent a termination signal and the time when the processes are forcibly
halted with a kill signal. Set this value longer than the expected
cleanup time for your process. Defaults to 30 seconds.
Example:
spec:
replicas: 1
selector:
...
template:
spec:
terminationGracePeriodSeconds: 0
containers:
...
Also, from here:
The kubectl delete command supports the --grace-period=
option which allows a user to override the default and specify their
own value. The value 0 force deletes the Pod. You must specify an
additional flag --force along with --grace-period=0 in order to
perform force deletions.
Example (specify the namespace, if needed):
kubectl delete pod <pod-name> --now
...or
kubectl delete pod <pod-name> --grace-period=0 --force
As part of our CI pipeline, we have a deployment script for a number of web services that looks something like this:
kubectl apply -f deployment1.yml
kubectl apply -f deployment2.yml
The problem is that the next stage of the pipeline sometimes fails because the services are not ready by the time it starts.
I would like to add a line in the script that says something like:
Wait until all deployments are in the Ready state, or fail if more than 30 seconds has elapsed.
I thought that the following would work but unfortunately it seems that the timeout flag is not available:
kubectl rollout status deployment deployment1 --timeout=30s
kubectl rollout status deployment deployment2 --timeout=30s
I don't want to run "kubectl rollout status" without a timeout as that will cause our build to hang if there is a failure in one of the deployments.
Recent versions of kubectl do support a timeout option:
$ kubectl create -f ds-overlaytest.yml
daemonset.apps/overlaytest created
$ kubectl rollout status ds/overlaytest --timeout=10s
Waiting for daemon set spec update to be observed...
error: timed out waiting for the condition
$
Check out the kubectl reference on how to use this option:
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-status-em-
I found a solution that works well. Set the property .spec.progressDeadlineSeconds to a value such as 30 (default is 600 or ten minutes) and kubectl rollout status deployment will wait for this amount of time before displaying an error message and exiting with a non zero exit code:
$ kubectl rollout status deploy/nginx
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
error: deployment "nginx" exceeded its progress deadline
Documentation is here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment
You could possibly handle this with pure bash i.e.:
ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/myapp -n namespace"
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 60 ]; do
$ROLLOUT_STATUS_CMD
ATTEMPTS=$((attempts + 1))
sleep 10
done
This is specified in this blog
However, I do not believe there is a Kubernetes Native way to wait for a deployments rollout, you could possibly accheive this with hooks in Helm or Webhooks if you want to get really fancy