As part of our CI pipeline, we have a deployment script for a number of web services that looks something like this:
kubectl apply -f deployment1.yml
kubectl apply -f deployment2.yml
The problem is that the next stage of the pipeline sometimes fails because the services are not ready by the time it starts.
I would like to add a line in the script that says something like:
Wait until all deployments are in the Ready state, or fail if more than 30 seconds has elapsed.
I thought that the following would work but unfortunately it seems that the timeout flag is not available:
kubectl rollout status deployment deployment1 --timeout=30s
kubectl rollout status deployment deployment2 --timeout=30s
I don't want to run "kubectl rollout status" without a timeout as that will cause our build to hang if there is a failure in one of the deployments.
Recent versions of kubectl do support a timeout option:
$ kubectl create -f ds-overlaytest.yml
daemonset.apps/overlaytest created
$ kubectl rollout status ds/overlaytest --timeout=10s
Waiting for daemon set spec update to be observed...
error: timed out waiting for the condition
$
Check out the kubectl reference on how to use this option:
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-status-em-
I found a solution that works well. Set the property .spec.progressDeadlineSeconds to a value such as 30 (default is 600 or ten minutes) and kubectl rollout status deployment will wait for this amount of time before displaying an error message and exiting with a non zero exit code:
$ kubectl rollout status deploy/nginx
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
error: deployment "nginx" exceeded its progress deadline
Documentation is here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment
You could possibly handle this with pure bash i.e.:
ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/myapp -n namespace"
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 60 ]; do
$ROLLOUT_STATUS_CMD
ATTEMPTS=$((attempts + 1))
sleep 10
done
This is specified in this blog
However, I do not believe there is a Kubernetes Native way to wait for a deployments rollout, you could possibly accheive this with hooks in Helm or Webhooks if you want to get really fancy
Related
I'm trying to add a step in the pipeline where a check is made if the PODS are ready before the pipeline moves forward.
I've read on forums that the kubectl wait command can work, however, every time the pipeline is executed, the PODS are created again, so I can't put a default name on the kubectl wait command.
Does anyone have any tips, please?
If your pods are part of a deployment:
kubectl wait deployment <deployment-name> -n <deployment-namespace> --for condition=Available=True --timeout=120s
I have a deployment that fails within one second. And the logs are destroyed as the deployment does a rollback.
Is there anything similar to logs -f that works before a deployment have started and wait until it starts?
Check previous logs with kubectl logs -p <pod-name> to spot application issues.
Also, check the exit code of your container with:
kubectl describe pod <pod-name> | grep "Exit Code"
Finally, if it is a scheduling problem, check out the event log of the corresponding ReplicaSet:
kubectl describe replicaset <name-of-deployment>
When deleting a pod manually kubectl delete wait for the pod to be deleted and one can include a kubectl wait --for.... condition in a script to wait for the pod to be deleted.
I would like to perform the same wait condition but when scaling down (replicas: 0) a deployment.
From the deployment json, the available/unavailable replicas doesn't count the "terminating" pods and as expected kubectl wait --for=delete deployment test-dep doesn't wait for pod termination but for deployment deletion.
So I would like to perform on my script:
kubectl scale --replicas=0 deployment foo-bar
kubectl wait --for=deletion-of-pod-from-deploymenent=foo-bar
Is there a way to do that?
Remarks: I would like to have the code as generic as possible so no hard writing the labels from deployment.
The easiest way would be to use labels and issue kubectl wait based on that.
kubectl wait --for delete pod --selector=<label>=<value>
but, since you don't want that, you can use the script below
#!/bin/bash
deployment_uid=$(kubectl get deployments ${1} -o=jsonpath='{.metadata.uid}')
rs_uids=$(kubectl get rs -o=json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=='\"${deployment_uid}\"') | .metadata.uid')
PODS=""
for i in $rs_uids; do
PODS+=$(kubectl get pods -o=json | jq -r '.items[] | select(.metadata.ownerReferences[].uid=='\"$i\"') | "pod/" + .metadata.name')
done
[ -z "$PODS" ] && echo "Pods not found" || kubectl wait --for=delete --timeout=-1s ${PODS}
This uses Deployment name (as a first argument), to get ownerRefereces UIDs, chaining down to Pods names.
It's much more complicated, and prone to failure, than just using labels.
I have progressDeadlineSeconds set to 120 seconds
I deploy and run kubectl rollout status deployment mydeployment
The deployment failed with 0 of 1 updated replicas available - CrashLoopBackOff
But the kubectl is still hanging forever with message: Waiting for deployment "mydeployment" rollout to finish: 0 of 1 updated replicas are available...
Why is this happening, progressDeadlineSeconds is suppose to force it to fail and cause kubectl rollout status deployment to exit with a non-zero return code right?
You are correct, kubectl rollout status returns a non-zero exit code if the Deployment has exceeded the progression deadline. Progress Deadline Seconds:
.spec.progressDeadlineSeconds is an optional field that specifies
the number of seconds you want to wait for your Deployment to progress
before the system reports back that the Deployment has failed
progressing - surfaced as a condition with Type=Progressing,
Status=False. and Reason=ProgressDeadlineExceeded in the status of
the resource. The Deployment controller will keep retrying the
Deployment. This defaults to 600. In the future, once automatic
rollback will be implemented, the Deployment controller will roll back
a Deployment as soon as it observes such a condition.
If specified, this field needs to be greater than
.spec.minReadySeconds.
Which brings us to Min Ready Seconds
.spec.minReadySeconds is an optional field that specifies the
minimum number of seconds for which a newly created Pod should be
ready without any of its containers crashing, for it to be considered
available. This defaults to 0 (the Pod will be considered available as
soon as it is ready).
Without your exact deployment configs it is hard to tell where the problem is but there are few things to check out:
Try setting both the progressDeadlineSeconds and minReadySeconds remembering that the later must have a smaller value.
progressDeadlineSeconds might not be respected when used with Replicas: 1. Check your maxUnavailable param in order to see if it allows deployment full unavailability.
As a workaround you can specify a timeout period for your kubectl rollout status command. For example: kubectl rollout status deployment mydeployment --timeout=120s
If you don't want to wait for the rollout to finish then you can use --watch=false. You will than have to check the status manually by running kubectl describe deployment and kubectl get deployment commands which might not be ideal.
Please let me know if that helps.
To wait for a certain pod to be completed the command is
kubectl wait --for=condition=Ready pod/pod-name
Similarly I want to wait for any one pod in the statefulset to be ready. I tried the command below which did not work,
kubectl wait --for=condition=Ready statefulset/statefulset-name
What should the command options look like?
I used following and it works for me
kubectl wait -l statefulset.kubernetes.io/pod-name=activemq-0 --for=condition=ready pod --timeout=-1s
kubectl rollout status --watch --timeout=600s statefulset/name-of-statefulset
from https://github.com/kubernetes/kubernetes/issues/79606#issuecomment-779779928