I have integration test, where i start StatefulSet, wait untill ready and then do some asserts.
My problem that if Application fails - it try to restart too fast.
And I can't get logs from failed pod.
SO my question how can i increase time between Restart of pod in StatefulSet?
Because K8s controllers do not support RestartPolicy: Never.
If all you want is to view the logs of the terminated pod, you can do
kubectl log <pod_name> --previous
I would try to run the service in question as a regular deployment and convert it to a StatefulSet after I analyze the issue with the application.
Why can't you get the logs from the terminated pods?
Maybe you should try to set terminationGracePeriodSeconds on the SS container to make the dying pods stay longer for analysis.
Related
I have a testing scenario to check if the API requests are being handled by another pod if one goes down. I know this is the default behaviour, but I want to stimulate the following scenario.
Pod replicas - 2 (pod A and B)
During my API requests, I want to kill/stop only pod A.
During downtime of A, requests should be handled by B.
I am aware that we can restart the deployment and also scale replicas to 0 and again to 2, but this won't work for me.
Is there any way to kill/stop/crash only pod A?
Any help will be appreciated.
If you want to simulate what happens if one of the pods just gets lost, you can scale down the deployment
kubectl scale deployment the-deployment-name --replicas=1
and Kubernetes will terminate all but one of the pods; you should almost immediately see all of the traffic going to the surviving pod.
But if instead you want to simulate what happens if one of the pods crashes and restarts, you can delete the pod
# kubectl scale deployment the-deployment-name --replicas=2
kubectl get pods
kubectl delete pod the-deployment-name-12345-f7h9j
Once the pod starts getting deleted, the Kubernetes Service should route all of the traffic to the surviving pod(s) (those with Running status). However, the pod is managed by a ReplicaSet that wants there to be 2 replicas, so if one of the pods is deleted, the ReplicaSet will immediately create a new one. This is similar to what would happen if the pod crashes and restarts (in this scenario you'd get the same pod and the same node, if you delete the pod it might come back in a different place).
As you mentioned you can manually kill or restart the pod that is the only solution to test the case or else you can try crashing the one single POD but in the end, it will create the same scenario POD will auto restart.
Or else may you can increase the Graceful shutdown period for deployment so this way POD might take time and stay in terminating state for a good amount of time and you can perform the test.
In kubernetes where pods are controlled by the replicaSet, if you kill a pod it will again be recreated. So the only way to do this is to scale down the number of replicas.
Let's say if your deployment had 4 replicas. You can scale down to 3 by running the command below
kubectl scale deployment <deployment-name> --replicas=3
My example is as show below
kubectl scale deployment hello-world --replicas=3
deployment.apps/hello-world scaled
I know I can view logs of a crash pod by using
kubectl logs --previous
But if a pod belongs to a deployment, when it crashes, a new pod with a different name is going to be created.
I can no longer know the crashed pod name.
Where can I find the log of the crashed pod?
And how can I know if/when/why the pod crashed?
If a Deployment-managed Pod crashes, the same Pod will restart, and you can look at its logs using kubectl logs --previous the same as before.
If you manually kubectl delete pod something a Deployment manages, you'll lose its logs and the Deployment will create a new one; but you have to explicitly do that, if a pod fails it will be the same pod restarting (or in CrashLoopBackOff state).
If you can't get the logs then try the below command to know the reason why the pod is failed to start
kubectl describe pod <pod-name>
I have a statefulset which constitutes of multiple pods. I have a use case where I need to invoke restart of the STS, I run this: kubectl rollout restart statefulset mysts
If I restart the statefulset at a time when one or more pods are in not-ready state, the restart action get queued up. Restart takes effect only after all the pods become ready. This could take long depending on the readiness threshold and the kind of issue the pod is facing.
Is there a way to force restart the statefulset, wherein I don't wait for pods to become ready? I don't want to terminate/delete the pods instead of restarting statefulset. A rolling restart works well for me as it helps avoid outage of the application.
I am trying to use Kubernetes Deployment , i would like to know whether this is same as kubectl apply -f deployment.yaml or does this wait for the deployments to be up and running . because when i used kubernetes deployment to create a basic pod which i know will not work, i got this error
Error: Waiting for rollout to finish: 0 of 1 updated replicas are available...
Is this just giving me the error from kubernetes or the entire terraform script fails because of this?
According to the documentation
A Deployment ensures that a specified number of pod “replicas” are running at any one time. In other words, a Deployment makes sure that a pod or homogeneous set of pods are always up and available. If there are too many pods, it will kill some. If there are too few, the Deployment will start more.
So, It will wait to ensure number of expected replicas are up
I have a simple Node.js server running on some Kubernetes pods. When I delete a deployment with:
kubectl delete deployment <deployment-name>
sometimes pods have a status of terminating for 5-8 seconds. Seems excessive - is there a way to kill them faster somehow?
If you really want to kill them instead of gracefully shutdown, then:
kubectl delete deployment <deployment-name> --grace-period=0
Also, if you have any preStop handlers configured, you could investigate if they are causing any unnecessary delay.