I have a pod with the following config:
apiVersion: v1
kind: Pod
metadata:
labels:
name: demo
name: demo
spec:
containers:
- name: demo
image: ubuntu:14.04
command:
- sleep
- "3600"
When I try to stop it, the SIGTERM is ignored by the sleep command, and it takes 30 seconds (the full default grace period) to stop. I can also get on the pod and send the signal to the process (pid 1) manually, and it does not kill the pod. How can I get sleep to die when a signal is sent to it?
Bash ignores SIGTERM when there are no traps. You can trap SIGTERM to force an exit. For example, trap 'exit 255' SIGTERM; sleep 3600
Related
Currently it takes quite a long time before the pod can be terminated after a kubectl delete command. I have the feeling that it could be because of the sleep command.
How can I make the container stop faster?
What best practices should I use here?
apiVersion: apps/v1
kind: Deployment
...
spec:
template:
spec:
containers:
- image: alpine
..
command:
- /bin/sh
- -c
- |
trap : TERM INT
while true; do
# some code to check something
sleep 10
done
Is my approach with "trap: TERM INT" correct? At the moment I don't see any positive effect...
When I terminate the pod it takes several seconds for the command to come back.
kubectl delete pod my-pod
Add terminationGracePeriodSeconds to your spec will do:
...
spec:
template:
spec:
terminationGracePeriodSeconds: 10 # <-- default is 30, can go as low as 0 to send SIGTERM immediately.
containers:
- image: alpine
The scenario is as follows:
Our pods have a terminationGracePeriodSeconds of 60, which gives them ~60 seconds to do any necessary cleanup before Kubernetes will kill them ungracefully. In the majority of cases the cleanup happens well within the 60 seconds. But every now and then we (manually) observe pods that didn't complete their gracefully termination and were killed by Kubernetes.
How does one monitor these situations? When I try replicating this scenario with a simple linux image and sleep, I don't see Kubernetes logging an additional event after the "Killed" one. Without an additional event this is impossible to monitor from the infrastructure side.
You can use container hooks and then you can monitor those hooks events. For example preStop hook which is called when a POD get destroyed, will fire FailedPreStopHook event if it can not complete its work until terminationGracePeriodSeconds
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop:
exec:
command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
I have just started with Kubernetes.
I need to run a Deployment in Kubernetes with a container that competes for execution after ~10-15 minutes.
When I tried, "restart Policy=Never" doesn't hold true with Deployments.
Reason for opting for Deployment is to use Replicas.
Please provide your inputs on how I can achieve multiple replicas of my Deployment with the container that completes execution and not keep running.
You can run a Job as below where the container runs a sleep command for 15m. After 15 minutes the container will exit and pod will be terminated.
apiVersion: batch/v1
kind: Job
metadata:
name: job
spec:
template:
spec:
containers:
- command:
- sh
- -c
- sleep 15m
image: bash:5.1.0
restartPolicy: Never
We are listening to multiple mailboxes on a single pod but if this pod goes down due to some reason need the other pod that is up to listen to these mailboxes. In order to keep recieving emails.
I would like to know if it is possible to find if a pod goes down like an event and trigger a script to perform above action on the go?
Approach 1:
kubernetes life cycle handler hook
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop:
exec:
command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
Approach2:
Write a script which monitors the health of for every say x seconds, when 3 consecutive health checks fail kubernetes deletes the pod. so in your script, if 3 consecutive rest call fails for health then the pod is deleted. you can trigger your event.
Approach3:
maintain 2 replicas => problem could be two pods processing same mail. you can avoid this if you use kafka.
I have a job definition based on example from kubernetes website.
apiVersion: batch/v1
kind: Job
metadata:
name: pi-with-timeout-6
spec:
activeDeadlineSeconds: 30
completions: 1
parallelism: 1
template:
metadata:
name: pi
spec:
containers:
- name: pi
image: perl
command: ["exit", "1"]
restartPolicy: Never
I would like run this job once and not restart if fails. With comand exit 1 kubernetes trying to run new pod to get exit 0 code until reach activeDeadlineSeconds timeout. How can avoid that? I would like run build commands in kubernetes to check compilation and if compilation fails I'll get exit code different than 0. I don't want run compilation again.
Is it possible? How?
By now this is possible by setting backoffLimit: 0 which tells the controller to do 0 retries. default is 6
If you want a one-try command runner, you probably should create bare pod, because the job will try to execute the command until it's successful or the active deadline is met.
Just create the pod from your template:
apiVersion: v1
kind: Pod
metadata:
name: pi
spec:
containers:
- name: pi
image: perl
command: ["exit", "1"]
restartPolicy: Never
Sadly there is currently no way to prevent the job controller to just respawn new pods when they fail, but the kubernetes community is working on a solution, see:
"Backoff policy and failed pod limit" https://github.com/kubernetes/community/pull/583