Are there any hooks available for Pod lifecycle events? Specifically, I want to run a command to upload logs on pod restart.
Edit: PreStop hook doesn't work for container restart - please see rest of answer below
As standing in documentation there are PreStop and PostStart events and you can attach to them.
Example from docs:
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop:
exec:
command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
Edit:
So I checked with following POC if that preStop hook is executed on container crash and conclusion is: NOT
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
volumeMounts:
- mountPath: /data
name: test-volume
image: nginx
command: ["/bin/sh"]
args: ["-c", "sleep 5; exit 1"]
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /data/postStart"]
preStop:
exec:
command: ["/bin/sh","-c","echo preStop handler! > /data/preStop"]
volumes:
- name: test-volume
hostPath:
path: /data
type: Directory
As solution for you I would recommend to override command section for you container this way:
command: ["/bin/sh"]
args: ["-c", "your-application-executable; your-logs-upload"]
so your-logs-upload executable will be executed after your-application-executable crash/end
Related
I am trying to execute some scripts as part of statefulset deployment kind. This script I have added as configmap and I use this as volumeMount inside the pod definition. I use the lifecycle poststart exec command to execute this script. It fails with the permission issue.
based on certain articles, I found that we should copy this file as part of InitContainer and then use that (I am not sure why should we do and what will make a difference)
Still, I tried it and that also gives the same error.
Here is my ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-configmap-initscripts
data:
poststart.sh: |
#!/bin/bash
echo "It`s done"
Here is my StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres-statefulset
spec:
....
serviceName: postgres-service
replicas: 1
template:
...
spec:
initContainers:
- name: "postgres-ghost"
image: alpine
volumeMounts:
- mountPath: /scripts
name: postgres-scripts
containers:
- name: postgres
image: postgres
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "/scripts/poststart.sh" ]
ports:
- containerPort: 5432
name: dbport
....
volumeMounts:
- mountPath: /scripts
name: postgres-scripts
volumes:
- name: postgres-scripts
configMap:
name: postgres-configmap-initscripts
items:
- key: poststart.sh
path: poststart.sh
The error I am getting:
postStart hook will be call at least once but may be call more than once, this is not a good place to run script.
The poststart.sh file that mounted as ConfigMap will not have execute mode hence the permission error.
It is better to run script in initContainers, here's an quick example that do a simple chmod; while in your case you can execute the script instead:
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: busybox
data:
test.sh: |
#!/bin/bash
echo "It's done"
---
apiVersion: v1
kind: Pod
metadata:
name: busybox
labels:
run: busybox
spec:
volumes:
- name: scripts
configMap:
name: busybox
items:
- key: test.sh
path: test.sh
- name: runnable
emptyDir: {}
initContainers:
- name: prepare
image: busybox
imagePullPolicy: IfNotPresent
command: ["ash","-c"]
args: ["cp /scripts/test.sh /runnable/test.sh && chmod +x /runnable/test.sh"]
volumeMounts:
- name: scripts
mountPath: /scripts
- name: runnable
mountPath: /runnable
containers:
- name: busybox
image: busybox
imagePullPolicy: IfNotPresent
command: ["ash","-c"]
args: ["while :; do . /runnable/test.sh; sleep 1; done"]
volumeMounts:
- name: scripts
mountPath: /scripts
- name: runnable
mountPath: /runnable
EOF
So I have a helm chart that deploys a pod, so the next task is to create another pod once the first pod is running.
So I created a simple pod.yaml in chart/templates which creates a simple pod-b, so next step to only create pod-b after pod-a is running.
So was only at helm hooks but don't think they care about pod status.
Another idea is to use Init container like below but not sure how to write command to lookup a pod is running?
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init-myservice
image: busybox
command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
Another idea is a simple script to check pod status something like:
y=`kubectl get po -l app=am -o 'jsonpath={.items[0].status.phase}'`
while [ $i -le 5 ]
do
if [[ "$y" == "Running" ]]; then
break
fi
sleep 5
done
Any advice would be great.
If you want your post-install/post-upgrade chart hooks to work, you should add readiness probes to your first pod and use --wait flag.
helm upgrade --install -n test --wait mychart .
pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: readiness-exec
labels:
test: readiness
spec:
containers:
- name: readiness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- sleep 30; touch /tmp/healthy; sleep 600
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 10
hook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: "post-deploy"
annotations:
"helm.sh/hook": post-upgrade,post-install
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 1
template:
metadata:
name: "post-deploy"
spec:
restartPolicy: Never
containers:
- name: post-deploy
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- echo "executed only after previous pod is ready"
I´m trying to retrieve some code from gitlab in my yaml.
Unfortunatly the job fails to initalize the pod. I have checked all the logs and it fails with the following message:
0 container "filter-human-genome-and-create-mapping-stats" in pod "create-git-container-5lhp2" is waiting to start: PodInitializing
Here is the yaml file:
apiVersion: batch/v1
kind: Job
metadata:
name: create-git-container
namespace: diag
spec:
template:
spec:
initContainers:
- name: bash-script-git-downloader
image: alpine/git
volumeMounts:
- mountPath: /bash_scripts
name: bash-script-source
command: ["/bin/sh","-c"]
args: ["git", "clone", "https://.......#gitlab.com/scripts.git" ]
containers:
- name: filter-human-genome-and-create-mapping-stats
image: myimage
env:
- name: OUTPUT
value: "/output"
command: ["ls"]
volumeMounts:
- mountPath: /bash_scripts
name: bash-script-source
- mountPath: /output
name: output
volumes:
- name: bash-script-source
emptyDir: {}
- name: output
persistentVolumeClaim:
claimName: output
restartPolicy: Never
If you use bash -c, it expects only one argument. So you have to pass your args[] as one argument. There are ways to do it:
command: ["/bin/sh","-c"]
args: ["git clone https://.......#gitlab.com/scripts.git"]
or
command: ["/bin/sh","-c", "git clone https://.......#gitlab.com/scripts.git"]
or
args: ["/bin/sh","-c", "git clone https://.......#gitlab.com/scripts.git"]
or
command:
- /bin/sh
- -c
- |
git clone https://.......#gitlab.com/scripts.git
or
args:
- /bin/sh
- -c
- |
git clone https://.......#gitlab.com/scripts.git
Can I use environment variable in lifecycle.preStop.exec.command? I have a script that has to be run in preStop command. The answer here states that it's possible to use env variables in postStart Can I use env in postStart command. It doesn't work with preStop though. Is it a bug or am I doing something wrong?
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: loap
spec:
replicas: 1
template:
metadata:
labels:
app: loap
spec:
containers:
-
command:
- sh
- "-c"
- "echo $(date +%s): START >> /loap/timing; sleep 10; echo $(date +%s): END >> /loap/timing;"
image: busybox
env:
- name: secretThing
valueFrom:
secretKeyRef:
name: supersecret
key: password
lifecycle:
preStop:
exec:
command:
- sh
- "-c"
- "echo ${secretThing} $(date +%s): PRE-HOOK >> /loap/timing"
livenessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): LIVENESS >> /loap/timing"
name: main
readinessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): READINESS >> /loap/timing"
volumeMounts:
-
mountPath: /loap
name: timing
initContainers:
-
command:
- sh
- "-c"
- "echo $(date +%s): INIT >> /loap/timing"
image: busybox
name: init
volumeMounts:
-
mountPath: /loap
name: timing
volumes:
-
hostPath:
path: /tmp/loap
name: timing
This is explained in the Kubernetes docs Working with objects - Names.
A client-provided string that refers to an object in a resource URL, such as /api/v1/pods/some-name.
Only one object of a given kind can have a given name at a time. However, if you delete the object, you can make a new object with the same name.
By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions.
Can I use environment variable in lifecycl.postStart.exe.command?
I have a script that has to be run in postStart command.
The command contains a secret, can I use valueFrom to get the secret to env, and use the env in postStart command?
Yes, it is possible.
Using the example from this post to create hooks, let's read a secret and pass it as environment variable to the container, to later read it in the postStart hook.
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: loap
spec:
replicas: 1
template:
metadata:
labels:
app: loap
spec:
containers:
-
command:
- sh
- "-c"
- "echo $(date +%s): START >> /loap/timing; sleep 10; echo $(date +%s): END >> /loap/timing;"
image: busybox
env:
- name: SECRET_THING
valueFrom:
secretKeyRef:
name: supersecret
key: password
lifecycle:
postStart:
exec:
command:
- sh
- "-c"
- "echo ${SECRET_THING} $(date +%s): POST-START >> /loap/timing"
preStop:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): PRE-HOOK >> /loap/timing"
livenessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): LIVENESS >> /loap/timing"
name: main
readinessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): READINESS >> /loap/timing"
volumeMounts:
-
mountPath: /loap
name: timing
initContainers:
-
command:
- sh
- "-c"
- "echo $(date +%s): INIT >> /loap/timing"
image: busybox
name: init
volumeMounts:
-
mountPath: /loap
name: timing
volumes:
-
hostPath:
path: /tmp/loap
name: timing
If you examine the contents of /tmp/loap/timings, you can see the secret being shown
my-secret-password 1515415872: POST-START
1515415873: READINESS
1515415879: LIVENESS
1515415882: END
1515415908: START
my-secret-password 1515415908: POST-START
1515415909: LIVENESS
1515415913: READINESS
1515415918: END