Can I use env in preStop command - kubernetes

Can I use environment variable in lifecycle.preStop.exec.command? I have a script that has to be run in preStop command. The answer here states that it's possible to use env variables in postStart Can I use env in postStart command. It doesn't work with preStop though. Is it a bug or am I doing something wrong?
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: loap
spec:
replicas: 1
template:
metadata:
labels:
app: loap
spec:
containers:
-
command:
- sh
- "-c"
- "echo $(date +%s): START >> /loap/timing; sleep 10; echo $(date +%s): END >> /loap/timing;"
image: busybox
env:
- name: secretThing
valueFrom:
secretKeyRef:
name: supersecret
key: password
lifecycle:
preStop:
exec:
command:
- sh
- "-c"
- "echo ${secretThing} $(date +%s): PRE-HOOK >> /loap/timing"
livenessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): LIVENESS >> /loap/timing"
name: main
readinessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): READINESS >> /loap/timing"
volumeMounts:
-
mountPath: /loap
name: timing
initContainers:
-
command:
- sh
- "-c"
- "echo $(date +%s): INIT >> /loap/timing"
image: busybox
name: init
volumeMounts:
-
mountPath: /loap
name: timing
volumes:
-
hostPath:
path: /tmp/loap
name: timing

This is explained in the Kubernetes docs Working with objects - Names.
A client-provided string that refers to an object in a resource URL, such as /api/v1/pods/some-name.
Only one object of a given kind can have a given name at a time. However, if you delete the object, you can make a new object with the same name.
By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions.

Related

Bash script in postStart is not executing

I'm trying to run Stateful set with my own scripts, and I'm able to run first script that will spin up mongodb and setup some users etc, but the second script in postStart block, named configure.sh is never executed for some reason.
Here's the StatefulSet manifest yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
labels:
component: mongo
spec:
selector:
matchLabels:
component: mongo
serviceName: mongo
replicas: 1
template:
metadata:
labels:
component: mongo
spec:
containers:
- name: mongo
image: mongo:latest
command: [ "/bin/bash", "-c" , "+m"]
workingDir: /mongo/scripts
args:
- /mongo/scripts/mongo-start.sh
livenessProbe:
exec:
command:
- "bin/bash"
- "-c"
- mongo -u $MONGO_USER -p $MONGO_PASSWORD --eval db.adminCommand\(\"ping\"\)
failureThreshold: 3
successThreshold: 1
periodSeconds: 10
timeoutSeconds: 5
ports:
- containerPort: 27017
imagePullPolicy: Always
lifecycle:
postStart:
exec:
command: ["/bin/bash", "-c", "/mongodb/configure.sh"]
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-scripts
mountPath: /mongo/scripts
- name: mongo-config
mountPath: /mongodb/configure.sh
subPath: configure.sh
env:
- name: MONGO_USER_APP_NAME
valueFrom:
configMapKeyRef:
key: MONGO_USER_APP_NAME
name: mongo-auth-env
- name: MONGO_USER_APP_PASSWORD
valueFrom:
configMapKeyRef:
key: MONGO_USER_APP_PASSWORD
name: mongo-auth-env
- name: MONGO_USER
valueFrom:
configMapKeyRef:
key: MONGO_USER
name: mongo-auth-env
- name: MONGO_PASSWORD
valueFrom:
configMapKeyRef:
key: MONGO_PASSWORD
name: mongo-auth-env
- name: MONGO_BIND_IP
valueFrom:
configMapKeyRef:
key: MONGO_BIND_IP
name: mongo-config-env
restartPolicy: Always
volumes:
- name: mongo-scripts
configMap:
name: mongo-scripts
defaultMode: 0777
- name: mongo-config
configMap:
name: mongo-config
defaultMode: 0777
- name: mongo-config-env
configMap:
name: mongo-config-env
defaultMode: 0777
- name: mongo-auth-env
configMap:
name: mongo-auth-env
defaultMode: 0777
volumeClaimTemplates:
- metadata:
name: mongo-persistent-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
mongo-start.sh which is in /scripts folder with another scripts, is being executed, but after Pod is up and running, configure.sh is never executed, logs are not helpful, kubectl describe pod returns it as recognized, but it never runs. ConfigMaps are all deployed and their content and paths are also ok. Is there any other way to run the script after another, or I'm doing something wrong, been searching on SO and official docs, that's the only examples I found. Tnx
EDIT
it started somehow, but with:
Exec lifecycle hook ([/bin/bash -c /mongodb/mongodb-config.sh]) for Container "mongo" in Pod "mongo-0_test(e9db216d-c1c2-4f19-b85e-19b210a22bbb)" failed - error: command '/bin/bash -c /mongodb/mongodb-config.sh' exited with 1: , message: "MongoDB shell version v4.2.12\nconnecting to: mongodb://mongo:27017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb\n2021-11-24T22:16:50.520+0000 E QUERY [js] Error: couldn't connect to server mongo:27017, connection attempt failed: SocketException: Error connecting to mongo:27017 (172.20.3.3:27017) :: caused by :: Connection refused :\nconnect#src/mongo/shell/mongo.js:353:17\n#(connect):2:6\n2021-11-24T22:16:50.522+0000
Content of the configure.sh:
#!/bin/bash
mongo --username $MONGO_USER_ROOT_NAME --password "$MONGO_USER_ROOT_PASSWORD" --authenticationDatabase "$MONGO_AUTH_SOURCE" --host mongo --port "$MONGO_PORT" < create.js
If I remove postStart part, and init into container, I can successfully run the script..
There is no guarantee that postStart hook will be call after container entry point. Also, postStart hook can be called more than once. The error was because by the time configure.sh was executed; the mongodb was not up and running yet. If your configure.sh script is idempotent, you can do a wait before proceed to next step:
#!/bin/bash
until mongo --nodb --disableImplicitSessions --host mongo --username $MONGO_USER_ROOT_NAME --password $MONGO_USER_ROOT_PASSWORD --eval 'db.adminCommand("ping")'; do sleep 1; done
mongo --username $MONGO_USER_ROOT_NAME --password "$MONGO_USER_ROOT_PASSWORD" --authenticationDatabase "$MONGO_AUTH_SOURCE" --host mongo --port "$MONGO_PORT" < create.js

How can I check whether K8s volume was mounted correctly?

I'm testing out whether I can mount data from S3 using initContainer. What I intended and expected was same volume being mounted to both initContainer and Container. Data from S3 gets downloaded using InitContainer to mountPath called /s3-data, and as the Container is run after the initContainer, it can read from the path the volume was mounted to.
However, the Container doesn't show me any logs, and just says 'stream closed'. The initContainer shows logs that data were successfully downloaded from S3.
What am I doing wrong? Thanks in advance.
apiVersion: batch/v1
kind: Job
metadata:
name: train-job
spec:
template:
spec:
initContainers:
- name: data-download
image: <My AWS-CLI Image>
command: ["/bin/sh", "-c"]
args:
- aws s3 cp s3://<Kubeflow Bucket>/kubeflowdata.tar.gz /s3-data
volumeMounts:
- mountPath: /s3-data
name: s3-data
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef: {key: AWS_ACCESS_KEY_ID, name: aws-secret}
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef: {key: AWS_SECRET_ACCESS_KEY, name: aws-secret}
containers:
- name: check-proper-data-mount
image: <My Image>
command: ["/bin/sh", "-c"]
args:
- cd /s3-data
- echo "Just s3-data dir"
- ls
- echo "After making a sample file"
- touch sample.txt
- ls
volumeMounts:
- mountPath: /s3-data
name: s3-data
volumes:
- name: s3-data
emptyDir: {}
restartPolicy: OnFailure
backoffLimit: 6
You can try like the below mentioned the argument part
---
apiVersion: v1
kind: Pod
metadata:
labels:
purpose: demonstrate-command
name: command-demo
spec:
containers:
-
args:
- cd /s3-data;
echo "Just s3-data dir";
ls;
echo "After making a sample file";
touch sample.txt;
ls;
command:
- /bin/sh
- -c
image: "<My Image>"
name: containername
for reference:
How to set multiple commands in one yaml file with Kubernetes?

Initcontainer not initializing in kubernetes

I´m trying to retrieve some code from gitlab in my yaml.
Unfortunatly the job fails to initalize the pod. I have checked all the logs and it fails with the following message:
0 container "filter-human-genome-and-create-mapping-stats" in pod "create-git-container-5lhp2" is waiting to start: PodInitializing
Here is the yaml file:
apiVersion: batch/v1
kind: Job
metadata:
name: create-git-container
namespace: diag
spec:
template:
spec:
initContainers:
- name: bash-script-git-downloader
image: alpine/git
volumeMounts:
- mountPath: /bash_scripts
name: bash-script-source
command: ["/bin/sh","-c"]
args: ["git", "clone", "https://.......#gitlab.com/scripts.git" ]
containers:
- name: filter-human-genome-and-create-mapping-stats
image: myimage
env:
- name: OUTPUT
value: "/output"
command: ["ls"]
volumeMounts:
- mountPath: /bash_scripts
name: bash-script-source
- mountPath: /output
name: output
volumes:
- name: bash-script-source
emptyDir: {}
- name: output
persistentVolumeClaim:
claimName: output
restartPolicy: Never
If you use bash -c, it expects only one argument. So you have to pass your args[] as one argument. There are ways to do it:
command: ["/bin/sh","-c"]
args: ["git clone https://.......#gitlab.com/scripts.git"]
or
command: ["/bin/sh","-c", "git clone https://.......#gitlab.com/scripts.git"]
or
args: ["/bin/sh","-c", "git clone https://.......#gitlab.com/scripts.git"]
or
command:
- /bin/sh
- -c
- |
git clone https://.......#gitlab.com/scripts.git
or
args:
- /bin/sh
- -c
- |
git clone https://.......#gitlab.com/scripts.git

Kubernetes | Any hooks available for Pod restarts?

Are there any hooks available for Pod lifecycle events? Specifically, I want to run a command to upload logs on pod restart.
Edit: PreStop hook doesn't work for container restart - please see rest of answer below
As standing in documentation there are PreStop and PostStart events and you can attach to them.
Example from docs:
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
preStop:
exec:
command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
Edit:
So I checked with following POC if that preStop hook is executed on container crash and conclusion is: NOT
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
volumeMounts:
- mountPath: /data
name: test-volume
image: nginx
command: ["/bin/sh"]
args: ["-c", "sleep 5; exit 1"]
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /data/postStart"]
preStop:
exec:
command: ["/bin/sh","-c","echo preStop handler! > /data/preStop"]
volumes:
- name: test-volume
hostPath:
path: /data
type: Directory
As solution for you I would recommend to override command section for you container this way:
command: ["/bin/sh"]
args: ["-c", "your-application-executable; your-logs-upload"]
so your-logs-upload executable will be executed after your-application-executable crash/end

Can I use env in postStart command

Can I use environment variable in lifecycl.postStart.exe.command?
I have a script that has to be run in postStart command.
The command contains a secret, can I use valueFrom to get the secret to env, and use the env in postStart command?
Yes, it is possible.
Using the example from this post to create hooks, let's read a secret and pass it as environment variable to the container, to later read it in the postStart hook.
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: loap
spec:
replicas: 1
template:
metadata:
labels:
app: loap
spec:
containers:
-
command:
- sh
- "-c"
- "echo $(date +%s): START >> /loap/timing; sleep 10; echo $(date +%s): END >> /loap/timing;"
image: busybox
env:
- name: SECRET_THING
valueFrom:
secretKeyRef:
name: supersecret
key: password
lifecycle:
postStart:
exec:
command:
- sh
- "-c"
- "echo ${SECRET_THING} $(date +%s): POST-START >> /loap/timing"
preStop:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): PRE-HOOK >> /loap/timing"
livenessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): LIVENESS >> /loap/timing"
name: main
readinessProbe:
exec:
command:
- sh
- "-c"
- "echo $(date +%s): READINESS >> /loap/timing"
volumeMounts:
-
mountPath: /loap
name: timing
initContainers:
-
command:
- sh
- "-c"
- "echo $(date +%s): INIT >> /loap/timing"
image: busybox
name: init
volumeMounts:
-
mountPath: /loap
name: timing
volumes:
-
hostPath:
path: /tmp/loap
name: timing
If you examine the contents of /tmp/loap/timings, you can see the secret being shown
my-secret-password 1515415872: POST-START
1515415873: READINESS
1515415879: LIVENESS
1515415882: END
1515415908: START
my-secret-password 1515415908: POST-START
1515415909: LIVENESS
1515415913: READINESS
1515415918: END