When should I use commands or args in readinessProbes - kubernetes

I am working my way through killer.sh.for the CKAD. I encountered a pod definition file that has a command field under the readiness probe and the container executes another command but uses args.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod6
name: pod6
spec:
containers:
- args:
- sh
- -c
- touch /tmp/ready && sleep 1d
image: busybox:1.31.0
name: pod6
resources: {}
readinessProbe: # add
exec: # add
command: # add
- sh # add
- -c # add
- cat /tmp/ready # add
initialDelaySeconds: 5 # add
periodSeconds: 10 # add
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
If the readiness probe weren't used and this pod were created implicitly, args wouldn't be utilized.
kubectl run pod6 --image=busybox:1.31.0 --dry-run=client --command -- sh -c "touch /tmp/ready && sleep 1d" > 6.yaml
The output YAML would look like this:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod69
name: pod69
spec:
containers:
- command:
- sh
- -c
- touch /tmp/ready && sleep 1d
image: busybox:1.31.9
name: pod69
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
Why is command not used on both the readinessProbe and the container?
When do commands become args?
Is there a way to tell?
I've read through this document: https://kubernetes.io/docs/tasks/inject-data-application/_print/
but I still haven't had much luck understanding this situation and when to switch to args.

The reason why you have both cmd + args in Kubernetes is because it gives you options to override the default Commands + Args from the image that you are trying to run.
In your specific case, the busybox image does not have any default Commands with the image so specifying the starting command in either cmd or args in the Pod.yaml file is essentially the same.
To your question of when do commands become args - they dont, when a container is spun up using your image, it simply executes cmd + args. And if the cmd is empty in (both the image & the yaml file) then only the args are executed.
The thread here may give you some more explanation

Related

livenessProbe seems not to be executed

A container defined inside a deployment has a livenessProbe set up: by definition, it calls a remote endpoint and checks, whether response contains useful information or an empty response (which should trigger the pod's restart).
The whole definition is as follows (I removed the further checks for better clarity of the markup):
apiVersion: apps/v1
kind: Deployment
metadata:
name: fc-backend-deployment
labels:
name: fc-backend-deployment
app: fc-test
spec:
replicas: 1
selector:
matchLabels:
name: fc-backend-pod
app: fc-test
template:
metadata:
name: fc-backend-pod
labels:
name: fc-backend-pod
app: fc-test
spec:
containers:
- name: fc-backend
image: localhost:5000/backend:1.3
ports:
- containerPort: 4044
env:
- name: NODE_ENV
value: "dev"
- name: REDIS_HOST
value: "redis"
livenessProbe:
exec:
command:
- curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log
initialDelaySeconds: 20
failureThreshold: 12
periodSeconds: 10
I also tried putting the command into an array:
command: ["sh", "-c", "curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats", "|", "head", "-c", "30", ">", "/app/out.log"]
and splitting into separate lines:
- /bin/bash
- -c
- curl
- -X
- GET
- $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats
- |
- head
- -c
- "30"
- >
- /app/out.log
and even like this:
command:
- |
curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log
All attempts were made with and without (/bin/ba)sh -c - with the same result.
But, as you're reading this, you already know that none of these worked.
I know it by exec'ing into running container and trying to find the /app/out.log file - it wasn't present any time I watched the directory contents. It looks like the probe gets never executed.
The command run inside running container works just fine: data gets fetched and written to the specified file.
What might be causing the probe not to get executed?
When using the exec type of probes, Kubernetes will not run a shell to process the command, it will just run the command directly. This means that you can only use a single command and that the | character is considered just another parameter of your curl.
To solve the problem, you need to use sh -c to exec shell code, something like the following:
livenessProbe:
exec:
command:
- sh
- -c
- >-
curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats |
head -c 30 > /app/out.log

Restart a Kubernetes Job or Pod with a different command

I'm looking for a way to quickly run/restart a Job/Pod from the command line and override the command to be executed in the created container.
For context, I have a Kubernetes Job that gets executed as a part of our deploy process. Sometimes that Job crashes and I need to run certain commands inside the container the Job creates to debug and fix the problem (subsequent Jobs then succeed).
The way I have done this so far is:
Copy the YAML of the Job, save into a file
Clean up the YAML (delete Kubernetes-managed fields)
Change the command: field to tail -f /dev/null (so that the container stays alive)
kubectl apply -f job.yaml && kubectl get all && kubectl exec -ti pod/foobar bash
Run commands inside the container
kubectl delete job/foobar when I am done
This is very tedious. I am looking for a way to do something like the following
kubectl restart job/foobar --command "tail -f /dev/null"
# or even better
kubectl run job/foobar --exec --interactive bash
I cannot use the run command to create a Pod:
kubectl run --image xxx -ti
because the Job I am trying to restart has certain volumeMounts and other configuration I need to reuse. So I would need something like kubectl run --from-config job/foobar.
Is there a way to achieve this or am I stuck with juggling the YAML definition file?
Edit: the Job YAML looks approx. like this:
apiVersion: batch/v1
kind: Job
metadata:
name: database-migrations
labels:
app: myapp
service: myapp-database-migrations
spec:
backoffLimit: 0
template:
metadata:
labels:
app: myapp
service: myapp-database-migrations
spec:
restartPolicy: Never
containers:
- name: migrations
image: registry.example.com/myapp:977b44c9
command:
- "bash"
- "-c"
- |
set -e -E
echo "Running database migrations..."
do-migration-stuff-here
echo "Migrations finished at $(date)"
imagePullPolicy: Always
volumeMounts:
- mountPath: /home/example/myapp/app/config/conf.yml
name: myapp-config-volume
subPath: conf.yml
- mountPath: /home/example/myapp/.env
name: myapp-config-volume
subPath: .env
volumes:
- name: myapp-config-volume
configMap:
name: myapp
imagePullSecrets:
- name: k8s-pull-project
The commands you suggested don't exist. Take a look at this reference where you can find all available commands.
Based on that documentation the task of the Job is to create one or more Pods and continue retrying execution them until the specified number of successfully terminated ones will be achieved. Then the Job tracks the successful completions. You cannot just update the Job because these fields are not updatable. To do what's you want you should delete current job and create one once again.
I recommend you to keep all your configurations in files. If you have a problem with configuring job commands, practice says that you should modify these settings in yaml and apply to the cluster - if your deployment crashes - by storing the configuration in files, you have a backup.
If you are interested how to improve this task, you can try those 2 examples describe below:
Firstly I've created several files:
example job (job.yaml):
apiVersion: batch/v1
kind: Job
metadata:
name: test1
spec:
template:
spec:
containers:
- name: test1
image: busybox
command: ["/bin/sh", "-c", "sleep 300"]
volumeMounts:
- name: foo
mountPath: "/script/foo"
volumes:
- name: foo
configMap:
name: my-conf
defaultMode: 0755
restartPolicy: OnFailure
patch-file.yaml:
spec:
template:
spec:
containers:
- name: test1
image: busybox
command: ["/bin/sh", "-c", "echo 'patching test' && sleep 500"]
and configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-conf
data:
test: |
#!/bin/sh
echo "skrypt test"
If you want to automate this process you can use plugin
A plugin is a standalone executable file, whose name begins with kubectl-. To install a plugin, move its executable file to anywhere on your PATH.
There is no plugin installation or pre-loading required. Plugin executables receive the inherited environment from the kubectl binary. A plugin determines which command path it wishes to implement based on its name.
Here is the file that can replace your job
A plugin determines the command path that it will implement based on its filename.
kubectl-job:
#!/bin/bash
kubectl patch -f job.yaml -p "$(cat patch-job.yaml)" --dry-run=client -o yaml | kubectl replace --force -f - && kubectl wait --for=condition=ready pod -l job-name=test1 && kubectl exec -it $(kubectl get pod -l job-name=test1 --no-headers -o custom-columns=":metadata.name") -- /bin/sh
This command uses an additional file (patch-job.yaml, see this link) - within we can put our changes for job.
Then you should change the permissions of this file and move it:
sudo chmod +x .kubectl-job
sudo mv ./kubectl-job /usr/local/bin
It's all done. Right now you can use it.
$ kubectl job
job.batch "test1" deleted
job.batch/test1 replaced
pod/test1-bdxtm condition met
pod/test1-nh2pv condition met
/ #
As you can see Job has been replaced (deleted and created).
You can also use single-line command, here is the example:
kubectl get job test1 -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl patch -f - --patch '{"spec": {"template": {"spec": {"containers": [{"name": "test1", "image": "busybox", "command": ["/bin/sh", "-c", "sleep 200"]}]}}}}' --dry-run=client -o yaml | kubectl replace --force -f -
With this command you can change your job entering parameters "by hand". Here is the output:
job.batch "test1" deleted
job.batch/test1 replaced
As you can see this solution works as well.

Checking result of command in helm chart (helm-hooks)

I am trying to execute a pre install job using helm charts. Can someone help getting result of command (parameter in yaml file) that I put in the below file:
apiVersion: batch/v1
kind: Job
metadata:
name: pre-install-job
annotations:
"helm.sh/hook": "pre-install"
spec:
template:
spec:
containers:
- name: pre-install
image: busybox
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'touch somefile.txt && echo $PWD && sleep 15']
restartPolicy: OnFailure
terminationGracePeriodSeconds: 0
backoffLimit: 3
completions: 1
parallelism: 1
I want to know where somefile.txt is created and echo is printed. And the reason I know it is working because "sleep 15" works. I see a 15 second difference in start and end time of pod creation.
Any file you create in a container environment is created inside the container filesystem. Unless you've mounted some storage into the container, the file will be lost as soon as the container exits.
Anything a Kubernetes process writes to its stdout will be captured by the Kubernetes log system. You can retrieve it using kubectl logs pre-install-job-... -c pre-install.

How to run one-off job?

I've found 2 different ways to run a one-off command in my kubernetes cluster:
Method 1
kubectl apply -f kubernetes/migrate.job.yaml
kubectl wait --for=condition=complete --timeout=600s job/migrate-job
kubectl delete job/migrate-job
The problem with this is (a) it doesn't show me the output which I like to see, and (b) it's 3 commands
Method 2
kubectl run migrate --stdin --tty --rm --restart=Never --image=example.org/app/php:v-$(VERSION) --command -- ./artisan -vvv migrate
This almost works except I also need a volume mount to run this command, which AFAIK would require a rather lengthy --overrides arg. If I could pull the override in from a file instead it'd probably work well. Can I do that?
I also need to to return the exit code if the command fails.
There's an open ticket for this: https://github.com/kubernetes/kubernetes/issues/63214
A short term solution is to run your job like this:
kubectl run migrate --stdin --tty --rm --restart=Never --image=example.org/app/php:v-$(VERSION) --overrides="$(cat kubernetes/migrate.pod.yaml | y2j)"
Using y2j to convert YAML to JSON so that I can use a standard pod manifest.
migrate.pod.yaml looks like:
apiVersion: v1
kind: Pod
metadata:
name: migrate-pod
spec:
volumes:
- name: migrate-secrets-volume
secret:
secretName: google-service-account
containers:
- name: migrate-container
image: example.org/app/php
command: ["./artisan", "-vvv", "migrate"]
stdin: true
stdinOnce: true
tty: true
envFrom:
- secretRef:
name: dev-env
volumeMounts:
- name: migrate-secrets-volume
mountPath: /app/secrets
readOnly: true
restartPolicy: Never
imagePullSecrets:
- name: regcred

Replication Controller replica ID in an environment variable?

I'm attempting to inject a ReplicationController's randomly generated pod ID extension (i.e. multiverse-{replicaID}) into a container's environment variables. I could manually get the hostname and extract it from there, but I'd prefer if I didn't have to add the special case into the script running inside the container, due to compatibility reasons.
If a pod is named multiverse-nffj1, INSTANCE_ID should equal nffj1. I've scoured the docs and found nothing.
apiVersion: v1
kind: ReplicationController
metadata:
name: multiverse
spec:
replicas: 3
template:
spec:
containers:
- env:
- name: INSTANCE_ID
value: $(replicaID)
I've tried adding a command into the controller's template configuration to create the environment variable from the hostname, but couldn't figure out how to make that environment variable available to the running script.
Is there a variable I'm missing, or does this feature not exist? If it doesn't, does anyone have any ideas on how to make this to work without editing the script inside of the container?
There is an answer provided by Anton Kostenko about inserting DB credentials into container environment variables, but it could be applied to your case also. It is all about the content of the InitContainer spec.
You can use InitContainer to get the hash from the container’s hostname and put it to the file on the shared volume that you mount to the container.
In this example InitContainer put the Pod name into the INSTANCE_ID environment variable, but you can modify it according to your needs:
Create the init.yaml file with the content:
apiVersion: v1
kind: Pod
metadata:
name: init-test
spec:
containers:
- name: init-test
image: ubuntu
args: [bash, -c, 'source /data/config && echo $INSTANCE_ID && while true ; do sleep 1000; done ']
volumeMounts:
- name: config-data
mountPath: /data
initContainers:
- name: init-init
image: busybox
command: ["sh","-c","echo -n INSTANCE_ID=$(hostname) > /data/config"]
volumeMounts:
- name: config-data
mountPath: /data
volumes:
- name: config-data
emptyDir: {}
Create the pod using following command:
kubectl create -f init.yaml
Check if Pod initialization is done and is Running:
kubectl get pod init-test
Check the logs to see the results of this example configuration:
$ kubectl logs init-test
init-test