I want to define a livenessProbe with an httpHeader whose value is secret.
This syntax is invalid:
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: X-Custom-Header
valueFrom:
secretKeyRef:
name: my-secret-key
value: secret
If I specify my-secret-key with value secret as an environment variable named MY_SECRET_KEY, the following could work:
livenessProbe:
exec:
command:
- curl
- --fail
- -H
- "X-Custom-Header: $MY_SECRET_KEY"
- 'http://localhost:8080/healthz'
Unfortunately it doesn't due to the way the quotations are being evaluated. If I type the command curl --fail -H "X-Custom-Header: $MY_SECRET_KEY" http://localhost:8080/healthz directly on the container, it works.
I've also tried many combinations of single quotes and escaping the double quotes.
Does anyone know of a workaround?
Here some examples with curl and wget:
exec:
command:
- /bin/sh
- -c
- "curl -H 'Authorization: Bearer $(AUTH_TOKEN)' 'http://example.com'"
exec:
command:
- /bin/sh
- -c
- "wget --spider --header \"Authorization: Bearer $AUTH_TOKEN\" http://some.api.com/spaces/${SPACE_ID}/entries"
One workaround I can think of is to create some bash script to run this health check, and put your secret data to the environment as usual.
Related
As per Kubernetes Doc , to reference a env variable in config use expr $(ENV_VAR).
But, this is not working for me.
In Readiness and Liveliness Probe API, I am getting token value as $(KUBERNETES_AUTH_TOKEN) instead of abcdefg
containers:
env:
- name: KUBERNETES_AUTH_TOKEN
value: "abcdefg"
lifecycle:
preStop:
exec:
command:
- "sh"
- "-c"
- |
curl -H "Authorization: $(KUBERNETES_AUTH_TOKEN)" -H "Content-Type: application/json" -X GET http://localhost:9443/pre-stop
readinessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 9853
scheme: HTTP
httpHeaders:
- name: Authorization
value: $(KUBERNETES_AUTH_TOKEN)
livenessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 9853
scheme: HTTP
httpHeaders:
- name: Authorization
value: $(KUBERNETES_AUTH_TOKEN)
There is an issue opened about a very similar question here
Anyway, i want to share some yaml lines.
This is not intended for production env, obviously it's just to play around with the commands.
apiVersion: v1
kind: Pod
metadata:
labels:
run: ngtest
name: ngtest
spec:
volumes:
- name: stackoverflow-volume
hostPath:
path: /k8s/stackoverflow-eliminare-vl
initContainers:
- name: initial-container
volumeMounts:
- mountPath: /status/
name: stackoverflow-volume
image: bash
env:
- name: KUBERNETES_AUTH_TOKEN
value: "abcdefg"
command:
# this only writes a text file only to show the purpose of initContainers (here you could create a bash script)
- "sh"
- "-c"
- "echo $(KUBERNETES_AUTH_TOKEN) > /status/$(KUBERNETES_AUTH_TOKEN).txt" # withi this line, substitution occurs with $() form
containers:
- image: nginx
name: ngtest
ports:
- containerPort: 80
env:
- name: KUBERNETES_AUTH_TOKEN
value: "abcdefg"
volumeMounts:
- mountPath: /status/
name: stackoverflow-volume
lifecycle:
preStop:
exec:
command:
- "sh"
- "-c"
- |
echo $KUBERNETES_AUTH_TOKEN > /status/$KUBERNETES_AUTH_TOKEN.txt &&
echo 'echo $KUBERNETES_AUTH_TOKEN > /status/anotherFile2.txt' > /status/exec-$KUBERNETES_AUTH_TOKEN-2.sh && . /status/exec-$KUBERNETES_AUTH_TOKEN-2.sh &&
echo 'echo $(KUBERNETES_AUTH_TOKEN) > /status/anotherFile3.txt' > /status/exec-$(KUBERNETES_AUTH_TOKEN)-3.sh && . /status/exec-$(KUBERNETES_AUTH_TOKEN)-3.sh &&
echo 'curl -H "Authorization: $KUBERNETES_AUTH_TOKEN" -k https://www.google.com/search?q=$KUBERNETES_AUTH_TOKEN > /status/googleresult.txt && exit 0' > /status/exec-a-query-on-google.sh && . /status/exec-a-query-on-google.sh
# first two lines are working
# the third one with $(KUBERNETES_AUTH_TOKEN), do not
# last one with a bash script creation works, and this could be a solution
resources: {}
# # so, instead of use http liveness
# livenessProbe:
# failureThreshold: 3
# httpGet:
# path: /status
# port: 80
# scheme: HTTP
# httpHeaders:
# - name: Authorization
# value: $(KUBERNETES_AUTH_TOKEN)
# u can use the exec to call the endpoint from the bash script.
# reads the file of the initContainer (optional)
# here i omitted the url call , but you can copy from the above examples
livenessProbe:
exec:
command:
- sh
- -c
- "cat /status/$KUBERNETES_AUTH_TOKEN.txt && echo -$KUBERNETES_AUTH_TOKEN- `date` top! >> /status/resultliveness.txt &&
exit 0"
initialDelaySeconds: 15
periodSeconds: 5
dnsPolicy: ClusterFirst
restartPolicy: Always
This creates a pod with an hostPath volume (only to show you the output of the files) where will be create files based on the command of the yaml.
More details on the yaml.
If you go in you cluster machine, you can view the files produced.
Anyway, you should use ConfigMap, Secrets and/or https://helm.sh/docs/chart_template_guide/values_files/, which allow us to create your own charts an separate you config values from the yaml templates.
Hopefully, it helps.
Ps. This is my first answer on StackOverflow, please don't be too rude with me!
I don't think that's possible (at least, not that way -- kiggyttass' answer has an alternative that may work), and really your readiness and liveness endpoints shouldn't require authentication.
However, there is a hacky way around this. I don't know if its available to you, but it works for me. First, you'll need to have the KUBERNETES_AUTH_TOKEN variable set in the environment in which you're running kubectl, then you need to use the bash tool envsubst, like so:
cat k8s.yaml | envsubst | kubectl apply -f -
If you use this approach, you'll have to remove the parenthesis around the env-var.
One final note, the header you're using requires a hint as to the nature of the token, so it should maybe be:
value: Bearer $KUBERNETES_AUTH_TOKEN
Or Basic, or whatever. Also, I believe the token itself must be base64 encoded.
A container defined inside a deployment has a livenessProbe set up: by definition, it calls a remote endpoint and checks, whether response contains useful information or an empty response (which should trigger the pod's restart).
The whole definition is as follows (I removed the further checks for better clarity of the markup):
apiVersion: apps/v1
kind: Deployment
metadata:
name: fc-backend-deployment
labels:
name: fc-backend-deployment
app: fc-test
spec:
replicas: 1
selector:
matchLabels:
name: fc-backend-pod
app: fc-test
template:
metadata:
name: fc-backend-pod
labels:
name: fc-backend-pod
app: fc-test
spec:
containers:
- name: fc-backend
image: localhost:5000/backend:1.3
ports:
- containerPort: 4044
env:
- name: NODE_ENV
value: "dev"
- name: REDIS_HOST
value: "redis"
livenessProbe:
exec:
command:
- curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log
initialDelaySeconds: 20
failureThreshold: 12
periodSeconds: 10
I also tried putting the command into an array:
command: ["sh", "-c", "curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats", "|", "head", "-c", "30", ">", "/app/out.log"]
and splitting into separate lines:
- /bin/bash
- -c
- curl
- -X
- GET
- $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats
- |
- head
- -c
- "30"
- >
- /app/out.log
and even like this:
command:
- |
curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats | head -c 30 > /app/out.log
All attempts were made with and without (/bin/ba)sh -c - with the same result.
But, as you're reading this, you already know that none of these worked.
I know it by exec'ing into running container and trying to find the /app/out.log file - it wasn't present any time I watched the directory contents. It looks like the probe gets never executed.
The command run inside running container works just fine: data gets fetched and written to the specified file.
What might be causing the probe not to get executed?
When using the exec type of probes, Kubernetes will not run a shell to process the command, it will just run the command directly. This means that you can only use a single command and that the | character is considered just another parameter of your curl.
To solve the problem, you need to use sh -c to exec shell code, something like the following:
livenessProbe:
exec:
command:
- sh
- -c
- >-
curl -X GET $BACKEND_SERVICE_HOST:$BACKEND_SERVICE_PORT/api/v3/stats |
head -c 30 > /app/out.log
I was trying to add a poststart hook for my pod using curl, say sending a message to my slack channel
in shell, the command looks like this
curl -d "text=Hi I am a bot that can post messages to any public channel." -d "channel=C1234567" -H "Authorization: Bearer xoxb-xxxxxxxxxxxxxxxx" -X POST https://slack.com/api/chat.postMessage
and in my pod definition, i tried sth like this
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: curlimages/curl
env:
- name: TOKEN
valueFrom:
configMapKeyRef:
name: my-config
key: token
command: ["sleep"]
args: ["3000"]
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- |
curl -d "text=Hi going to start." -d "channel=C1234567" -H "Authorization: Bearer $(TOKEN)" -X POST https://slack.com/api/chat.postMessage
Unlike the container->command, it has args parameter which i could pass multi line command with quote, but in lifecycle->poststart->exec->command it doesn't support args parameter
I also tried sth like but no luck
command: ["curl","-d","text=Hi going to start.",....]
but i never got my slack message
My question is, how can i pass long curl command with quote in lifecycle->poststart->exec->command?
it finally solved by replacing () with {}
to use a env variable in command, it should be ${TOKEN}
I am deploying my application in kubernetes using helm chart with 2 sub-charts app and test.
I have the pod of app chart properly running.
But test pod will be running only if it can properly authenticate to app container.
That means, i have to generate an auth_token using a curl request to app service and then add that token as Environment variable AUTH_TOKEN for test container.
I tried different ways to achieve this:
Added an init-container generate-token for test pod, that will generate the token and will save it in a shared volume. And test container will have access to that volume. But the problem here is, the test container doesn't have a code to set env for the container by reading from the shared volume.
Added a sidecar-container sidecar-generate-token instead of an init-container for the same setup as mentioned above. Here also problem is, the test container doesn't have a code to set env for the container by reading from the shared volume. And also, the test pod got into a crashloopbackoff state. If you check the content of volume by getting into the container, there are multiple tokens in the volume file which are generated on each pod restart of crashloopbackoff.
Third plan was that an init-container generate-token should create a kubernetes secret in the cluster, after generating the auth_token. Then the main container test can set Environment variable from that secret. For that, the init container generate-token should have a kubectl setup in it first.
If i am proceeding with the third plan, How can i setup and use kubectl from init-container to generate secret in the cluster?
Is there any other alternative plan to achieve this goal?
EDIT:
This is the yaml part for the first option:
initContainers:
- name: generate-service-token
image: app.mycr.io/alpine-network-troubleshooting:dev-latest
command:
- /bin/sh
- -c
- |
BEARER_TOKEN=$(curl -k -X POST -H "Content-Type:application/json" --data '{"user":"dynizer","password":"xxxx"}' "https://app:50051/api/v2/login" | jq -r '.jwt')
SERVICE_TOKEN=$(curl -k -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "https://app:50051/api/v2/servicetoken/issue" | jq -r '.token')
echo $SERVICE_TOKEN
mkdir -p /vol
touch /vol/token.txt
echo $SERVICE_TOKEN >> /vol/token.txt
volumeMounts:
- mountPath: /vol
name: token-vol
containers:
- name: nginx-container
image: nginx
volumeMounts:
- name: token-vol
mountPath: /vol
volumes:
- name: token-vol
emptyDir: {}
Trying to answer your question:
But still the same problem of container not having the code to set env by reading from the shared volume, will be there.
Let's try to read this env from other container. Here Is what I have come up with.
First you need to know what command your container is running. In case of nginx that is /docker-entrypoint.sh nginx -g "daemon off;" (source code)
Then you use command field where you read the token value from file and use env to set it and run the actual applciation.
Example:
initContainers:
- name: generate-service-token
image: app.mycr.io/alpine-network-troubleshooting:dev-latest
command:
- /bin/sh
- -c
- |
BEARER_TOKEN=$(curl -k -X POST -H "Content-Type:application/json" --data '{"user":"dynizer","password":"xxxx"}' "https://app:50051/api/v2/login" | jq -r '.jwt')
SERVICE_TOKEN=$(curl -k -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "https://app:50051/api/v2/servicetoken/issue" | jq -r '.token')
echo $SERVICE_TOKEN
mkdir -p /vol
touch /vol/token.txt
echo $SERVICE_TOKEN >> /vol/token.txt
volumeMounts:
- mountPath: /vol
name: token-vol
containers:
- name: nginx-container
image: nginx
command:
- sh
- -c
- exec env SERVICE_TOKEN=$(cat /vol/token.txt) /docker-entrypoint.sh nginx -g "daemon off;"
volumeMounts:
- name: token-vol
mountPath: /vol
volumes:
- name: token-vol
emptyDir: {}
More general example:
command:
- sh
- -c
- exec env SERVICE_TOKEN=$(cat /vol/token.txt) <<any command>>
I am not sure if this is the best example, but I hope that at least it gives you an idea how you can approach this problem.
I am trying to use below command to curl to the pod itself using podip as currently I don't want kubelet to request my pod for health check.
livenessProbe:
exec:
command:
- curl
- $POD_IP:9990/admin/ping
initialDelaySeconds: 3
periodSeconds: 5
but the env variable $POD_IP is not recognized here,
Could not resolve host: $POD_IP
How to configure this so that env var can be read by curl in the command.
reference:
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#execaction-v1-core
try following :
command:
- bash
- -c
- curl $POD_IP:9990/admin/ping
Try using escaping $ with \$ this has worked for me.
livenessProbe:
exec:
command:
- curl
- \$POD_IP:9990/admin/ping
initialDelaySeconds: 3
periodSeconds: 5