Set node label to the pod environment variable - kubernetes

How to set Node label to Pod environment variable? I need to know the label topology.kubernetes.io/zone value inside the pod.

The Downward API currently does not support exposing node labels to pods/containers. There is an open issue about that on GitHib, but it is unclear when it will be implemented if at all.
That leaves the only option to get node labels from Kubernetes API, just as kubectl does. It is not easy to implement, especially if you want labels as environment variables. I'll give you an example how it can be done with an initContainer, curl, and jq but if possible, I suggest you rather implement this in your application, for it will be easier and cleaner.
To make a request for labels you need permissions to do that. Therefore, the example below creates a service account with permissions to get (describe) nodes. Then, the script in the initContainer uses the service account to make a request and extract labels from json. The test container reads environment variables from the file and echoes one.
Example:
# Create a service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: describe-nodes
namespace: <insert-namespace-name-where-the-app-is>
---
# Create a cluster role that allowed to perform describe ("get") over ["nodes"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: describe-nodes
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
---
# Associate the cluster role with the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: describe-nodes
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: describe-nodes
subjects:
- kind: ServiceAccount
name: describe-nodes
namespace: <insert-namespace-name-where-the-app-is>
---
# Proof of concept pod
apiVersion: v1
kind: Pod
metadata:
name: get-node-labels
spec:
# Service account to get node labels from Kubernetes API
serviceAccountName: describe-nodes
# A volume to keep the extracted labels
volumes:
- name: node-info
emptyDir: {}
initContainers:
# The container that extracts the labels
- name: get-node-labels
# The image needs 'curl' and 'jq' apps in it
# I used curl image and run it as root to install 'jq'
# during runtime
# THIS IS A BAD PRACTICE UNSUITABLE FOR PRODUCTION
# Make an image where both present.
image: curlimages/curl
# Remove securityContext if you have an image with both curl and jq
securityContext:
runAsUser: 0
# It'll put labels here
volumeMounts:
- mountPath: /node
name: node-info
env:
# pass node name to the environment
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: APISERVER
value: https://kubernetes.default.svc
- name: SERVICEACCOUNT
value: /var/run/secrets/kubernetes.io/serviceaccount
- name: SCRIPT
value: |
set -eo pipefail
# install jq; you don't need this line if the image has it
apk add jq
TOKEN=$(cat ${SERVICEACCOUNT}/token)
CACERT=${SERVICEACCOUNT}/ca.crt
# Get node labels into a json
curl --cacert ${CACERT} \
--header "Authorization: Bearer ${TOKEN}" \
-X GET ${APISERVER}/api/v1/nodes/${NODENAME} | jq .metadata.labels > /node/labels.json
# Extract 'topology.kubernetes.io/zone' from json
NODE_ZONE=$(jq '."topology.kubernetes.io/zone"' -r /node/labels.json)
# and save it into a file in the format suitable for sourcing
echo "export NODE_ZONE=${NODE_ZONE}" > /node/zone
command: ["/bin/ash", "-c"]
args:
- 'echo "$$SCRIPT" > /tmp/script && ash /tmp/script'
containers:
# A container that needs the label value
- name: test
image: debian:buster
command: ["/bin/bash", "-c"]
# source ENV variable from file, echo NODE_ZONE, and keep running doing nothing
args: ["source /node/zone && echo $$NODE_ZONE && cat /dev/stdout"]
volumeMounts:
- mountPath: /node
name: node-info

You could use InitContainer
...
spec:
initContainers:
- name: node2pod
image: <image-with-k8s-access>
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
...
Ref: Node Label to Pod
Edit Update
A similar Solution could be Inject node labels into Kubernetes pod

Related

Why cannot K8s pod read stored secret?

I cannot access a secret I created. I inserted a secret in K8s secret store and am simply trying to test access to it with this yaml...
apiVersion: v1
kind: Namespace
metadata:
name: space1
---
apiVersion: v1
kind: Pod
metadata:
name: space1
namespace: space1
spec:
containers:
- name: space1-pod
image: repo/python-image:latest
imagePullPolicy: Always
command: ['sh', '-c', 'echo "Username: $USER" "Password: $PASSWORD"']
env:
- name: USER
valueFrom:
secretKeyRef:
name: tool-user
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name:tool-user
key: password
restartPolicy: Always
The status of the "pod is waiting to start: CreateContainerConfigError". And I receive this error...
Error: secret "tool-user" not found
Despite the result I get from "kubectl get secrets" which clearly shows...
NAME TYPE DATA AGE
tool-user Opaque 2 4d1h
kubectl get secrets shows secrets from a default namespace, add -n space1 to see secrets from the namespace your pod runs in.
secrets are namespaced objects. Make sure the secret "tool-user" is created on the "secret1" namespace.

Add to configmap pod name in Kubernetes

I'm trying to figure it out, how to change one string inside configmap in Kubernetes. I have pretty simple configmap:
apiVersion: v1
data:
config.cfg: |-
[authentication]
USERNAME=user
PASSWORD=password
[podname]
PODNAME=metadata.podName
kind: ConfigMap
metadata:
name: name_here
And I need to mount the configmap inside a couple of pods. But PODNAME should be matched to current podname. Is it possible in any another way? thanks!
I do not think it could be done with ConfigMap. But you can set environment variables in your pod spec that references a pod fields.
apiVersion: v1
kind: Pod
metadata:
name: test-ref-pod-name
spec:
containers:
- name: test-container
image: busybox
command: [ "sh", "-c"]
args:
- env | grep PODNAME
env:
- name: PODNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
restartPolicy: Never
See official documentation: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables
This doesn't answer you question exactly but the pod name normally ends up as the host name inside the pod and can be accessed as a standard environment variable
echo $HOSTNAME

how does K8S handles multiple remote docker registeries in POD definition using imagePullSecrets list

I would like to access multiple remote registries to pull images.
In the k8s documentation they say:
(If you need access to multiple registries, you can create one secret
for each registry. Kubelet will merge any imagePullSecrets into a
single virtual .docker/config.json)
and so the POD definition should be something like this:
apiVersion: v1
kind: Pod
spec:
containers:
- name: ...
imagePullSecrets:
- name: secret1
- name: secret2
- ....
- name: secretN
Now I am not sure how K8S will pick the right secret for each image? will all secrets be verified one by one each time? and how K8S will handle the failed retries? and if a specific amount of unauthorized retries could lead to some lock state in k8sor docker registries?
/ Thanks
You can use following script to add two authentications in one secret
#!/bin/bash
u1="user_1_here"
p1="password_1_here"
auth1=$(echo -n "$u1:$p1" | base64 -w0)
u2="user_2_here"
p2="password_2_here"
auth2=$(echo -n "$u2:$p2" | base64 -w0)
cat <<EOF > docker_config.json
{
"auths": {
"repo1_name_here": {
"auth": "$auth1"
},
"repo2_name_here": {
"auth": "$auth2"
}
}
}
EOF
base64 -w0 docker_config.json > docker_config_b64.json
cat <<EOF | kubectl apply -f -
apiVersion: v1
type: kubernetes.io/dockerconfigjson
kind: Secret
data:
.dockerconfigjson: $(cat docker_config_b64.json)
metadata:
name: specify_secret_name_here
namespace: specify_namespace_here
EOF
Kubernetes isn't going to try all secrets until find the correct. When you create the secret, you are referencing that it's a docker registry:
$ kubectl create secret docker-registry user1-secret --docker-server=https://index.docker.io/v1/ --docker-username=user1 --docker-password=PASSWORD456 --docker-email=user1#email.com
$ kubectl create secret docker-registry user2-secret --docker-server=https://index.docker.io/v1/ --docker-username=user2 --docker-password=PASSWORD123 --docker-email=user2#email.com
$ kubectl get secrets user1-secret -o yaml
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXIuZXhhbXBsZS5jb20iOnsidXNlcm5hbWUiOiJrdWJlIiwicGFzc3dvcmQiOiJQV19TVFJJTkciLCJlbWFpbCI6Im15QGVtYWlsLmNvbSIsImF1dGgiOiJhM1ZpWlRwUVYxOVRWRkpKVGtjPSJ9fX0=
kind: Secret
metadata:
creationTimestamp: "2020-01-13T13:15:52Z"
name: user1-secret
namespace: default
resourceVersion: "1515301"
selfLink: /api/v1/namespaces/default/secrets/user1-secret
uid: d2f3bb0c-3606-11ea-a202-42010a8000ad
type: kubernetes.io/dockerconfigjson
As you can see, type is kubernetes.io/dockerconfigjson is telling Kubernetes to treat this differently.
So, when you reference the address of your container as magic.example.com/magic-image on your yaml, Kubernetes will have enough information to connect the dots and use the right secret to pull your image.
apiVersion: v1
kind: Pod
metadata:
name: busyboxes
namespace: default
spec:
imagePullSecrets:
- name: user1-secret
- name: user2-secret
containers:
- name: jenkins
image: user1/jenkins
imagePullPolicy: Always
- name: busybox
image: user2/busybox
imagePullPolicy: Always
So as this example describes, it's possible to have 2 or more docker registry secrets with the same --docker-server value. Kubernetes will manage to take care of it seamlessly.

Terminate istio sidecar istio-proxy for a kubernetes job / cronjob

We recently started using istio Istio to establish a service-mesh within out Kubernetes landscape.
We now have the problem that jobs and cronjobs do not terminate and keep running forever if we inject the istio istio-proxy sidecar container into them. The istio-proxy should be injected though to establish proper mTLS connections to the services the job needs to talk to and comply with our security regulations.
I also noticed the open issues within Istio (istio/issues/6324) and kubernetes (kubernetes/issues/25908), but both do not seem to provide a valid solution anytime soon.
At first a pre-stop hook seemed suitable to solve this issue, but there is some confusion about this conecpt itself: kubernetes/issues/55807
lifecycle:
preStop:
exec:
command:
...
Bottomline: Those hooks will not be executed if the the container successfully completed.
There are also some relatively new projects on GitHub trying to solve this with a dedicated controller (which I think is the most preferrable approach), but to our team they do not feel mature enough to put them right away into production:
k8s-controller-sidecars
K8S-job-sidecar-terminator
In the meantime, we ourselves ended up with the following workaround that execs into the sidecar and sends a SIGTERM signal, but only if the main container finished successfully:
apiVersion: v1
kind: ServiceAccount
metadata:
name: terminate-sidecar-example-service-account
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: terminate-sidecar-example-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: terminate-sidecar-example-rolebinding
subjects:
- kind: ServiceAccount
name: terminate-sidecar-example-service-account
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: terminate-sidecar-example-role
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: terminate-sidecar-example-cronjob
labels:
app: terminate-sidecar-example
spec:
schedule: "30 2 * * *"
jobTemplate:
metadata:
labels:
app: terminate-sidecar-example
spec:
template:
metadata:
labels:
app: terminate-sidecar-example
annotations:
sidecar.istio.io/inject: "true"
spec:
serviceAccountName: terminate-sidecar-example-service-account
containers:
- name: ****
image: ****
command:
- "/bin/ash"
- "-c"
args:
- node index.js && kubectl exec -n ${POD_NAMESPACE} ${POD_NAME} -c istio-proxy -- bash -c "sleep 5 && /bin/kill -s TERM 1 &"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
So, the ultimate question to all of you is: Do you know of any better workaround, solution, controller, ... that would be less hacky / more suitable to terminate the istio-proxy container once the main container finished its work?
- command:
- /bin/sh
- -c
- |
until curl -fsI http://localhost:15021/healthz/ready; do echo \"Waiting for Sidecar...\"; sleep 3; done;
echo \"Sidecar available. Running the command...\";
<YOUR_COMMAND>;
x=$(echo $?); curl -fsI -X POST http://localhost:15020/quitquitquit && exit $x
Update: sleep loop can be omitted if holdApplicationUntilProxyStarts is set to true (globally or as an annotation) starting with istio 1.7
This was not a misconfiguration, this was a bug in upstream Kubernetes. As of September of 2019, this has been resolved by Istio by introducing a /quitquitquit endpoint to the Pilot agent.
Unfortunately, Kubernetes has not been so steadfast in solving this issue themselves. So it still does exist in some facets. However, the /quitquitquit endpoint in Istio should have resolved the problem for this specific use case.
I have found a work around by editing the configmap of istio-sidecar-injector as per the link Istio documentation
https://istio.io/docs/setup/additional-setup/sidecar-injection/
apiVersion: v1
kind: ConfigMap
metadata:
name: istio-sidecar-injector
data:
config: |-
policy: enabled
neverInjectSelector:
- matchExpressions:
- {key: job-name, operator: Exists}
But with this changes in our cronjob sidecar will not inject and istio policy will not apply on the cronjob job, and in our case we dont want any policy to be enforced by istio
Note :- job-name is by default label gets added in the pod creation
For those for whom curl is a luxury my wget version of the Dimitri's code:
command:
- /bin/sh
- -c
- |
until wget -q --spider http://127.0.0.1:15021/healthz/ready 2>/dev/null; do echo "Waiting for Istio sidecar..."; sleep 3; done;
echo \"Sidecar available. Running...\";
<COMMAND>;
x=$?; wget -q --post-data='' -S -O /dev/null http://127.0.0.1:15020/quitquitquit && exit $x

can i use a configmap created from an init container in the pod

I am trying to "pass" a value from the init container to a container. Since values in a configmap are shared across the namespace, I figured I can use it for this purpose. Here is my job.yaml (with faked-out info):
apiVersion: batch/v1
kind: Job
metadata:
name: installer-test
spec:
template:
spec:
containers:
- name: installer-test
image: installer-test:latest
env:
- name: clusterId
value: "some_cluster_id"
- name: in_artifactoryUrl
valueFrom:
configMapKeyRef:
name: test-config
key: artifactorySnapshotUrl
initContainers:
- name: artifactory-snapshot
image: busybox
command: ['kubectl', 'create configmap test-config --from-literal=artifactorySnapshotUrl=http://artifactory.com/some/url']
restartPolicy: Never
backoffLimit: 0
This does not seem to work (EDIT: although the statements following this edit note may still be correct, this is not working because kubectl is not a recognizable command in the busybox image), and I am assuming that the pod can only read values from a configmap created BEFORE the pod is created. Has anyone else come across the difficulty of passing values between containers, and what did you do to solve this?
Should I deploy the configmap in another pod and wait to deploy this one until the configmap exists?
(I know I can write files to a volume, but I'd rather not go that route unless it's absolutely necessary, since it essentially means our docker images must be coupled to an environment where some specific files exist)
You can create an EmptyDir volume, and mount this volume onto both containers. Unlike persistent volume, EmptyDir has no portability issue.
apiVersion: batch/v1
kind: Job
metadata:
name: installer-test
spec:
template:
spec:
containers:
- name: installer-test
image: installer-test:latest
env:
- name: clusterId
value: "some_cluster_id"
volumeMounts:
- name: tmp
mountPath: /tmp/artifact
initContainers:
- name: artifactory-snapshot
image: busybox
command: ['/bin/sh', '-c', 'cp x /tmp/artifact/x']
volumeMounts:
- name: tmp
mountPath: /tmp/artifact
restartPolicy: Never
volumes:
- name: tmp
emptyDir: {}
backoffLimit: 0
If for various reasons, you don't want to use share volume. And you want to create a configmap or a secret, here is a solution.
First you need to use a docker image which contains kubectl : gcr.io/cloud-builders/kubectl:latest for example. (docker image which contains kubectl manage by Google).
Then this (init)container needs enough rights to create resource on Kubernetes cluster. Ok by default, kubernetes inject a token of default service account named : "default" in container, but I prefer to make more explicit, then add this line :
...
initContainers:
- # Already true by default but if use it, prefer to make it explicit
automountServiceAccountToken: true
name: artifactory-snapshot
And add "edit" role to "default" service account:
kubectl create rolebinding default-edit-rb --clusterrole=edit --serviceaccount=default:myapp --namespace=default
Then complete example :
apiVersion: batch/v1
kind: Job
metadata:
name: installer-test
spec:
template:
spec:
initContainers:
- # Already true by default but if use it, prefer to make it explicit.
automountServiceAccountToken: true
name: artifactory-snapshot
# You need to use docker image which contains kubectl
image: gcr.io/cloud-builders/kubectl:latest
command:
- sh
- -c
# the "--dry-run -o yaml | kubectl apply -f -" is to make command idempotent
- kubectl create configmap test-config --from-literal=artifactorySnapshotUrl=http://artifactory.com/some/url --dry-run -o yaml | kubectl apply -f -
containers:
- name: installer-test
image: installer-test:latest
env:
- name: clusterId
value: "some_cluster_id"
- name: in_artifactoryUrl
valueFrom:
configMapKeyRef:
name: test-config
key: artifactorySnapshotUrl
First of all, kubectl is a binary. It was downloaded in your machine before you could use the command. But, In your POD, the kubectl binary doesn't exist. So, you can't use kubectl command from a busybox image.
Furthermore, kubectl uses some credential that is saved in your machine (probably in ~/.kube path). So, If you try to use kubectl from inside an image, this will fail because of missing credentials.
For your scenario, I will suggest the same as #ccshih, use volume sharing.
Here is the official doc about volume sharing between init-container and container.
The yaml that is used here is ,
apiVersion: v1
kind: Pod
metadata:
name: init-demo
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: workdir
mountPath: /usr/share/nginx/html
# These containers are run during pod initialization
initContainers:
- name: install
image: busybox
command:
- wget
- "-O"
- "/work-dir/index.html"
- http://kubernetes.io
volumeMounts:
- name: workdir
mountPath: "/work-dir"
dnsPolicy: Default
volumes:
- name: workdir
emptyDir: {}
Here init-containers saves a file in the volume and later the file was available in inside the container. Try the tutorial by yourself for better understanding.