Below is how I am using kunbernetes on google.
I have one node application let's say Book-portal.
node app is using environment variables for configurations.
Step1: I created docker file and pushed
gcr.io/<project-id>/book-portal:v1
Step2: deployed with following commands
kubectl run book-portal --image=gcr.io/<project-id>/book-portal:v1 --port=5555 --env ENV_VAR_KEY1=value1 --env ENV_VAR_KEY2=value2 --env ENV_VAR_KEY3=value3
Step3:
kubectl expose deployment book-portal --type="LoadBalancer"
Step4: Get public ip with
kubectl get services book-portal
now assume I added new features and new configurations in next release.
So to roll out new version v2
Step1: I created docker file and pushed
gcr.io/<project-id>/book-portal:v2
Step2: Edit deployment
kubectl edit deployment book-portal
---------------yaml---------------
...
spec:
replicas: 1
selector:
matchLabels:
run: book-portal
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: book-portal
spec:
containers:
- env:
- name: ENV_VAR_KEY1
value: value1
- name: ENV_VAR_KEY2
value: value2
- name: ENV_VAR_KEY3
value: value3
image: gcr.io/<project-id>/book-portal:v1
imagePullPolicy: IfNotPresent
name: book-portal
...
----------------------------------
I am successfully able to change
image:gcr.io/<project-id>/book-portal:v1
to
image:gcr.io/<project-id>/book-portal:v2
But I can not add/change environment variables
- env:
- name: ENV_VAR_KEY1
value: value1
- name: ENV_VAR_KEY2
value: value2
- name: ENV_VAR_KEY3
value: value3
- name: ENV_VAR_KEY4
value: value4
Can anyone guide with what is best practices to pass configurations
in node app on kubernetes?
how should I handle environment variable
changes during rolling updates?
I think your best bet is to use configmaps in k8s and then change you pod template to get env variable values from the configmap see Consuming ConfigMap in pods
edit: I appologize I put the wrong link here. I have updated but for the TL;DR
you can do the following.
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
namespace: default
data:
special.how: very
special.type: charm
and then pod usage can look like this.
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
name: special-config
key: special.how
- name: SPECIAL_TYPE_KEY
valueFrom:
configMapKeyRef:
name: special-config
key: special.type
restartPolicy: Never
Related
I'm currently learning Kubernetes and recently learnt about using ConfigMaps for a Containers environment variables.
Let's say I have the following simple ConfigMap:
apiVersion: v1
data:
MYSQL_ROOT_PASSWORD: password
kind: ConfigMap
metadata:
creationTimestamp: null
name: mycm
I know that a container of some deployment can consume this environment variable via:
kubectl set env deployment mydb --from=configmap/mycm
or by specifying it manually in the manifest like so:
containers:
- env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
configMapKeyRef:
key: MYSQL_ROOT_PASSWORD
name: mycm
However, this isn't what I am after, since I'd to manually change the environment variables each time the ConfigMap changes.
I am aware that mounting a ConfigMap to the Pod's volume allows for the auto-updating of ConfigMap values. I'm currently trying to find a way to set a Container's environment variables to those stored in the mounted config map.
So far I have the following YAML manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: mydb
name: mydb
spec:
replicas: 1
selector:
matchLabels:
app: mydb
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: mydb
spec:
containers:
- image: mariadb
name: mariadb
resources: {}
args: ["export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD)"]
volumeMounts:
- name: config-volume
mountPath: /etc/config
env:
- name: MYSQL_ROOT_PASSWORD
value: temp
volumes:
- name: config-volume
configMap:
name: mycm
status: {}
I'm attempting to set the MYSQL_ROOT_PASSWORD to some temporary value, and then update it to mounted value as soon as the container starts via args: ["export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD)"]
As I somewhat expected, this didn't work, resulting in the following error:
/usr/local/bin/docker-entrypoint.sh: line 539: /export MYSQL_ROOT_PASSWORD=$(cat /etc/config/MYSQL_ROOT_PASSWORD): No such file or directory
I assume this is because the volume is mounted after the entrypoint. I tried adding a readiness probe to wait for the mount but this didn't work either:
readinessProbe:
exec:
command: ["sh", "-c", "test -f /etc/config/MYSQL_ROOT_PASSWORD"]
initialDelaySeconds: 5
periodSeconds: 5
Is there any easy way to achieve what I'm trying to do, or is it impossible?
So I managed to find a solution, with a lot of inspiration from this answer.
Essentially, what I did was create a sidecar container based on the alpine K8s image that mounts the configmap and constantly watches for any changes, since the K8s API automatically updates the mounted configmap when the configmap is changed. This required the following script, watch_passwd.sh, which makes use of inotifywait to watch for changes and then uses the K8s API to rollout the changes accordingly:
update_passwd() {
kubectl delete secret mysql-root-passwd > /dev/null 2>&1
kubectl create secret generic mysql-root-passwd --from-file=/etc/config/MYSQL_ROOT_PASSWORD
}
update_passwd
while true
do
inotifywait -e modify "/etc/config/MYSQL_ROOT_PASSWORD"
update_passwd
kubectl rollout restart deployment $1
done
The Dockerfile is then:
FROM docker.io/alpine/k8s:1.25.6
RUN apk update && apk add inotify-tools
COPY watch_passwd.sh .
After building the image (locally in this case) as mysidecar, I create the ServiceAccount, Role, and RoleBinding outlined here, adding rules for deployments so that they can be restarted by the sidecar.
After this, I piece it all together to create the following YAML Manifest (note that imagePullPolicy is set to Never, since I created the image locally):
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: mydb
name: mydb
spec:
replicas: 3
selector:
matchLabels:
app: mydb
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: mydb
spec:
serviceAccountName: secretmaker
containers:
- image: mysidecar
name: mysidecar
imagePullPolicy: Never
command:
- /bin/sh
- -c
- |
./watch_passwd.sh $(DEPLOYMENT_NAME)
env:
- name: DEPLOYMENT_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['app']
volumeMounts:
- name: config-volume
mountPath: /etc/config
- image: mariadb
name: mariadb
resources: {}
envFrom:
- secretRef:
name: mysql-root-passwd
volumes:
- name: config-volume
configMap:
name: mycm
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: secretmaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: mydb
name: secretmaker
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete", "list"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: mydb
name: secretmaker
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: secretmaker
subjects:
- kind: ServiceAccount
name: secretmaker
namespace: default
---
It all works as expected! Hopefully this is able to help someone out in the future. Also, if anybody comes across this and has a better solution please feel free to let me know :)
I am unable to deploy this file by using
kubectl apply -f command
Deployment YAML image
I have provided the YAML file required for your deployment. It is important that all the lines are indented correctly. Hyphens (-) indicate a list item. Therefore, it is not required to use them on every line.
apiVersion: apps/v1
kind: Deployment
metadata:
name: abc-deployment
namespace: abc
spec:
replicas: 3
selector:
matchLabels:
app: abc-deployment
template:
metadata:
labels:
app: abc-deployment
spec:
containers:
- name: abc-deployment
image: anyimage
ports:
- containerPort: 80
env:
- name: APP_VERSION
value: v1
- name: ENVIRONMENT
value: "123"
- name: DATA
valueFrom:
configMapKeyRef:
name: abc-configmap
key: data
imagePullPolicy: IfNotPresent
restartPolicy: Always
imagePullSecrets:
- name: abc-secret
As a side note, the way envFrom was used is incorrect. It must be within the container env section, and formatted as such in the example above (see the DATA env variable).
If you are using Visual Studio Code, there is an official Kubernetes extension from Microsoft that provides Intellisense (suggestions) and alerts you to errors.
Hope this helps.
We ran into an issue recently as to using environment variables inside container.
OS: windows 10 pro
k8s cluster: minikube
k8s version: 1.18.3
1. The way that doesn't work, though it's preferred way for us
Here is the deployment.yaml using 'envFrom':
apiVersion: apps/v1
kind: Deployment
metadata:
name: db
labels:
app.kubernetes.io/name: db
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: db
template:
metadata:
labels:
app.kubernetes.io/name: db
spec:
serviceAccountName: default
securityContext:
{}
containers:
- name: db
image: "postgres:9.4"
ports:
- name: http
containerPort: 5432
protocol: TCP
envFrom:
- configMapRef:
name: db-configmap
here is the db.properties:
POSTGRES_HOST_AUTH_METHOD=trust
step 1:
kubectl create configmap db-configmap ./db.properties
step 2:
kebuctl apply -f ./deployment.yaml
step 3:
kubectl get pod
Run the above command, get the following result:
db-8d7f7bcb9-7l788 0/1 CrashLoopBackOff 1 9s
That indicates the environment variables POSTGRES_HOST_AUTH_METHOD is not injected.
2. The way that works (we can't work with this approach)
Here is the deployment.yaml using 'env':
apiVersion: apps/v1
kind: Deployment
metadata:
name: db
labels:
app.kubernetes.io/name: db
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: db
template:
metadata:
labels:
app.kubernetes.io/name: db
spec:
serviceAccountName: default
securityContext:
{}
containers:
- name: db
image: "postgres:9.4"
ports:
- name: http
containerPort: 5432
protocol: TCP
env:
- name: POSTGRES_HOST_AUTH_METHOD
value: trust
step 1:
kubectl apply -f ./deployment.yaml
step 2:
kubectl get pod
Run the above command, get the following result:
db-fc58f998d-nxgnn 1/1 Running 0 32s
the above indicates the environment is injected so that the db starts.
What did I do wrong in the first case?
Thank you in advance for the help.
Update:
Provide the configmap:
kubectl describe configmap db-configmap
Name: db-configmap
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
db.properties:
----
POSTGRES_HOST_AUTH_METHOD=trust
For creating config-maps for usecase-1. please use the below command
kubectl create configmap db-configmap --from-env-file db.properties
Are you missing the key? (see "key:" (no quotes) below) And I think you need to provide the name of the env-variable...which people usually use the key-name, but you don't have to. I've repeated the same value ("POSTGRES_HOST_AUTH_METHOD") below as the environment variable NAME and the keyname of the config-map.
#start env .. where we add environment variables
env:
# Define the environment variable
- name: POSTGRES_HOST_AUTH_METHOD
#value: "UseHardCodedValueToDebugSometimes"
valueFrom:
configMapKeyRef:
# The ConfigMap containing the value you want to assign to environment variable (above "name:")
name: db-configmap
# Specify the key associated with the value
key: POSTGRES_HOST_AUTH_METHOD
My example (trying to use your values)....comes from this generic example:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-container-environment-variables-using-configmap-data
pods/pod-single-configmap-env-variable.yaml
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
# Define the environment variable
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
# The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY
name: special-config
# Specify the key associated with the value
key: special.how
restartPolicy: Never
PS
You can use "describe" to take a looksie at your config-map, after you (think:) ) you have set it up correctly.
kubectl describe configmap db-configmap --namespace=IfNotDefaultNameSpaceHere
See when you do it like you described.
deployment# exb db-7785cdd5d8-6cstw
root#db-7785cdd5d8-6cstw:/# env | grep -i TRUST
db.properties=POSTGRES_HOST_AUTH_METHOD=trust
the env set is not exactly POSTGRES_HOST_AUTH_METHOD its actually taking filename in env.
create configmap via
kubectl create cm db-configmap --from-env-file db.properties and it will actually put env POSTGRES_HOST_AUTH_METHOD in pod.
We are starting a project from scratch that will be managed on Google Cloud Services. I'd like to use Google Kubernetes Engine. Our application will have multiple environments (Dev, Staging, Production). Each environment is setup as a new Project on Google Cloud.
What is unclear to me is how to parameterize our service/manifest files. For instance our deploy file below, anything in {} I'd like to pull from a list of variables per environment. In a previous post someone mentioned using Helm, but I cannot find much documentation supporting the use of helm this way.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: webapp
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: {max-surge}
maxUnavailable: 0
selector:
matchLabels:
run: webapp
template:
metadata:
labels:
run: webapp
spec:
containers:
- name: webapp
image: {gcr-image-url}
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: app-secrets
key: DATABASE_URL
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: app-secrets
key: SECRET_KEY_BASE
What tools are available to manage my GKE environments? We'll use terraform for our infrastructure management, but again is there a larger wrapper I can use to set parameters per environment?
Helm would work for this, as would kustomize. In the case of helm, you'll have separate values.yaml files (e.g. dev-values.yaml) with e.g.:
max-surge: 2
gcr-image-url: project-23456/test
And then reference them in the yaml via:
{{ .Values.max-surge }}
The when installing you would use helm upgrade --install my-app . --values=dev-values.yaml
https://get-ytt.io could be a solution.
Particularly if you look at this github discussion you will notice that you can configure your environment and then pass in values in the form of flags or environment variables.
In case of your example, given the following config.yml:
## load("#ytt:data", "data")
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: webapp
annotations:
environment: ## data.values.env
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: ## data.values.max_surge
maxUnavailable: 0
selector:
matchLabels:
run: webapp
template:
metadata:
labels:
run: webapp
spec:
containers:
- name: webapp
image: ## data.values.gcr_image_url
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: app-secrets
key: DATABASE_URL
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: app-secrets
key: SECRET_KEY_BASE
and values.yml:
##data/values
---
env: staging
max-surge: 1
gcr-image-url: some/other-image:latest
assuming that everything is in the same directory, you can template config.yml like:
ytt -f .
or customize the values on the fly from env vars and command line arguments:
export CUSTOM_env=production
ytt -f . \
--data-value max_surge=10 \
--data-value gcr_image_url=some/image:1.0 \
--data-values-env CUSTOM
Im on kubernetes 1.3.5, we are using Deployments with rollingupdates to update the pods in our cluster. However, on rollingupdate, the newly added environment variable never gets added to the pod, is it by design ? what are the ways to get around that ?
Following is the sample deployment yaml files. Basically the deployment was deployed with first version then we updated the yaml with newly added env variable NEW_KEY and basically run through the rolling updated. But the new env does not show up in the PODS.
first version yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: APP_NAME-deployment
labels:
name: APP_NAME
environment: DEV
spec:
revisionHistoryLimit: 2
strategy:
type: RollingUpdate
replicas: 2
template:
metadata:
labels:
name: APP_NAME
environment: DEV
spec:
containers:
- name: APP_NAME
image: repo.app_name:latest
env:
- name: NODE_ENV
value: 'development'
- name: APP_KEY
value: '123'
updated yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: APP_NAME-deployment
labels:
name: APP_NAME
environment: DEV
spec:
revisionHistoryLimit: 2
strategy:
type: RollingUpdate
replicas: 2
template:
metadata:
labels:
name: APP_NAME
environment: DEV
spec:
containers:
- name: APP_NAME
image: repo.app_name:latest
env:
- name: NODE_ENV
value: 'development'
- name: APP_KEY
value: '123'
- name: NEW_KEY
value: 'new'
You can store the env variable in either a ConfigMap or secretKeyRef. For a ConfigMap you would do:
env:
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
name: node_env
key: node.dev
Or with a secretKeyRef:
env:
- name: SPECIAL_LEVEL_KEY
valueFrom:
secretKeyRef:
name: node_env
key: node.dev