I plan to upgrade my project to HELM.
I have many environment variables that I have defined in deployment.yaml.
Best practice is it best to define the environment variables in the values.yaml file or the templates / deployment.yaml drop?
Can you help if there is a sample application you use?
Disclaimer: My answers are based on Helm 3. So let's get to it:
#1: No, in your values.yaml you define the static/default values. It's not the best approach to let static values in your template files (like deployment.yaml). To override the values of values.yaml The best practice is to use --set KEY=VALUE file. In this case, is totally possible to get the environment variable.
#2: Can you give an example? Yes, sure.
For example, I want to install Elasticsearch on my cluster using helm so I use the command:
helm install elastic/elasticsearch --version 7.8.0
But I do not want to use the default values of the chart. So I went to https://hub.helm.sh/charts/elastic/elasticsearch and https://github.com/elastic/helm-charts/blob/7.8/elasticsearch/values.yaml, saw what's possible to change, then I create the command:
helm install elastic/elasticsearch --set minimumMasterNodes=1 --set protocol=https --version 7.8.0
But in my CD tool, the minimum master nodes are different values and since this is an environment variable I changed my command line to this:
helm install elastic/elasticsearch --set minimumMasterNodes=$MIN_MASTER_NODES --set protocol=https --version 7.8.0
So, as a result, the command above will run with no problem in your CD tool once the MIN_MASTER_NODES environment variable is provided correctly.
Your use of values.yaml to define environment vars is totally up to you. Is the value static? I'd have no problem leaving it in the deployment yaml. If it's a secret you should manage it either with k8s secrets or input it when you use helm install --set-value.. If the value is dynamic and is changed often or could be changed in the future that is the true use for values.yaml imo
There's four possible places you could set environment variables, and each has a use.
Is the value essentially fixed, any time you'd run the container in any environment? For example, consider setting a Python application to not buffer its log output or specifying the container-internal port number. Set these in the image's Dockerfile:
ENV PYTHONUNBUFFERED=1
ENV PORT=8000
Is the value more or less fixed, any time you'd run the container in Kubernetes? Or, can you reliably calculate the value? In these cases, you can set the value directly in your templates/deployment.yaml file, maybe with Helm templating.
env:
- name: COORDINATOR_HOST
value: {{ .Release.Name }}-coordinator # another Service in the same chart
- name: DATABASE_DRIVER
value: postgresql # this chart doesn't support MySQL
Does the value have a sensible default, but needs to be overridden sometimes? Put it in your chart's values.yaml.
# concurrency specifies the maximum number of concurrent tasks
# to launch.
concurrency: 4
This needs to be repeated in the templates/deployment.yaml as well
env:
- name: CONCURRENCY
value: {{ quote .Values.concurrency }}
Is the value only available at deployment time; or do you need to override one of these defaults? Use the helm install -f option to provide a per-environment value
databaseHost: myapp-pg.qa.example.com
or the similar helm install --set option. If it's reasonable to include a default value, also do so as in the previous example, but if not, you can use the required template function to give a reasonable error.
env:
- name: DATABASE_HOST
value: {{ .Values.databaseHost | required "a databaseHost must be provided" }}
You can use any or all of these options, depending on the specific values, even within the same chart.
The one pattern I don't particularly recommend is giving an open-ended list of environment variables (or other raw Kubernetes YAML) in the values file. As an operator, this is hard to consume, and it especially doesn't interact well with the helm install --set option. I tend to prefer listing out each configurable option in the Helm values file, and would modify the templates/*.yaml (maybe behind a deploy-time flag) if I needed more advanced customization.
Related
We deploy our microservices in multiple AWS regions. I therefore want to be able to do this in a Helm chart values.yaml file.
# Default region
aws_region: us-east-1
aws_ecrs:
us-east-1: 01234567890.dkr.ecr.us-east-1.amazonaws.com
eu-north-1: 01234567890.dkr.ecr.eu-nort-1.amazonaws.com
image:
name: microservice0
repository: {{ .Values.aws_ecrs.{{ .Values.aws_region }} }} # I know this is incorrect
So now when I install the chart, I just want to do
$ helm install microservice0 myChart/ --set aws_region=eu-north-1
and the appropriate repository will be assigned to .Values.image.repository. Can I do this? If so what is the correct syntax?
NOTE: The image repository is just one value that depends on the AWS region, we have many more other values that also depend on the AWS region.
Pass the repository name as an ordinary Helm value.
# templates/deployment.yaml
image: {{ .Values.repository }}/my-image:{{ .Values.tag }}
Create a separate file per region. This does not necessarily need to be in the same place as the Helm chart. Provide the regional values as ordinary top-level values. You'll have multiple files that provide the same values and that's fine.
# eu-north-1.yaml
repository: 01234567890.dkr.ecr.eu-nort-1.amazonaws.com
Then when you deploy the chart, use the helm install -f option to use the correct per-region values. These values will override anything in the chart's values.yaml file, but anything you don't specifically set here will use those default values from the chart.
helm install microservice0 myChart/ \
--set-string tag=20220201 \
-f eu-north-1.yaml
You can in principle use the Go template index function to do the lookup as you describe; the top-level structure in Variable value as yaml key in helm chart is similar to what you show in the question. This is more complex to implement in the templating code, though, and it means you have different setups for the values that must vary per region and those that can't.
Why write non-secret metadata as PLACEHOLDER in kubernetes yaml files? (from here):
namespace: PLACEHOLDER
Any reason to replace it with a later sed command? why not simply write it inside the yaml file?
Those values are normally replaced or overridden later, either by sed within a CI/CD pipeline like Jenkins, or (as indicated by the question you linked to) by Helm.
With helm, you can override values within the yamls either with a second yaml, or at the command line using --set switches.
So I could have
helm install nginx --values values.yaml --values values2.yaml
and the placeholder value in values.yaml would get overridden by the value in values2.yaml
Placeholders are often used to deliberately break an install is someone tries to install it without passing in the right values.yaml
For example:
helm install my-chart
Could break because of the placeholder value, but
helm install my-chart --values production.values.yaml
would install because the placeholder values are overridden and the chart can install correctly.
Based on my experience, when you see files like this, they might be used in some sort of pipelines to deploy them in multiple clusters. These files, from my prior experience, act like templates when you don't want to go all in with Helm charts for a single ConfigMap or Secret or a different isolated resource. This allows you to replace these placeholders with values corresponding with you cluster, region etc.
Hope this makes sense.
I want to deploy nextcloud with helm and a custom value.yaml file that fits my needs. Do i have to specify all values given from the original value.yaml or is it possible to only change the values needed, Eg if the only thing I want to change is the host adress my file can look like this:
nextlcoud:
host: 192.168.178.10
instead of copying this file and changing only a few values.
As the underlying issue was resolved by the answer of user #Kun Li, I wanted to add some examples when customizing Helm charts as well as some additional reference.
As asked in the question:
Do i have to specify all values given from the original value.yaml or is it possible to only change the values needed
In short you don't need to specify all of the values. You can change only some of them (like the host from your question).
The ways to change the values are following:
Individual parameters passed with --set (such as helm install --set foo=bar ./mychart)
A values file if passed into helm install or helm upgrade with the -f flag (helm install -f myvals.yaml ./mychart)
If this is a subchart, the values.yaml file of a parent chart
The values.yaml file in the chart
You can read more about it by following official Helm documentation:
Helm.sh: Docs: Chart template guide: Values files
A side note!
Above points are set in the order of priority. The first one (--set) will have the highest priority to override the values.
Example
A side note!
This examples assume that you are in the directory of a pulled Helm chart and you are using Helm v3
Using the nextcloud Helm chart used in the question you can set the nextcloud.host value by:
Pulling the Helm chart and editing the values.yaml
Creating additional new-values.yaml to pass it in (the values.yaml from Helm chart will be used regardless with lower priority):
$ helm install NAME . -f new-values.yaml
new-values.yaml
nextcloud:
host: 192.168.0.2
Setting the value with helm install NAME . --set nextcloud.host=192.168.0.2
You can check if the changes were done correctly by either:
$ helm template . - as pointed by user #David Maze
$ helm install NAME . --dry-run --debug
you misspelled the nextcloud to nextlcoud in your value file.
I want to use the pre-install hook of helm,
https://github.com/helm/helm/blob/master/docs/charts_hooks.md
in the docs its written that you need to use annotation which is clear but
what is not clear how to combine it ?
apiVersion: ...
kind: ....
metadata:
annotations:
"helm.sh/hook": "pre-install"
for my case I need to execute a bash script which create some env variable , where should I put this pre-hook script inside my chart that helm can use
before installation ?
I guess I need to create inside the templates folder a file which called: pre-install.yaml is it true? if yes where should I put the commands which create the env variables during the installation of the chart?
UPDATE
The command which I need to execute in the pre-install is like:
export DB=prod_sales
export DOMAIN=www.test.com
export THENANT=VBAS
A Helm hook launches some other Kubernetes object, most often a Job, which will launch a separate Pod. Environment variable settings will only effect the current process and children it launches later, in the same Docker container, in the same Pod. That is: you can't use mechanisms like Helm pre-install hooks or Kubernetes initContainers to set environment variables like this.
If you just want to set environment variables to fixed strings like you show in the question, you can directly set that in a Pod spec. If the variables are, well, variable, but you don't want to hard-code them in your Pod spec, you can also put them in a ConfigMap and then set environment variables from that ConfigMap. You can also use Helm templating to inject settings from install-time configuration.
env:
- name: A_FIXED_VARIABLE
value: A fixed value
- name: SET_FROM_A_CONFIG_MAP
valueFrom:
configMapKeyRef:
name: the-config-map-name
key: someKey
- name: SET_FROM_HELM
value: {{ .Values.environmentValue | quote }}
With the specific values you're showing, the Helm values path is probably easiest. You can run a command like
helm install --set db=prod_sales --set domain=www.test.com ...
and then have access to .Values.db, .Values.domain, etc. in your templates.
If the value is really truly dynamic and you can't set it any other way, you can use a Docker entrypoint script to set it at container startup time. In this answer I describe the generic-Docker equivalents to this, including the entrypoint script setup.
You can take as an example the built-in helm-chart from arc* project, here is the source code.
*Arc - kind of bootstraper for Laravel projects, that can Dockerize/Kubernetize existing apps written in this PHP framework.
You can place ENV in POD.yaml under the template folder. That will be the easiest option.
First, a little context: I have 4 Kubernetes clusters, one for each environment (develop, staging, testing, prod). Each cluster has its own values.yaml file with env-specific configuration of all helm charts that we've written.
So, when our CD pipeline deploys mychart to the develop cluster, it essentially does the following:
helm install -f base-values.yaml -f develop-values.yaml ./mychart.tgz
Now, let's presume mychart has a requirements.yaml file which specifies the mongodb chart as a subchart dependency. The mongodb chart references, for example, .Values.mongodbRootPassword. When included as a subchart by mychart, I can set .Values.mongodb.mongodbRootPassword in mychart's default values.yaml to change this value.
My problem is that given my current CD pipeline, if I set .Values.mongodb.mongodbRootPassword in develop-values.yaml, it will be taken for all mongodb instances that are deployed to the develop cluster - not just mychart's.
So, my questions:
using per-environment values.yaml files, how would I go about setting mychart's mongodb's root password in one of the cluster-specific values.yaml files?
is this even possible? Or is my current CD approach (per-environment values files) an anti-pattern?
if this is not possible or an anti-pattern, how would you go about setting the values of helm individual charts and subcharts on a per-environment basis?
using per-environment values.yaml files, how would I go about setting mychart's mongodb's root password in one of the cluster-specific values.yaml files?
You can override YAML files with --set option from install
Example:
helm install -f base-values.yaml -f develop-values.yaml --set someSection.someValue=1234 ./mychart.tgz
Then, you can set CI command call with environment settings and have just one YAML.
is this even possible? Or is my current CD approach (per-environment values files) an anti-pattern?
Nope, that's a feature :-)
if this is not possible or an anti-pattern, how would you go about setting the values of helm individual charts and subcharts on a per-environment basis?
It would be nice also. Perhaps, if your infrastructure grow fast and you have a lot of environments, apps or else, this could be better to manage.
I created a named template (within _helpers.tpl) that combined a set of default values defined in values.yaml with a set of env specific values defined in their own file. This allows me to set 20+ default values and then override them with the env file WITHOUT setting them as part of the cli command. I wanted to avoid using the cli to override values because that gets complicated and doesn't provide tracking (ie my env specific files are in git).