I would like to use variables (vars.yaml) to replace the content in a config file (config.yaml) and automatise the process in the playbook (playbook.yaml). But I can't found a type of process that automaticly search for vars inside an external file through the playbook.yaml.
My files look like this :
vars.yaml
---
var1: content1
var2: content2
var3: content3
config.yaml:
apiVersion: 1
datasources:
- name: {{ var1 }}
type: {{ var2 }}
access: proxy
url: {{ var3 }}
playbook.yaml:
---
- name: Deploy
hosts: localhost
tasks:
- include_vars: ./vars.yaml
- name: Update config file
# Something to replace all vars in the config file using the content of vars.yaml
Can you help me and tell me if there is something possible instead of doing a regex for each variable ...?
The file with the unresolved variables is actually a template. Let's name it this way
shell> cat config.yaml.j2
apiVersion: 1
datasources:
- name: {{ var1 }}
type: {{ var2 }}
access: proxy
url: {{ var3 }}
Use the template module to update config.yaml, e.g.
- include_vars: vars.yaml
- name: Update config file
template:
src: config.yaml.j2
dest: config.yaml
gives
shell> cat config.yaml
apiVersion: 1
datasources:
- name: content1
type: content2
access: proxy
url: content3
Related
We have some services that can be installed in multiple locations with differing configurations. We've been asked to support multi-level configuration options using environment variables set with defaults, configmaps, secrets, and command-line options passed in via helm install --set. The following works, but is very cumbersome as the number of parameters for some of the services are numerous and the Values dot-notation goes a few levels deeper.
env:
# Set default values
- name: MY_VAR
value: default-value
- name: OTHER_VAR
value: default-other-value
# Allow configmap to override
- name: MY_VAR
valueFrom:
configMapKeyRef:
name: env-configmap
key: MY_VAR
optional: true
- name: OTHER_VAR
valueFrom:
configMapKeyRef:
name: env-configmap
key: OTHER_VAR
optional: true
# Allow secrets to override
- name: MY_VAR
valueFrom:
secretsKeyRef:
name: env-secrets
key: MY_VAR
optional: true
- name: OTHER_VAR
valueFrom:
secretsKeyRef:
name: env-secrets
key: OTHER_VAR
optional: true
# Allow 'helm install --set' to override
{{- if .Values.env }}
{{- if .Values.env.my }}
{{- if .Values.env.my.var }}
- name: MY_VAR
value: {{ .Values.env.my.var }}
{{- end }}
{{- end }}
{{- if .Values.env.other }}
{{- if .Values.env.other.var }}
- name: OTHER_VAR
value: {{ .Values.env.other.var }}
{{- end }}
{{- end }}
{{- end }}
Using envFrom for the ConfigMap and Secrets would be nice, but tests and docs show this would not allow the command-line override, since env: and envFrom: doesn't mix in the way that's needed. As the v1.9 and v2.1 Kubernetes API states:
envFrom: List of sources to populate environment variables in the
container. The keys defined within a source must be a C_IDENTIFIER.
All invalid keys will be reported as an event when the container is
starting. When a key exists in multiple sources, the value associated
with the last source will take precedence. Values defined by an Env
with a duplicate key will take precedence. Cannot be updated.
Is there a better way to provide this default->configmap->secrets->cmd-line override precedence?
I found a solution that I mostly like. My issue was caused by giving too much weight to the "Values defined by an Env with a duplicate key will take precedence" comment in the docs, and thinking I needed to exclusively use Env. The defined precedence is exactly what I needed.
Here's the helm chart files for my current solution.
configmap/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: my-configmap
data:
{{- if .Values.env }}
{{- toYaml $.Values.env | nindent 2 }}
{{- end }}
{{- if .Values.applicationYaml }}
application.yml: |
{{- toYaml $.Values.applicationYaml | nindent 4 }}
{{- end }}
secrets/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: my-secrets
type: Opaque
data:
{{- range $key, $val := .Values.env }}
{{ $key }}: {{ $val | b64enc }}
{{- end }}
stringData:
{{- if .Values.applicationYaml }}
application.yml: |
{{- toYaml $.Values.applicationYaml | nindent 4 }}
{{- end }}
deployment.yaml
apiVersion: apps/v1
kind: Deployment
...
spec:
...
containers:
- name: my-deployment
{{- if .Values.env }}
env:
{{- range $key, $val := .Values.env }}
- name: {{ $key }}
value: {{ $val }}
{{- end }}
{{- end }}
envFrom:
- configMapRef:
name: my-configmap
- secretRef:
name: my-secrets
volumeMounts:
- name: configmap-application-config
mountPath: /application/config/configmap/
- name: secrets-application-config
mountPath: /application/config/secrets/
volumes:
- name: configmap-application-config
configMap:
name: my-configmap
optional: true
- name: secrets-application-config
secret:
secretName: my-secrets
optional: true
Since this is a Spring Boot app, I used volumeMounts to allow the application.yml default values to be overridden in the ConfigMap and Secrets. The order of precedence from lowest to highest is:
the application's application.yml (v1 in following examples)
the configmap's applicationYaml (v2)
the secret's applicationYaml (v3)
the configmap env (v4)
the secret env (v5)
the helm install/uninstall --set (v6)
To complete the example, here's test values yaml files and the command-line.
app/src/main/resources/application.yml
applicationYaml:
test:
v1: set-from-this-value
v2: overridden
v3: overridden
v4: overridden
v5: overridden
v6: overridden
configmap/values.yaml
applicationYaml:
test:
v2: set-from-this-value
v3: overridden
v4: overridden
v5: overridden
v6: overridden
env:
TEST_V4: set-from-this-value
TEST_V5: overridden
TEST_V6: overridden
secrets/values.yaml
applicationYaml:
test:
v3: set-from-this-value
v4: overridden
v5: overridden
v6: overridden
env:
TEST_V5: set-from-this-value
TEST_V6: overridden
command-line
helm install --set env.TEST_V6=set-from-this-value ...
Ideally, I'd like to be able to use dot-notation instead of TEST_V6 in the env and --set fields, but I'm not finding a way in helm to operate only on the leaves of yaml. In other words, I'd like something like range $key, $val, but where the key is equal to "test.v6". If that was possible, the key could be internally converted to an environment variable name with {{ $key | upper | replace "-" "_" | replace "." "_" }}.
The helm values look like in the following example. Here appdata can scale to any number but will contain the same set of keys.
data:
appdata:
config0:
url: 'https://example1.com'
port: '80'
config1:
url: 'https://example2.com'
port: '8673'
someotherconfig:
url: 'https://example3.com'
port: '9887'
...
...
This is what I have so far. This keeps updating the last config's data from someotherconfig key and also I want to have the config map name to contain the config name for each iteration, like {{ template "app" $ }}-config0, {{ template "app" $ }}-config1 and so on based on iteration.
{{- range $Mydata := $.Values.data.appdata }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "app" $ }}
labels:
data:
url: {{ $Mydata.url }}
port: {{ $Mydata.port }}
{{- end }}
You're almost there. You need to use the key/value notation to get the key name. Try the following.
{{- range $configuration, $Mydata := $.Values.data.appdata }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "app" $ }}-{{ $configuration }}
data:
url: {{ $Mydata.url }}
port: {{ $Mydata.port }}
{{- end }}
Below are files that I have:
users-values.yaml file :
users:
- foo
- baz
other-values.yaml file:
foo_engine=postgres
foo_url=some_url
foo_username=foofoo
baz_engine=postgres
baz_url=some_url
baz_username=bazbaz
config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: test-catalog
data:
{{- range $user := .Values.users }}
{{ . }}: |
engine.name={{ printf ".Values.%s_engine" ($user) }}
url={{ printf ".Values.%s_url" ($user) }}
username={{ printf".Values.%s_username" ($user) }}
{{- end }}
deployment-coordinator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: "{{ .Release.Name }}-coordinator"
labels:
app.kubernetes.io/name: "{{ .Release.Name }}-coordinator"
spec:
replicas: 1
...
template:
metadata:
labels:
app.kubernetes.io/name: "{{ .Release.Name }}-coordinator"
spec:
volumes:
- name: config
configMap:
name: test-catalog
...
volumeMounts:
- name: config
mountPath: "/etc/config"
Then, I do a helm install test mychart.
When I exec into the pod, and cd to /etc/config, I expect to see foo.properties and baz.properties files in there, and each file looks like:
foo.properties: |
engine.name=postgres
url=some_url
username=foofoo
baz.properties: |
engine.name=postgres
url=some_url
username=bazbaz
The answer from Pawel below solved the error I got previously
unexpected bad character U+0022 '"' in command
But, the files are still not created in the /etc/config directory.
So, I was wondering if it's even possible to create the .properties files using helm range as I mentioned in my config.yaml file above.
The reason I wanted to do it the below way is because I have more than 10 users to create .properties files on, not just foo and baz. Just thought it'll be easier if I can do a for loop on it if possible.
data:
{{- range $user := .Values.users }}
{{ . }}: |
engine.name={{ printf ".Values.%s_engine" ($user) }}
url={{ printf ".Values.%s_url" ($user) }}
username={{ printf".Values.%s_username" ($user) }}
{{- end }}
I have created a template to deploy a compute instance content of template is given below:
resources:
- name: {{ properties["name"] }}
type: compute.v1.instance
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ properties["project"] }}/zones/{{ properties["zone"] }}/machineTypes/{{ properties["machinetype"] }}
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: {{ properties["sourceimage"] }}
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ properties["project"] }}/global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
outputs:
- name: var1
value: 'testing'
- name: var2
value: 88
Deploying template using gcloud I am expecting the output in outputs field But, After Successful Deployment of template, I am getting outputs field blank as given below:
{
"outputs": [],
"resources": [
{
"finalProperties":.....
}
}
Please suggest if I am missing out something.
It looks weird/impossible to use {{ properties["name"] }} with properties["name"] as a variable.
I think you should create a parameter as it is shown here
In the helm-template I'm trying to retrieve a value of the map by key.
I've tried to use the index from the go-templates, as suggested here:
Access a map value using a variable key in a Go template
However it doesn't work for me (see later test). Any idea for the alternative solution?
Chart.yaml:
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Kubernetes
name: foochart
version: 0.1.0
values.yaml:
label:
- name: foo
value: foo1
- name: bar
value: bar2
templates/test.txt
label: {{ .Values.label }}
Works OK for helm template .:
---
# Source: foochart/templates/test.txt
label: [map[value:foo1 name:foo] map[name:bar value:bar2]]
However once trying to use the index:
templates/test.txt
label: {{ .Values.label }}
foolabel: {{ index .Values.label "foo" }}
It won't work - helm template .:
Error: render error in "foochart/templates/test.txt": template: foochart/templates/test.txt:2:13: executing "foochart/templates/test.txt" at <index .Values.label ...>: error calling index: cannot index slice/array with type string
label is an array, so the index function will only work with integers, this is a working example:
foolabel: {{ index .Values.label 0 }}
The 0 selects the first element of the array.
A better option is to avoid using an array and replace it with a map:
label:
foo:
name: foo
value: foo1
bar:
name: bar
value: bar2
And you dont even need the index function:
foolabel: {{ .Values.label.foo }}
values.yaml
coins:
ether:
host: 10.11.0.50
port: 123
btc:
host: 10.11.0.10
port: 321
template.yaml
{{- range $key, $val := .Values.coins }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ $key }}
- env:
- name: SSH_HOSTNAME
value: {{ $val.host | quote }}
- name: SSH_TUNNEL_HOST
value: {{ $val.port | quote }}
---
{{- end }}
run $ helm template ./helm
---
# Source: test/templates/ether.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: btc
- env:
- name: SSH_HOSTNAME
value: "10.11.0.10"
- name: SSH_TUNNEL_HOST
value: "321"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ether
- env:
- name: SSH_HOSTNAME
value: "10.11.0.50"
- name: SSH_TUNNEL_HOST
value: "123"
---