Replacing variables in external file through playbook - deployment

I would like to use variables (vars.yaml) to replace the content in a config file (config.yaml) and automatise the process in the playbook (playbook.yaml). But I can't found a type of process that automaticly search for vars inside an external file through the playbook.yaml.
My files look like this :
vars.yaml
---
var1: content1
var2: content2
var3: content3
config.yaml:
apiVersion: 1
datasources:
- name: {{ var1 }}
type: {{ var2 }}
access: proxy
url: {{ var3 }}
playbook.yaml:
---
- name: Deploy
hosts: localhost
tasks:
- include_vars: ./vars.yaml
- name: Update config file
# Something to replace all vars in the config file using the content of vars.yaml
Can you help me and tell me if there is something possible instead of doing a regex for each variable ...?

The file with the unresolved variables is actually a template. Let's name it this way
shell> cat config.yaml.j2
apiVersion: 1
datasources:
- name: {{ var1 }}
type: {{ var2 }}
access: proxy
url: {{ var3 }}
Use the template module to update config.yaml, e.g.
- include_vars: vars.yaml
- name: Update config file
template:
src: config.yaml.j2
dest: config.yaml
gives
shell> cat config.yaml
apiVersion: 1
datasources:
- name: content1
type: content2
access: proxy
url: content3

Related

Is there a clean way to provide env var overrides using defaults, configmap, secrets, and command-line options?

We have some services that can be installed in multiple locations with differing configurations. We've been asked to support multi-level configuration options using environment variables set with defaults, configmaps, secrets, and command-line options passed in via helm install --set. The following works, but is very cumbersome as the number of parameters for some of the services are numerous and the Values dot-notation goes a few levels deeper.
env:
# Set default values
- name: MY_VAR
value: default-value
- name: OTHER_VAR
value: default-other-value
# Allow configmap to override
- name: MY_VAR
valueFrom:
configMapKeyRef:
name: env-configmap
key: MY_VAR
optional: true
- name: OTHER_VAR
valueFrom:
configMapKeyRef:
name: env-configmap
key: OTHER_VAR
optional: true
# Allow secrets to override
- name: MY_VAR
valueFrom:
secretsKeyRef:
name: env-secrets
key: MY_VAR
optional: true
- name: OTHER_VAR
valueFrom:
secretsKeyRef:
name: env-secrets
key: OTHER_VAR
optional: true
# Allow 'helm install --set' to override
{{- if .Values.env }}
{{- if .Values.env.my }}
{{- if .Values.env.my.var }}
- name: MY_VAR
value: {{ .Values.env.my.var }}
{{- end }}
{{- end }}
{{- if .Values.env.other }}
{{- if .Values.env.other.var }}
- name: OTHER_VAR
value: {{ .Values.env.other.var }}
{{- end }}
{{- end }}
{{- end }}
Using envFrom for the ConfigMap and Secrets would be nice, but tests and docs show this would not allow the command-line override, since env: and envFrom: doesn't mix in the way that's needed. As the v1.9 and v2.1 Kubernetes API states:
envFrom: List of sources to populate environment variables in the
container. The keys defined within a source must be a C_IDENTIFIER.
All invalid keys will be reported as an event when the container is
starting. When a key exists in multiple sources, the value associated
with the last source will take precedence. Values defined by an Env
with a duplicate key will take precedence. Cannot be updated.
Is there a better way to provide this default->configmap->secrets->cmd-line override precedence?
I found a solution that I mostly like. My issue was caused by giving too much weight to the "Values defined by an Env with a duplicate key will take precedence" comment in the docs, and thinking I needed to exclusively use Env. The defined precedence is exactly what I needed.
Here's the helm chart files for my current solution.
configmap/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: my-configmap
data:
{{- if .Values.env }}
{{- toYaml $.Values.env | nindent 2 }}
{{- end }}
{{- if .Values.applicationYaml }}
application.yml: |
{{- toYaml $.Values.applicationYaml | nindent 4 }}
{{- end }}
secrets/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: my-secrets
type: Opaque
data:
{{- range $key, $val := .Values.env }}
{{ $key }}: {{ $val | b64enc }}
{{- end }}
stringData:
{{- if .Values.applicationYaml }}
application.yml: |
{{- toYaml $.Values.applicationYaml | nindent 4 }}
{{- end }}
deployment.yaml
apiVersion: apps/v1
kind: Deployment
...
spec:
...
containers:
- name: my-deployment
{{- if .Values.env }}
env:
{{- range $key, $val := .Values.env }}
- name: {{ $key }}
value: {{ $val }}
{{- end }}
{{- end }}
envFrom:
- configMapRef:
name: my-configmap
- secretRef:
name: my-secrets
volumeMounts:
- name: configmap-application-config
mountPath: /application/config/configmap/
- name: secrets-application-config
mountPath: /application/config/secrets/
volumes:
- name: configmap-application-config
configMap:
name: my-configmap
optional: true
- name: secrets-application-config
secret:
secretName: my-secrets
optional: true
Since this is a Spring Boot app, I used volumeMounts to allow the application.yml default values to be overridden in the ConfigMap and Secrets. The order of precedence from lowest to highest is:
the application's application.yml (v1 in following examples)
the configmap's applicationYaml (v2)
the secret's applicationYaml (v3)
the configmap env (v4)
the secret env (v5)
the helm install/uninstall --set (v6)
To complete the example, here's test values yaml files and the command-line.
app/src/main/resources/application.yml
applicationYaml:
test:
v1: set-from-this-value
v2: overridden
v3: overridden
v4: overridden
v5: overridden
v6: overridden
configmap/values.yaml
applicationYaml:
test:
v2: set-from-this-value
v3: overridden
v4: overridden
v5: overridden
v6: overridden
env:
TEST_V4: set-from-this-value
TEST_V5: overridden
TEST_V6: overridden
secrets/values.yaml
applicationYaml:
test:
v3: set-from-this-value
v4: overridden
v5: overridden
v6: overridden
env:
TEST_V5: set-from-this-value
TEST_V6: overridden
command-line
helm install --set env.TEST_V6=set-from-this-value ...
Ideally, I'd like to be able to use dot-notation instead of TEST_V6 in the env and --set fields, but I'm not finding a way in helm to operate only on the leaves of yaml. In other words, I'd like something like range $key, $val, but where the key is equal to "test.v6". If that was possible, the key could be internally converted to an environment variable name with {{ $key | upper | replace "-" "_" | replace "." "_" }}.

How to create a k8s configmap by looping over nested helm values

The helm values look like in the following example. Here appdata can scale to any number but will contain the same set of keys.
data:
appdata:
config0:
url: 'https://example1.com'
port: '80'
config1:
url: 'https://example2.com'
port: '8673'
someotherconfig:
url: 'https://example3.com'
port: '9887'
...
...
This is what I have so far. This keeps updating the last config's data from someotherconfig key and also I want to have the config map name to contain the config name for each iteration, like {{ template "app" $ }}-config0, {{ template "app" $ }}-config1 and so on based on iteration.
{{- range $Mydata := $.Values.data.appdata }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "app" $ }}
labels:
data:
url: {{ $Mydata.url }}
port: {{ $Mydata.port }}
{{- end }}
You're almost there. You need to use the key/value notation to get the key name. Try the following.
{{- range $configuration, $Mydata := $.Values.data.appdata }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "app" $ }}-{{ $configuration }}
data:
url: {{ $Mydata.url }}
port: {{ $Mydata.port }}
{{- end }}

Helm3: Create .properties files recursively in Configmap

Below are files that I have:
users-values.yaml file :
users:
- foo
- baz
other-values.yaml file:
foo_engine=postgres
foo_url=some_url
foo_username=foofoo
baz_engine=postgres
baz_url=some_url
baz_username=bazbaz
config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: test-catalog
data:
{{- range $user := .Values.users }}
{{ . }}: |
engine.name={{ printf ".Values.%s_engine" ($user) }}
url={{ printf ".Values.%s_url" ($user) }}
username={{ printf".Values.%s_username" ($user) }}
{{- end }}
deployment-coordinator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: "{{ .Release.Name }}-coordinator"
labels:
app.kubernetes.io/name: "{{ .Release.Name }}-coordinator"
spec:
replicas: 1
...
template:
metadata:
labels:
app.kubernetes.io/name: "{{ .Release.Name }}-coordinator"
spec:
volumes:
- name: config
configMap:
name: test-catalog
...
volumeMounts:
- name: config
mountPath: "/etc/config"
Then, I do a helm install test mychart.
When I exec into the pod, and cd to /etc/config, I expect to see foo.properties and baz.properties files in there, and each file looks like:
foo.properties: |
engine.name=postgres
url=some_url
username=foofoo
baz.properties: |
engine.name=postgres
url=some_url
username=bazbaz
The answer from Pawel below solved the error I got previously
unexpected bad character U+0022 '"' in command
But, the files are still not created in the /etc/config directory.
So, I was wondering if it's even possible to create the .properties files using helm range as I mentioned in my config.yaml file above.
The reason I wanted to do it the below way is because I have more than 10 users to create .properties files on, not just foo and baz. Just thought it'll be easier if I can do a for loop on it if possible.
data:
{{- range $user := .Values.users }}
{{ . }}: |
engine.name={{ printf ".Values.%s_engine" ($user) }}
url={{ printf ".Values.%s_url" ($user) }}
username={{ printf".Values.%s_username" ($user) }}
{{- end }}

Gcloud : Getting output from a template

I have created a template to deploy a compute instance content of template is given below:
resources:
- name: {{ properties["name"] }}
type: compute.v1.instance
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ properties["project"] }}/zones/{{ properties["zone"] }}/machineTypes/{{ properties["machinetype"] }}
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: {{ properties["sourceimage"] }}
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ properties["project"] }}/global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
outputs:
- name: var1
value: 'testing'
- name: var2
value: 88
Deploying template using gcloud I am expecting the output in outputs field But, After Successful Deployment of template, I am getting outputs field blank as given below:
{
"outputs": [],
"resources": [
{
"finalProperties":.....
}
}
Please suggest if I am missing out something.
It looks weird/impossible to use {{ properties["name"] }} with properties["name"] as a variable.
I think you should create a parameter as it is shown here

helm-template get value of the map by key

In the helm-template I'm trying to retrieve a value of the map by key.
I've tried to use the index from the go-templates, as suggested here:
Access a map value using a variable key in a Go template
However it doesn't work for me (see later test). Any idea for the alternative solution?
Chart.yaml:
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Kubernetes
name: foochart
version: 0.1.0
values.yaml:
label:
- name: foo
value: foo1
- name: bar
value: bar2
templates/test.txt
label: {{ .Values.label }}
Works OK for helm template .:
---
# Source: foochart/templates/test.txt
label: [map[value:foo1 name:foo] map[name:bar value:bar2]]
However once trying to use the index:
templates/test.txt
label: {{ .Values.label }}
foolabel: {{ index .Values.label "foo" }}
It won't work - helm template .:
Error: render error in "foochart/templates/test.txt": template: foochart/templates/test.txt:2:13: executing "foochart/templates/test.txt" at <index .Values.label ...>: error calling index: cannot index slice/array with type string
label is an array, so the index function will only work with integers, this is a working example:
foolabel: {{ index .Values.label 0 }}
The 0 selects the first element of the array.
A better option is to avoid using an array and replace it with a map:
label:
foo:
name: foo
value: foo1
bar:
name: bar
value: bar2
And you dont even need the index function:
foolabel: {{ .Values.label.foo }}
values.yaml
coins:
ether:
host: 10.11.0.50
port: 123
btc:
host: 10.11.0.10
port: 321
template.yaml
{{- range $key, $val := .Values.coins }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ $key }}
- env:
- name: SSH_HOSTNAME
value: {{ $val.host | quote }}
- name: SSH_TUNNEL_HOST
value: {{ $val.port | quote }}
---
{{- end }}
run $ helm template ./helm
---
# Source: test/templates/ether.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: btc
- env:
- name: SSH_HOSTNAME
value: "10.11.0.10"
- name: SSH_TUNNEL_HOST
value: "321"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ether
- env:
- name: SSH_HOSTNAME
value: "10.11.0.50"
- name: SSH_TUNNEL_HOST
value: "123"
---