how to deep merge arrays of helm named templates - kubernetes-helm

Question

I am trying to merge 2 named templates.
In simplified form it looks like this:
{{- define "common.t1" -}}
a1: "t1-a1"
a2: "t1-a2"
b1:
c1: t1-b1-c1
c2: t1-b1-c2
d1:
- name: container1
e1: t1-d1-e1
e2: t1-d1-e2
- name: container2
{{- end -}}
{{- define "this.t2" -}}
a1: "t2-a1"
a3: "t2-a3"
b1:
c1: t2-b1-c1
c3: t2-b1-c3
d1:
- name: container1
e1: t2-d1-e1
e3: t2-d1-e3
- name: container3
{{- end -}}
{{- $t1 := fromYaml (include "common.t1" .) -}}
{{- $t2 := fromYaml (include "this.t2" .) -}}
{{ toYaml (merge $t2 $t1) }}
The result is
a1: t2-a1
a2: t1-a2
a3: t2-a3
b1:
c1: t2-b1-c1
c2: t1-b1-c2
c3: t2-b1-c3
d1:
- e1: t2-d1-e1
e3: t2-d1-e3
name: container1
- name: container3
the d1 array objects are fully overriden by t2 template, but i want it to be merged with d1 array from the t1.
What i want to get is
a1: t2-a1
a2: t1-a2
a3: t2-a3
b1:
c1: t2-b1-c1
c2: t1-b1-c2
c3: t2-b1-c3
d1:
- e1: t2-d1-e1
e2: t1-d1-e2
e3: t2-d1-e3
name: container1
- name: container2
- name: container3
The purpose of merging is to have templates in common library chart that may be extended by templates in consumer chart and simplify development and maintainance of several dozens of helm charts.
For example put all common kube deployment yaml boilerplate into base template and then just write
{{ magicMerge "common.deployment" "mychart548.deployment" . }}
{{- define "mychart548.deployment" -}}
spec:
template:
spec:
containers:
- name: {{ include "common.names.name" . }}
envFrom:
- configMapRef:
name: {{ include "common.names.envConfigmapName" . }}
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "common.names.fullname" . }}-postgress
key: postgres-password
{{- end -}}

Answer 1

I don't think this is possible without writing your own merge function in Helm template code, which will be unpleasant.
At a low level, the Helm merge function actually comes from an extension library called Sprig. The Sprig merge function delegates to another Go library, Mergo. At a code level there happens to be a config options for merging Go slices (lists), but there are only two options: append one to the other, or try to merge each corresponding pair of elements. Sprig, and therefore Helm, uses the latter (default) option, and therefore what I'd expect to see from your example is that, inside d1, in the second list item, in the pair of maps, in the name: key, the different strings container1 and container2 can't be merged.
More abstractly: you're trying to merge two lists by deep-comparing the individual items for equality and taking their union; you don't want to do a deeper merge. That's a little more application-specific knowledge than a generic merge function can have.
If it will be exactly two lists like this, and you're trying to output the result and not process it further (or use the include ... | toYaml trick you have already), then one possible approach is to
Write out the first list as-is
Also convert the first list to a list of serialized strings
For each item in the second list,
Serialize it
Check to see if it's in the first list (serialized)
If not, emit it
{{/*- Convert a list to a list of JSON-serialized strings. */}}
{{- define "list-to-string-list" -}}
[
{{- range $index, $item := . -}}
{{- if ne $index 0 }},{{- end -}}
{{- toJson $item | toJson -}}
{{- end -}}
]
{{- end -}}
{{- define "merge-d1" -}}
{{- $first := index . 0 -}}
{{- $second := index . 1 -}}
{{/*- emit the first list -*/}}
{{- toYaml $first -}}
{{/*- index the first list -*/}}
{{- $firstStrings := include "list-to-string-list" $first | fromJson -}}
{{/*- for each item in the second list... -*/}}
{{- range $second -}}
{{/*- ...if it's not in the indexed list... -*/}}
{{- if not (has (toJson .) $firstStrings) -*/}}
{{/*- ...emit it too -*/}}
- {{ . | toYaml | nindent 2 | trim }}
{{ end -}}
{{- end -}}
{{- end -}}
d1:
{{ include "merge-d1" (list $first.d1 $second.d1) | indent 2 }}
If you're starting to write logic like this, consider whether writing it in a Kubernetes operator in a more mainstream language like Go might be more maintainable long-term.