Helm escape string in nested json - kubernetes-helm

I am using helm to deploy apps on kubernetes. And I'm trying to figure out how to include an escaped multiline string in a nested json generated in a kubernetes secret.
The goal is to inline a certificate provided in helm chart values in nested json (for dev. purposes only)
Example values.yaml
certificate: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
What I tried:
configmap.yaml
apiVersion: v1
kind: Secret
stringData:
vaultfile.json: |-
[
{
"key": "myapp-config",
"value": "{\"certificate\": {{ .Values.certificate | quote | quote }} }"
}
]
Expected result:
apiVersion: v1
kind: Secret
stringData:
vaultfile.json: |-
[
{
"key": "myapp-config",
"value": "{\"certificate\": \"escaped-certificate\\ngoes\\nhere\"}"
}
]
What I got:
[
{
"key": "myapp-config",
"value": "{\"certificate\": "\"escaped-certificate\\ngoes\\nhere\"" }"
}
]
I am opened to any suggestions on how it could be done differently. But I have 2 constraints to respect:
This chart will be used as a subchart of a parent chart that won't be edited by users (making it difficult to provide certificate as a file)
Configuration format depends on an external tool and can't be modified.

Helm contains a toJson template function that can convert an arbitrary object to JSON. That will handle all of the string quoting and quote escaping that's necessary.
In the final structure:
The value of vaultfile.json is a JSON-encoded string.
Its value is a list.
The list contains a single object, with "key" and "value" fields.
The "value" is itself a JSON-encoded string.
That innermost string has an object, with key "certificate" and the value being the certificate.
You can build up this structure using toJson, along with dict and list to create mappings and lists. That would roughly look like:
apiVersion: v1
kind: Secret
metadata: { ... }
{{- $certObj := dict "certificate" .Values.certificate -}}
{{- $certStr := toJson $certObj -}}
{{- $kvp := "key" "myapp-config" "value" $certStr -}}
{{- $kvps := list $kvp -}}
data:
vaultfile.json: {{ $kvps | toJson | b64enc }}

Related

Using toJson or toRawJson without having it automatically adding quotation marks?

So I have a values.yaml file with an string variable representing a database connection string with no quotes looking like this (don't worry, not the real password):
ActionLogsConnectionString: Database=ActionLogs;Server=SQL_DEV;User Id=sa;Password=Y19|yx\dySh53&h
My goal is to print it inside a ConfigMap resource so that it can then be injected in my pod as a .json configuration file for a dotnet app. I also want to append the application name in the connection string:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "asp.fullname" . }}
labels:
{{- include "asp.labels" . | nindent 4 }}
data:
appsettings.k8s.json: |-
{
"ConnectionStrings": {
"ActionLogsConnectionString": "{{ .Values.ActionLogsConnectionString }};Application Name=Asp;"
}
}
This produce this result:
"ActionLogsConnectionString": "Database=ActionLogs;Server=SQL_DEV;User Id=sa;Password=Y19|yx\dySh53&h;Application Name=Asp;"
Look great! And at this point I don't have a quote problem.
Now problem, the slash isn't escaped for the json file format. Good thing, helm provide a toJson function. Unfortunately, it also transform the "&" to unicode value. I then found toRawJson and it gives the expected results.
My problem is that, when using either toJson or toRawJson, it adds extra quotes and break my result:
so this yalm file:
"ActionLogsConnectionString": "{{ .Values.ActionLogsConnectionString | toRawJson }};Application Name=Asp;"
results in this json file (note the extra quotes):
"ActionLogsConnectionString": ""Database=ActionLogs;Server=SQL_DEV;User Id=sa;Password=Y19|yx\\dySh53&h";Application Name=Asp;"
I see there's a function called | quote, but this only add some. No way to use toRawJson without adding any?
Using toJson or toRawJson is the wrong solution here, because the JSON representation of a string by definition includes the double quotes. "foo" is a JSON string, foo isn't valid JSON.
But you're only working with a scalar value, so there's not much point in marshaling it to JSON in the first place. I think the following gets you what you want:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "asp.fullname" . }}
labels:
{{- include "asp.labels" . | nindent 4 }}
data:
appsettings.k8s.json: |-
{
"ConnectionStrings": {
"ActionLogsConnectionString": {{ printf "%s;Application Name=asp" .Values.ActionLogsConnectionString | quote }}
}
}
Here, we're using the printf to produce the desired string (and then passing it to the quote function for proper quoting).
This produces:
---
# Source: example/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: my-example-fullname
labels:
# This is a test
data:
appsettings.k8s.json: |-
{
"ConnectionStrings": {
"ActionLogsConnectionString": "Database=ActionLogs;Server=SQL_DEV;User Id=sa;Password=Y19|yx\\dySh53&h;Application Name=asp"
}
}

Helm template converting to json

I would like to be able from Helm template file as below to insert template with toJson helm function as explained in the documentation :
value: {{ include "mytpl" . | lower | quote }}
https://helm.sh/docs/howto/charts_tips_and_tricks/#know-your-template-functions
My configuration :
_helper.tpl
{{- define "my_tpl" -}}
key1: value1
key2: value2
{{- end -}}
dep.yaml
template:
metadata:
annotations:
test: >-
{{ include "my_tpl" . | toJson }}
This should return
template:
metadata:
annotations:
test: >-
{"key1":"value1","key2":"value2"}
but it return
template:
metadata:
annotations:
test: >-
"key1:value1\nkey2:value2"
I'm using Helm v3.
Anyone have an idea please ?
A defined template always produces a string; the Helm-specific include function always returns a string.
In your example, you have a string that happens to be valid YAML. Helm has an undocumented fromYaml function that converts the string to object form, and then you can serialize that again with toJson.
{{ include "my_tpl" . | fromYaml | toJson }}
You may find it easier to have the template itself produce the correct JSON serialization. That could look something like
{{- define "my_tpl" -}}
{{- $dict := dict "key1" "value1" "key2" "value2" -}}
{{- toJson $dict -}}
{{- end -}}
{{ include "my_tpl" . }}
where the "key1", "value1", etc. can be any valid template expression (you do not need nested {{ ... }}).

accessing dynamic file in helm

I am new to kubernetes helm chart. There is a yaml file named configmap. This file containing all the configuration which are related to the application. Since this file containing a lot of data, I was trying to put some data to the new file and access using FILE object. So created two different file with the name:
data1.yaml and data2.yaml
data1.yaml file have only static data. On the other hand data2.yaml file contains dynamic data (some variables also like $.Values.appUrl).
I am able to read static file (data1.yaml) to the configmap.yaml file using FILE object. I am also able to read data2.yaml file but since this file containing some variable also, the variable value is not replacing by actual value. It printing the same variable in configmap file. So my question is,
Is there any way to access dynamic file (which contains variable type data) ?
Below is the example data shown.
configmap.yaml file is->
kind: ConfigMap
apiVersion: v1
metadata:
name: example-configmap
namespace: default
data1: {{ .File.Get "data1.yaml" | indent 2 }}
data2: {{ .File.Get "data2.yaml" | indent 2 }}
data1.yaml file is ->
data1:
ui.type:2
ui.color1:red
ui.color2:green
data2.yaml file is ->
system.data.name: "app-name"
system.data.url: {{ $.Values.appUrl }} # variable
system.data.type_one: "app-type-xxx"
system.data.value: "3"
system.interface.properties: |
Values.yaml file is ->
appUrl: "https://app-name.com"
Output:
kind: ConfigMap
apiVersion: v1
metadata:
name: example-configmap
namespace: default
data1:
ui.type:2
ui.color1:red
ui.color2:green
data2:
system.data.name: "app-name"
system.data.url: {{ $.Values.appUrl }} # here should be "https://app-name.com"
system.data.type_one: "app-type-xxx"
system.data.value: "3"
system.interface.properties: |
{{ (tpl (.Files.Glob "data2.yaml").AsConfig . ) | indent 2 }}
Using above syntax it's picking the actual value of variables. But it also printing the file name like below:
data2.yaml: |-
So I resolve the issue by using below syntax:
{{ (tpl (.Files.Get "data2.yaml") . ) | indent 2 }}
You can use the tpl function to process the templated file like this:
configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: example-configmap
data:
{{ (tpl (.Files.Get "data2.yaml") . ) | indent 2 }}
Note that configmap has data key by default. I think you should capture your data under this key, instead of creating data1 and data2 but not sure.
values.yaml:
appUrl: "https://app-name.com"
data2.yaml: (indentations corrected, otherwise it fails)
system.data.name: "app-name"
system.data.url: {{ .Values.appUrl }} # variable
system.data.type_one: "app-type-xxx"
system.data.value: "3"
system.interface.properties: |

How to parse a JSON secret to YAML in Vault?

I have a secret in Vault which is really a "complex" structure of JSON, meaning it's not just a key/value but there are several keys at different sublevels.
I need to somehow get this secret and convert it to the YAML representation of that JSON. If it was a simple structure (like several k/v at the same level), I could use something as simple as
{{- with secret "secret/foo" -}}
{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{- end }}
{{- end }}
however, as this is not the case, and the structure of the JSON is complex, trying to come up with a template is rather impossible.
However, I found that Vault uses Consul templates, and Consul has a parseYAML function, so my question is, how can I template this so that I get all the content of .Data.data and translate it into YAML?
I have tried several approaches similar to this one below:
{{- with secret "secret/foo" -}}
{{ .Data.data| parseYAML }}
{{- end }}
but I'm always getting the same error wrong type for value; expected string; got map[string]interface {}"
UPDATE
Sample yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-secret-foo: 'secret/foo'
vault.hashicorp.com/agent-inject-template-secret-foo: |
{{- with secret "secret/foo" -}}
{{ .Data.data| parseYAML }}
{{- end }}
vault.hashicorp.com/role: 'app'
spec:
containers:
- name: app
image: 'app:1.0.0'
serviceAccountName: app
where secret/foo is a long JSON with no clear structure. A random example (actual JSON is about 300 lines I think).
{
"a": {
"a": "a",
"b": "b",
"c": {
"a": "a",
"b": {
"c": "c"
},
"d": "a"
},
"e": {
"a": {
"b": {
"c": {
"a": "a",
"b": "b"
}
}
}
}
}
}
Ok, just figure this out.
{{- with secret "secret/foo" -}}
{{ .Data.data| toYAML }}
{{- end }}
this is correct, but instead of parseYAML the right function is toYAML.

helm upgrade fails with "function "X" not defined"

I'm trying to upgrade a helm chart,
I get the error function "pod" not defined which make sense because I really have no such function.
The "pod" is coming from a json file which I convert into a configmap and helm is reading this value as a function and not as a straight string which is part of the json file.
This is a snippet of my configmap:
# Generated from 'pods' from https://raw.githubusercontent.com/coreos/prometheus-operator/master/contrib/kube-prometheus/manifests/grafana-dashboardDefinitions.yaml
# Do not change in-place! In order to change this file first read following link:
# https://github.com/helm/charts/tree/master/stable/prometheus-operator/hack
{{- if and .Values.grafana.enabled .Values.grafana.defaultDashboardsEnabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ printf "%s-%s" (include "prometheus-operator.fullname" $) "services-health" | trunc 63 | trimSuffix "-" }}
labels:
{{- if $.Values.grafana.sidecar.dashboards.label }}
{{ $.Values.grafana.sidecar.dashboards.label }}: "1"
{{- end }}
app: {{ template "prometheus-operator.name" $ }}-grafana
{{ include "prometheus-operator.labels" $ | indent 4 }}
data:
services-health.json: |-
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"targets": [
{
"expr": "{__name__=~\"kube_pod_container_status_ready\", container=\"aggregation\",kubernetes_namespace=\"default\",chart=\"\"}",
"format": "time_series",
"instant": false,
"intervalFactor": 2,
"legendFormat": "{{pod}}",
"refId": "A"
}
}
{{- end }}
The error I get is coming from this line: "legendFormat": "{{pod}}",
And this is the error I get:
helm upgrade --dry-run prometheus-operator-chart
/home/ubuntu/infra-devops/helm/vector-chart/prometheus-operator-chart/
Error: UPGRADE FAILED: parse error in "prometheus-operator/templates/grafana/dashboards/services-health.yaml":
template:
prometheus-operator/templates/grafana/dashboards/services-health.yaml:1213:
function "pod" not defined
I tried to escape it but nothing worked.
Anyone get idea about how I can work around this issue?
Escaping gotpl placeholders is possible using backticks. For example, in your scenario, instead of using {{ pod }} you could write {{` {{ pod }} `}}.
Move your dashboard json to a separate file, let's say name it dashboard.json.
Then in your configmap file: instead of listing the json down inline, reference the dashboard.json file by typing the following:
data:
services-health.json: |-
{{ .Files.Get "dashboard.json" | indent 4 }}
That would solve the problem!
In the case of my experiments, I replaced
"legendFormat": "{{ pod }}",
with
"legendFormat": "{{ "{{ pod }}" }}",
and it was very happy to return the syntax I needed (Specifically for the grafana-operator GrafanaDashboard CRD).
Keeping json file out of config map and sourcing it within config map works, but make sure to keep the json file out of template directory while using with helm, or else it will try to search for {{ pod }} .