I use Vault to retrieve some secrets that I put inside a configuration file. All works fine until this configuration gets bigger and I want it to be saved in sub configs in a folder. The issue is that those files can't get imported using go templating used to fill passwords..
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: my-app
spec:
...
template:
metadata:
annotations:
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/secret-volume-path-my-config: "/my-path/etc"
vault.hashicorp.com/agent-inject-file-my-config: "my-app.conf"
vault.hashicorp.com/agent-inject-secret-my-config: secret/data/my-app/config
vault.hashicorp.com/agent-inject-template-my-config: |
{{- $file := .Files }}
{{ .Files.Get "configurations/init.conf" }}
{{- with secret "secret/data/my-app/config" -}}
...
{{- end }}
The file configurations/init.conf for example doesn't seem to be visible by the vault injector and so gets simply replaced by <no value>. Is there a way to make those files in configurations/* visible to vault injector maybe by mounting them somewhere?
Related
My ingress.yaml looks like so:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ .Values.name }}-a
namespace: {{ .Release.Namespace }}
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "{{ .Values.canary.weight }}"
nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
spec:
tls:
- hosts:
- {{ .Values.urlFormat | quote }}
secretName: {{ .Values.name }}-cert // <-------------- This Line
ingressClassName: nginx-customer-wildcard
rules:
- host: {{ .Values.urlFormat | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ .Values.name }}-a
port:
number: {{ .Values.backendPort }}
Assume Values.name = customer-tls then secretName will become customer-tls-cert.
On removing secretName: {{ .Values.name }}-cert the the nginx ingress start to use default certificate which is fine as I expect it to be but this also results in the customer-tls-cert certificate still hanging around in the cluster though unused. Is there a way that when I delete the cert from helm config it also removed the certificate from the cluster.
Otherwise, some mechanism that will will figure out the certificates that are no longer in use and will get deleted automatically ?
My nginx version is nginx/1.19.9
K8s versions:
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.24.6
I experimented with --enable-dynamic-certificates a lil bit but that's not supported anymore on the versions that I am using. I am not even sure if that would have solved my problem.
For now I have just manually deleted the certificate from the cluster using kubectl delete secret customer-tls-cert -n edge where edge is the namespace where cert resides.
Edit: This is how my certificate.yaml looks like,
{{- if eq .Values.certificate.enabled true }}
apiVersion: v1
kind: Secret
metadata:
name: {{ .Values.name }}-cert
namespace: edge
annotations:
vault.security.banzaicloud.io/vault-addr: {{ .Values.vault.vaultAddress | quote }}
vault.security.banzaicloud.io/vault-role: {{ .Values.vault.vaultRole | quote }}
vault.security.banzaicloud.io/vault-path: {{ .Values.vault.vaultPath | quote }}
vault.security.banzaicloud.io/vault-namespace : {{ .Values.vault.vaultNamespace | quote }}
type: kubernetes.io/tls
data:
tls.crt: {{ .Values.certificate.cert }}
tls.key: {{ .Values.certificate.key }}
{{- end }}
Kubernetes in general will not delete things simply because they are not referenced. There is a notion of ownership which doesn't apply here (if you delete a Job, the cluster also deletes the corresponding Pod). If you have a Secret or a ConfigMap that's referenced by name, the object will still remain even if you delete the last reference to it.
In Helm, if a chart contains some object, and then you upgrade the chart to a newer version or values that don't include that object, then Helm will delete the object. This would require that the Secret actually be part of the chart, like
{{/* templates/cert-secret.yaml */}}
{{- if .Values.createSecret -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ .Values.name }}-cert
...
{{ end -}}
If your chart already included this, and you ran helm upgrade with values that set createSecret to false, then Helm would delete the Secret.
If you're not in this situation, though – your chart references the Secret by name, but you expect something else to create it – then you'll also need to manually destroy it, maybe with kubectl delete.
I don't know Golang at all but need to implement Go template syntax in my kubernetes config (where hishicorp vault is configured). What I'm trying to do is to modify file in order to change its format. So source looks like this:
data: map[key1:value1]
metadata: map[created_time:2021-10-06T21:02:18.41643371Z deletion_time: destroyed:false version:1]
The Kubernetes config part with go template is used in order to format file is here:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test
spec:
replicas: ${REPLICAS}
selector:
matchLabels:
component: test
template:
metadata:
labels:
component: test
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-status: 'update'
vault.hashicorp.com/role: 'test'
vault.hashicorp.com/agent-inject-secret-config: 'secret/data/test/config'
vault.hashicorp.com/agent-inject-template-config: |
{{- with secret "secret/data/test/config" -}}
{{ range $k, $v := .Data.data }}
export {{ $k }}={{ $v | quote }}
{{ end }}
{{- end}}
spec:
serviceAccountName: test
containers:
- name: test
image: ${IMAGE}
ports:
- containerPort: 3000
But the error I'm getting is:
runtime error encountered: error="template server: (dynamic): parse: template: :2: unexpected "," in range"
EDIT:
To deploy vault on k8s I'm using vault helm chart
For what I can see you have env variables in your yaml file (${REPLICAS}, ${IMAGE}), which makes me think that you are using something like cat file.yml | envsubst | kubectl apply --wait=true -f - in order to replace those env vars for the real values.
The issue with this is that $k and $v are also being replaced for '' (since you do not have that env var in your system).
One ugly but effective solution is to export v="$v" and export k="$k" which whill generate your yaml file correctly.
I am writing helm charts and it creates one deployment and one statefulset component.
Now I want to generate uuid and send the value to both k8s components.
I a using uuid function to generate the uuid. But need help how I can send this value to both components.
Here is my chart folder structure --
projectdir
chart1
templates
statefulset.yaml
chart2
templates
deployment.yaml
helperchart
templates
_helpers.tpl
I have to write the logic to generate the uuid in _helpers.tpl.
Edit: It seems defining it in the _helpers.tpl does not work - thank you for pointing it out.
I have lookup it up a bit, and it seems currently the only way to achieve that is to put both of the manifests, separated by --- to the same file under the templates/. See the following example, where the UUID is defined in the first line and then used in the both Deployment and the StatefulSet:
{{- $mySharedUuid := uuidv4 -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "uuid-test.fullname" . }}-1
labels:
{{- include "uuid-test.labels" . | nindent 4 }}
annotations:
my-uuid: {{ $mySharedUuid }}
spec:
...
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "uuid-test.fullname" . }}-2
labels:
{{- include "uuid-test.labels" . | nindent 4 }}
annotations:
my-uuid: {{ $mySharedUuid }}
spec:
...
After templating, the output is:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: uuid-test-app-1
labels:
helm.sh/chart: uuid-test-0.1.0
app.kubernetes.io/name: uuid-test
app.kubernetes.io/instance: uuid-test-app
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
annotations:
my-uuid: fe0346f5-a963-4ca1-ada0-af17405f3155
spec:
...
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: uuid-test-app-2
labels:
helm.sh/chart: uuid-test-0.1.0
app.kubernetes.io/name: uuid-test
app.kubernetes.io/instance: uuid-test-app
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
annotations:
my-uuid: fe0346f5-a963-4ca1-ada0-af17405f3155
spec:
...
See the same issue: https://github.com/helm/helm/issues/6456
Note that this approach will still cause the UUID to be regenerated when you do a helm upgrade. To circumvent that, you would need to use another workaround along with this one.
You should explicitly pass the value in as a Helm value; don't try to generate it in the chart.
The other answers to this question highlight a couple of the issues you'll run into. #UtkuÖzdemir notes that every time you call the Helm uuidv4 function it will create a new random UUID, so you can only call that function once in the chart ever; and #srr further notes that there's no way to persist a generated value like this, so if you helm upgrade the chart the UUID value will be regenerated, which will cause all of the involved Kubernetes objects to be redeployed.
The Bitnami RabbitMQ chart has an interesting middle road here. One of its configuration options is an "Erlang cookie", also a random string that needs to be consistent across all replicas and upgrades. On an initial install it generates a random value if one isn't provided, and tells you how to retrieve it from a Secret; but if .Release.IsUpgrade then you must provide the value directly, and the error message explains how to get it from your existing deployment.
You may be able to get around the "only call uuidv4 once ever" problem by putting the value into a ConfigMap or Secret, and then referencing it from elsewhere. This works only if the only place you use the UUID value is in an environment variable, or something else that can have a value injected from a secret; it won't help if you need it in an annotation or label.
apiVersion: v1
kind: Secret
metadata:
name: {{ template "chart.name" . }}
data:
the-uuid: {{ .Values.theUuid | default uuidv4 | b64enc }}
{{-/* this is the only place uuidv4 ^^^^^^ is called at all */}}
env:
- name: THE_UUID
valueFrom:
secretKeyRef:
name: {{ template "chart.name" . }}
key: the-uuid
As suggested in helm issue tracker https://github.com/helm/helm/issues/6456, we have to put both components in same file and looks like thats the only solution right now.
Its a surprise, Helm not supporting cache the value to share across charts/components. I wish Helm support this feature in future.
I want to append the hash of my Secret or ConfigMap contents to the name of the resource in order to trigger a rolling update and keep the old version of that resource around in case there is a mistake in the new configuration.
This can almost be achieved using "helm.sh/resource-policy": keep on the Secret/ConfigMap but these will never be cleaned up. Is there a way of saying 'keep all but the last two' in Helm or an alternative way of achieving this behaviour?
$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}
Automatically Roll Deployments
In order to update resource when Secret or Configmap changes, you can add checksum annotation to your deployment
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
You can revert to your previous configuration with helm rollback command
Update:
A ssuming that your Configmap is generated using values.yaml file, you can add a _helper.tpl function
{{- define "mychart.configmapChecksum" -}}
{{ printf "configmap-%s" (.Values.bar | sha256sum) }}
{{- end }}
And use {{ include "mychart.configmapChecksumed" . }} both as configmap name and reference in deployment.
configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "mychart.configmapChecksumed" . }}
annotations:
"helm.sh/resource-policy": keep
data:
config.properties: |
foo={{ .Values.bar }}
deployment.yaml
...
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: {{ include "mychart.configmapChecksumed" . }}
Please note that you have to keep "helm.sh/resource-policy": keep annotation on Configmap telling helm to not delete the previous versions.
You can not use {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} as a configmap name directly because helm rendering will fail with
error calling include: rendering template has a nested reference name
I have close to 25 deployments and services. Each Helm template has the following:
apiVersion: v1
kind: Deployment
metadata:
name: aa1
spec:
replicas: 1
strategy: {}
template:
metadata:
labels:
app: aa1
spec:
containers:
- env:
- name: {{ .Values.env1 }}
value: "yes"
volumeMounts:
- name: {{ .Values.v1 }}
mountPath: {{ .Values.v1monthPath}}
subPath: {{ .Values.v1subpath }}
Similarly I have same env and volumeMount defined for all 25 templates.
Is there any way I can use template inheritance in Helm?
A single Helm .tpl file can contain multiple YAML documents, separated with the --- start-of-document separator. Also, you can use all of the standard Go text/template capabilities, including its looping constructs.
That means that you could write a template like this:
{{- $top := . -}}
{{- range .Values.names -}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ . }}
spec:
...
template:
spec:
containers:
- env:
- name: {{ $top.Values.env1 }}
value: "yes"
---
apiVersion: v1
kind: Service
metadata:
name: {{ . }}
spec: { ... }
{{ end -}}
What this does is first to save away the current context; since range resets the current template default context . to the iterator value, we need to remember what its current value is. We then iterate through every value in a list from the values file. (The Sprig list function could create the list in the template.) For each item, we create a Deployment and a Service, each starting with a --- on its own line. When we need the current name, it is .; when we need something from the Helm values, we need to explicitly look it up in $top.Values.
Another possible approach is to write separate templates for the deployment and service, and then have files invoke each of them. Technically a template only takes one parameter, but you can use list and index to have that one parameter be a list.
{{/* _deployment.tpl */}}
{{- define "deployment" -}}
{{- $name := index 0 . -}}
{{- $top := index 1 . -}}
---
apiVersion: apps/v1
kind: Deployment
...
name: {{ $name }}
...
name: {{ $top.Values.env1 }}
...
{{ end -}}
{{/* aa1.yaml */}}
{{- template "deployment" (list "aa1" .) -}}
{{- template "service" (list "aa1" .) -}}
This can get almost arbitrarily complex (you can have conditionals based on the current object name; you can use the template index function to look up values in the values object; and so on). helm template will do some validation and write the rendered template to stdout without invoking Kubernetes, which is helpful for eyeballing it and can be a basis for automated tests.
The text/template "inheritance" capabilities (using e.g. block) aren't a good match here. The documentation for (*text/template.Template).Parse notes that, if Parse is called multiple times, the last define found wins, which gives you a room to Parse a base template with a placeholder block and then Parse a refined template that redefines that. Helm doesn't really use this functionality; while it loads all of the YAML and supporting template files into the same template instance, it renders all of the top-level files separately instead of rendering some root template that allows for overrides.