Yaml dynamic variables - kubernetes

So, I'm just starting with YAML and k8s and maybe this questions comes from lack of understanding how YAML and Helm works together.
But I was wondering if I can declare a variable inside Values.YAML file that will be changed during the run of the scripts?
I was thinking about accumulating value for each pod I am starting, that will be saved as an environment variable in each pod. I can manually create different value for each pod but I was wondering if there is an automatic way to do so?
Hope my question is clear :)

Helm allows for conditionals using its templating. For example, I can have this in my values.yaml
environment: preprod
And then this inside a yaml within my helm chart
{{ if eq .Values.environment "preprod" }}
## Do preprod stuff here
{{ end }}
{{ if eq .Values.environment "prod" }}
## Do prod stuff here
{{ end }}
This means if I ran helm install, then .Values.environment would resolve to "preprod" and the block within the {{ if eq .Values.environment "preprod" }}...{{ end }} would be printed in the yaml.
If I wanted to override that default, I can by adding the --set switch (details here)
helm install --set environment=prod
Which would cause the .Values.environment variable to resolve to "prod" instead, and the block within {{ if eq .Values.environment "prod" }} ... {{ end }} would be output instead.

Helm templates are stateless, and the variables structure is immutable.
Can I declare a variable inside the values.yaml file that will be changed during the run of the scripts?
That's not possible, no.
If you have experience with functional programming, there are some related tricks you can use in the context of Helm templates. A template only takes one parameter, but you can make that parameter be a list, and then pass some state forward through a series of recursive template calls. If you're tempted to do this, consider writing a Kubernetes operator instead: even if you have to learn Go to do it, the language is much more mainstream and practical than the template language, and it's much easier to test.
This having been said:
... accumulating value for each pod I am starting ...
If all you're asking for is a set of Pods that are very similar except that they have sequential names, this is one of the things a StatefulSet provides.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: some-name
spec:
replicas: 5
The Pods generated by this StatefulSet will be named some-name-0, some-name-1, and so on. Your application code can see these names via the hostname command and language-specific equivalents. That could meet your needs.
If you need something more complex, you can also use a template range loop to generate a series of documents. Each document needs to begin with a --- YAML start-of-document marker. You need to be aware that range rebinds the . special template variable as appears in constructs like .Values, and I tend to save that away.
{{- $top := . }}
{{- $i := range until 5 }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "mychart.fullname" $top }}-{{ $i }}
spec: { ... }
{{- end }}
You should almost always use higher-level constructs like Deployments, StatefulSets, or Jobs instead of creating bare Pods. Trying to fit within their patterns will usually be a little easier than trying to manually create several very slightly different Pods.

Related

Install single Kubernetes deployment multiple times

I have a Helm chart that installs different kubernetes resources to deploy my application.
One of those resources is a deployment that has two flavors, one for a client part of the app and one for the server part, so actually they are two deployments. Most of their manifests (yaml files) are exactly the same, the only important difference is that each refers to a different configmap in order to have specific values for some of the configmap properties (particularly the type: client/server and number of replicas). This doesn't seem to be very efficient since I'm duplicating code for the deployments, but it's the way I found to do it.
On the other hand, for the configmaps I made use of Helm's template feature ({{ include }}) so I have a "main" configmap template which has all the common content, and two separate configmaps specifying the differences for each deployment and including the main template.
So far so good, even though there may be some unnecessary code duplication, in which case I wouldn't know how to improve.
The problem is that multiple variants of the above two deployments came into play. For example, I may want to deploy a client-type pod with property X having a certain value, and two server-type pods with property X having a different value. So following my approach, I would have to start creating more deployment yaml files to cover all possible combinations: type=client & X=Y, type=client & X=Z, type=server & X=Y, type=server & X=Z and so on. And the only purpose of this is to be able to specify how many replicas I want for each kind or combination.
Is there any way (using Helm or other Kubernetes related framework) to have a single deployment yaml file and be able to install it multiple times specifying only the properties that vary and the number of replicas for that variation?
For example:
I want:
3 replicas that have "type=client" and "X=1"
2 replicas that have "type=server" and "X=1"
4 replicas that have "type=client" and "X=2"
1 replicas that have "type=server" and "X=3"
where type and X are properties (data) in some configmap.
Hope it's clear enough, otherwise please let me know, thanks.
In Helm there are a couple of ways to approach this. You need to bring the settings up to Helm's configuration layer (they would be in values.yaml or provided via a mechanism like helm install --set); you can't extract them out of the ConfigMap.
One approach is to have your Helm chart install only a single instance of the Deployment and the corresponding ConfigMap. Have a single templates/deployment.yaml file that includes lines like:
name: {{ .Release.Name }}-{{ .Chart.Name }}-{{ .Values.type }}-{{ .Values.X }}
replicas: {{ .Values.replicas }}
env:
- name: TYPE
value: {{ .Values.type }}
- name: X
value: {{ quote .Values.X }}
Then you can deploy multiple copies of it:
helm install c1 . --set type=client --set X=1 --set replicas=3
helm install s1 . --set type=server --set X=1 --set replicas=2
You mention that you're generating similar ConfigMaps using templates already, and you can also use that same approach for any YAML structure. A template takes a single parameter, and one trick that's possible is to pass a list as that parameter. The other important detail to remember is that the top-level names like .Values are actually field lookups in a special object ., which can get reassigned in several contexts, so you may need to explicitly pass around and reference the top-level object.
Say your template needs the top-level values, and also some extra configuration settings:
{{- define "a.deployment" -}}
{{- $top := index . 0 -}}
{{- $config := index . 1 -}}
metadata:
name: {{ include "chart.name" $top }}-{{ $config.type }}-{{ $config.X }}
{{ end -}}
Note that we unpack the two values from the single list parameter, then pass $top in places where we might expect to pass . as a parameter.
You can have a top-level file per variant of this. For example, templates/deployment-server-1.yaml might contain:
{{- $config := dict "type" "server" "X" "1" -}}
{{- include "a.deployment" (list . $config) -}}
Here . is the top-level object; we're embedding that and the config dictionary into a single list parameter to match what the template expects. You could use any templating constructs in the dict call if some of the values were specified in Helm configuration.
Finally, there's not actually a rule that a YAML file contains only a single object. If your Helm configuration just lists out the variants, you can loop through them and emit them all:
{{-/* range will reassign . so save its current value */-}}
{{- $top := . -}}
{{- range .Values.installations -}}
{{-/* Now . is one item from the installations list */-}}
{{-/* This is the YAML start-of-document marker: */-}}
---
{{ include "a.deployment" (list $top .) -}}
{{- end -}}
You'd just list out all of the variants and settings in the Helm values.yaml (or, again, an externally provided helm install -f more-values.yaml file):
installations:
- type: client
X: 1
replicas: 3
- type: server
X: 1
replicas: 2

Import parent template with subchart values

I have multiple sucharts with applications and a parent chart that will deploy them.
All subcharts have the same manifests for the underlying application. Therefore I decided to create a library and put general variables from subcharts in it.
Example from lib:
{{- define "app.connect.common.release.common_libs.servicetemplate" -}}
apiVersion: v1
kind: Service
metadata:
labels:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
name: {{ .Values.application.name }}-service
namespace: {{ .Values.global.environment.namespace }}
spec:
type: LoadBalancer
ports:
- name: https
port: 443
targetPort: 8080
- name: http
port: 80
targetPort: 8080
selector:
app: {{ .Values.application.name }}
status:
loadBalancer: {}
{{- end }}
I declared a dependency in Chart.yaml and executed helm dep up. Then in my subchart I'm importing this template. But when I'm trying to run --dry-run on parent chart I'm receiving the following error:
Error: template: app.connect.common.release/charts/app.connect.common.release.chtmgr/templates/service.yaml:1:4: executing "app.connect.common.release/charts/app.connect.common.release.chtmgr/templates/service.yaml" at <include "app.connect.common.release.common_libs.servicetemplate" .>: error calling include: template: app.connect.common.release/charts/app.connect.common.release.chtmgr/charts/app.connect.common.release.common_libs/templates/_helpers.tpl:169:18: executing "app.connect.common.release.common_libs.servicetemplate" at <.Values.application.name>: nil pointer evaluating interface {}.name
My values values.yaml in the subchart:
application:
name: chtmgr-api
image: cht-mgr-api
The same error with named template.
Is it possible to put general values from subchart in parent template(example _helper.tpl) and import it in subchart?
If not, how do you implement this?
I've checked a lot of resources but still don't have an idea am I going in the right direction.
The Helm template define action creates a "function". It implicitly takes a single "parameter", using the special variable name ., and .Values is actually a lookup in .. It does not "capture" .Values at the point where it is defined; it uses the Values property of the parameter that's passed to it.
This means the template will behave differently when it's called in different contexts. As the Helm documentation on Subcharts and Global Variables describes, when executing the subchart, the top-level . parameter will have its Values replaced by the subchart's key in the primary values.
There's three ways to work around this:
If you're using Helm 3, you can directly import a value from the subchart into the parent. (I'm not clear what version of Helm exactly this was added, or if the syntax works in a separate requirements.yaml file.) Declare the subchart dependency in your Chart.yaml as
dependencies:
- name: subchart
import-values:
- child: application
parent: application
and the template you show should work unmodified.
(There's a more involved path that involves the subchart explicitly exporting values to the parent. I'm not sure if this is that useful to you: the paths will still be different in the two charts, and in any case Helm values can't contain computed values.)
You already use .Values.global in your example; this will have the same value in the parent chart and all included charts.
# vvvvvv inserted "global" here
name: {{ .Values.global.application.name }}-service
namespace: {{ .Values.global.environment.namespace }}
You can also use template logic to try to look in more places for the application dictionary. This will only work from the parent chart and the subchart proper and not any other sibling charts; I believe it will also only work if the configuration is embedded in the parent chart's values.yaml or an external helm install -f values file, but won't find content in the included chart's values.yaml.
{{/* Get the subchart configuration, or an empty dict (assuming we're in the parent) */}}
{{- $subchart := .Values.subchart | default dict -}}
{{/* Find the application values (assuming first we're in the subchart) */}}
{{- $application := .Values.application | default $subchart.application | default dict -}}
{{/* Then use it */}}
name: {{ $application.name }}-service
This logic sets the variable $application, normally, to .Values.application. If that's not set (you're in the parent chart) then it in effect looks up .Values.subchart.application, and if that's not available either then it uses an empty dictionary. This is (again, using programming terminology) eagerly evaluated – even if you're not falling back to the default, Helm will always look up .Values.subchart – so we use a second variable to have the .Values.subchart, or an empty dict (if you're in the subchart). In both cases looking up a nonexistent key in an empty dict isn't an error but looking up on an unset value is.
Found a solution. Problem was related to the chart/template names all of them include '.' in the names what causes problem. Don't include dots in your templates/charts names. Example how not to do: "ibext.common.connect.etc".
My assumption
I'm quite newbie in helm and my assumption that helm when template engine is looking in the subchart it goes through the following way to find variables .Values.subchartName in my case it was .Values.ibext.common.connect.etc So when it faced with .Values.ibext it can't understand what it is(but helm doesn't show anything).
But this is only my assumption. If someone there understands the behaviour of template engine in this case please reveal the secret.
Be aware the *.tpl files must be in the templates folder. Double check all folder names and the structure.

Templating external files in helm

I want to use application.yaml file to be passed as a config map.
So I have written this.
apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
{{ (.Files.Glob "foo/*").AsConfig | indent 2 }}
my application.yaml is present in foo folder and
contains a service name which I need it to be dynamically populated via helm interpolation.
foo:
service:
name: {{.Release.Name}}-service
When I dry run , I am getting this
apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
application.yaml: "ei:\r\n service:\r\n name: {{.Release.Name}}-service"
but I want name: {{.Release.Name}}-service to contain actual helm release name.
Is it possible to do templating for external files using helm , if yes then how to do it ?
I have gone through https://v2-14-0.helm.sh/docs/chart_template_guide/#accessing-files-inside-templates
I didn't find something which solves my use case.
I can also copy the content to config map yaml and can do interpolation but I don't want to do it. I want application.yml to be in a separate file, so that, it will be simple to deal with config changes..
Helm includes a tpl function that can be used to expand an arbitrary string as a Go template. In your case the output of ...AsConfig is a string that you can feed into the template engine.
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-conf
data:
{{ tpl (.Files.Glob "foo/*").AsConfig . | indent 2 }}
Once you do that you can invoke arbitrary template code from within the config file. For example, it's common enough to have a defined template that produces the name prefix of the current chart as configured, and so your config file could instead specify
foo:
service:
name: {{ template "mychart.name" . }}-service
As best I can tell, there is no recursive template evaluation available in helm (nor in Sprig), likely by design
However, in your specific case, if you aren't expecting the full power of golang templates, you can cheat and use Sprig's regexReplaceAllLiteral:
kind: ConfigMap
data:
{{/* here I have used character classes rather that a sea of backslashes
you can use the style you find most legible */}}
{{ $myRx := "[{][{] *[.]Release[.]Name *[}][}]" }}
{{ regexReplaceAllLiteral $myRx (.Files.Glob "foo/*").AsConfig .Release.Name }}
If you genuinely need the full power of golang templates for your config files, then helm, itself, is not the mechanism for doing that -- but helmfile has a lot of fancy tricks for generating the ultimate helm chart that helm will install

Create kubernetes resources with helm only if custom resource definition exists

I have a helm chart that deploys a number of Kubernetes resources. One of them is a resource that is of a Custom Resource Definition (CRD) type (ServiceMonitor used by prometheus-operator).
I am looking for a way, how to "tell" helm that I'd want to create this resource only if such a CRD is defined in the cluster OR to ignore errors only caused by the fact that such a CRD is missing.
Is that possible and how can I achieve that?
Helm's Capabilities object can tell you if an entire API class is installed in the cluster. I don't think it can test for a specific custom resource type.
In your .tpl files, you can wrap the entire file in a {{ if }}...{{ end }} block. Helm doesn't especially care if the rendered version of a file is empty.
That would lead you to a file like:
{{ if .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" -}}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
{{ end -}}
That would get installed if the operator is installed in the cluster, and skipped if not.
If you are on Helm 3 you can put your CRD in the crds/ directory. Helm will treat it differently, see the docs here.
In Helm 2 there is another mechanism using the crd-install hook. You can add the following to your CRD:
annotations:
"helm.sh/hook": crd-install
There are some limitations with this approach so if you are using Helm 3 that would be preferred.
In Helm v3, you can test for specific resources:
{{ if .Capabilities.APIVersions.Has "monitoring.coreos.com/v1/ServiceMonitor" -}}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
...
spec:
...
{{- end }}
https://helm.sh/docs/chart_template_guide/builtin_objects/

Best way to DRY up deployments that all depend on a very similar init-container

I have 10 applications to deploy to Kubernetes. Each of the deployments depends on an init container that is basically identical except for a single parameter (and it doesn't make conceptual sense for me to decouple this init container from the application). So far I've been copy-pasting this init container into each deployment.yaml file, but I feel like that's got to be a better way of doing this!
I haven't seen a great solution from my research, though the only thing I can think of so far is to use something like Helm to package up the init container and deploy it as part of some dependency-based way (Argo?).
Has anyone else with this issue found a solution they were satisfied with?
A Helm template can contain an arbitrary amount of text, just so long as when all of the macros are expanded it produces a valid YAML Kubernetes manifest. ("Valid YAML" is trickier than it sounds because the indentation matters.)
The simplest way to do this would be to write a shared Helm template that included the definition for the init container:
_init_container.tpl:
{{- define "common.myinit" -}}
name: myinit
image: myname/myinit:{{ .Values.initTag }}
# Other things from a container spec
{{ end -}}
Then in your deployment, include this:
deployment.yaml:
apiVersion: v1
kind: Deployment
spec:
template:
spec:
initContainers:
- {{ include "common.myinit" . | indent 10 | strip }}
Then you can copy the _init_container.tpl file into each of your individual services.
If you want to avoid the copy-and-paste (reasonable enough) you can create a Helm chart that contains only templates and no actual Kubernetes resources. You need to set up some sort of repository to hold this chart. Put the _init_container.tpl into that shared chart, declare it as a dependency is the chart metadata, and reference the template in your deployment YAML in the same way (Go template names are shared across all included charts).