What is the best way to use multiple containers with Helm charts? - kubernetes

I have a single values.yaml file with the following two containers:
...
nginx:
image:
repository: _ADDRESS_
tag: stable
pullPolicy: IfNotPresent
flask:
image:
repository: _ADDRESS_
tag: stable
pullPolicy: IfNotPresent
...
Is it best practice to duplicate the following code manually within the deployment file for each container and just change the variables or is there a more standard way of dealing with this? Feels as if it limits the abstraction so was just curious as to whether I am doing this wrong.
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{{- toYaml .Values.resources | nindent 12 }}
Thank you for your time. If there are any details I can add to simplify my question, please don't hesitate to mention them and I will edit the post.

Is it best practice to duplicate the following code manually within the deployment file for each container
Yes.
In principle it's possible to write a generic Helm template that produces a Deployment YAML specification. But then you'll run into a problem where the Flask application listens on port 5000 but the Nginx server uses port 80, and the Flask application has a dedicated /health endpoint but the Nginx server should just probe /, and so on, and you'd need to make these differences visible in the values.yaml. This has two problems: you're exposing fixed details of the application as configuration, and in effect you're republishing the entire Kubernetes YAML structure as Helm values.
For some things that really do get repeated over and over, you could use a helper template; for example
{{- define "container.common" -}}
securityContext: {{- toYaml .Values.securityContext | nindent 2 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
resources: {{- toYaml .Values.resources | nindent 2 }}
{{- end -}}
- name: {{ .Chart.Name }}
{{ include "container.common" . | indent 2 }}
ports: { ... }
livenessProbe: { ... }
readinessProbe: { ... }
But for details like ports: and the probes, again, these are fixed properties of the image (the Flask application will always listen on port 5000 and the end user will never need to configure it) and it's appropriate to write it in the templates/flask-deployment.yaml file, even if it looks very similar to settings in other files.

I believe this covers your need, omit the image,tag and pull policy in your values and you can use the default function as
.
.
image: "{{ .Values.image.repository | default "_ADDRESS_" }}:{{ .Values.image.tag | default "stable" }}"
imagePullPolicy: {{ .Values.image.pullPolicy | default "IfNotPresent" }}
.
.
Additional info, you can even provide a more dynamical behavior using Template function list

Related

Kubernetes cluster unable to pull images from DigitalOcean Registry

My DigitalOcean kubernetes cluster is unable to pull images from the DigitalOcean registry. I get the following error message:
Failed to pull image "registry.digitalocean.com/XXXX/php:1.1.39": rpc error: code = Unknown desc = failed to pull and unpack image
"registry.digitalocean.com/XXXXXXX/php:1.1.39": failed to resolve reference
"registry.digitalocean.com/XXXXXXX/php:1.1.39": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
I have added the kubernetes cluster using DigitalOcean Container Registry Integration, which shows there successfully both on the registry and the settings for the kubernetes cluster.
I can confirm the above address `registry.digitalocean.com/XXXX/php:1.1.39 matches the one in the registry. I wonder if I’m misunderstanding how the token / login integration works with the registry, but I’m under the impression that this was a “one click” thing and that the cluster would automatically get the connection to the registry after that.
I have tried by logging helm into a registry before pushing, but this did not work (and I wouldn't really expect it to, the cluster should be pulling the image).
It's not completely clear to me how the image pull secrets are supposed to be used.
My helm deployment chart is basically the default for API Platform:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "api-platform.fullname" . }}
labels:
{{- include "api-platform.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "api-platform.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "api-platform.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "api-platform.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}-caddy
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.caddy.image.repository }}:{{ .Values.caddy.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.caddy.image.pullPolicy }}
env:
- name: SERVER_NAME
value: :80
- name: PWA_UPSTREAM
value: {{ include "api-platform.fullname" . }}-pwa:3000
- name: MERCURE_PUBLISHER_JWT_KEY
valueFrom:
secretKeyRef:
name: {{ include "api-platform.fullname" . }}
key: mercure-publisher-jwt-key
- name: MERCURE_SUBSCRIBER_JWT_KEY
valueFrom:
secretKeyRef:
name: {{ include "api-platform.fullname" . }}
key: mercure-subscriber-jwt-key
ports:
- name: http
containerPort: 80
protocol: TCP
- name: admin
containerPort: 2019
protocol: TCP
volumeMounts:
- mountPath: /var/run/php
name: php-socket
#livenessProbe:
# httpGet:
# path: /
# port: admin
#readinessProbe:
# httpGet:
# path: /
# port: admin
resources:
{{- toYaml .Values.resources | nindent 12 }}
- name: {{ .Chart.Name }}-php
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.php.image.repository }}:{{ .Values.php.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.php.image.pullPolicy }}
env:
{{ include "api-platform.env" . | nindent 12 }}
volumeMounts:
- mountPath: /var/run/php
name: php-socket
readinessProbe:
exec:
command:
- docker-healthcheck
initialDelaySeconds: 120
periodSeconds: 3
livenessProbe:
exec:
command:
- docker-healthcheck
initialDelaySeconds: 120
periodSeconds: 3
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumes:
- name: php-socket
emptyDir: {}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
How do I authorize the kubernetes cluster to pull from the registry? Is this a helm thing or a kubernetes only thing?
Thanks!
The problem that you have is that you do not have an image pull secret for your cluster to use to pull from the registry.
You will need to add this to give your cluster a way to authorize its requests to the cluster.
Using the DigitalOcean kubernetes Integration for Container Registry
Digital ocean provides a way to add image pull secrets to a kubernetes cluster in your account. You can link the registry to the cluster in the settings of the registry. Under "DigitalOcean Kuberentes Integration" select edit, then select the cluster you want to link the registry to.
This action adds an image pull secret to all namespaces within the cluster and will be used by default (unless you specify otherwise).
The issue was that API Platform automatically has a default value for imagePullSecrets in the helm chart, which is
imagePullSecrets: []
in values.yaml
So this seems to override kubernetes from accessing imagePullSecrets in the way that I expected. The solution was to add the name of the imagePullSecrets directly to the helm deployment command, like this:
--set "imagePullSecrets[0].name=registry-secret-name-goes-here"
You can view the name of your secret using kubectl get secrets like this:
kubectl get secrets
And the output should look something like this:
NAME TYPE DATA AGE
default-token-lz2ck kubernetes.io/service-account-token 3 38d
registry-secret-name-goes-here kubernetes.io/dockerconfigjson 1 2d16h

Non Root User Helm & AKS

I'm attempting to connect and run a pod in an AKS cluster (v1.19.6) as a non-root user with Helm (v3.5.2), and getting a crashback loop with the error I have no name!. The docker image and service runs locally without an issue as the correct user at runtime.
After helm create mychart I set up my security in the values.yaml as :
podSecurityContext:
runAsNonRoot: true
runAsUser: 123
securityContext:
# capabilities:
# drop:
# - ALL
#readOnlyRootFilesystem: false
runAsNonRoot: true
runAsUser: 123
The deployment.yaml is below. I've not modified anything else other than the parameters to connect to my AKS cluster:
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "mychart.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
My Dockerfile ends with
USER 123
EXPOSE 8080
CMD [ "sh", "-c", "./blah; bash"]
Am I correct that this is most likely the issue? How do I go about resolving the problem? Supporting documentation would be very helpful everything I'm finding is outdated.
I created a startup script to start the service with the user declared. Not sure if this is the K8s methodology but it worked. Will leave it unanswered in the event someone has a better solution.

Kubernetes requests not balanced

We've just had an increase in traffic to our kubernetes cluster and I've noticed that of our 6 application pods, 2 of them are seemingly not used very much. kubectl top pods returns the following
You can see of the 6 pods, 4 of them are using more than 50% of the CPU (2 vCPU nodes), but two of them aren't really doing much at all.
Our cluster is setup on AWS, using the ALB ingress controller. The load balancer is configured to use the Least outstanding requests rather than Round robin in an attempt to balance things out a bit more, but we're still seeing this imbalance.
Is there any way of determining why this is happening, or if indeed it actually is a problem? I'm hoping it's normal behaviour rather than an issue but I'd rather investigate it.
App deployment config
This is the configuration of the main application pods. Nothing fancy really
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "app.fullname" . }}
labels:
{{- include "app.labels" . | nindent 4 }}
app.kubernetes.io/component: web
spec:
replicas: {{ .Values.app.replicaCount }}
selector:
matchLabels:
app: {{ include "app.fullname" . }}-web
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/config_maps/app-env-vars.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 10 }}
{{- end }}
labels:
{{- include "app.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: web
app: {{ include "app.fullname" . }}-web
spec:
serviceAccountName: {{ .Values.serviceAccount.web }}
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- {{ include "app.fullname" . }}-web
topologyKey: failure-domain.beta.kubernetes.io/zone
containers:
- name: {{ .Values.image.name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- bundle
args: ["exec", "puma", "-p", "{{ .Values.app.containerPort }}"]
ports:
- name: http
containerPort: {{ .Values.app.containerPort }}
protocol: TCP
readinessProbe:
httpGet:
path: /healthcheck
port: {{ .Values.app.containerPort }}
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
resources:
{{- toYaml .Values.resources | nindent 12 }}
envFrom:
- configMapRef:
name: {{ include "app.fullname" . }}-cm-env-vars
- secretRef:
name: {{ include "app.fullname" . }}-secret-rails-master-key
- secretRef:
name: {{ include "app.fullname" . }}-secret-db-credentials
- secretRef:
name: {{ include "app.fullname" . }}-secret-db-url-app
- secretRef:
name: {{ include "app.fullname" . }}-secret-redis-credentials
That's a known issue with Kubernetes.
In short, Kubernetes doesn't load balance long-lived TCP connections.
This excellent article covers it in details.
The load distribution you service is showing complies exactly with the case.

Knative service with Keycloak gatekeeper sidecar

I am trying to deploy the following service:
{{- if .Values.knativeDeploy }}
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
{{- if .Values.service.name }}
name: {{ .Values.service.name }}
{{- else }}
name: {{ template "fullname" . }}
{{- end }}
labels:
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
spec:
template:
spec:
containers:
- image: quay.io/keycloak/keycloak-gatekeeper:9.0.3
name: gatekeeper-sidecar
ports:
- containerPort: {{ .Values.keycloak.proxyPort }}
env:
- name: KEYCLOAK_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: {{ template "keycloakclient" . }}
key: secret
args:
- --resources=uri=/*
- --discovery-url={{ .Values.keycloak.url }}/auth/realms/{{ .Values.keycloak.realm }}
- --client-id={{ template "keycloakclient" . }}
- --client-secret=$(KEYCLOAK_CLIENT_SECRET)
- --listen=0.0.0.0:{{ .Values.keycloak.proxyPort }} # listen on all interfaces
- --enable-logging=true
- --enable-json-logging=true
- --upstream-url=http://127.0.0.1:{{ .Values.service.internalPort }} # To connect with the main container's port
resources:
{{ toYaml .Values.gatekeeper.resources | indent 12 }}
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
{{- range $pkey, $pval := .Values.env }}
- name: {{ $pkey }}
value: {{ quote $pval }}
{{- end }}
envFrom:
{{ toYaml .Values.envFrom | indent 10 }}
ports:
- containerPort: {{ .Values.service.internalPort }}
livenessProbe:
httpGet:
path: {{ .Values.probePath }}
port: {{ .Values.service.internalPort }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
readinessProbe:
httpGet:
path: {{ .Values.probePath }}
port: {{ .Values.service.internalPort }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
resources:
{{ toYaml .Values.resources | indent 12 }}
terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }}
{{- end }}
Which fails with the following error:
Error from server (BadRequest): error when creating "/tmp/helm-template-workdir-290082188/jx/output/namespaces/jx-staging/env/charts/docs/templates/part0-ksvc.yaml": admission webhook "webhook.serving.knative.dev" denied the request: mutation failed: expected exactly one, got both: spec.template.spec.containers'
Now, if I read the specs (https://knative.dev/v0.15-docs/serving/getting-started-knative-app/), I can see this example:
apiVersion: serving.knative.dev/v1 # Current version of Knative
kind: Service
metadata:
name: helloworld-go # The name of the app
namespace: default # The namespace the app will use
spec:
template:
spec:
containers:
- image: gcr.io/knative-samples/helloworld-go # The URL to the image of the app
env:
- name: TARGET # The environment variable printed out by the sample app
value: "Go Sample v1"
Which has exactly the same structure. Now, my questions are:
How can I validate my yam without waiting for a deployment? Intellij has a k8n plugin, but I can't find the CRD schema for serving.knative.dev/v1 that are machine consumable. (https://knative.dev/docs/serving/spec/knative-api-specification-1.0/)
Is it allowed with knative to have multiple container? (that configuration works perfectly with apiVersion: apps/v1 kind: Deployment)
Multi container is alpha feature in knative version 0.16.
This feature need to be enabled by setting multi-container to enabled in the config-features ConfigMap. So edit the configmap using
kubectl edit cm config-features and enable that feature.
apiVersion: v1
kind: ConfigMap
metadata:
name: config-features
namespace: knative-serving
labels:
serving.knative.dev/release: devel
annotations:
knative.dev/example-checksum: "983ddf13"
data:
_example: |
...
# Indicates whether multi container support is enabled
multi-container: "enabled"
...
What version of Knative are you using?
Support for multiple containers was added as an alpha feature in 0.16. If you're not using 0.16 or later or don't have the alpha flag enabled, the request will probably be blocked.
There were a number of edge cases to define for multi-container support in Knative, so the default was to be conservative and only allow one container until the constraints had been explored.

How to refer whole structure from values.yaml instead of specifying one by one?

I am trying to deploy helm chart in local vritual box on minikube using helm command shown below.
I am referring livenessProbe , readinessProbe configuration directly from values.yam in the deployment.yaml as shown below. However following this approach gives me the error specified below , if i change this to refer each attribute value independently i see the chart is getting installed , pod deploys successfully.
livenessProbe:
- {{ .Values.monitorConfig.liveness }}
readinessProbe:
- {{ .Values.monitorConfig.readiness }}
Can anyone please let me know what can be done to avoid the error and why??
Thank you.
Helm Command
helm install --debug -n pspk ./pkg/helm/my-service/
Error
Error: release pspk failed: Deployment in version "v1beta1" cannot be
handled as a Deployment: v1beta1.Deployment.Spec:
v1beta1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec:
v1.PodSpec.Containers: []v1.Container: v1.Container.LivenessProbe:
readObjectStart: expect { or n, but found [, error found in #10 byte
of ...|ssProbe":["map[failu|..., bigger context
...|"imagePullPolicy":"IfNotPresent","livenessProbe":["map[failureThreshold:3
httpGet:map[path:/greeting|...
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "fullname" . }}
labels:
app: {{ template "fullname" .}}
chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
release: "{{ .Release.Name }}"
spec:
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app: {{ template "fullname" . }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 50443
protocol: TCP
- name: grpc
containerPort: 50051
protocol: TCP
livenessProbe:
- {{ .Values.monitorConfig.liveness }}
readinessProbe:
- {{ .Values.monitorConfig.readiness }}
resources:
{{ toYaml .Values.resources | indent 12 }}
values.yaml
replicaCount: 2
application:
track: stable
image:
repository: test/go-k8s
tag: 0.1.1
pullPolicy: IfNotPresent
# SQL migration scripts
service:
enabled: false
type: NodePort
port: 80
grpc_port: 50051
env:
# POSTGRES_HOST
postgresHost: localhost
# POSTGRES_PORT
postgresPort: "5432"
# POSTGRES_SSL_MODE
postgresSSLMode: "disable"
# POSTGRES_DB
postgresDB: test
# POSTGRES_USER
postgresUser: test
# POSTGRES_PASSWORD
postgresPassword: "test"
monitorConfig:
liveness:
httpGet:
path: "/greeting"
port: 50443
periodSeconds: 2
timeoutSeconds: 10
initialDelaySeconds: 5
failureThreshold: 3
successThreshold: 1
readiness:
httpGet:
path: "/greeting"
port: 50443
periodSeconds: 2
timeoutSeconds: 10
initialDelaySeconds: 5
failureThreshold: 3
successThreshold: 1
resources: {}
nodeSelector: {}
tolerations: []
affinity: {}
You need to do two things to make this work correctly: explicitly serialize the value as YAML, and make the indentation correct. This tends to look something like
livenessProbe:
- {{ .Values.monitorConfig.liveness | toYaml | indent 8 | trim }}
The default serialization will be a Go-native dump format, which isn't YAML and leads to the weird map[failureThreshold:1] output; toYaml fixes this. indent 8 puts spaces at the front of every line in the resulting block (you will need to adjust the "8"). trim removes leading and trailing spaces. (toYaml is Helm-specific and isn't documented well; the other two functions come from the Sprig support library.)
You should double-check this output with
helm template -n pspk ./pkg/helm/my-service/
and if it doesn't look like valid YAML, adjust it further.
In your YAML:
livenessProbe:
- {{ .Values.monitorConfig.liveness }}
readinessProbe:
- {{ .Values.monitorConfig.readiness }}
You insert your values into sequence items. Sequence items in YAML are started with -. However, the contents of livenessProbe is expected to be a YAML mapping. The error message is poor but tells you what goes wrong:
expect { or n, but found [,
{ starts a YAML mapping (in flow style), [ starts a YAML sequence (in flow style). The message tells you that the start of a YAML mapping is expected, but the start of a YAML sequence is found. Note that since you're using block style, you don't actually use { and [ here.
So to fix it, simply remove the - so that your inserted mapping (as seen in your values.yaml) is the direct value of livenessProbe and not contained in a sequence:
livenessProbe:
{{ .Values.monitorConfig.liveness }}
readinessProbe:
{{ .Values.monitorConfig.readiness }}
Thanks to the community answers/comments and helm template guide,
it can be combined into:
{{- if .Values.monitorConfig.liveness }}
livenessProbe:
{{ toYaml .Values.monitorConfig.liveness | indent 12 }}
{{- end }}
This will give more flexibility.