Helm chart overriding values - kubernetes

There is such structure of the project, I'm trying to understand:
charts/
spark-0.0.1-100.tgz
templates/
Chart.yaml
values.yaml
Chart.yaml
appVersion: 0.1.0
dependencies:
- name: spark
version: "0.0.1-100"
repository: https://helm.<corporation>.com/<project>
condition: spark.enabled
values.yaml (some values are omitted for simplicity)
spark:
enabled: true
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/spark-service-account
image:
tag: "3.3.0-dev-28"
extraEnv:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
Master:
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
Worker:
Replicas: 3
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
zookeeper:
host: "project-zookeeper"
port: 2181
Then, I have unzipped charts/spark-0.0.1-100.tgz into folder charts/spark/:
charts/
spark/
templates/
Chart.yaml
values.yaml
charts/spark/values.yaml:
global:
aci:
sdrAppname: spark
image:
repository: "docker.<corporation>.com/<project>/spark"
tag: "1.0.1"
spark:
path: "/opt/spark"
user: 1000
group: 1000
config: |
SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=21600"
Master:
Name: master
Replicas: 1
Component: "spark-core"
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
ServicePort: <port>
ContainerPort: <port>
RestEnabled: "true"
RestPort: <port>
ServiceType: LoadBalancer
WebUi:
Name: webui
ServicePort: <port>
ContainerPort: <port>
The question is, how values from values.yaml and charts/spark/values.yaml are being corresponded?
Are values from root values.yaml are replaced with values from charts/spark/values.yaml?
Thank you in advance.

The question is, how values from values.yaml and charts/spark/values.yaml are being corresponded?
The short version is they have almost nothing to do with one another, they're just the defaults used by their respective charts. The medium version is that the outer chart can supersede values in the subordinate charts if it chooses to, but ultimately the user has the final word in that discussion because helm --values win out over the defaults (same for usage of --set, but --values are far easier to discuss due to not having to delve into that --set DSL)
The subcharts are not aware of the parent chart's defaults. The parent chart doesn't have to be aware of the child chart's values, and cannot -- itself -- refer to the subchart's defaults
Are values from root values.yaml are replaced with values from charts/spark/values.yaml?
For the most part, no, they're completely separate namespaces. However, as we saw above, for every - name: key in the dependencies: list, those top-level keys in the parent chart's values.yaml becomes special in that it overlays the values on top of the subordinate chart's defaults (you see in your example of { spark: { path: "/opt/spark" } }; that top level spark: key matches up with - name: in the dependencies: list)
It's kind of like "duck typing" though, because the top level key is free to use any random structure and the child chart will use ones that it is aware of. For example:
# values.yaml
spark:
path: /opt/spark
value-of-pi: 3.1415
is perfectly legal in the top chart's values.yaml even though the child spark chart will only notice the { spark: { path: "" } } because it does not have any {{ .Values.value-of-pi }} in its templates
But, again, for clarity: the top level values.yaml, even if it includes spark: { path: /alpha } can be superseded by the user with --values <(echo '{spark: {path: /beta } }') and the resulting spark install will have path: /beta when that chart is templated out

Related

How can I use multiple Community Mongo replica sets in one Kubernetes namespace?

I've been running a community version of the MongoDB replica set in Kubernetes for over a year now. For reference, I am deploying with these Helm charts:
https://github.com/mongodb/helm-charts/tree/main/charts/community-operator
https://github.com/mongodb/helm-charts/tree/main/charts/community-operator-crds
I now have a need to spin up a couple extra replica sets. I cannot reuse the existing rs because this new deployment is software from one of our subcontractors and we want to keep it separate. When I try to start a new rs, all the rs on the namespace get into a bad state, failing readiness probes.
Do the different replica sets each require their own operator? In an attempt to test that theory, I modified the values.yaml and deployed an operator for each rs but I'm still getting the error.
I think I am missing a config on the DB deployment that tells it which operator to use, but I can't find that config option in the helm chart (referencing the 2nd link from earlier https://github.com/mongodb/helm-charts/blob/main/charts/community-operator-crds/templates/mongodbcommunity.mongodb.com_mongodbcommunity.yaml)
EDIT: For a little extra info, it seems like the mongodb can be used without issue. Kubernetes is just showing an error, saying the readiness probes have failed.
EDIT: Actually, this didn't fix the entire problem. It seems to work for a few hours then all the readiness probes for all rs start to fail again.
It looks like you can deploy multiple operators and rs in a single namespace. What I was missing was linking the operators and rs together.
I also left out an important detail. I don't actually deploy the mongodb using that helm chart because the config options are too limited. At the bottom of the values.yaml there's a setting called createResource: false which I have set to false. I then deploy a separate yaml that defines the mongo replicaset like so:
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: my-mongo-rs
spec:
members: 3
type: ReplicaSet
version: "5.0.5"
security:
authentication:
modes: ["SCRAM"]
# tls:
# enabled: true
# certificateKeySecretRef:
# name: mongo-tls
# caConfigMapRef:
# name: mongo-ca
statefulSet:
spec:
template:
spec:
containers:
- name: "mongodb-agent"
resources:
requests:
cpu: 200m
memory: 500M
limits: {}
- name: "mongod"
resources:
requests:
cpu: 1000m
memory: 5G
limits: {}
serviceAccountName: my-mongodb-database
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
storageClassName: my-retainer-sc
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 20G
# replicaSetHorizons:
# - external: myhostname.com:9000
# - external: myhostname.com:9001
# - external: myhostname.com:9002
users:
- name: my-mongo-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-mongo-user-creds
roles:
- db: admin
name: clusterAdmin
- db: admin
name: readWriteAnyDatabase
- db: admin
name: dbAdminAnyDatabase
- db: admin
name: userAdminAnyDatabase
- db: admin
name: root
scramCredentialsSecretName: my-user-scram
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
Anywhere in this config where you see "my", I actually use my app name but I've genericized it for this post.
To link the operator and rs together, it's done using a name. In the operator yaml it's this line:
database:
name: my-mongodb-database
That name creates a serviceaccount in kubernetes and you must define your database pod to use that specific serviceaccount. Otherwise, it tries to default to a serviceaccount named mongodb-database which either won't exist, or you'll end up with multiple rs using the same serviceaccount (therefore, the same operator).
And in the rs yaml it's this line:
serviceAccountName: my-mongodb-database
This will link it to the correct serviceaccount.

I created a serviceemointer using jsonexporter in Prometheus environment, but the metrics could not be verified. Is there a way to check the metric?

I am a beginner who is using Prometheus and Grapana to monitor the value of REST API.
Prometheus, json-exporrter, and grafana both used the Helm chart, Prometheus installed as default values.yaml, and json-exporter installed as custom values.yaml.
I checked that the prometheus set the service monitor of json-exporter as a target, but I couldn't check its metrics.
How can I check the metrics? Below is the environment , screenshots and code.
environment :
kubernetes : v1.22.9
helm : v3.9.2
prometheus-json-exporter helm chart : v0.5.0
kube-prometheus-stack helm chart : 0.58.0
screenshots :
https://drive.google.com/drive/folders/1vfjbidNpE2_yXfxdX8oX5eWh4-wAx7Ql?usp=sharing
values.yaml
in custom_jsonexporter_values.yaml
# Default values for prometheus-json-exporter.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: quay.io/prometheuscommunity/json-exporter
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: []
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: []
podSecurityContext: {}
# fsGroup: 2000
# podLabels:
# Custom labels for the pod
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: ClusterIP
port: 7979
targetPort: http
name: http
serviceMonitor:
## If true, a ServiceMonitor CRD is created for a prometheus operator
## https://github.com/coreos/prometheus-operator
##
enabled: true
namespace: monitoring
scheme: http
# Default values that will be used for all ServiceMonitors created by `targets`
defaults:
additionalMetricsRelabels: {}
interval: 60s
labels:
release: prometheus
scrapeTimeout: 60s
targets:
- name : pi2
url: http://xxx.xxx.xxx.xxx:xxxx
labels: {} # Map of labels for ServiceMonitor. Overrides value set in `defaults`
interval: 60s # Scraping interval. Overrides value set in `defaults`
scrapeTimeout: 60s # Scrape timeout. Overrides value set in `defaults`
additionalMetricsRelabels: {} # Map of metric labels and values to add
ingress:
enabled: false
className: ""
annotations: []
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: []
tolerations: []
affinity: []
configuration:
config: |
---
modules:
default:
metrics:
- name: used_storage_byte
path: '{ .used }'
help: used storage byte
values:
used : '{ .used }'
labels: {}
- name: free_storage_byte
path: '{ .free }'
help: free storage byte
labels: {}
values :
free : '{ .free }'
- name: total_storage_byte
path: '{ .total }'
help: total storage byte
labels: {}
values :
total : '{ .total }'
prometheusRule:
enabled: false
additionalLabels: {}
namespace: ""
rules: []
additionalVolumes: []
# - name: password-file
# secret:
# secretName: secret-name
additionalVolumeMounts: []
# - name: password-file
# mountPath: "/tmp/mysecret.txt"
# subPath: mysecret.txt
Firstly you can check the targets page on the Prometheus UI to see if a) your desired target is even defined and b) if the endpoint is reachable and being scraped.
However, you may need to troubleshoot a little if either of the above is not the case:
It is important to understand what is happening. You have deployed a Prometheus Operator to the cluster. If you have used the default values from the helm chart, you also deployed a Prometheus custom resource(CR). This instance is what is telling the Prometheus Operator how to ultimately configure the Prometheus running inside the pod. Certain things are static, like global metric relabeling for example, but most are dynamic, such as picking up new targets to actually scrape. Inside the Prometheus CR you will find options to specify serviceMonitorSelector and serviceMonitorNamespaceSelector (The behaviour is the same also for probes and podmonitors so I'm just going over it once). Assuming you leave the default set like serviceMonitorNamespaceSelector: {}, Prometheus Operator will look for ServiceMonitors in all namespaces on the cluster to which it has access via its serviceAccount. The serviceMonitorSelector field lets you specify a label and value combination that must be present on a serviceMonitor that must be present for it to be picked up. Once a or multiple serviceMonitors are found, that match the criteria in the selectors, Prometheus Operator adjusts the configuration in the actual Prometheus instance(tl;dr version) so you end up with proper scrape targets.
Step 1 for trouble shooting: Do your selectors match the labels and namespace of the serviceMonitors? Actually check those. The default on the prometheus operator helm chart expects a label release: prometheus-operator and in your config, you don't seem to add that to your json-exporter's serviceMonitor.
Step 2: The same behaviour as outline for how serviceMonitors are picked up, is happening in turn inside the serviceMonitor itself, so make sure that your service actually matches what is specced out in the serviceMonitor.
To deep dive further into the options you have and what the fields do, check the API documentation.

Helm function to set value based on a variable?

I'm learning Helm to setup my 3 AWS EKS clusters - sandbox, staging, and production.
How can I set up my templates so some values are derived based on which cluster the chart is being installed at? For example, in my myapp/templates/deployment.yaml I may want
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
I may want replicas to be either 1, 2, or 4 depending if I'm installing the chart in my sandbox, staging, or production cluster respectively? I wanna do same trick for cpu and memory requests and limits for my pods for example.
I was thinking of having something like this in my values.yaml file
environments:
- sandbox
- staging
- production
perClusterValues:
replicas:
- 1
- 2
- 4
cpu:
requests:
- 256m
- 512m
- 1024m
limits:
- 512m
- 1024m
- 2048m
memory:
requests:
- 1024Mi
- 1024Mi
- 2048Mi
limits:
- 2048Mi
- 2048Mi
- 3072Mi
So if I install a helm chart in the sandbox environment, I want to be able to do
$ helm install myapp myapp --set environment=sandbox
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
{{- if not .Values.autoscaling.enabled }}
# In pseudo-code, in my YAML files
# Get the index value from .Values.environments list
# based on pass-in environment parameter
{{ $myIndex = indexOf .Values.environments .Value.environment }}
replicas: {{ .Values.perClusterValues.replicas $myIndex }}
{{- end }}
I hope you understand my logic, but what is the correct syntax? Or is this even a good approach?
You can use the helm install -f option to pass an extra YAML values file in, and this takes precedence over the chart's own values.yaml file. So using exactly the template structure you already have, you can provide alternate values files
# sandbox.yaml
autoscaling:
enabled: false
replicaCount: 1
# production.yaml
autoscaling:
enabled: true
replicaCount: 5
And then when you go to deploy the chart, run it with
helm install myapp . -f production.yaml
(You can also helm install --set replicaCount=3 to override specific values, but the --set syntax is finicky and unusual; using a separate YAML file per environment is probably easier. Some tooling might be able to take advantage of JSON files also being valid YAML to write out additional deploy-time customizations.)

Dask Helm Chart - How to create Dask-CUDA-Worker nodes

I installed the Dask Helm Chart with a revised values.yaml to have 10 workers, however instead of Dask Workers I want to create Dash CUDA Workers to take advantage of the NVIDIA GPUs on my multi-node cluster.
I tried to modify the values.yaml as follows to get Dask CUDA workers instead of Dask Workers, but the worker pods are able to start. I did already install the NVIDIA GPUs on all my nodes on the Kubernetes per the official instructions so I'm not sure what DASK needs to see in order to create the Dask-Cuda-Workers.
worker:
name: worker
image:
repository: "daskdev/dask"
tag: 2.19.0
pullPolicy: IfNotPresent
dask_worker: "dask-cuda-worker"
#dask_worker: "dask-worker"
pullSecrets:
# - name: regcred
replicas: 15
default_resources: # overwritten by resource limits if they exist
cpu: 1
memory: "4GiB"
env:
# - name: EXTRA_APT_PACKAGES
# value: build-essential openssl
# - name: EXTRA_CONDA_PACKAGES
# value: numba xarray -c conda-forge
# - name: EXTRA_PIP_PACKAGES
# value: s3fs dask-ml --upgrade
resources: {}
# limits:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
# requests:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
As dask-cuda-worker is not yet officially in the dask image you will need to pull the image a different image: rapidsai/rapidsai:latest

How can I make an HELM UPGRADE with specific tag container version?

I am trying through Azure DevOps to launch a Pipeline that specifies the label of a specific version of the container (not latest). How can I do that?
previously to this requirement, I used:
helm upgrade --values=$(System.DefaultWorkingDirectory)/<FOLDER/NAME>.yaml --namespace <NAMESPACE> --install --reset-values --wait <NAME> .
I am trying through Azure DevOps to launch a Pipeline that specifies the label of a specific version of the container (not latest). How can I do that?
At the moment, it gives me errors with the flag "--app-version":
2020-06-25T15:43:51.9947356Z Error: unknown flag: --app-version
2020-06-25T15:43:51.9990453Z
2020-06-25T15:43:52.0054964Z ##[error]Bash exited with code '1'.
Maybe, another way is download from the harbor repository and make a helm roll to a version with these TAG. But I can´t find the way. I can´t see that clear.
YML:
# Default values for consent-sandbox.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
nameSpace: <NAME>-pre
image:
repository: <REPO>
pullPolicy: Always
## Uncomment and remove [] to download image private
imagePullSecrets: []
# - name: <namePullSecret>
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: false
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name:
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
containers:
portName: http
port: 8080
protocol: TCP
env:
APP_NAME: <NAME>
JAVA_OPTS_EXT: -Djava.security.egd=file:/dev/./urandom -Dcom.sun.net.ssl.checkRevocation=false -Djavax.net.ssl.trustStore=/etc/truststore/jssecacerts -Djavax.net.ssl.trustStorePassword=changeit
WILY_MOM_PORT: 5001
TZ: Europe/Madrid
spring_cloud_config_uri: https://<CONF>.local
spring_application_name: <NAME>
SPRING_CLOUD_CONFIG_PROFILE: pre
envSecrets: {}
livenessProbe: {}
# path: /
# port: 8080
readinessProbe: {}
# path: /
# port: 8080
service:
type: ClusterIP
portName: http
port: 8080
targetPort: 8080
containerPort: 8080
secret:
jks: <JKS>-jks
jssecacerts: jssecacerts
ingress:
enabled: false
route:
enabled: true
status: ""
# Default values for openshift-route.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
annotations:
# kubernetes.io/acme-tls: "true"
# haproxy.router.openshift.io/timeout: 5000ms
# haproxy.router.openshift.io/ip_whitelist: <IP>
labels:
host: "<HOST>.paas.cloudcenter.corp"
path: ""
wildcardPolicy: None
nameOverride: ""
fullnameOverride: ""
tls:
enabled: true
termination: edge
insecureEdgeTerminationPolicy: "None"
key:
certificate:
caCertificate:
destinationCACertificate:
service:
name: "<NAME"
targetPort: http
weight: 100
alternateBackends: []
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
cpu: 150m
memory: 1444Mi
requests:
cpu: 100m
memory: 1024Mi
nodeSelector: {}
tolerations: []
affinity: {}
Probably, I need add in the YML:
containers:
- name: my_container
image: my_image:latest
imagePullPolicy: "Always"
CHART:
apiVersion: v2
name: examplename
description: testing
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 1.0.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: latest
but...what can I do if I can´t change the YML?
Finally, I use other way with OC client:
oc patch deploy push-engine --type='json' -p '[{ "op": "replace", "path": "/spec/template/spec/containers/0/image", "value": "registry.sdi.dev.weu.azure.paas.cloudcenter.corp/test-dev/test:0.0.1" }]'