I have a query related with prometheus-operator helm chart & alert manager combination.
Currently we are using prometheus-operator helm chart:
https://github.com/helm/charts/tree/master/stable/prometheus-operator
and I wrote a simple rule in values.yml (this is just a sample code) to generate an alert:
further I am using alertmanager config/routes/receivers to send alerts. It's working perfectly fine.
But as part of real-time implementation, I may be having so many alert rules. Is there any way where I can bring these all rules in separate rules file & configure the path (rule file path) in values.yml (under: additionalPrometheusRules section)
I also saw kube-prometheus-stack & additionalPrometheusRulesMap (in values.yml):
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
But didn't fine any solution. Anyone can help me on this?
So helm doesn't allow includes in values.yaml files typically. I read that there's a way to do it, but it depends on how the chart is built and typically upstream maintainers don't use templates that way afaik (could be wrong there, but I've never noticed it).
Your problem is exactly the same problem I've been trying to solve adequately, and I think I came up with something. It's not perfect, but it is better than having one huge monolithic values.yaml file.
helm allows the operator to specify multiple values.yaml files using the paradigm, -f values1.yaml -f values2.yaml -f some-more-values.yaml, so I broke my values file up into multiple logically-divided yaml files.
There might be gotchas, so be aware, but so far for this use-case, it seems to be working. I'm still testing things out. https://helm.sh/docs/helm/helm_install/
You can also add your own custom rules file using config maps. In this way, you can avoid over alerting and get notified for specific alerts only.
Related
Currently I'm working with Kubeflow. It is a quite large set up with about 30 different deployments. The default manifests of the Kubeflow team is using the standard Kustomize tool to provide patches for different environments like on-prem, cloud, testing, etc.
However, I still feel quite overwhelmed and limited with all those configurations. The only way that I can quickly navigate and manipulate variables for the whole project is to use the search and/or replace function from IDE (yes I know Kustomize can do variables in each environment but I want to do it for all deployments). This sucks, as it is not reversible once replaced. Another problem is that many folders are not just named base or reverse so many folders are named base, making it quite hard to findout some original fields. I also struggled when I want to combine just a few parts of different environment. For example, they provided 3 different environment for dex auth, 1 using email, 1 using GitHub and 1 using Google. I want my setup to have all 3 of them but now I cannot reuse the config from those environment easily. I had to create my own environment and dig up each of those environment to see what changes did they make in the patches.
I have never tried setting up Jsonnet configurations (maybe with tools like Tanka) on a similar cluster. I'm seeing a few big companies using Jsonnet. The two that I know are Grafana (they even created Tanka) and Databricks (they create their own Jsonnet compiler). What are the pros and cons of doing k8s configurations management in Jsonnet compared to Kustomize - the currently most popular choice? Is it worth learning and managing k8s config using Jsonnet (maybe with Tanka)?
What about ArgoCD?
Argo CD is a git-opt tool for managing projects like you describe.
It support Kustomization, Jsonnet and more
It can manage your resources, shows you what is deployed, synced, edit yor YAML file, and more.
lately I was experimenting with Skaffold with our Helm Charts and I am in little bit in a dilemma that our Helm Chart \ Sub Charts are compatible with Skaffold or not.
Our helm Charts are looking like the following
my-helm-charts
+-charts
+-project1
+-project2
+-project3
+-project4
+-infrastructure_kafka
+-charts
+-kafka
+-zookeeper
+-infrastructure_cassandra
+-infrastructure_elasticsearch
+-Charts.yaml
+-Values.yaml
The reason we choosed to structure the Helm Charts this way, is that if necessary to spin up extra stages for our project.
Now when I want to develop project2 with Google Cloud Code / Skaffold (which I configured correctly and I can start without problem in IntelliJ) I have to start whole my-helm-charts.
That is actually Ok but the problem is, if I use Debug in Kubernetes, I have a feeling Google Cloud Code/Skaffold can really locate the project2 and no debugging occurs.
My feeling is Google Cloud/Skaffold is more oriented to work with following contruct...
project2-helm
+-templates
+-Charts.yaml
+-Values.yaml
My Subcharts contructs starts in Google Cloud Code/Skaffold without any exception but I can't debug, is it possible to achieve want I want with my structure and if yes, how?
Or is it not possible at all...
Thx for answers...
We recently added a feature called config dependencies which might help here. It allows you to create more specific skaffold.yamls and then map them together with a "requires" field:
https://skaffold.dev/docs/design/config/#configuration-dependencies
Once you have the skaffold.yamls created and the right dependency mapping you can run skaffold with the -m flag to choose once slice of your services:
skaffold dev -m project3
Cloud Code support for modules is incoming.
Cloud Code IntelliJ and Cloud Code VS Code recently added preview level support for deploying and debugging modules of a larger application which uses Skaffold. See more here https://cloud.google.com/code/docs/intellij/skaffold-modules
In a Kubernetes operator based on operator-sdk, do you know how to write code to synchronize CR resource when CR specification is updated with kubectl apply? Could you please provide some code samples?
It is mostly up to how you deploy things. The default skeleton gives you a Kustomize-based deployment structure so kustomize build config/default | kubectl apply -f. This is also wrapped up for you behind make deploy. There is also make install for just installing the generated CRD files.
Implementing it a go-lang based operator is pretty complex, and I would recommand studying the kubebuilder documentation and example in order to achieve that: https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html#implementing-a-controller
I have just started experimenting with Helm kubernetes package manager.
But chart vs template topic seems a bit confusing to me.
I understand that by template I will create kubernetes yaml, which will create the objects and install them.
However the same is true for charts as well, but this latter is an abstraction over the yamls. And ./Charts containns standalone charts, while ./templates is valid only for the base chart. So I know that. But when should I include an other chart or just create a template?
Looking for different kind of charts through the web I still don't know which to use.
Say I have a project called MyApp, which has one component named MyServer which will communicate to MySql.
So I created a chart and put in it MyServer as a template :
./MyApp/templates/MyServer.yaml
What should I do with MySql?
I have seen both solutions in different projects, one just creates an other template:
./MyApp/templates/MySQL.yaml
on other project I saw a chart for MySql from a chart repository:
./MyApp/charts/mysql-version.tgz
On the top of that I have seen a bigdata project (hdfs,kafka,zookeeper,ELK,oracle db..etc) and one component was included as chart in ./charts other was created as a template in./templates.
This whole decision between chart and template seems random and confusing to me.
Could you explain it please when to use which?
A chart is a collection of templates, plus a little extra information like the metadata in the Chart.yaml file and the default values.yaml. In your example, MyApp is itself a chart.
For well-known dependencies (particularly things in the Helm charts repository and especially the stable charts) you're probably better off using the external chart; declare the dependency in your requirements.yaml or (Helm v3) Chart.yaml file and run helm dependency update. This lets you import the chart with two lines, rather than reproducing the StatefulSet, PersistentVolumeClaim, etc. that are included in the chart.
I'd like to check that my kubernetes helm chart does not define unused values in values.yaml. This should include any subcharts such that if you've defined subchart.foo.bar: ??? in the top-level values.yaml that key is definitely used in the subchart, or possibly as a short-cut mentioned in the subchart/values.yaml.
This is needed to prevent us from shipping bogus "documentation" in the values.yaml, for example if a key in a subchart has been changed or removed.
Ideally there would also be some possibility to report on which subchart values have not been overridden in the top-level chart, though this is less concerning.
Are there any existing tools that can help with this?
Since the Helm v3 release you can now define a schema for your values. On commands like helm install your provided values are automatically validated against the schema.
Please see the official documentation: https://helm.sh/docs/topics/charts/#schema-files
Schema validation works for subcharts too, this is also mentioned in the documentation on the link above.
AFAIK, there isn't a tool for that. However, it shouldn't be that hard to make one, even using bash. For example, you need to export all key/value pairs like this test.test1.test2 and grep for that string recursively in the templates folder. If you want to read yaml using bash, you can install shyaml. If you know how to code in Python, even better.
helm lint --detect-unused-values