Using the Helm Charts, is there a trick to get the Grafana to be in Persistence Mode. I have tried PVC and Stateful Sets. Neither will allow the pod to spin up. I have tried starting with no PV or PVC pre-created and with. Any pointers?
Related
I recently learned about helm and how easy it is to deploy the whole prometheus stack for monitoring a Kubernetes cluster, so I decided to try it out on a staging cluster at my work.
I started by creating a dedicates namespace on the cluster for monitoring with:
kubectl create namespace monitoring
Then, with helm, I added the prometheus-community repo with:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, I installed the chart with a prometheus release name:
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
At this time I didn't pass any custom configuration because I'm still trying it out.
After the install is finished, it all looks good. I can access the prometheus dashboard with:
kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring
There, I see a bunch of pre-defined alerts and rules that are monitoring but the problem is that I don't quite understand how to create new rules to check the pods in the default namespace, where I actually have my services deployed.
I am looking at http://localhost:9090/graph to play around with the queries and I can't seem to use any that will give me metrics on my pods in the default namespace.
I am a bit overwhelmed with the amount of information so I would like to know what did I miss or what am I doing wrong here?
The Prometheus Operator includes several Custom Resource Definitions (CRDs) including ServiceMonitor (and PodMonitor). ServiceMonitor's are used to define services to the Operator to be monitored.
I'm familiar with the Operator although not the Helm deployment but I suspect you'll want to create ServiceMonitors to generate metrics for your apps in any (including default) namespace.
See: https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions
ServiceMonitors and PodMonitors are CRDs for Prometheus Operator. When working directly with Prometheus helm chart (without operator), you need have to configure your targets directly in values.yaml by editing the scrape_configs section.
It is more complex to do it, so take a deep breath and start by reading this: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
I need to start by saying that I have no experience using Cassandra and I am not the one who who created this deployment.
I have Cassandra running in a cluster in AKS. The PVC as configured in the statefulset is 1000Gi. Currently the pods are out of storage and are in a constant unhealthy state.
I am looking to expand the volumes attached to the pods. The problem I am facing is that I cannot scale down the statefulset because the statefulsets only scale down when all their pods are healthy.
I even tried deleting the statefulset and then recreateing it with a larger PVC (as recomended here)
Howerver, I can't seem to delete the statefulset. It looks to me like the CassandraDatacenter CRD keeps recreating the statefulset as soon as I delete it. Giving me no time to change anything.
My question are as follows:
Is there a standard way to expand the volume without losing data?
What would happen if I scale down the replicas in the CassandraDatacenter? Will it delete the PVC or keep it?
If there is no standard, does anyone have any ideas on how to accomplish expanding the volume size without losing storage?
Ordinarily in a Cassandra cluster, the best practice is to scale horizontally (not vertically). You want more Cassandra nodes to spread the load out to achieve maximum throughput.
The equivalent in Kubernetes is to scale up your deployment. As you increase the node count, the amount of data on each individual Cassandra node will decrease proportionally.
If you really want to resize the PVC, you will only be able to do it dynamically if you have enabled allowVolumeExpansion. You won't lose data as you do this.
Deleting a STS isn't going to work because by design it will be automatically replaced as you already know. You also won't be able to scale down because there isn't enough capacity (disk space) in your cluster if you do. Cheers!
Answer for: How can I expand a PVC for StatefulSet on AKS without loosing data?
While the answer of #Erick Raminez is a very good advice for Cassandra specific, I would like to answers the more general question "How can I expand a PVC for my StatefulSet on AKS without loosing data?".
If downtime is allowed, you can follow these 4 steps:
# Delete StatefulSet
# This is required on AKS since the azure disk must have status "Unattached"
kubectl delete statefulsets.apps STATEFULSET_NAME
# Edit capacity in
# - your statefulset yaml file
# - each pvc
kubectl patch pvc PVC_NAME -p '{"spec": {"resources": {"requests": {"storage": "3Gi"}}}}'
# Deploy updated statefulset yaml (or helm chart)
kubectl apply -f statefulset.yaml
# Verify Capacity
kubectl get pvc
If you don't want downtime, check the first reference for some additional steps.
References:
https://adamrushuk.github.io/increasing-the-volumeclaimtemplates-disk-size-in-a-statefulset-on-aks/
https://serverfault.com/a/989665/561107
I need to deploy NGINX to a Kubernetes cluster, for which I can either use a Helm chart or a Docker image. But I am not clear of the benefits of using a Helm chart. I guess my question is not specific to NGINX but in general.
A helm chart and a container image aren't equivalent things to compare in Kubernetes
A container image is the basic building block of what kubernetes runs. An image will always be required to run an application on kubernetes, no matter how it is deployed.
Helm is a packaging and deployment tool. It makes management of deployments to kubernetes easier. This deployment would normally include a container image. It is possible to write a helm chart that just manages other kubernetes resources but fairly rare.
Other tools in the same arena as helm are kustomize, kompose, or using kubectl to apply or create resources. These are all clients of the kubernetes API.
Helm Charts: making it simple to package and deploy common applications on Kubernetes [1]. Helm brings three major benefits to your service deployments [2]:
Deployment speed
Helm chart on Kubernetes for application configuration templates
Application testing
Use of Helm charts is recommended, because they are maintained and typically kept up to date by the Kubernetes community [3].
[1] https://kubernetes.io/blog/2016/10/helm-charts-making-it-simple-to-package-and-deploy-apps-on-kubernetes/
[2] https://www.nebulaworks.com/blog/2019/10/30/three-benefits-to-using-a-helm-chart-on-kubernetes/
[3] https://cloud.google.com/community/tutorials/nginx-ingress-gke
When installing consul using Helm, it expects the cluster to dynamic provison the PersistentVolume requested by consul-helm chart. It is the default behavior.
I have the PV and PVC created manually and need to use this PV to be used by consul-helm charts. Is it posisble to install consul using helm to use manually created PV in kubernetes.
As #coderanger said
For this to be directly supported the chart author would have to provide helm variables you could set. Check the docs.
As showed on github docs there is no variables to change that.
If You have to change it, You would have to work with consul-statefulset.yaml, this chart provide dynamically volumes for each statefulset pod created.
volumeMounts
volumeClaimTemplates
Use helm fetch to download consul files to your local directory
helm fetch stable/consul --untar
Then i found a github answer with good explain and example about using one PV & PVC in all replicas of Statefulset, so I think it could actually work in consul chart.
I've deployed Prometheus and Grafana via helm by issuing the following command.
helm install --name prom --namespace monitoring stable/Prometheus-operator
I've managed to use kubectl to port forward to grafana, and I can view the interface just fine.
However, most of my pods are not generating any data points. I've tried re-deploying the workloads as I figured maybe that was necessary for Prometheus to start picking up the metrics. I can select the pods in the workloads in Grafana drop-down menus so they are being detected, but they are not generating any data points or populating the graphs.
What do I need to do to make this happen?
Thanks in advance.