Prometheus deployed using kube-prometheus can't scrape resources on namespaces other than default, monitoring and kube-system. I added additional namespaces on my jsonnet as described in kube-prometheus README but no success...
I also tried to create a new ServiceMonitor manually, but no success...
I appreciate any help.
Thanks.
If you used the pre-compiled manifests here you will only have your service account with 3 rolebindings allowing access to the namespaces you mentioned.
You can add more namespaces for example by applying the same roleBinding in more namespaces.
This is more secure as opposed to using a clusterRoleBinding since it allows for more finegrained permissions.
Related
I recently learned about helm and how easy it is to deploy the whole prometheus stack for monitoring a Kubernetes cluster, so I decided to try it out on a staging cluster at my work.
I started by creating a dedicates namespace on the cluster for monitoring with:
kubectl create namespace monitoring
Then, with helm, I added the prometheus-community repo with:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, I installed the chart with a prometheus release name:
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
At this time I didn't pass any custom configuration because I'm still trying it out.
After the install is finished, it all looks good. I can access the prometheus dashboard with:
kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring
There, I see a bunch of pre-defined alerts and rules that are monitoring but the problem is that I don't quite understand how to create new rules to check the pods in the default namespace, where I actually have my services deployed.
I am looking at http://localhost:9090/graph to play around with the queries and I can't seem to use any that will give me metrics on my pods in the default namespace.
I am a bit overwhelmed with the amount of information so I would like to know what did I miss or what am I doing wrong here?
The Prometheus Operator includes several Custom Resource Definitions (CRDs) including ServiceMonitor (and PodMonitor). ServiceMonitor's are used to define services to the Operator to be monitored.
I'm familiar with the Operator although not the Helm deployment but I suspect you'll want to create ServiceMonitors to generate metrics for your apps in any (including default) namespace.
See: https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions
ServiceMonitors and PodMonitors are CRDs for Prometheus Operator. When working directly with Prometheus helm chart (without operator), you need have to configure your targets directly in values.yaml by editing the scrape_configs section.
It is more complex to do it, so take a deep breath and start by reading this: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
I am trying to use Helm 3 to install Kubeflow 1.3 with Istio 1.9 on Kubernetes 1.16. Kubeflow does not provide official Helm chart so I figured it out by myself.
But Helm does not guarantee order. Pods of other deployments and statefulsets could be up before Istio mutating webhook and istiod are up. For example, if A pod is up earlier without istio-proxy, B pod is up later with a istio-proxy, they cannot communicate with each other.
Are there any simple best practices so I can work this out as expected each time I deploy? That is say, make sure my installation with Helm is atomic?
Thank you in advance.
UPDATE:
I tried for three ways:
mark resources as pre-install, post-install, etc.
using subcharts
decouple one chart into several charts
And I adopted the third. The issue of the first is that helm hook is designed for Job, a resource could be marked as helm hook but it would not be deleted when using helm uninstall since a resource cannot hold two helm hooks at the same time(key conflict in annotations). The issue of the second is that helm installs subcharts and charts at the same time, helm call hooks of subcharts and charts at the same time as well.
Helm does not guarantee order.
Not completely. Helm collects all of the resources in a given Chart and it's dependencies, groups them by resource type, and then installs them in the following order:
Namespace
NetworkPolicy
ResourceQuota
LimitRange
PodSecurityPolicy
PodDisruptionBudget
ServiceAccount
Secret
SecretList
ConfigMap
StorageClass
PersistentVolume
PersistentVolumeClaim
CustomResourceDefinition
ClusterRole
ClusterRoleList
ClusterRoleBinding
ClusterRoleBindingList
Role
RoleList
RoleBinding
RoleBindingList
Service
DaemonSet
Pod
ReplicationController
ReplicaSet
Deployment
HorizontalPodAutoscaler
StatefulSet
Job
CronJob
Ingress
APIService
Additionally:
That is say, make sure my installation with Helm is atomic
you should to know that:
Helm does not wait until all of the resources are running before it exits.
You generally have no control over the order if you are using Helm. You can try to use Init Containers to validate your pods to check if they have all dependencies before they run. You can read more about it here. Another workaround will be to install a health check to make sure everything is okay. If not, it will restart until it is successful.
See also:
this article about checking your helm deployments.
question Helm Subchart order of execution in an umbrella chart with good explanation
this question
related topic on github
I found a lot of information on how to give helm permission to create resources in a particular namespace.
I am trying to see if I can create namespaces on the fly(with random names) and then use helm to install and delete resources inside the namespace.
My idea is to create a namespace with name such as Fixedsuffix-randomprefix and then allow helm to create all resources inside it. Is this possible ?
I can create a clusterrole and clusterrolebinding to allow tiller's serviceaccount to create namespaces, but I am not able to figure out how to have a serviceaccount that could create resources in the particular namespace( mainly because this serviceaccount to create resources cant would have to be created when the namespace is created and then assigned to tiller pod).
TIA
My question is why would you create sa, clusterrole and rolebinding to do that? Helm has it´s own resources which allow him to install and delete resources inside new namespace.
My idea is to create a namespace with name such as Fixedsuffix-randomprefix and then allow helm to create all resources inside it. Is this possible ?
Yes, you can create your new namespace and use helm to install everything in this namespace.Or even better you can just use helm install and it will create new namespace for you. For that purpose helm have helm install --namespace.
-n, --namespace string namespace scope for this request
For example you can install traefik chart in namespace tla.
helm install stable/traefik --namespace=tla
NAME: oily-beetle
LAST DEPLOYED: Tue Mar 24 07:33:03 2020
NAMESPACE: tla
STATUS: DEPLOYED
Another idea which came to my mind is you might want tiller not to use cluster-admin credentials, then this link could help.
We have multiple development teams who work and deploy their applications on kuberenetes. We use helm to deploy our application on kubernetes.
Currently the challenge we are facing with one of our shared clusters. We would like to deploy tiller separate for each team. So they have access to their resources. default Cluster-admin role will not help us and we don't want that.
Let's say we have multiple namespaces for one team. I would want to deploy tiller which has permission to work with resources exist or need to be created in these namespaces.
Team > multiple namespaces
tiller using the service account that has the role ( having full access to namespaces - not all ) associated with it.
I would want to deploy tiller which has permission to work with resources exist or need to be created in these namespaces
According to the fine manual, you'll need a ClusterRole per team, defining the kinds of operations on the kinds of resources, but then use a RoleBinding to scope those rules to a specific namespace. The two ends of the binding target will be the team's tiller's ServiceAccount and the team's ClusterRole, and then one RoleBinding instance per Namespace (even though they will be textually identical except for the namespace: portion)
I actually would expect you could make an internal helm chart that would automate the specifics of that relationship, and then helm install --name team-alpha --set team-namespaces=ns-alpha,ns-beta my-awesome-chart and then grant your tiller cluster-admin or whatever more restrictive ClusterRole you wish.
When I deploy a deamonset in kubernetes(1.7+), i.e, nginx ingress as daemonset, do I need to set some rbac rule ??? I know I need to set some rbac rule if I use deployment.
To deploy ingress, you need to enable some RBAC rules. In the nginx controller repository you can find the RBAC rules: https://github.com/kubernetes/ingress/blob/master/examples/rbac/nginx/nginx-ingress-controller-rbac.yml
To create daemonset you don't need to create RBAC rules for it. You might need RBAC for what is running in your Pod, be it via Deployment, Daemonset or whatever. It is the software you're running inside that might want to interact with kubernetes API, like it is in case of Ingress Controller. So, it is in fact irrelevant how you make the Pod happen, the RBAC (Cluster)Role, bindings etc. It is what soft you deploy, that defines what access rules it needs.
I was able to enable RBAC using helm (--set rbac.create=true) and this error is not seen anymore, and the nginx ingress controller is working as expected!
helm install --name my-release stable/nginx-ingress --set rbac.create=true