I am trying to setup Prometheus for my Kubernetes cluster. I am using https://github.com/coreos/kube-prometheus . I am using default namespace only. When I apply resources from manifests (https://github.com/coreos/kube-prometheus/tree/master/manifests) folder, It creates resources but in targets, it doesn't show my pod service.
I am new to Kubernetes so I need help with this. How do I configure my pod to show up in Prometheus targets?
I tried https://github.com/coreos/kube-prometheus#quickstart
For better manageability use Prometheus Operator.
You need to define ServiceMonitor or PodMonitor
ServiceMonitor which declaratively specifies how groups of services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition and the targets will have the IPs of all the pods behind the service.
PodMonitor, which declaratively specifies how groups of pods should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition and targets will have the Pod IP
Refer to the examples here
Related
Is it possible to create a Kubernetes service and pod in different namespaces, for example, having myweb-svc pointing to the actual running myweb-pod, while myweb-svc and myweb-pod are in different namespaces?
YAML manifest to create both the pod and the service in their respective namespaces. You need to specify the ‘namespace’ field in the ‘metadata’ section of both the ‘pod’ and ‘service’ objects to specify the namespace in which they should be created.
Also, if you want to point your Service to a Service in a different namespace or on another cluster you can use service without a pod selector.
Refer to this link on Understanding kubernetes Object for more information.
Kubernetes API objects that are connected together at the API layer generally need to be in the same namespace. So a Service can only connect to Pods in its own namespace; if a Pod references a ConfigMap or a Secret or a PersistentVolumeClaim, those need to be in the same namespace as well.
I recently learned about helm and how easy it is to deploy the whole prometheus stack for monitoring a Kubernetes cluster, so I decided to try it out on a staging cluster at my work.
I started by creating a dedicates namespace on the cluster for monitoring with:
kubectl create namespace monitoring
Then, with helm, I added the prometheus-community repo with:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Next, I installed the chart with a prometheus release name:
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
At this time I didn't pass any custom configuration because I'm still trying it out.
After the install is finished, it all looks good. I can access the prometheus dashboard with:
kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring
There, I see a bunch of pre-defined alerts and rules that are monitoring but the problem is that I don't quite understand how to create new rules to check the pods in the default namespace, where I actually have my services deployed.
I am looking at http://localhost:9090/graph to play around with the queries and I can't seem to use any that will give me metrics on my pods in the default namespace.
I am a bit overwhelmed with the amount of information so I would like to know what did I miss or what am I doing wrong here?
The Prometheus Operator includes several Custom Resource Definitions (CRDs) including ServiceMonitor (and PodMonitor). ServiceMonitor's are used to define services to the Operator to be monitored.
I'm familiar with the Operator although not the Helm deployment but I suspect you'll want to create ServiceMonitors to generate metrics for your apps in any (including default) namespace.
See: https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions
ServiceMonitors and PodMonitors are CRDs for Prometheus Operator. When working directly with Prometheus helm chart (without operator), you need have to configure your targets directly in values.yaml by editing the scrape_configs section.
It is more complex to do it, so take a deep breath and start by reading this: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
Manifest(yml) with kubernetes resource type(kind: Namespace) can be applied through kubectl to create a virtual cluster
In our environment, manifest yaml's are applied using kubectl to create kubernetes resource types(deployment, service, autoscaling, ingress) under the given namespace
But, rancher is used to create kubernetes resource type(kind: Namespace virtual cluster).
What is the advantage of creating kubernetes resource type(Namespace) using rancher? instead of a manifest yaml applied through kubectl
Rancher uses concept of "Project" which is not present in "vanilla" kubernetes, which allows you to assign RBAC roles, PodSecurityPolicy etc to a group of namespaces in easy way.
If you are not using rancher to create projects and namespaces - you have to assign all these Roles and PSPs by yourself. For example, if you have default restricted policy on your cluster, namespace created by kubectl create namespace foo won't be able to run any pods by default, see https://rancher.com/docs/rancher/v2.5/en/admin-settings/pod-security-policies/
Namespaces that are not assigned to projects do not inherit PSPs, regardless of whether the PSP is assigned to a cluster or project. Because these namespaces have no PSPs, workload deployments to these namespaces will fail, which is the default Kubernetes behavior.
To sum it up, namespaces can be created using kubectl create namespace or manifests, but it might be cumbersome to make it all work well. Using rancher to provision namespaces is easier to maintain and troubleshoot.
As for advantages, having ability to group namespaces under "project" and assign resources, PSP and roles to a group of namespaces with rancher UI support is one of the main selling points of having rancher in a first place. Namespace objects themselves are basically the same as anywhere else.
I have a pod running in a statefulset but it needs to know the hostname or address of all pods running in another statefulset to communicate with them. The second statefulset is being created by a separate helm chart. Can the pod work this out dynamically? Can I inject this information into the pod through an env similar to setting .Status.ip?
Edit: Each statefulSet has its own headless service
As discussed in the comments, the way to go here is to use a service-resource as this will give you a static DNS within the cluster to reach all the pods that a targeted by that service.
The DNS for the service is:
the services name if you access it from within the same namespace
<my-service-name>.<namespace-name>.svc.cluster.local if you access it from another namespace, and where cluster.local is the clusters domain that might differ from cluster to cluster depending on the clusters configuration
If you further need more configuration options, e.g. when you want to deploy your chart into different cloud environments where the clusters-domain might actually differ, you can use kustomize.io to adjust your configuration at apply time.
We have a kubernetes cluster for the entire organisation. And there are namespaces for individual teams to host their projects in that cluster. The problem seems to be monitoring individual namespace. Since we have only access to our namespace we can't setup any monitoring for the PODs, containers, or the nodes on which our pods are residing. Is there any way to accomplish monitoring of only the things within the namespace without bothering the kubernetes cluster?
Monitoring should be provided on central level for all the basics of the cluster and more, but sure, you just need limited scope. If you deploy prometheus and configure only targets in your namespace you should be ok. Same goes for any other solutions.
For reference see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E
# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
names:
[ - <string> ]