Cannot see the target added to service monitor for Prometheus Operator - service

Question

I am trying to set up to add the target to my service monitor for Prometheus Operator (inside my terraform that is using helm chart to deploy prometheus, prometheus operator and service monitor and a bunch of stuff).
After I successfully deployed service monitor, I cannot see the new target app.kubernetes.io/instance: jobs-manager in prometheus. I am not sure what I did wrong in my configuration. I am also checking this document to see what is missing but cannot figure it out yet.
Here are some configuration files concerned:
/helm/charts/prometheus-abcd/templates/service_monitor.tpl
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: jobs-manager-servicemonitor
# Change this to the namespace the Prometheus instance is running in
namespace: prometheus
labels:
app: jobs-manager
release: prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/instance: jobs-manager # Targets jobs-manager service
endpoints:
- port: http
interval: 15s
/helm/charts/prometheus-abcd/Chart.yaml
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
apiVersion: v1
appVersion: "1.0.0"
description: Prometheus Service monitor, customized for abcd
name: prometheus-abcd
version: 1.0.0
/terraform/kubernetes/helm_values/prometheus.yaml
prometheus:
podMetadata:
annotations:
container.apparmor.security.beta.kubernetes.io/prometheus-operator: runtime/default
seccomp.security.alpha.kubernetes.io/pod: runtime/default
nodeAffinityPreset:
## Node affinity type
## Allowed values: soft, hard
##
type: "hard"
## Node label key to match
## E.g.
## key: "kubernetes.io/e2e-az-name"
##
key: "cloud.google.com/gke-nodepool"
## Node label values to match
## E.g.
## values:
## - e2e-az1
## - e2e-az2
##
values: [
"abcd-primary-pool"
]
prometheus:
configMaps:
- prometheus-config
## ServiceMonitors to be selected for target discovery.
## If {}, select all ServiceMonitors
##
serviceMonitorSelector: {
jobs-manager-servicemonitor
}
# matchLabels:
# foo: bar
## Namespaces to be selected for ServiceMonitor discovery.
## See https://github.com/prometheusoperator/prometheusoperator/blob/master/
## Documentation/api.md#namespaceselector for usage
##
serviceMonitorNamespaceSelector: {
matchNames: prometheus
}
When running this command:
kubectl get -n prometheus prometheuses.monitoring.coreos.com prometheus-kube-prometheus-prometheus
I can see that the service monitor was successfully deployed:
But when I run this command:
kubectl describe -n prometheus prometheuses.monitoring.coreos.com prometheus-kube-prometheus-prometheus
I see that many parameters still have missing values such as serviceMonitorSelector
Name: prometheus-kube-prometheus-prometheus
Namespace: prometheus
Labels: app.kubernetes.io/component=prometheus
app.kubernetes.io/instance=prometheus
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-prometheus
helm.sh/chart=kube-prometheus-3.4.0
Annotations: meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: prometheus
API Version: monitoring.coreos.com/v1
Kind: Prometheus
Metadata:
Creation Timestamp: 2021-05-26T15:19:42Z
Generation: 1
Managed Fields:
API Version: monitoring.coreos.com/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/component:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/managed-by:
f:app.kubernetes.io/name:
f:helm.sh/chart:
f:spec:
.:
f:affinity:
.:
f:podAntiAffinity:
.:
f:preferredDuringSchedulingIgnoredDuringExecution:
f:alerting:
.:
f:alertmanagers:
f:configMaps:
f:enableAdminAPI:
f:externalUrl:
f:image:
f:listenLocal:
f:logFormat:
f:logLevel:
f:paused:
f:podMetadata:
.:
f:labels:
.:
f:app.kubernetes.io/component:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/name:
f:podMonitorNamespaceSelector:
f:podMonitorSelector:
f:probeNamespaceSelector:
f:probeSelector:
f:replicas:
f:retention:
f:routePrefix:
f:ruleNamespaceSelector:
f:ruleSelector:
f:securityContext:
.:
f:fsGroup:
f:runAsUser:
f:serviceAccountName:
f:serviceMonitorNamespaceSelector:
f:serviceMonitorSelector:
Manager: Go-http-client
Operation: Update
Time: 2021-05-26T15:19:42Z
Resource Version: 11485229
Self Link: /apis/monitoring.coreos.com/v1/namespaces/prometheus/prometheuses/prometheus-kube-prometheus-prometheus
UID: xxxxxxxxxxxxxxxxxxxx
Spec:
Affinity:
Pod Anti Affinity:
Preferred During Scheduling Ignored During Execution:
Pod Affinity Term:
Label Selector:
Match Labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: kube-prometheus
Namespaces:
prometheus
Topology Key: kubernetes.io/hostname
Weight: 1
Alerting:
Alertmanagers:
Name: prometheus-kube-prometheus-alertmanager
Namespace: prometheus
Path Prefix: /
Port: http
Config Maps:
prometheus-config
Enable Admin API: false
External URL: http://prometheus-kube-prometheus-prometheus.prometheus:9090/
Image: docker.io/bitnami/prometheus:2.24.0-debian-10-r1
Listen Local: false
Log Format: logfmt
Log Level: info
Paused: false
Pod Metadata:
Labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/name: kube-prometheus
Pod Monitor Namespace Selector:
Pod Monitor Selector:
Probe Namespace Selector:
Probe Selector:
Replicas: 1
Retention: 10d
Route Prefix: /
Rule Namespace Selector:
Rule Selector:
Security Context:
Fs Group: 1001
Run As User: 1001
Service Account Name: prometheus-kube-prometheus-prometheus
Service Monitor Namespace Selector:
Service Monitor Selector:
Events: <none>
This is why I check this document to get the template for serviceMonitorSelector and also serviceMonitorNamespaceSelector and added them to the prometheus.yaml file above but not sure if it is correctly added.
Anyone with experience setting up service monitor with helm and terraform, could you please help me check what I did wrong? Thank you in advance.

Answer 1

the way you have passed value in prometheus.yaml is wrong
serviceMonitorNamespaceSelector: {
matchNames: prometheus
} #this is wrong way
you should have to set the values like :
serviceMonitorNamespaceSelector:
matchLabels:
prometheus: somelabel
also same for
serviceMonitorSelector: {
jobs-manager-servicemonitor
}
it's not set proper way.
for reference please check : https://github.com/prometheus-community/helm-charts/blob/83aa113f52e5f45fd04b4dd909172a6da1826592/charts/kube-prometheus-stack/values.yaml#L2034
checkout this nice example : https://rtfm.co.ua/en/kubernetes-a-clusters-monitoring-with-the-prometheus-operator/
Prometheus operator with Terraform & helm : https://github.com/OpenQAI/terraform-helm-release-prometheus-operator