Monitoring Kubernetes cluster using prometheus outside the k8 cluster - kubernetes

Question

We have kubernetes cluster where I have service account "kube", namespace "monitoring" with cluster role binding created to monitor cluster
We have prometheus installed on a linux system (on prem) outside the cluster and is installed using "root"
When I try to connect to the k8 cluster with the https api using ca.crt and user token (given by kubernetes admin), it throws multiple errors.
Error messages:
component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig
component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig
Prometheus configuration:
- job_name: 'kubernetes-apiservers'
scheme: https
tls_config:
ca_file: /root/prometheus/ca.crt
bearer_token_file: /root/prometheus/user_token
kubernetes_sd_configs:
- role: endpoints
api_server: https://example.com:1234
bearer_token_file: /root/prometheus/user_token
tls_config:
ca_file: /root/prometheus/prometheus-2.12.0.linux-amd64/ca.crt
relabel_configs:
- source_labels: [monitoring, monitoring-sa, 6443]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /root/prometheus/ca.crt
bearer_token_file: /root/prometheus/user_token
kubernetes_sd_configs:
- role: node
api_server: https://example.com:1234
bearer_token_file: /root/prometheus/user_token
tls_config:
ca_file: /root/prometheus/ca.crt
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: https://example.com:1234
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics

Answer 1

The main problem you're facing is: "unable to use specified CA cert /root/prometheus/ca.crt"
Someone recently faced the same problem:
https://github.com/prometheus/prometheus/issues/6015#issuecomment-532058465
He solved it by reinstalling the new version.
Version 2.13.1 is out. Try installing the latest version, it might solve your problem too.

Answer 2

Your ca.crt is most probably still in base64 format since secrets are encoded that way when describing them, as explained here.

Answer 3

Maybe your ca.crt have some error, check your ca cert file, make sure this file format like this:
-----BEGIN CERTIFICATE-----
xxxxx
-----END CERTIFICATE-----
I think your ca.crt is get by kubectl get serviceaccount -o yaml, but this is a public key with your kubernetes cluster, so, if you want to get the token, you can specify the serviceAccountName in the yaml file with a new Deployment, like this:
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: test
spec:
replicas: 1
template:
metadata:
labels:
app: test
version: v1
spec:
serviceAccountName: prometheus
containers:
- name: test
image: alpine
imagePullPolicy: Always
command: ["ping", "127.0.0.1"]
imagePullSecrets:
- name: harbor-secret
restartPolicy: Always
Then, get your token and ca.crt under /var/run/secrets/kubernetes.io/serviceaccount/.