I'm trying to configure Prometheus outside Kubernetes Cluster.
Below is my Prometheus config.
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
api_server: https://10.0.4.155:6443
scheme: https
tls_config:
insecure_skip_verify: true
basic_auth:
username: kube
password: Superkube01
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
These is how it looks:
root#master01:~# kubectl cluster-info
Kubernetes master is running at https://10.0.4.155:6443
root#master01:~# kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 10.0.4.103:6443,10.0.4.138:6443,10.0.4.155:6443 11h
netchecker-service 10.2.0.10:8081 11h
root#master01:~#
But, when starting Prometheus, i'm getting below error.
level=error ts=2018-05-29T13:55:08.171451623Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:270: Failed to list *v1.Pod: Get https://10.0.4.155:6443/api/v1/pods?resourceVersion=0: x509: certificate signed by unknown authority"
Could anyone please tell me, what wrong i'm doing here?
Thanks,
Pavanasam R
The error indicates that Prometheus is using a different certificate to sign its metric collection request than the one expected by your apiserver.
You really need to format your code in a code block so we can see the yaml formatting. kubernetes_sd_configs seems to be the wrong home for insecure_skip_verify and basic_auth according to this link. Might want to move them and try scraping again.
As of now your insecure_skip_verify is a part of kubernetes_sd_configs:. Add it in api_server context as well.
kubernetes_sd_configs:
- api_server: https://<ip>:6443
role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
in order to access kubernetes api endpoint you need to authenticate the client either through basic_auth, bearer_token, tls_config. please go through this , it will be helpful.
Related
I have configured Prometheus on one of the kubernetes cluster nodes using [this][1]. After that I added following prometheus.yml file. I can list nodes and apiservers but for pods, all the pods shows down and error:
Get "https:// xx.xx.xx:443 /metrics": dial tcp xx.xx.xx:443: connect: connection refused and for some pods the status is unknown.
Can someone point me what am I doing wrong here?
Cat prometheus.yml
global:
scrape_interval: 1m
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: \['localhost:9090'\]
# metrics for default/kubernetes api's from the kubernetes master
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
bearer_token_file: /dfgdjk/token
api_server: https://masterapi.com:3343
tls_config:
insecure_skip_verify: true
tls_config:
insecure_skip_verify: true
bearer_token_file: /dfgdjk/token
scheme: https
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: \[__meta_kubernetes_namespace\]
action: replace
target_label: kubernetes_namespace
- source_labels: \[__meta_kubernetes_pod_name\]
action: replace
target_label: kubernetes_pod_name
# metrics for default/kubernetes api's from the kubernetes master
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
api_server: https://masterapi.com:3343
bearer_token_file: /dfgdjk/token
tls_config:
insecure_skip_verify: true
tls_config:
insecure_skip_verify: true
bearer_token_file: /dfgdjk/token
scheme: https
relabel_configs:
- source_labels: \[__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name\]
action: keep
regex: default;kubernetes;https][1]
[1]: https://devopscube.com/install-configure-prometheus-linux/
It's impossible to get metrics to external prometheus server without having any prometheus components inside the kubernetes cluster. This happens because cluster network is isolated with host's network and it's not possible to scrape metrics from pods directly from outside the cluster.
Please refer to Monitoring kubernetes with prometheus from outside of k8s cluster GitHub issue
There options which can be done:
install prometheus inside the cluster using prometheus operator or manually - example
use proxy solutions, for example this one from the same thread on GitHub - k8s-prometheus-proxy
on top of the prometheus installed within the cluster, it's possible to have external prometheus in federation so all logs are saved outside of the cluster. Please refer to prometheus federation.
Also important part is kube state metrics should be installed as well in kubernetes cluster. How to set it up.
Edit: also you can refer to another SO question/answer which confirms that only with additional steps or OP resolved it by another proxy solution.
I have setup prometheus, running in my kubernetes cluster , And I configured the certificate of kubernetes in the configuration file of Prometheus, but for some targets I am getting back a "server returned HTTP status 403 Forbidden". this is part of my config:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /etc/k8spem/ca.pem
cert_file: /etc/k8spem/admin.pem
key_file: /etc/k8spem/admin.key
bearer_token_file: /etc/k8spem//token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
I have already configured the certificate, why still 403 ?
by the way, I can get results on CLI by executing this command curl -k --cacert /work/deploy/kubernetes/security/ca.pem --cert /work/deploy/kubernetes/security/admin.pem --key /work/deploy/kubernetes/security/admin.key --cert-type PEM https://172.16.5.150:6443/metrics
I don't know why, I just mount a new directory, delete the old configMap and recreate it. And it' work. I think maybe i just forgot to reapply the configMap.
I have set up a v1.13 Kubernetes cluster using Kube spray.
Our etcd is running as docker containers outside the K8s cluster. If I check the etcd certificates, I can see each etcd has its own ca, client cert and key.
If I want to scrape the /metrics endpoints of these etcd conatiners for Prometheus, which certificates to use for the HTTPS endpoints?
I am not yet sure, if this is the most secured way or not.
But I took the ca.pem, cert and key that one of the etcd uses.
I created a Kubernetes secret object out of the three:
kubectl create secret generic etcd-metrics -n monitoring --from-file=etcd-secrets/
Then I added the secrets as configmaps in Prometheus config and below as my scrape
targets:
- job_name: etcd
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
static_configs:
- targets:
- 172.xxxxx:2379
- 172.xxxxx:2379
- 172.xxxxx:2379
tls_config:
ca_file: /etc/ssl/etcd/ca.pem
cert_file: /etc/ssl/etcd/etcd-node.pem
key_file: /etc/ssl/etcd/etcd-key.pem
insecure_skip_verify: false
While not exactly what you asked, I had great success pushing that authentication down onto the actual machine by using socat running in a sidecar container listening on etcd's prometheus port :9379 and then you can just point prometheus at http://${etcd_hostname}:9379/metrics without having to deal with authentication for those metrics endpoints.
I don't have the socat invocation in front of me, but something like:
socat tc4-listen:9379,reuseaddr,fork \
openssl:127.0.0.1:2379,capath=/etc/kubernetes/pki/etcd/cacert.crt,key=/etc/kubernetes/pki/etcd/peer.key,cert=/etc/kubernetes/pki/etcd/peer.crt
I am using Promethues to monitor my Kubernetes cluster. All my microservices can be accessed using my HA Proxy.
My base Promethues config is :
- job_name: 'kubernetes_pods'
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- api_server: http://172.29.219.102:8080
role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_host_ip]
target_label: __address__
regex: (.*)
replacement: 172.29.219.110:8080
Where 172.29.219.110:8080 is the IP & Port of my standalone HA Proxy.
The endpoint that I am trying to monitor using Prometheus is /auth/health.
When I do a simple curl command from anywhere, I see :
# curl http://172.29.219.110:8080/auth/health
{"status":"UP"}
But when Prometheus tries to do it, the logs indicate :
level=warn ts=2017-12-15T16:40:48.301741927Z caller=scrape.go:673 component="target manager" scrape_pool=kubernetes_pods target=http://172.29.219.110:8080/auth/health msg="append failed" err="no token found"
This endpoint is publicly exposed and requires no authentication what so ever. So why does Promethues say :
{"status":"UP"}
Prometheus requires data to be in its format, and cannot handle other arbitrary data. The error you are getting is a parse error due to this.
You should instrument your code using a client library, and have it expose data in the Prometheus text format.
I am trying to setup prometheus to monitor nodes, services and endpoints for my kubernetes cluster [1 master, 7 minions ] . For that I have a very basic promethus.yml file :
scrape_configs:
- job_name: 'kubernetes-pods'
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- role: pod
Before starting the Prometheus application , I ran the below 2 commands :
export KUBERNETES_SERVICE_HOST=172.9.25.6
export KUBERNETES_SERVICE_PORT=8080
I can access the Kubernetes API server using http://172.9.25.6:8080
The connect is formed over http and NOT https.
Now when I start the application, I get the below ERROR :
level=info ts=2017-12-13T20:39:05.312987614Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-12-13T20:39:05.313443232Z caller=main.go:371 msg="Server is ready to receive requests."
level=error ts=2017-12-13T20:39:05.316618074Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:205: Failed to list *v1.Pod: Get https://172.9.25.6:8080/api/v1/pods?resourceVersion=0: http: server gave HTTP response to HTTPS client"
I also tried to add scheme: http to my prometheus.yml config but it does not work. How can I configure the client to accept HTTP responses ?
Try specifying api_server inside kubernetes_sd_configs:
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
api_server: http://172.9.25.6:8080