I am trying to setup prometheus to monitor nodes, services and endpoints for my kubernetes cluster [1 master, 7 minions ] . For that I have a very basic promethus.yml file :
scrape_configs:
- job_name: 'kubernetes-pods'
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- role: pod
Before starting the Prometheus application , I ran the below 2 commands :
export KUBERNETES_SERVICE_HOST=172.9.25.6
export KUBERNETES_SERVICE_PORT=8080
I can access the Kubernetes API server using http://172.9.25.6:8080
The connect is formed over http and NOT https.
Now when I start the application, I get the below ERROR :
level=info ts=2017-12-13T20:39:05.312987614Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2017-12-13T20:39:05.313443232Z caller=main.go:371 msg="Server is ready to receive requests."
level=error ts=2017-12-13T20:39:05.316618074Z caller=main.go:211 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:205: Failed to list *v1.Pod: Get https://172.9.25.6:8080/api/v1/pods?resourceVersion=0: http: server gave HTTP response to HTTPS client"
I also tried to add scheme: http to my prometheus.yml config but it does not work. How can I configure the client to accept HTTP responses ?
Try specifying api_server inside kubernetes_sd_configs:
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
api_server: http://172.9.25.6:8080
Related
Suppose there is an Application which is located in namespace called "API" and the Prometheus Server which is located in namespace "prometheus", how can I access my Application from Prometheus Server if both of the Server and Application are in different namespaces?
I've tried to specify following construction <application-service-name>.API.svc.cluster.local:<application-service-port> as a reference to the Application, but it does not seems to work
And the Prometheus responds in the UI with Connection Refused.
scrape_configs:
- job_name: 'some-job'
kubernetes_sd_configs:
namespaces:
names: 'API'
scrape_interval: 10s
scrape_timeout: 5s
static_configs:
- targets: ['application-service-name>.API.svc.cluster.local:<application-service-port>']
I have setup prometheus, running in my kubernetes cluster , And I configured the certificate of kubernetes in the configuration file of Prometheus, but for some targets I am getting back a "server returned HTTP status 403 Forbidden". this is part of my config:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /etc/k8spem/ca.pem
cert_file: /etc/k8spem/admin.pem
key_file: /etc/k8spem/admin.key
bearer_token_file: /etc/k8spem//token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
I have already configured the certificate, why still 403 ?
by the way, I can get results on CLI by executing this command curl -k --cacert /work/deploy/kubernetes/security/ca.pem --cert /work/deploy/kubernetes/security/admin.pem --key /work/deploy/kubernetes/security/admin.key --cert-type PEM https://172.16.5.150:6443/metrics
I don't know why, I just mount a new directory, delete the old configMap and recreate it. And it' work. I think maybe i just forgot to reapply the configMap.
I have set up a v1.13 Kubernetes cluster using Kube spray.
Our etcd is running as docker containers outside the K8s cluster. If I check the etcd certificates, I can see each etcd has its own ca, client cert and key.
If I want to scrape the /metrics endpoints of these etcd conatiners for Prometheus, which certificates to use for the HTTPS endpoints?
I am not yet sure, if this is the most secured way or not.
But I took the ca.pem, cert and key that one of the etcd uses.
I created a Kubernetes secret object out of the three:
kubectl create secret generic etcd-metrics -n monitoring --from-file=etcd-secrets/
Then I added the secrets as configmaps in Prometheus config and below as my scrape
targets:
- job_name: etcd
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
static_configs:
- targets:
- 172.xxxxx:2379
- 172.xxxxx:2379
- 172.xxxxx:2379
tls_config:
ca_file: /etc/ssl/etcd/ca.pem
cert_file: /etc/ssl/etcd/etcd-node.pem
key_file: /etc/ssl/etcd/etcd-key.pem
insecure_skip_verify: false
While not exactly what you asked, I had great success pushing that authentication down onto the actual machine by using socat running in a sidecar container listening on etcd's prometheus port :9379 and then you can just point prometheus at http://${etcd_hostname}:9379/metrics without having to deal with authentication for those metrics endpoints.
I don't have the socat invocation in front of me, but something like:
socat tc4-listen:9379,reuseaddr,fork \
openssl:127.0.0.1:2379,capath=/etc/kubernetes/pki/etcd/cacert.crt,key=/etc/kubernetes/pki/etcd/peer.key,cert=/etc/kubernetes/pki/etcd/peer.crt
I'm trying to configure Prometheus outside Kubernetes Cluster.
Below is my Prometheus config.
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
api_server: https://10.0.4.155:6443
scheme: https
tls_config:
insecure_skip_verify: true
basic_auth:
username: kube
password: Superkube01
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
These is how it looks:
root#master01:~# kubectl cluster-info
Kubernetes master is running at https://10.0.4.155:6443
root#master01:~# kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 10.0.4.103:6443,10.0.4.138:6443,10.0.4.155:6443 11h
netchecker-service 10.2.0.10:8081 11h
root#master01:~#
But, when starting Prometheus, i'm getting below error.
level=error ts=2018-05-29T13:55:08.171451623Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:270: Failed to list *v1.Pod: Get https://10.0.4.155:6443/api/v1/pods?resourceVersion=0: x509: certificate signed by unknown authority"
Could anyone please tell me, what wrong i'm doing here?
Thanks,
Pavanasam R
The error indicates that Prometheus is using a different certificate to sign its metric collection request than the one expected by your apiserver.
You really need to format your code in a code block so we can see the yaml formatting. kubernetes_sd_configs seems to be the wrong home for insecure_skip_verify and basic_auth according to this link. Might want to move them and try scraping again.
As of now your insecure_skip_verify is a part of kubernetes_sd_configs:. Add it in api_server context as well.
kubernetes_sd_configs:
- api_server: https://<ip>:6443
role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
in order to access kubernetes api endpoint you need to authenticate the client either through basic_auth, bearer_token, tls_config. please go through this , it will be helpful.
I am using Promethues to monitor my Kubernetes cluster. All my microservices can be accessed using my HA Proxy.
My base Promethues config is :
- job_name: 'kubernetes_pods'
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- api_server: http://172.29.219.102:8080
role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_host_ip]
target_label: __address__
regex: (.*)
replacement: 172.29.219.110:8080
Where 172.29.219.110:8080 is the IP & Port of my standalone HA Proxy.
The endpoint that I am trying to monitor using Prometheus is /auth/health.
When I do a simple curl command from anywhere, I see :
# curl http://172.29.219.110:8080/auth/health
{"status":"UP"}
But when Prometheus tries to do it, the logs indicate :
level=warn ts=2017-12-15T16:40:48.301741927Z caller=scrape.go:673 component="target manager" scrape_pool=kubernetes_pods target=http://172.29.219.110:8080/auth/health msg="append failed" err="no token found"
This endpoint is publicly exposed and requires no authentication what so ever. So why does Promethues say :
{"status":"UP"}
Prometheus requires data to be in its format, and cannot handle other arbitrary data. The error you are getting is a parse error due to this.
You should instrument your code using a client library, and have it expose data in the Prometheus text format.