How to get k8s controller manager's metrics? - kubernetes

I have deploy a k8s cluster with kubeadm, I want to get controller manager's metrics with following command:
curl -k https://localhost:10257/metrics
but got the following error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
So my question is, how to get k8s controller manager's metrics?

This is a forbidden error due to permission issues which need to be authenticated with a valid user. For this,You need to create a service account, then give that service account access permissions to the metrics Path through RBAC, then this will make that service account to get the metrics.
As per this Role and Cluster Binding doc, you need to allow metrics path(replace with /healthz) as below and give a try.
Allow GET and POST requests to the non-resource endpoint /healthz and all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding to be effective):
rules:
- nonResourceURLs: ["/healthz", "/healthz/*"] # '*' in a nonResourceURL is a suffix glob match
verbs: ["get", "post"]

Related

Can't access deployed Kubernetes-Dashboard - Error 503

i folowed guide on official Kubernetes Dashboard github (https://github.com/kubernetes/dashboard) and now I'm facing problem with accessing it. I used kubectl proxy to redirect internal port outside, but when I try to open address:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
it just ends up with this error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "error trying to reach service: dial tcp 192.168.23.7:8443: i/o timeout",
"reason": "ServiceUnavailable",
"code": 503
}
What am I supposed to do?
You get a timeout. Check if dashboard pods are working (kubectl get pods -n kubernetes-dashboard)
Check u have enough access control
Check here https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md

Kubernetes pull infomation from endpoints REST API

I trying to create a service account with permissions to get information about endpoints pod IPs and getting back a permissions problem.
Creating the service account and give it the right permissions:
$ kubectl create role endpoints-reader --verb=get --verb=list --resource=endpoints
$ kubectl create serviceaccount endpoints-reader-sa
$ kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=endpoints-reader-sa:endpoints-reader-sa
Adding this sa to the deployment YAML file:
...
spec:
serviceAccountName: endpoints-reader-sa
containers:
- name: ...
I stated the pod and logged in into it (ssh). Now I want to run a REST call to pull the information:
$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token);
$ curl https://kubernetes.default.svc/api/v1/namespaces/XXX/endpoints --silent --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "endpoints is forbidden: User \"system:serviceaccount:XXX:endpoints-reader-sa\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"XXX\"",
"reason": "Forbidden",
"details": {
"kind": "endpoints"
},
"code": 403
}
What I'm doing wrong?
OK... Found the issue.
So, this line:
kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=endpoints-reader-sa:endpoints-reader-sa
should be changed to this:
kubectl create rolebinding default-endpoints-reader --role=endpoints-reader --serviceaccount=XXX:endpoints-reader-sa
Where XXX is the namespace name.
Similar problem you can find here.

Scaling and listing deployments via Kubernetes HTTP API?

I'm trying to scale up/down some deployments over HTTP and also list the deployments on my cluster. I'm able to list pods, but can't figure out the deployments piece.
http://localhost:8080/api/v1/namespaces/default/deployments
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {},
"code": 404
}
Deployments are in the apps/v1 namespace, and you need to include apps in the URL. The API documentation for the "list deployments" endpoint gives the URL as
GET /apis/apps/v1/namespaces/{namespace}/deployments
You can use the normal read-modify-write sequence to change the replicas: field in a deployment spec to scale it.
There is also a dedicated endpoint to scale deployments, though it's slightly underdocumented. Manage replicas count for deployment using Kubernetes API
suggests reading and patching the scale resource, or there is an example with a minimal JSON payload.

Error while accessing Web UI Dashboard using RBAC

I created a cluster role "try-usr"
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: try-usr
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- get
- list
- watch
While accessing the Web UI(Dashboard), it's throwing an error as follows:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"xyz\" cannot get services/proxy in the namespace \"kube-system\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}
Depending on the kubernetes version, the dashboard will require different permissions according to the docs
v1.7
create and watch permissions for secrets in kube-system namespace required to - create and watch for changes of kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.
v1.8
create permission for secrets in kube-system namespace required to create kubernetes-dashboard-key-holder secret.
get, update and delete permissions for secrets named kubernetes-dashboard-key-holder and kubernetes-dashboard-certs in kube-system namespace.
get and update permissions for config map named kubernetes-dashboard-settings in kube-system namespace.
proxy permission to heapster service in kube-system namespace required to allow getting metrics from heapster.

kubernetes dashboard is not found

http://10.199.135.36:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui/#/dashboard/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "endpoints \"kube-ui\" not found",
"reason": "NotFound",
"details": {
"name": "kube-ui",
"kind": "endpoints"
},
"code": 404
}
In the newer version of kubernetes,dashboard is a alternative solution for kube-ui.Using endpoint named kube-ui may lead to endpoints \"kube-ui\" not found(404).To solve this problem,you can choose an new endpoint named kubernetes-dashboard.For more details,see:
http://kubernetes.io/docs/user-guide/ui/
All in all,if you use kube-ui(example:v3),the automatic redirection maybe not correct and the 404 error appears because of the unlocation of the resources.
Good luck!
You're hitting a 10-dot ip which is only routable within your cluster, so I'm going to assume you're eg: curling that url from a node.
Please debug the service and report as to what fails: http://kubernetes.io/docs/user-guide/debugging-services/, I'm guessing kubectl --namespace=kube-system get ep kube-ui shows nothing.
If you modify your service name or namespace in dashboard.yaml, you should change your URL:
http://cluster_ip_address:8080/api/v1/proxy/namespaces/modify-namespace/services/modify-service-name/#/dashboard/
dashboard-controller.yaml
dashboard-service.yaml
standard configure file
...
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: kubernetes-dashboard
#image: gcr.io/google_containers/kube-ui:v3
**image: index.tenxcloud.com/google_containers/kubernetes-dashboard-amd64:v1.0.1**
resources:
...
after run and test logs info such as
[root#test-ops-node1 pods]# kubectl logs kubernetes-dashboard-v1.0.1-mhz6w --namespace=kube-system
2016/05/20 08:54:10 Starting HTTP server on port 9090
2016/05/20 08:54:10 Creating API server client for http://localhost:8080
2016/05/20 08:54:10 Creating in-cluster Heapster client
2016/05/20 09:09:56 Incoming HTTP/1.1 GET /api/v1/replicationcontrollers request from 172.17.80.0:39277
2016/05/20 09:09:56 Getting list of all replication controllers in the cluster
2016/05/20 09:09:56 Get http://localhost:8080/api/v1/replicationcontrollers: dial tcp [::1]:8080: getsockopt: connection refused
2016/05/20 09:09:56 Outcoming response to 172.17.80.0:39277 with 500 status code
Kubernetes-UI is addon. You can create it in Ubuntu follow this link
http://kubernetes.io/docs/getting-started-guides/ubuntu/
on Deploy addons section.