i folowed guide on official Kubernetes Dashboard github (https://github.com/kubernetes/dashboard) and now I'm facing problem with accessing it. I used kubectl proxy to redirect internal port outside, but when I try to open address:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
it just ends up with this error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "error trying to reach service: dial tcp 192.168.23.7:8443: i/o timeout",
"reason": "ServiceUnavailable",
"code": 503
}
What am I supposed to do?
You get a timeout. Check if dashboard pods are working (kubectl get pods -n kubernetes-dashboard)
Check u have enough access control
Check here https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md
Related
I have deploy a k8s cluster with kubeadm, I want to get controller manager's metrics with following command:
curl -k https://localhost:10257/metrics
but got the following error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
So my question is, how to get k8s controller manager's metrics?
This is a forbidden error due to permission issues which need to be authenticated with a valid user. For this,You need to create a service account, then give that service account access permissions to the metrics Path through RBAC, then this will make that service account to get the metrics.
As per this Role and Cluster Binding doc, you need to allow metrics path(replace with /healthz) as below and give a try.
Allow GET and POST requests to the non-resource endpoint /healthz and all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding to be effective):
rules:
- nonResourceURLs: ["/healthz", "/healthz/*"] # '*' in a nonResourceURL is a suffix glob match
verbs: ["get", "post"]
I have a baremetal k8s cluster (1.23.5, calico as CNI), and after some uptime I encountered that kubectl proxy is unable to reach pod services:
proxying:
kubectl proxy --port=8008
Starting to serve on 127.0.0.1:8008
requesting:
curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr:80/proxy/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "error trying to reach service: dial tcp 10.200.3.21:80: i/o timeout",
"reason": "ServiceUnavailable",
"code": 503
}
No matter what pod, service or namespace - the result is the same. Pods are running without errors, pods can ping each other among different namespaces, DNS resolve is working inside pods, and api objects are browseable inside k8s api (I omit most of the sample output because it's fully normal and cumbersome):
> curl http://localhost:8008/api/v1/namespaces/emz/pods/nginx-6b78d5b64c-km5lr
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "nginx-6b78d5b64c-km5lr",
"generateName": "nginx-6b78d5b64c-",
"namespace": "emz",
"uid": "819b94de-7aa6-441a-a40e-805a0f79e9a8",
"resourceVersion": "8070751",
[...]
"containers": [
{
"name": "nginx",
"image": "nginx",
"ports": [
{
"name": "nginx",
"containerPort": 80,
"protocol": "TCP"
}
],
[...]
"status": {
"phase": "Running",
[...]
"podIP": "10.200.3.21",
"podIPs": [
{
"ip": "10.200.3.21"
}
],
What is even more weird - is that kubectl port-forward is able to reach said services:
port-forward:
> kubectl port-forward -n emz nginx-6b78d5b64c-km5lr 8008:80
Forwarding from 127.0.0.1:8008 -> 80
Forwarding from [::1]:8008 -> 80
request:
> curl 127.0.0.1:8008
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
nginx.org.<br/>
Commercial support is available at
nginx.com.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Can anyone please point me to the direction on how to diagnose/investigate the reason for the kubectl proxy to stuck in these timeouts ? Cluster seems to be fully operating besides that.
Thanks.
Update: Seems like the reason of this, direct or relatedб may be in the fact that all of the three control nodes lost connectivity via the internal pod network to other pods. I have drained and rebooted one node, but this didn't help at all.
Update 2: In fact it did help to restore the connectivity from this node to the indernal network. Weird, but calico pods/deployments were not complaining about loss of connectivity.
I am trying to delete (and recreate) the Argo namespace, but it won't fully delete because I tried launching an eventsource and eventbus there. Now these will not delete.
I have tried to delete them via yaml and individually - no success yet.
The frustrating result is that I cannot re-launch argo
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtasksets.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": serviceaccounts "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": roles.rbac.authorization.k8s.io "argo-role" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": rolebindings.rbac.authorization.k8s.io "argo-binding" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": configmaps "workflow-controller-configmap" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": services "workflow-controller-metrics" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "argo-server" is forbidden: unable to create new content in namespace argo because it is being terminated
Error from server (Forbidden): error when creating "https://github.com/argoproj/argo-workflows/releases/download/v3.2.0/install.yaml": deployments.apps "workflow-controller" is forbidden: unable to create new content in namespace argo because it is being terminated
Here is what is going on in the argo namespace itself
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2021-10-21T14:52:51Z",
"deletionTimestamp": "2021-12-10T18:48:17Z",
"labels": {
"kubernetes.io/metadata.name": "argo"
},
"name": "argo",
"resourceVersion": "9222845",
"uid": "2ce61352-74a5-40d1-b35e-fe1efa39c3af"
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"conditions": [
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All resources successfully discovered",
"reason": "ResourcesDiscovered",
"status": "False",
"type": "NamespaceDeletionDiscoveryFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All legacy kube types successfully parsed",
"reason": "ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "All content successfully deleted, may be waiting on finalization",
"reason": "ContentDeleted",
"status": "False",
"type": "NamespaceDeletionContentFailure"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "Some resources are remaining: eventbus.argoproj.io has 1 resource instances, eventsources.argoproj.io has 1 resource instances",
"reason": "SomeResourcesRemain",
"status": "True",
"type": "NamespaceContentRemaining"
},
{
"lastTransitionTime": "2021-12-10T18:48:23Z",
"message": "Some content in the namespace has finalizers remaining: eventbus-controller in 1 resource instances, eventsource-controller in 1 resource instances",
"reason": "SomeFinalizersRemain",
"status": "True",
"type": "NamespaceFinalizersRemaining"
}
],
"phase": "Terminating"
}
}
UPDATE:
Even though one test of Kubernetes shows that the eventsource persists, another test shows that it does not
This is the response to
kubectl get eventsources argo
Error from server (NotFound): eventsources.argoproj.io "argo" not found
For anyone who stumbles onto this question, it is a permissions issue. Make certain your service account has permissions to work in both namespaces (argo and argo-events).
I'm trying to scale up/down some deployments over HTTP and also list the deployments on my cluster. I'm able to list pods, but can't figure out the deployments piece.
http://localhost:8080/api/v1/namespaces/default/deployments
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource",
"reason": "NotFound",
"details": {},
"code": 404
}
Deployments are in the apps/v1 namespace, and you need to include apps in the URL. The API documentation for the "list deployments" endpoint gives the URL as
GET /apis/apps/v1/namespaces/{namespace}/deployments
You can use the normal read-modify-write sequence to change the replicas: field in a deployment spec to scale it.
There is also a dedicated endpoint to scale deployments, though it's slightly underdocumented. Manage replicas count for deployment using Kubernetes API
suggests reading and patching the scale resource, or there is an example with a minimal JSON payload.
http://10.199.135.36:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui/#/dashboard/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "endpoints \"kube-ui\" not found",
"reason": "NotFound",
"details": {
"name": "kube-ui",
"kind": "endpoints"
},
"code": 404
}
In the newer version of kubernetes,dashboard is a alternative solution for kube-ui.Using endpoint named kube-ui may lead to endpoints \"kube-ui\" not found(404).To solve this problem,you can choose an new endpoint named kubernetes-dashboard.For more details,see:
http://kubernetes.io/docs/user-guide/ui/
All in all,if you use kube-ui(example:v3),the automatic redirection maybe not correct and the 404 error appears because of the unlocation of the resources.
Good luck!
You're hitting a 10-dot ip which is only routable within your cluster, so I'm going to assume you're eg: curling that url from a node.
Please debug the service and report as to what fails: http://kubernetes.io/docs/user-guide/debugging-services/, I'm guessing kubectl --namespace=kube-system get ep kube-ui shows nothing.
If you modify your service name or namespace in dashboard.yaml, you should change your URL:
http://cluster_ip_address:8080/api/v1/proxy/namespaces/modify-namespace/services/modify-service-name/#/dashboard/
dashboard-controller.yaml
dashboard-service.yaml
standard configure file
...
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: kubernetes-dashboard
#image: gcr.io/google_containers/kube-ui:v3
**image: index.tenxcloud.com/google_containers/kubernetes-dashboard-amd64:v1.0.1**
resources:
...
after run and test logs info such as
[root#test-ops-node1 pods]# kubectl logs kubernetes-dashboard-v1.0.1-mhz6w --namespace=kube-system
2016/05/20 08:54:10 Starting HTTP server on port 9090
2016/05/20 08:54:10 Creating API server client for http://localhost:8080
2016/05/20 08:54:10 Creating in-cluster Heapster client
2016/05/20 09:09:56 Incoming HTTP/1.1 GET /api/v1/replicationcontrollers request from 172.17.80.0:39277
2016/05/20 09:09:56 Getting list of all replication controllers in the cluster
2016/05/20 09:09:56 Get http://localhost:8080/api/v1/replicationcontrollers: dial tcp [::1]:8080: getsockopt: connection refused
2016/05/20 09:09:56 Outcoming response to 172.17.80.0:39277 with 500 status code
Kubernetes-UI is addon. You can create it in Ubuntu follow this link
http://kubernetes.io/docs/getting-started-guides/ubuntu/
on Deploy addons section.