I'm trying to add my custom scheduler with scheduler plugin in Kubernetes.
Due to no detailed example in Kubernetes document (scheduling-framework, 20180409-scheduling-framework)
So I simply made it for testing. But, I got error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1".
How could I configure and use scheduler plugin?
In my case, test configurations are
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
plugins:
queueSort:
enabled:
- name: test
pluginConfig:
- name: test-
kubectl command
$ kubectl apply -f test.yaml
error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1"
When I created as pod,
$ kubectl create -f test.yaml
$ kubectl logs pod test
E0920 15:55:47.970707 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:47.971808 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.962810 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:48.963476 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.964615 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:48.965674 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:48.966800 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:48.967967 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:48.969030 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "pods" in API group "" at the cluster scope
E0920 15:55:48.970095 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0920 15:55:48.971150 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0920 15:55:48.972252 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:48.973396 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.964805 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:49.965338 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.966313 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:49.967467 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:49.968448 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:49.969534 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:49.970775 1 reflector.go:123] pkg/mod/k8s.io/client-go#v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is f^C
Enviroment
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
go
$ go version
go version go1.13 linux/amd64
go module
$ cat go.sum
...
k8s.io/kubernetes v1.16.0 h1:WPaqle2JWogVzLxhN6IK67u62IHKKrtYF7MS4FVR4/E=
k8s.io/kube-scheduler v0.0.0-20190918162820-3b5c1246eb18 h1:WnbesbtFrWlZ8wwgDuP1KN+6E+XLDGzG4ztojMxfHsc=
...
KubeSchedulerConfiguration is a kind of configmap, if you want to apply if you have to command like below -
kubectl create -n kube-system configmap my-scheduler-config --from-file=/root/my-scheduler-config.yaml
Here -
kube-system = your target namespace
my-scheduler-config = the name of your config map, it can be any name
/root/my-scheduler-config.yaml = replace with your yaml config file name & location
The scheduler configuration is not applied to the API server with kubectl apply command, it is passed to the scheduler process with kube-scheduler --config=/path/to/config. Read more about it on Github.
--config is the entry where you should specify which parameters the scheduler will take. From the API’s perspective, that config file is supposed to contain a KubeSchedulerConfiguration object:
# content of the file passed to "--config"
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/var/run/kubernetes/scheduler.kubeconfig"
algorithmSource:
policy:
file:
path: "/root/config/scheduler-extender-policy.json"
Refer to this article, which describe how to create custom Kubernetes scheduler and configuration of Scheduler.
Related
I am using terraform to deploy a Kubernetes cluster with an nginx ingress controller which is hosted on EKS. The controller pod however fails and enters a CrashLoopBackOff status.
Based on the logs, it seems that the ingress controller is failing to watch and list *v1beta1.Ingress resources. The error message "failed to list *v1beta1.Ingress: the server could not find the requested resource" is repeated multiple times in the logs.
Is there a solution to this that I can apply in my terraform template?
main.tf
provider "kubernetes" {
host = data.aws_eks_cluster.default.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.default.token
config_path = "~/.kube/config"
}
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.default.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.default.token
}
}
resource "local_sensitive_file" "kubeconfig" {
content = templatefile("${path.module}/kubeconfig.tpl", {
cluster_name = var.cluster_name,
clusterca = data.aws_eks_cluster.default.certificate_authority[0].data,
endpoint = data.aws_eks_cluster.default.endpoint,
})
filename = "./kubeconfig-${var.cluster_name}"
}
resource "kubernetes_namespace" "test" {
metadata {
name = "test"
}
}
resource "helm_release" "nginx_ingress" {
namespace = kubernetes_namespace.test.metadata.0.name
wait = true
timeout = 600
name = "ingress-nginx"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
version = "3.30.0"
}
Error:
Warning: Helm release "ingress-nginx" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│ with helm_release.nginx_ingress,
│ on main.tf line 39, in resource "helm_release" "nginx_ingress":
│ 39: resource "helm_release" "nginx_ingress" {
│
╵
╷
│ Error: timed out waiting for the condition
│
│ with helm_release.nginx_ingress,
│ on main.tf line 39, in resource "helm_release" "nginx_ingress":
│ 39: resource "helm_release" "nginx_ingress" {
Logs:
➜ ~ kubectl get pods --namespace=test
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-554bcbbc94-d89bh 0/1 CrashLoopBackOff 13 (88s ago) 37m
➜ ~ kubectl logs -f ingress-nginx-controller-554bcbbc94-d89bh --namespace=test
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.46.0
Build: 6348dde672588d5495f70ec77257c230dc8da134
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
I0210 15:15:37.180002 7 flags.go:208] "Watching for Ingress" class="nginx"
W0210 15:15:37.180037 7 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0210 15:15:37.180266 7 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0210 15:15:37.180719 7 main.go:241] "Creating API client" host="https://XXX:443"
I0210 15:15:37.198460 7 main.go:285] "Running in Kubernetes cluster" major="1" minor="24+" git="v1.24.8-eks-ffeb93d" state="clean" commit="abb98ec0631dfe573ec5eae40dc48fd8f2017424" platform="linux/amd64"
I0210 15:15:37.203187 7 main.go:87] "Valid default backend" service="test/ingress-nginx-defaultbackend"
I0210 15:15:37.985389 7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0210 15:15:37.989186 7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0210 15:15:37.991201 7 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0210 15:15:38.007946 7 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0210 15:15:38.038537 7 nginx.go:254] "Starting NGINX Ingress controller"
I0210 15:15:38.048874 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"test", Name:"ingress-nginx-controller", UID:"b764504c-fcef-478e-aae3-684821ee8568", APIVersion:"v1", ResourceVersion:"54262", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap test/ingress-nginx-controller
E0210 15:15:39.148154 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:40.495320 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:42.544137 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:46.797783 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0210 15:15:54.929779 7 reflector.go:138] k8s.io/client-go#v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
^C
➜ ~ kubectl describe pod ingress-nginx-controller-554bcbbc94-d89bh --namespace=test
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 42m default-scheduler Successfully assigned test/ingress-nginx-controller-554bcbbc94-d89bh to ip-XXX.eu-west-1.compute.internal
Normal Killing 41m kubelet Container controller failed liveness probe, will be restarted
Normal Pulled 41m (x2 over 42m) kubelet Container image "k8s.gcr.io/ingress-nginx/controller:v0.46.0#sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a" already present on machine
Normal Created 41m (x2 over 42m) kubelet Created container controller
Normal Started 41m (x2 over 42m) kubelet Started container controller
Warning Unhealthy 22m (x46 over 41m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Warning Unhealthy 7m4s (x94 over 41m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Warning BackOff 2m7s (x118 over 35m) kubelet Back-off restarting failed container
Your Ingress-nginx Helm chart is seems outdated as it is trying to find Ingress resources with API version v1beta1.Ingress which have been deprecated in newer k8s versions and moved to networking.k8s.io/v1.
So you need to update your helm-chart.
Trying to install kubernetes dashboard in Ubuntu 16.04 resulting in x509: certificate signed by unknown authority error.
Kubernetes cluster with a single node is running fine and deployments are happening too.
Tried enabling apiserver-host property in kubernetes-dashboard.yaml file without any lock.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml
Unable to connect to the server: x509: certificate signed by unknown authority
Any suggestions.
Output from kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/calico-node-6dgkc 2/2 Running 4 4d23h
pod/calico-node-v8xjr 2/2 Running 0 2d4h
pod/coredns-fb8b8dccf-8jznp 1/1 Running 2 4d23h
pod/coredns-fb8b8dccf-pl87d 1/1 Running 2 4d23h
pod/etcd-ets-kubernetes 1/1 Running 2 4d23h
pod/kube-apiserver-ets-kubernetes 1/1 Running 2 4d23h
pod/kube-controller-manager-ets-kubernetes 1/1 Running 2 4d23h
pod/kube-proxy-24qjz 1/1 Running 0 2d4h
pod/kube-proxy-ccqpn 1/1 Running 2 4d23h
pod/kube-scheduler-ets-kubernetes 1/1 Running 2 4d23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/calico-typha ClusterIP 10.110.39.31 <none> 5473/TCP 4d23h
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 4d23h
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:ets-kubernetes" cannot list resource "replicationcontrollers" in API group "" in the namespace "kube-system"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:ets-kubernetes" cannot list resource "daemonsets" in API group "apps" in the namespace "kube-system"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:ets-kubernetes" cannot list resource "deployments" in API group "apps" in the namespace "kube-system"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:ets-kubernetes" cannot list resource "replicasets" in API group "apps" in the namespace "kube-system"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:ets-kubernetes" cannot list resource "statefulsets" in API group "apps" in the namespace "kube-system"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:ets-kubernetes" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "kube-system"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:ets-kubernetes" cannot list resource "jobs" in API group "batch" in the namespace "kube-system"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:ets-kubernetes" cannot list resource "cronjobs" in API group "batch" in the namespace "kube-system"
Output from kubectl get nodes
NAME STATUS ROLES AGE VERSION
ets-kubernetes Ready master 4d23h v1.14.1
ets-node Ready <none> 2d4h v1.14.1
Kubectl output.PNG
Certificate Error.PNG
It would be better if you would specify how did you deploy your cluster but, try to regenerate your cluster certificates. If you used kubeadm then from control plane node you can run
kubeadm alpha certs renew
For more info check this
EDIT according to update on original post:
According your updated output, as you can see from the events, somehow there are many lines like:
User "system:node:ets-kubernetes" cannot list resource .........
It means, above user doesn't have relevant role to do those actions on specified resources.
To fix this you have to create relevant Role and RoleBindings for this user.
You can get more info from official Using RBAC Authorization documentation
Had the same issue after resetting k8s to defaults while having kubectl proxy running.
Simply restarting kubectl proxy fixed the issue :)
I had an incorrect date set (by a few years) I've modified this but now I'm getting errors with some pods:
kube-controller-manager-master
E0803 01:06:31.311871 1 leaderelection.go:234] error retrieving
resource lock kube-system/kube-controller-manager: Get
https://192.168.0.33:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager:
x509: certificate has expired or is not yet valid
kube-scheduler-master
E0803 01:06:24.507668 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1.StorageClass: Get
https://192.168.0.33:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0:
x509: certificate has expired or is not yet valid E0803
01:06:24.511785 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1beta1.ReplicaSet: Get
https://192.168.0.33:6443/apis/extensions/v1beta1/replicasets?limit=500&resourceVersion=0:
x509: certificate has expired or is not yet valid E0803
01:06:24.532539 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1.Node: Get
https://192.168.0.33:6443/api/v1/nodes?limit=500&resourceVersion=0:
x509: certificate has expired or is not yet valid E0803
01:06:24.543719 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1beta1.PodDisruptionBudget: Get
https://192.168.0.33:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0:
x509: certificate has expired or is not yet valid E0803
01:06:24.547678 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1.ReplicationController: Get
https://192.168.0.33:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid E0803
01:06:24.554880 1 reflector.go:205]
k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130:
Failed to list *v1.PersistentVolume: Get
https://192.168.0.33:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid E0803
01:06:24.559708 1 reflector.go:205]
k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:176: Failed to list
*v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0:
x509: certificate has expired or is not yet valid
How could I fix this?
One solution was to delete the whole cluster and start again, which I had to do now...
I deploy kube-scheduler using https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ .
I followed the steps exactly at the beginning however it does not schedule the node using "my-scheduler" the node is pending instead.
The log of "my-scheduler" pod is
E0207 20:35:43.079477 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:default" cannot list poddisruptionbudgets.policy at the cluster scope
E0207 20:35:43.080416 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope
E0207 20:35:43.081490 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot list persistentvolumes at the cluster scope
E0207 20:35:43.082515 1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:593: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:default" cannot list pods at the cluster scope
E0207 20:35:43.083566 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list nodes at the cluster scope
E0207 20:35:43.084795 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list replicationcontrollers at the cluster scope
E0207 20:35:44.077899 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list persistentvolumeclaims at the cluster scope
E0207 20:35:44.078410 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:serviceaccount:kube-system:default" cannot list replicasets.extensions at the cluster scope
E0207 20:35:44.079496 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list statefulsets.apps at the cluster scope
E0207 20:35:44.080585 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:default" cannot list poddisruptionbudgets.policy at the cluster scope
E0207 20:35:44.081675 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope
E0207 20:35:44.082726 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot list persistentvolumes at the cluster scope
E0207 20:35:44.083811 1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:593: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:default" cannot list pods at the cluster scope
E0207 20:35:44.084887 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list nodes at the cluster scope
E0207 20:35:44.085921 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list replicationcontrollers at the cluster scope
It seems it does not have permission to access resources. I tried configured RBAC as the link says but it does not help.
Please help me if you ever tried this.
I don't know why the new scheduler use "system:serviceaccount:kube-system:default" instead of "system:kube-system".
The quick solution is:
kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin
When I follow these instructions to create custom scheduler, the pods assigned to my-scheduler (pod annotation-second-scheduler in the example) keep status Pending and are never scheduled.
I think this is because the kube-scheduler cannot access the master from within the pod. I don't know how to get this working. How can the master be accessed from within a pod? I tried running kubectl proxy -p 8001 in the pod, but this doesn't work.
There are few issues with the instructions mentioned in https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ for local clusters that I created using the instructions mentioned in https://blog.tekspace.io/setup-kubernetes-cluster-with-ubuntu-16-04/
These errors were reported from custom scheduler container (kubect logs command):
E0628 21:05:29.128618 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumeclaims at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.129945 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list services at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.132968 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.151367 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumes at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.152097 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list replicasets.extensions at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153187 1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:594: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list pods at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153201 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list nodes at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153300 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list replicationcontrollers at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153338 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list poddisruptionbudgets.policy at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153757 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list statefulsets.apps at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.147954 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.149547 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumeclaims at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.149562 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list services at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
The issue is in the my-scheduler.yaml file: in roleref change the name field from kube-scheduler to system:kube-scheduler. Verify it using this command before changing the yaml file:
kubectl get clusterrole --all-namespaces | grep -i kube
It should list system:kube-scheduler instead of kube-scheduler only.
Then, it might print these errors in the custom scheduler container:
E0628 21:22:39.937271 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:40.940461 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:41.943323 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:42.946263 1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
In this case, please append these lines:
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- watch
- list
- get
to the end of the output of this command (this opens a file for you to edit):
kubectl edit clusterrole system:kube-scheduler
From the user guide section on accessing the cluster API from a pod at kubernetes.io:
When accessing the API from a pod, locating and authenticating to the
api server are somewhat different.
The recommended way to locate the apiserver within the pod is with the
kubernetes DNS name, which resolves to a Service IP which in turn will
be routed to an apiserver.
The recommended way to authenticate to the apiserver is with a service
account credential. By kube-system, a pod is associated with a service
account, and a credential (token) for that service account is placed
into the filesystem tree of each container in that pod, at
/var/run/secrets/kubernetes.io/serviceaccount/token.
If available, a certificate bundle is placed into the filesystem tree
of each container at
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be
used to verify the serving certificate of the apiserver.
Finally, the default namespace to be used for namespaced API
operations is placed in a file at
/var/run/secrets/kubernetes.io/serviceaccount/namespace in each
container.
From within a pod the recommended ways to connect to API are:
run a kubectl proxy as one of the containers in the pod, or as a background process within a container. This proxies the Kubernetes
API to the localhost interface of the pod, so that other processes in
any container of the pod can access it. See this example of using
kubectl proxy in a pod.
use the Go client library, and create a client using the client.NewInCluster() factory. This handles locating and
authenticating to the apiserver.
In each case, the credentials of the pod are used to communicate
securely with the apiserver.