No resources found when installing kubernetes dashboard - kubernetes

I am install kubernetes dashboard using this command:
[root#iZuf63refzweg1d9dh94t8Z ~]# kubectl create -f kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
this is my kubernetes yaml config:
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: registry.cn-beijing.aliyuncs.com/minminmsn/kubernetes-dashboard:v1.10.1
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
get the result:
[root#iZuf63refzweg1d9dh94t8Z ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 102d
kubernetes-dashboard NodePort 10.254.180.117 <none> 443:31720/TCP 58s
metrics-server ClusterIP 10.43.96.112 <none> 443/TCP 102d
[root#iZuf63refzweg1d9dh94t8Z ~]# kubectl get pods -n kube-system
No resources found.
but when I check the port 31720:
lsof -i:31720
the output is empty.Is the service deploy success? How to check the deploy log? Why the port not binding success?

it is under its own namespace - "kubernetes-dashboard". So, just use kubectl get all -n kubernetes-dashboard to see everything

Related

Nginx minikube ingress : 503 Server error

I am trying to use minikube to deploy a sample flask app. But getting 503 nginx error. Please note I am able to access the app using the Nodeport service config.
I checked with minikube IP which is mapped to local host and tried to access the app, but getting 503 error. Not sure if I missed anything. I enable the minikube addons for nginx.
Here are my files -
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: flaskapp-deployment
labels:
app: flaskapp
spec:
replicas: 1
selector:
matchLabels:
app: flaskapp
template:
metadata:
labels:
app: flaskapp
spec:
containers:
- name: flaskapp
image: <repo>/sample-flask-app:1.0
ports:
- containerPort: 5000
env:
- name: APPLICATION_SETTINGS
value: prd_config.py
imagePullSecrets:
- name: jfrog-secret
Service.yaml
apiVersion: v1
kind: Service
metadata:
name: flaskapp-service
labels:
app: flaskapp
spec:
selector:
app: flaskapp
ports:
- protocol: TCP
port: 5000
targetPort: 5000
Ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: flaskapp-ingress
labels:
app: flaskapp
spec:
defaultBackend:
service:
name: default-http-backend
port:
number: 80
rules:
- host: mydashboard.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: flaskapp-service
port:
number: 5000
Ingress status :
minikube kubectl -- get ingress flaskapp-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
flaskapp-ingress nginx mydashboard.com localhost 80 18m
Cluster status:
minikube kubectl -- get all
NAME READY STATUS RESTARTS AGE
pod/flaskapp-deployment-7f59f96fd5-j9mv9 1/1 Running 1 (103m ago) 15h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/flaskapp-deployment ClusterIP 10.103.143.58 <none> 5000/TCP 34m
service/flaskapp-service ClusterIP 10.111.242.99 <none> 5000/TCP 15h
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/flaskapp-deployment 1/1 1 1 15h
NAME DESIRED CURRENT READY AGE
replicaset.apps/flaskapp-deployment-7f59f96fd5 1 1 1 15h

Accessing an external InfluxDb Database from a microk8s pod using selectorless service and manual endpoint?

Gist: I am struggling to get a pod to connect to a service outside the cluster.
Basically the pod manages to resolve the ClusterIp of the selectorless service, but traffic does not go through. Traffic does go through if i hit the ClusterIp of the selectorless service from the cluster host.
I'm fairly new with microk8s and k8s in general. I hope i am making some sense though...
Background:
I am attempting to move parts of my infrastructure from a docker-compose setup on one virtual machine, to a microk8s cluster (with 2 nodes).
In the docker compose, i have a Grafana Container, connecting to an InfluxDb container.
kubectl version:
Client Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+9ad9ee77396805", GitCommit:"9ad9ee77396805781cd0ae076d638b9da93477fd", GitTreeState:"clean", BuildDate:"2021-09-30T09:52:57Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
I now want to setup a Grafana container on the microk8s cluster, and have it connect to the InfluxDb that is still running on the docker-compose vm.
All of these VM's are running on an ESXi host.
InfluxDb is exposed at 10.1.2.220:8086
microk8s-master has ip 10.1.2.50
microk8s-slave-1 has ip 10.1.2.51
I have enabled ingress and dns. I have also enabled metallb, though i don't intend to use it here.
I have configured a selectorless service, a remote endpoint and an egress Network Policy (currently allowing all).
From microk8s-master and slave-1, i can
telnet directly to 10.1.2.220:8086 successfully
telnet to the ClusterIP(10.152.183.26):8086 of the service, successfully reaching influxdb
wget ClusterIp:8086
Inside the Pod, if i do a wget to influxdb-service:8086, it will resolve to the ClusterIP, but after that it times out.
I can however reach (wget), services pointing to other pods in the same namespace
Update:
I have been able to get it to work through a workaround, but i dont think this is the correct way.
My temporary solution is to expose the selectorless service on metallb, then use that exposed ip inside the pod.
Service and Endpoints for InfluxDb
---
apiVersion: v1
kind: Service
metadata:
name: influxdb-service
labels:
app: grafana
spec:
ports:
- protocol: TCP
port: 8086
targetPort: 8086
---
apiVersion: v1
kind: Endpoints
metadata:
name: influxdb-service
subsets:
- addresses:
- ip: 10.1.2.220
ports:
- port: 8086
The service and endpoint shows up fine
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl get endpoints
NAME ENDPOINTS AGE
neo4j-service-lb 10.1.166.176:7687,10.1.166.176:7474 25h
influxdb-service 10.1.2.220:8086 127m
questrest-service 10.1.166.178:80 5d
kubernetes 10.1.2.50:16443,10.1.2.51:16443 26d
grafana-service 10.1.237.120:3000 3h11m
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 26d
questrest-service ClusterIP 10.152.183.56 <none> 80/TCP 5d
neo4j-service-lb LoadBalancer 10.152.183.166 10.1.2.60 7474:31974/TCP,7687:32688/TCP 25h
grafana-service ClusterIP 10.152.183.75 <none> 3000/TCP 3h13m
influxdb-service ClusterIP 10.152.183.26 <none> 8086/TCP 129m
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl get networkpolicy
NAME POD-SELECTOR AGE
grafana-allow-egress-influxdb app=grafana 129m
test-egress-influxdb app=questrest 128m
Describe:
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl describe svc influxdb-service
Name: influxdb-service
Namespace: default
Labels: app=grafana
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.152.183.26
IPs: 10.152.183.26
Port: <unset> 8086/TCP
TargetPort: 8086/TCP
Endpoints: 10.1.2.220:8086
Session Affinity: None
Events: <none>
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl describe endpoints influxdb-service
Name: influxdb-service
Namespace: default
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 10.1.2.220
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 8086 TCP
Events: <none>
eso#microk8s-master:~/k8s-grafana$ microk8s.kubectl describe networkpolicy grafana-allow-egress-influxdb
Name: grafana-allow-egress-influxdb
Namespace: default
Created on: 2021-11-03 20:53:00 +0000 UTC
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=grafana
Not affecting ingress traffic
Allowing egress traffic:
To Port: <any> (traffic allowed to all ports)
To: <any> (traffic not restricted by destination)
Policy Types: Egress
Grafana.yml:
eso#microk8s-master:~/k8s-grafana$ cat grafana.yml
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: grafana-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
storageClassName: ""
claimRef:
name: grafana-pvc
namespace: default
persistentVolumeReclaimPolicy: Retain
nfs:
path: /mnt/MainVol/grafana
server: 10.2.0.1
readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
volumeName: grafana-pv
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: grafana
name: grafana
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
fsGroup: 472
supplementalGroups:
- 0
containers:
- name: grafana
image: grafana/grafana:7.5.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
name: http-grafana
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /robots.txt
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 1
resources:
requests:
cpu: 250m
memory: 750Mi
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-pv
volumes:
- name: grafana-pv
persistentVolumeClaim:
claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
name: grafana-service
spec:
ports:
- port: 3000
protocol: TCP
targetPort: http-grafana
selector:
app: grafana
#sessionAffinity: None
#type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: "g2.some.domain.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana-service
port:
number: 3000
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: grafana-allow-egress-influxdb
namespace: default
spec:
podSelector:
matchLabels:
app: grafana
ingress:
- {}
egress:
- {}
policyTypes:
- Egress
As I haven't gotten much response, i'll answer the question with my "workaround". I am still not sure this is the best way to do it though.
I got it to work by exposing the selectorless service on metallb, then using that exposed ip inside grafana
kind: Service
apiVersion: v1
metadata:
name: influxdb-service-lb
#namespace: ingress
spec:
type: LoadBalancer
loadBalancerIP: 10.1.2.61
# selector:
# app: grafana
ports:
- name: http
protocol: TCP
port: 8086
targetPort: 8086
---
apiVersion: v1
kind: Endpoints
metadata:
name: influxdb-service-lb
subsets:
- addresses:
- ip: 10.1.2.220
ports:
- name: influx
protocol: TCP
port: 8086
I then use the loadbalancer ip in grafana (10.1.2.61)
Update October 2022
As a response to a comment above, I have added a diagram of how i believe this to work

I have deployed a Drupal Instance but i see that the instance Endpoint are not visible although the containers deployed successfully

I have deployed a Drupal Instance but i see that the instance Endpoint are not visible although the containers deployed successfully.
Container logs don't point to any direction
apiVersion: apps/v1
kind: Deployment
metadata:
name: drupal-deployment
spec:
replicas: 1
selector:
matchLabels:
app: drupal
type: frontend
template:
metadata:
labels:
app: drupal
spec:
containers:
- name: drupal
image: drupal
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
**********************
apiVersion: v1
kind: Service
metadata:
name: drupal-service
spec:
type: NodePort
ports:
- targetPort: 80
port: 80
nodePort: 30010
selector:
app: drupal
type: frontend
************************`
root#ip-172-31-32-54:~# microk8s.kubectl get pods
NAME READY STATUS RESTARTS AGE
drupal-deployment-6fdd7975f-l4j2z 1/1 Running 0 9h
drupal-deployment-6fdd7975f-p7sfz 1/1 Running 0 9h
root#ip-172-31-32-54:~# microk8s.kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
drupal-service NodePort 10.152.183.6 <none> 80:30010/TCP 9h
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 34h
***********************
root#ip-172-31-32-54:~# microk8s.kubectl describe service drupal-service
Name: drupal-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=drupal,type=frontend
Type: NodePort
IP: 10.152.183.6
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30010/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Any directions is really helpful.
NOTE: This works perfectly when running a container using the command
docker run --name some-drupal -p 8080:80 -d drupal
Thank you,
Anish
Your service selector has two values:
Selector: app=drupal,type=frontend
but your pod has only one of these:
spec:
template:
metadata:
labels:
app: drupal
Just make sure that all labels required by the service actually exist on the pod.
Like following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: drupal-deployment
spec:
replicas: 1
selector:
matchLabels:
app: drupal
type: frontend
template:
metadata:
labels:
app: drupal
type: frontend # <--------- look here
spec:
containers:
- name: drupal
image: drupal
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80

Service does not work after deleting one pod in a deployment

I apply a deployment of 2 some http pods, and a service for it, and it works fine. I can curl the serviceip or servicename. The service did round robin well.
But after I delete one pod, k8s create a new one to replace it. When I curl the service, the new pod doesn't return, only the other old one is OK.
The question is why k8s not update new pod to the service so I can curl the serviceip or servicename as before?
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mvn-dp
spec:
selector:
matchLabels:
run: mvn-demo
replicas: 2
template:
metadata:
labels:
run: mvn-demo
spec:
containers:
- name: mvndemo
image: 192.168.0.193:59999/mvndemo
ports:
- containerPort: 8080
Service:
apiVersion: v1
kind: Service
metadata:
name: mvn-svc
labels:
run: mvn-demo
spec:
ports:
- port: 8080
protocol: TCP
#type: NodePort
selector:
run: mvn-demo
kdes svc mvn-svc
Name: mvn-svc
Namespace: default
Labels: run=mvn-demo
Annotations: Selector: run=mvn-demo
Type: ClusterIP
IP: 10.97.21.218
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
Endpoints: 100.101.153.220:8080,100.79.233.220:8080
Session Affinity: None
Events: <none>
kpod
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mvn-dp-8f59c694f-2mwq8 1/1 Running 0 81m 100.79.233.220 worker2 <none> <none>
mvn-dp-8f59c694f-xmt6m 1/1 Running 0 87m 100.101.153.220 worker3 <none> <none>
[root#master1 k8s-yaml]# curl http://10.97.21.218:8080
Hello Docker World, from: mvn-dp-8f59c694f-xmt6m
[root#master1 k8s-yaml]# curl http://10.97.21.218:8080
**curl: (7) Failed connect to 10.97.21.218:8080; 连接超时(connetion timeout)**
As u can see the age of mvn-dp-8f59c694f-2mwq8 is newer than the other one, because I deleted one pod and k8s replace it with this new one.
Set a label to your deployment metadata
apiVersion: apps/v1
kind: Deployment
metadata:
name: mvn-dp
labels:
run: mvn-demo
it will work

Kubernetes metrics-server FailedDiscoveryCheck

was hoping to get a little help, my Google-Fu didnt get me much closer. I'm trying to install the metrics server for my fedora-coreos kubernetes 4 node cluster like so:
kubectl apply -f deploy/kubernetes/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
the service seems to never start
kubectl describe apiservice v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io"},"spec":{"...
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2020-03-04T16:53:33Z
Resource Version: 1611816
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
UID: 65d9a56a-c548-4d7e-a647-8ce7a865a266
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2020-03-04T16:53:33Z
Message: failing or missing response from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: bad status from https://10.3.230.59:443/apis/metrics.k8s.io/v1beta1: 403
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events: <none>
Diagnosing I have found googling around:
kubectl get deploy,svc -n kube-system |egrep metrics-server
deployment.apps/metrics-server 1/1 1 1 8m7s
service/metrics-server ClusterIP 10.3.230.59 <none> 443/TCP 8m7s
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
Error from server (ServiceUnavailable): the server is currently unable to handle the request
kubectl get all --all-namespaces | grep -i metrics-server
kube-system pod/metrics-server-75b5d446cd-zj4jm 1/1 Running 0 9m11s
kube-system service/metrics-server ClusterIP 10.3.230.59 <none> 443/TCP 9m11s
kube-system deployment.apps/metrics-server 1/1 1 1 9m11s
kube-system replicaset.apps/metrics-server-75b5d446cd 1 1 1 9m11s
kubectl logs -f metrics-server-75b5d446cd-zj4jm -n kube-system
I0304 16:53:36.475657 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
W0304 16:53:38.229267 1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
I0304 16:53:38.267760 1 secure_serving.go:116] Serving securely on [::]:4443
kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-server","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"metrics-server"}},"template":{"metadata":{"labels":{"k8s-app":"metrics-server"},"name":"metrics-server"},"spec":{"containers":[{"args":["--cert-dir=/tmp","--secure-port=4443","--kubelet-insecure-tls","--kubelet-preferred-address-types=InternalIP"],"image":"k8s.gcr.io/metrics-server-amd64:v0.3.6","imagePullPolicy":"IfNotPresent","name":"metrics-server","ports":[{"containerPort":4443,"name":"main-port","protocol":"TCP"}],"securityContext":{"readOnlyRootFilesystem":true,"runAsNonRoot":true,"runAsUser":1000},"volumeMounts":[{"mountPath":"/tmp","name":"tmp-dir"}]}],"nodeSelector":{"beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64"},"serviceAccountName":"metrics-server","volumes":[{"emptyDir":{},"name":"tmp-dir"}]}}}}
creationTimestamp: "2020-03-04T16:53:33Z"
generation: 1
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
resourceVersion: "1611810"
selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
uid: 006e758e-bd33-47d7-8378-d3a8081ee8a8
spec:
--
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
image: k8s.gcr.io/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
name: metrics-server
ports:
- containerPort: 4443
name: main-port
finally my deployment config:
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.6
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
imagePullPolicy: IfNotPresent
volumeMounts:
- name: tmp-dir
mountPath: /tmp
hostNetwork: true
nodeSelector:
beta.kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
I'm at a loss of what it could be getting the metrics service to start and just get the basic kubectl top node to display any info all I get is
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
I have searched the internet and tried adding the args: and command: lines but no luck
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
Can anyone shed light on how to fix this? Thanks
Pastebin log file
Log File
I've reproduced your issue. I have used Calico as CNI.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
fedora-master Ready master 6m27s v1.17.3
fedora-worker-1 Ready <none> 4m48s v1.17.3
fedora-worker-2 Ready <none> 4m46s v1.17.3
fedora-master:~/metrics-server$ kubectl describe apiservice v1beta1.metrics.k8s.io
Status:
Conditions:
Last Transition Time: 2020-03-12T16:04:59Z
Message: failing or missing response from https://10.99.122.196:443/apis/metrics.k8s.io/v
1beta1: Get https://10.99.122.196:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting
for connection (Client.Timeout exceeded while awaiting headers)
fedora-master:~/metrics-server$ kubectl top pod
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
When you have only one node in cluster, default settings in metrics-server repo works correctly. Issue occurs when you have more than 2 nodes. Ive used 1 master and 2 workers to reproduce. Below example deployment which works correct (have all required args). Before, please remove your current metrics-server YAMLs (kubectl delete -f deploy/kubernetes) and execute:
$ git clone https://github.com/kubernetes-sigs/metrics-server
$ cd metrics-server/deploy/kubernetes/
$ vi metrics-server-deployment.yaml
Paste below YAML:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
hostNetwork: true
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
args:
- /metrics-server
- --kubelet-preferred-address-types=InternalIP
- --kubelet-insecure-tls
- --cert-dir=/tmp
- --secure-port=4443
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
save and quit using :wq
$ cd ~/metrics-server
$ kubectl apply -f deploy/kubernetes/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
Wait a while for metrics-server to gather a few metrics from nodes.
$ kubectl describe apiservice v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
...
Metadata:
Creation Timestamp: 2020-03-12T16:57:58Z
...
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2020-03-12T16:58:01Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events: <none>
after a few minutes you can use top.
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
fedora-master 188m 9% 1315Mi 17%
fedora-worker-1 109m 5% 982Mi 13%
fedora-worker-2 84m 4% 969Mi 13%
If you will still encounter some issues, please add - --v=6 to deployment and provide logs from metrics-server pod.
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.1
args:
- /metrics-server
- --v=6
- --kubelet-preferred-address-types=InternalIP
- --kubelet-insecure-tls
You need to carefully check logs for calico-node pods. In my case i have some other network interfaces and the autodetection mechanism in calico was detecting wrong interface (ip address). You need to consult this documentation https://projectcalico.docs.tigera.io/reference/node/configuration.
What i did in my case, was simply:
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=cidr=172.16.8.0/24
cidr is my "working network". After this, all calico-nodes restarted and suddenly everything was fine.