Getting Service Unavailable for service metrices command in EKS - kubernetes

Question

I am getting ServiceUnavailable error when I try to run kubectl top nodes or kubectl top pods command in EKS. I am running my cluster in EKS , and I am not finding any solution for this online. If any one have faced this issue in EKS please let me know how we can resolve this issue
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
out put of kubectl get apiservices v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"v1beta1.metrics.k8s.io"},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-server","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
creationTimestamp: "2022-02-03T08:22:59Z"
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
resourceVersion: "1373088"
uid: 2066d4cb-8105-4aea-9678-8303595dc47b
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2022-02-03T08:22:59Z"
message: 'failing or missing response from https://10.16.55.204:4443/apis/metrics.k8s.io/v1beta1:
Get "https://10.16.55.204:4443/apis/metrics.k8s.io/v1beta1": dial tcp 10.16.55.204:4443:
i/o timeout'
reason: FailedDiscoveryCheck
status: "False"
type: Available
metrics-server 1/1 1 1 3d22h
kubectl describe deployment metrics-server -n kube-system
Name: metrics-server
Namespace: kube-system
CreationTimestamp: Thu, 03 Feb 2022 09:22:59 +0100
Labels: k8s-app=metrics-server
Annotations: deployment.kubernetes.io/revision: 2
Selector: k8s-app=metrics-server
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 0 max unavailable, 25% max surge
Pod Template:
Labels: k8s-app=metrics-server
Service Account: metrics-server
Containers:
metrics-server:
Image: k8s.gcr.io/metrics-server/metrics-server:v0.6.0
Port: 4443/TCP
Host Port: 0/TCP
Args:
--cert-dir=/tmp
--secure-port=4443
--kubelet-insecure-tls=true
--kubelet-preferred-address-types=InternalIP
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--kubelet-use-node-status-port
--metric-resolution=15s
Requests:
cpu: 100m
memory: 200Mi
Liveness: http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:https/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/tmp from tmp-dir (rw)
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
Priority Class Name: system-cluster-critical
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: metrics-server-5dcd6cbcb9 (1/1 replicas created)
Events: <none>

Answer 1

Download the components.yaml, find and replace 4443 to 443 and do a kubectl replace -f components.yaml -n kube-system --force.