metrics not available yet metrics-server kubernetes on aws - kubernetes

Question

I am trying to enable Metrics Server on AWS, and I followed these steps
Clone or download the Metrics Server project.
Open the deploy/1.8+/metrics-server-deployment.yaml file in an editor.
Add the following command values into the containers property (it should be at the same level as the image property).
command:
- /metrics-server
- --kubelet-insecure-tls
Run kubectl create -f deploy/1.8+ as shown on the Metrics Server repo to create the deployment, services, etc.
Till this point everything is working fine, my metrics-server-pod is running fine, but when I do kubectl top nodes I am getting the following error
error: metrics not available yet
when i did kubectl logs [metrics-server-pod-name] -n kube-system i am getting this
E0611 14:36:57.527048 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xxx.ec2.internal": no metrics known for node
E0611 14:36:57.527069 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xx.ec2.internal": no metrics known for node
E0611 14:36:57.527075 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-114-xxx.ec2.internal": no metrics known for node
E0611 14:36:57.527079 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-xx-xxx.ec2.internal": no metrics known for node
E0611 14:36:57.527084 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-91-xx.ec2.internal": no metrics known for node
E0611 14:36:57.527088 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xxx.ec2.internal": no metrics known for node
E0611 14:37:26.006830 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-xx-36-103.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-36-103.ec2.internal (172.20.36.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-65-xx.ec2.internal: unable to fetch metrics from Kubelet ip-172-20-65-xxx.ec2.internal (172.xx.65.xx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-114-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-20-114-223.ec2.internal (172.xx.114.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-63-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-63-xxx.ec2.internal (172.xx.63.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xxx-91-xx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-91-xxx.ec2.internal (172.xx.91.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xxx-96-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-96-xxx.ec2.internal (172.xx.96.xxx): request failed - "401 Unauthorized", response: "Unauthorized"]

Answer 1

Option I.
Add - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
to your metrics-server-deployment.yaml, e.g.
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
If still doesn't work, try with
Option II.
Because of metrics-server resolving the hostname from coredns, add node IPs to coredns configmap
kubectl edit configmap coredns -n kube-system
and add
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
hosts {
192.168.199.100 master.qls.com
192.168.199.220 worker.qls.com
fallthrough
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2019-05-17T12:32:08Z"
name: coredns
namespace: kube-system
resourceVersion: "180"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: c93e5274-789f-11e9-a0ea-42010a9c0003