Kubernetes dashboard: TLS handshake error - kubernetes

Question

I have an EKS 1.18 cluster. When I tried to deploy a k8s dashboard, it's failing with the error below.
Also, my dashboard svc uses a loadBalancer.
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
annotations:
external-dns.alpha.kubernetes.io/hostname: "test.xxx.net"
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: LoadBalancer
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
Pls let me know what does the TLS handshake error mean? What should I do to fix this error?
logs:
2021/03/18 22:03:08 http: TLS handshake error from xx.xxx.x.x:8279: EOF
2021/03/18 22:03:08 http: TLS handshake error from xx.xxx.x.x:34935: EOF
2021/03/18 22:03:08 http: TLS handshake error from xx.xxx.x.x:24437: EOF
2021/03/18 22:03:08 http: TLS handshake error from xx.xxx.x.x:64552: EOF
2021/03/18 22:03:10 http: TLS handshake error from xx.xxx.x.x:5481: EOF
code:
https://github.com/kubernetes/dashboard/releases/tag/v2.0.3
https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml
kubectl version
+ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T23:49:20Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Answer 1

Explanation:
zerkms already shed some light on the cause of your issue in his comment:
Your ingress presumably does not understand the dashboard terminates
tls itself and tries to send naked http requests there. So, whatever
you use to ingress/load balance traffic to the app - should be
configured to connect via tls. – zerkms Mar 19 at 2:55
You also told us that there isn't any ingress configured but you use for connection the domain name, pointing to your LoadBalancer's IP. That's fine but keep in mind that when you create a Service of LoadBalancer type on your EKS cluster, by default Classic Load Balancer is created. It works on layer 7 of the OSI model so it recognizes the https traffic, terminates TLS connection and then sends to your backend pods naked http request. As zerkms already explained, your backend is not prepared to handle such connection as it terminates TLS itself.
Solution:
As I already mentioned ,by default, when you create your LoadBalancer service, Classic Load Balancer is created. However you can change this default behaviour by adding to your Service the following annotation as mentioned here:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
As Network Load Balancer operates on layer 4 of the OSI model, it simply passes TCP packets to your backend pods without inspecting their content, terminating TLS etc and expects https traffic.
Alternatively you may set up some ingress controller which is configured to support SSL-passthrough like ngix-ingress as the AWS's ALB unfortunatelly doesn't support it.