authentication handshake failed: x509: certificate signed by unknown authority - kubernetes

Question

I am starting kubernetes api server(v1.15.3) using this command:
systemctl start kube-apiserver.service
this is the log output:
● kube-apiserver.service - Kubernetes API Service
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: activating (start) since 六 2019-08-24 20:12:18 CST; 4s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 9563 (kube-apiserver)
Tasks: 13
Memory: 11.0M
CGroup: /system.slice/kube-apiserver.service
└─9563 /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=https://172.19.104.231:2379,https://172.19.104.230:2379,https://172.19.150.82:2379 --advertise-address=172.19.104.231 --bind-address=172.19.104.231 --insecure-bind-address=172.19.104.231 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h
8月 24 20:12:19 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:19.994504 9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.150.82:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
8月 24 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.985988 9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.231:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
8月 24 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.986331 9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.230:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
this CA certificate of kubernetes config(kubernetes-csr.json):
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"172.19.104.230",
"172.19.150.82",
"172.19.104.231"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
What should I do to fix this problem?I have tried self sign certificate in CentOS 7:
openssl x509 -outform der -in kubernetes.pem -out kubernetes.crt
cp /data/k8s/ssl/kubernetes.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust
My etcd cluster using the same certification file.This is the generate certificate command:
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
this is the etcd list:
[root#iZuf63refzweg1d9dh94t8Z ssl]# etcdctl member list
55a782166ce91d01, started, infra3, https://172.19.150.82:2380, https://172.19.150.82:2379
67bca27e43a8258a, started, infra2, https://172.19.104.230:2380,
696a771758a889c4, started, infra1, https://172.19.104.231:2380, https://172.19.104.231:2379

Answer 1

This may caused by your certificate file generate encount warning,you should use new version of cfssl(above v1.2),and make sure have no warning.This is cause by this tip when using cfssl(v1.3) to generate certificate:
This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements")
try to upgrade the cfssl to v1.3.4 and regenerate certificate.
/usr/local/go/bin/go get -u github.com/cloudflare/cfssl/cmd/cfssl
verify the version.
[root#iZuf63refzweg1d9dh94t8Z ssl]# /root/go/bin/cfssl version
Version: 1.3.4
Revision: dev
Runtime: go1.12.9