I have been running K8s cluster(v1.13.5) for a year and the control plane certs and Kubelet certs are about to expire. I found a way to rotate all the control plane certs and I wanted to know how to rotate the Kubelet certs. Can someone help me to understand how to rotate the K certs for worker node and master (if needed)? This K8s cluster is deployed using Kubespray.
From Kubernetes version 1.8.0 a beta feature is available Certificate Rotation.
The kubelet uses certificates for authenticating to the Kubernetes API. By default, these certificates are issued with one year expiration so that they do not need to be renewed too frequently.
Kubernetes 1.8 contains kubelet certificate rotation, a beta feature that will automatically generate a new key and request a new certificate from the Kubernetes API as the current certificate approaches expiration. Once the new certificate is available, it will be used for authenticating connections to the Kubernetes API.
This needs to be enabled with Feature Gates because this is a beta feature. So you need to add
--feature-gates=RotateKubeletClientCertificate=true
When a kubelet starts up, if it is configured to bootstrap (using the --bootstrap-kubeconfig flag), it will use its initial certificate to connect to the Kubernetes API and issue a certificate signing request. You can view the status of certificate signing requests using:
kubectl get csr
Initially a certificate signing request from the kubelet on a node will have a status of Pending. If the certificate signing requests meets specific criteria, it will be auto approved by the controller manager, then it will have a status of Approved. Next, the controller manager will sign a certificate, issued for the duration specified by the --experimental-cluster-signing-duration parameter, and the signed certificate will be attached to the certificate signing requests.
The kubelet will retrieve the signed certificate from the Kubernetes API and write that to disk, in the location specified by --cert-dir. Then the kubelet will use the new certificate to connect to the Kubernetes API.
As the expiration of the signed certificate approaches, the kubelet will automatically issue a new certificate signing request, using the Kubernetes API. Again, the controller manager will automatically approve the certificate request and attach a signed certificate to the certificate signing request. The kubelet will retrieve the new signed certificate from the Kubernetes API and write that to disk. Then it will update the connections it has to the Kubernetes API to reconnect using the new certificate.
Related
My service (with no ingress) is running in the amazon EKS cluster and I was asked to provide a CA signed cert for a third party that consumes the API hosted in the service. I have tried provisioning my cert using certificates.k8s.io API but it is still self-signed I believe. Is there a CA that provides certification for services in the Kubernetes cluster?
Yes, Certificates created using the certificates.k8s.io API are signed by a dedicated CA. It is possible to configure your cluster to use the cluster root CA for this purpose, but you should never rely on this. Do not assume that these certificates will validate against the cluster root CA.
Refer this Certificate Signing Request Process
I am getting confused with all the certificates and keys in Kubernetes.
Inside the master I have the following certificates and keys:
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.key
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key
/var/lib/kubelet/pki/kubelet.crt
/var/lib/kubelet/pki/kubelet.key
The apiserver-kubelet-client (crt\key) is used for secure communication between the apiserver and the kubelet inside the Master.
Why I have also the kubelet (crt\key)? Isn't the apiserver-kubelet-client (crt\key) is enough ?
Why I need apiserver (crt\key) if I have apiserver-kubelet-client (crt\key) ?
Hope someone can make some order with this.
I read this article about Kubernetes PKI but I am still don't understand what is difference between the above certificates and keys.
While preparing certs, you will end up with at least 1 file.
CA_CERT
put in on node where apiserver runs, for example in /srv/kubernetes/ca.crt.
MASTER_CERT
signed by CA_CERT
put in on node where apiserver runs, for example in /srv/kubernetes/server.crt
MASTER_KEY
put in on node where apiserver runs, for example in /srv/kubernetes/server.key
You can read about it in Kubernetes docs here.
It’s hard to say by the names of your certs what are they used for.
You should refer to the standard configuration.
Besides, you can have a look at kelseyhightower/kubernetes-the-hard-way as he’s showing how and when to use certificates.
You can check this comment posted on Github regarding what kind of certifications are needed between which services.
I was also asking my self similar question.
The first thing u need to know is that both the kubelet and the api-server communicate with each other [0].
The second thing is that Kubernetes uses client certificates to authenticate a client identity to the server and a server certificate to authenticate the server’s identity to the client and establish an encrypted connection .
In case of kubelet -> api-server kubelet needs a client certificate and api-server a server certificate and for api-server -> kubelet then the otherway around.
you can check if a certificate is a server or client one using:
openssl x509 -noout -text -in apiserver.crt
Now back to the question :
/etc/kubernetes/pki/apiserver.crt and /var/lib/kubelet/pki/kubelet.crt are server certificates
/etc/kubernetes/pki/apiserver-kubelet-client.crt and /var/lib/kubelet/pki/kubelet-client-current.pem are client certificates
https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/
Is there a specific method or process to replace all of the certificates required in a Kubernetes 1.7 cluster? Is this even possible?
Client is worried about using certificate auth and not being able to revoke/blacklist certs properly if someone leaves.
We're using Fabric secure cluster and need client certificate for CI/CD tools.
I've created both Cluster primary certificate and client certificate with this script https://gist.github.com/kagarlickij/d63a4061a1066d3a85abcc658f0856f5
so both have been uploaded to the same Kay vault and both have been installed to local keystore on my machine.
I've added client certificate to my Fabric security settings (Authentication type = Admin client, Authorization method = Certificate thumbprint).
The problem is that I can connect (I'm using Connect-ServiceFabricCluster in PowerShell) to Fabric cluster with Cluster primary certificate but can't with Client certificate.
I'm getting this error: Connect-ServiceFabricCluster : FABRIC_E_SERVER_AUTHENTICATION_FAILED: 0x800b0109
Please advice what can be done?
Based on this link the corresponding error code for 0x800b0109 is:
A certificate chain processed, but terminated in a root certificate
which is not trusted by the trust provider.
You're using a self-signed certificate as client cert. I'm not sure it's supported as explained in the Service Fabric Security documentation, moreover you'll have to make sure the SSL certificate has been added inside your local Store.
Client X.509 certificates
Client certificates typically are not issued by a third-party CA.
Instead, the Personal store of the current user location typically
contains client certificates placed there by a root authority, with an
Intended Purposes value of Client Authentication. The client can use
this certificate when mutual authentication is required. Note
All management operations on a Service Fabric cluster require server certificates. Client certificates cannot be used for management.
I had the same issue managing my cluster through powershell, I only had 1 cert on the cluster (the one azure generates when creating the cluster) and I believe it is a client cert since I have to select it in my browser when managing the cluster.
Ultimately I had to add the self signed cert to my Root certificate store (in addition to my personal store where I already had it) to get the powershell module to stop complaining about it.
I have a Kubernetes cluster running in High Availability mode with 3 master nodes. When I try to run the DNS cluster add-on as-is, the kube2sky application errors with an x509 signed by unknown certificate authority message for the API Server service address (which in my case is 10.100.0.1). Reading through some of the GitHub issues, it looked like Tim Hockin had fixed this type of issue via using the default service account tokens available.
All 3 of my master nodes generate their own certificates for the secured API port, so is there something special I need to do configuration-wise on the API servers to get the CA certificate included in the default service account token?
It would be ideal to have the service IP of the API in the SAN field of all your server certificates.
If this is not possible in your setup, set the clusters{}.cluster.insecure-skip-tls-verify field to true in your kubeconfig file, or the pass the --insecure-skip-tls-verify flag to kubectl.
If you are trying to reach the API from within a pod you could use the secrets mounted via the Service Account. By default, if you use the default secret, the CA certificate and a signed token are mounted to /var/run/secrets/kubernetes.io/serviceaccount/ in every pod, and any client can use them from within the pod to communicate with the API. This would help you solving the unknown certificate authority error and provide you with an easy way to authenticate against your API servers at the same time.