I'm trying to deploy an instance of HashiCorp Vault with TLS and integrated storage using the official Helm chart. I've run through the official tutorial using minikube without any issues. I also tested this tutorial with a cluster created with kind. The tutorial went as expected on both minikube and kind, however when I tried on a production cluster created with TKGI (Tanzu Kubernetes Grid Integrated) I ran into x509 errors running vault commands in the server pods. I can get by some of them by using -tls-skip-verify, but what may be different between these two clusters to cause the warning? It seems to be causing additional problems when I try to join the replicas to the raft pool.
Here's an example showing the x509 error,
bash-3.2$ kubectl exec -n vault vault-0 -- vault operator init \
> -key-shares=1 \
> -key-threshold=1 \
> -format=json > /tmp/vault/cluster-keys.json
Get "https://127.0.0.1:8200/v1/sys/seal-status": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca")
Is there something that could be updated on the TKGI clusters so that these x509 errors could be avoided?
Related
I have deployed DigitalOcean Chatwoot. Now I need to add the certificate created in the cluster config so that it can be accessed using HTTPS.
I am following this documentation, to manage TLS Certificates in the Kubernetes Cluster.
After following the steps, mentioned in the documentation, I got these
But, as mentioned in the last point of the documentation, I need to add the certificate keys in the Kubernetes controller manager. To do that I used this command
kube-controller-manager --cluster-signing-cert-file ca.pem --cluster-signing-key-file ca-key.pem
But I am getting the below error :
I0422 16:02:37.871541 1032467 serving.go:348] Generated self-signed cert in-memory
W0422 16:02:37.871592 1032467 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
W0422 16:02:37.871598 1032467 client_config.go:622] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Can anyone please tell me how can I configure the cluster to provide security (HTTPS), to the cluster?
Suddenly an entire Kubernetes cluster (Azure's AKS-solution) became unresponsive.
When running kubectl commands, the result is kubectl x509 certificate has expired or is not yet valid.
Nothing in Azure Portal indicates an unhealthy state.
The quick solution:
az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
When certificates have been rotated, you can use kubectl again.
Be ready to wait 30 minutes before the cluster fully recovers.
Full explanation can be found in this article:
https://learn.microsoft.com/en-us/azure/aks/certificate-rotation
AKS clusters created prior to May 2019 have certificates that expire after two years. Any cluster created after May 2019 or any cluster that has its certificates rotated have Cluster CA certificates that expire after 30 years. All other AKS certificates, which use the Cluster CA for signing, will expire after two years and are automatically rotated during an AKS version upgrade which happened after 8/1/2021. To verify when your cluster was created, use kubectl get nodes to see the Age of your node pools.
Here is the commands you can resolve the issue by rotate certificates and
az account set --subscription
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
Note: get-credentials is mandatory to rotate certificates.
I am getting the following error when i try to deploy a kubernetes service using my bitbucket pipeline to my kubernetes cluster. I am using deploying services method to deploy the service which works fine on my local machine so i am not able to reproduce the issue.
Is it a certificate issue or some configuration issue ?
How can i resolve this ?
1s
+ kubectl apply -f dashboard/
unable to recognize "dashboard/deployment.yml": Get https://kube1.mywebsitedomain.com:6443/api?timeout=32s: x509: certificate is valid for kube1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not kube1.mywebsitedomain.com
unable to recognize "dashboard/ingress.yml": Get https://kube1.mywebsitedomain.com:6443/api?timeout=32s: x509: certificate is valid for kube1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not kube1.mywebsitedomain.com
unable to recognize "dashboard/secret.yml": Get https://kube1.mywebsitedomain.com:6443/api?timeout=32s: x509: certificate is valid for kube1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not kube1.mywebsitedomain.com
unable to recognize "dashboard/service.yml": Get https://kube1.mywebsitedomain.com:6443/api?timeout=32s: x509: certificate is valid for kube1, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not kube1.mywebsitedomain.com
Before running the apply command I did set the cluster using the kubectl config and i get the following on the console.
+ kubectl config set-cluster kubernetes --server=https://kube1.mywebsitedomain.com:6443
Cluster "kubernetes" set.
It was the certificate issue. Using the right certificate will definitely solve this problem but in my case the certificate verification wasn't necessary as secure connection is not required for this spike.
So here is my work around
I used the flag --insecure-skip-tls-verify with kubectl and it worked fine
+ kubectl --insecure-skip-tls-verify apply -f dashboard/
deployment.extensions/kubernetes-dashboard unchanged
ingress.extensions/kubernetes-dashboard unchanged
secret/kubernetes-dashboard-auth unchanged
service/kubernetes-dashboard unchanged
I am having tiller panic crash as detailed in helm FAQ (https://docs.helm.sh/using_helm/) under the question
Q: Tiller crashes with a panic:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation...
The FAQ answer suggests
To fix this, you will need to change your Kubernetes configuration.
Make sure that --service-account-private-key-file from
controller-manager and --service-account-key-file from apiserver point
to the same x509 RSA key.
I've tried to search online and read the docs at (https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) which states
You must pass a service account private key file to the token
controller in the controller-manager by using the
--service-account-private-key-file option. The private key will be used to sign generated service account tokens. Similarly, you must
pass the corresponding public key to the kube-apiserver using the
--service-account-key-file option. The public key will be used to verify the tokens during authentication.
and the docs at https://kubernetes.io/docs/reference/access-authn-authz/authentication/
The docs explain concepts well, but no specifics on actually changing the config.
How do I change my Kubernetes configuration as the FAQ answer suggests?
Make sure that --service-account-private-key-file from
controller-manager and --service-account-key-file from apiserver point
to the same x509 RSA key.
Details:
using kops and gossip based k8s cluster
I have found through trial and error that helm ls seems to have a bug for my setup, or maybe helm ls is expected to show the error if there are no releases to be shown.
Right after helm init, helm ls would show the error panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation...,
but if you just went ahead and used helm,
e.g. helm install stable/<package>, the chart is deployed successfully.
And after the charts are deployed, calling helm ls no longer returns the error, and correctly shows the deployed charts (releases).
I have confirmed it using kubectl get pods, kubectl get services and kubectl get deployments, the released objects are all running.
We have running instance on GKE. Till this afternoon we started to receive "x509: certificate signed by unknown authority" error from new created kubernetes cluster. The kubectl is working with old clusters but not with new ones.
What we tried:
gcloud update
kubectl update
gcloud re authenticate
clean gcloud install & auth
.kube/config cert remove & gcloud container clusters get-credentials
remove and add new clusters
Thanks.