I'm facing some issue even with the latest cert-manager (). I'm running on k8s v1.22 and the same chart was working as expected on v1.21
error:
Not ready: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
Not ready: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
This happens on the pod pod/cert-manager-startupapicheck-l4ccx started by job.batch/cert-manager-startupapicheck.
I am not sure why this is happening and how to fix this as it looks like to be a k8s issue rather than cert-manager.
Please can anyone point me to some documentation or some similar case as I was not able to find anything related to this. I read the documentation from cert-manager, all github issues I could find and this was not able to get this fixed.
Related
I have a really weird issue with one of my Linode K8s clusters running 1.23, there are multiple issues occuring and I can't quite pinpoint the root cause.
Linode have let me know it is not a issue with the master and nothing on there end, let me highlight all the identified problems to start.
Logs not Working
When trying to pull logs from any pods I get this error (which makes it very hard to troubleshoot)
root#aidan:~# kubectl logs <pod-name> -n revwhois-subdomain-enum
Error from server: Get "https://192.168.150.102:10250/containerLogs/revwhois-subdomain-enum/tldbrr-revwhois-worker12-twppv/tldbrr-revwhois-worker12": dial tcp 192.168.150.102:10250: i/o timeout
Metrics not Working
root#m0chan:~# kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
Pod Deletion not Working
When deleting a pod with kubectl delete pod <pod-name> - <namespace> - it will delete the pod however it is stuck in a terminating state, the old pod is not deleted and anew pod is not launched.
Errors Editing Ingress
Error from server (InternalError): error when creating "yaml/xxx/xxx-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: Temporary Redirect
I also have errors on Metrics logs and Cert-Manager logs relating to failed calling webhook
This is all for now and I would really appreciate some help resolving this.
Aidan
I am trying to get logs from a pod that is running using kubectl logs grafana-6bfd846fbd-nbv8r
and I am getting the following output:
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy)
I tried to figure why I would not have this specific authorisation even though I can manage everything with this user, no clue. The weirdest is that when I run kubectl auth can-i get pod/logs I get:
yes
After a few hours of going through ClusterRoles and ClusterRoleBindings, I am getting stuck and do know what to do to be authorized. Thanks for your help!
The failure is kube-apiserver trying to access the kubelet, not related to your user. This indicates your core system RBAC rules might be corrupted, check if your installer or K8s distro has a way to validate or repair them (most don't) or make a new cluster and compare them to that.
I’m k8s beginner, and struggling with below error.
E0117 18:24:47.596238 53015 portforward.go:400]
an error occurred forwarding 9999 -> 80: error forwarding port 80 to pod XXX,
uid : exit status 1: 2020/01/17 09:24:47 socat[840136] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
I don’t even know what the error stands for, needless to say for its cause. Does anyone know of which situation below error occurs?
This error is occuring while processing GCP's deployment manager tutorial according to tutorial project GCP provides.
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/gke
Error occurs when typing this command.
curl localhost:9999
Any ambiguous expression or extra information is required, please notify me.
Thanks in advance!
The error is telling you, that there's nothing listening to port 80 inside the pod. You should check the pod state:
kubectl get pods
It will also tell you which port(s) the pod (its containers) is listening to.
Maybe it has crashed. Also check the log of the pod:
kubectl logs <pod-name>
Btw. Google's Deployment Manager is a very special kind of a tool. Google itself suggests to use Terraform instead. It's nevertheless part of their certification exams.
I'm trying to use Pulumi to create a Deployment with a linked Service in a Kubesail cluster. The Deployment is created fine but when Pulumi tries to create the Service an error is returned:
kubernetes:core:Service (service):
error: Plan apply failed: resource service was not successfully created by the Kubernetes API server : Could not create watcher for Endpoint objects associated with Service "service": unknown
The Service is correctly created in Kubesail and the error seems to be glaringly obvious that it can't do Pulumi's neat monitoring but the unknown error isn't so neat!
What might be being denied on the Kubernetes cluster such that Pulumi can't do the monitoring that would be different between a Deployment and a Service? Is there a way to skip the watching that I missed in the docs to get me past this?
I dug a little into the Pulumi source code and found the resource kinds it uses to track and used kubectl auth can-i and low and behold watching an endpoint is currently denied but watching replicaSets and the service themselves is not.
Yesterday, my kubernetes cluster is suddenly down
I tried to investigate as the follows but not sure what the reason was:
Unable to access Kube Dashboard, it returns HTTP ERROR 502
Unable to access deployed apps on cluster, it also returns 502 error
Cannot use kubectl command, it shows the message: "Unable to connect
to the server: x509: certificate has expired or is not yet valid"
With this error, I googled and got the article.
But I'm not sure if this is correct or not.
Can you please help to advise.
Thank you so much.
Environment:
Kubernetes 1.5
Kube-aws