I’m k8s beginner, and struggling with below error.
E0117 18:24:47.596238 53015 portforward.go:400]
an error occurred forwarding 9999 -> 80: error forwarding port 80 to pod XXX,
uid : exit status 1: 2020/01/17 09:24:47 socat[840136] E connect(5, AF=2 127.0.0.1:80, 16): Connection refused
I don’t even know what the error stands for, needless to say for its cause. Does anyone know of which situation below error occurs?
This error is occuring while processing GCP's deployment manager tutorial according to tutorial project GCP provides.
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/examples/v2/gke
Error occurs when typing this command.
curl localhost:9999
Any ambiguous expression or extra information is required, please notify me.
Thanks in advance!
The error is telling you, that there's nothing listening to port 80 inside the pod. You should check the pod state:
kubectl get pods
It will also tell you which port(s) the pod (its containers) is listening to.
Maybe it has crashed. Also check the log of the pod:
kubectl logs <pod-name>
Btw. Google's Deployment Manager is a very special kind of a tool. Google itself suggests to use Terraform instead. It's nevertheless part of their certification exams.
Related
I have a really weird issue with one of my Linode K8s clusters running 1.23, there are multiple issues occuring and I can't quite pinpoint the root cause.
Linode have let me know it is not a issue with the master and nothing on there end, let me highlight all the identified problems to start.
Logs not Working
When trying to pull logs from any pods I get this error (which makes it very hard to troubleshoot)
root#aidan:~# kubectl logs <pod-name> -n revwhois-subdomain-enum
Error from server: Get "https://192.168.150.102:10250/containerLogs/revwhois-subdomain-enum/tldbrr-revwhois-worker12-twppv/tldbrr-revwhois-worker12": dial tcp 192.168.150.102:10250: i/o timeout
Metrics not Working
root#m0chan:~# kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
Pod Deletion not Working
When deleting a pod with kubectl delete pod <pod-name> - <namespace> - it will delete the pod however it is stuck in a terminating state, the old pod is not deleted and anew pod is not launched.
Errors Editing Ingress
Error from server (InternalError): error when creating "yaml/xxx/xxx-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: Temporary Redirect
I also have errors on Metrics logs and Cert-Manager logs relating to failed calling webhook
This is all for now and I would really appreciate some help resolving this.
Aidan
I have pods running on eks, and pulling the container logs worked fine couple days ago. but today when i tried to run kubectl logs podname i get a tls error.
Error from server: Get "https://host:10250/containerLogs/dev/pod-748b649458-bczdq/server": remote error: tls: internal error
does anyone know how to fix this? the other answers on stackoverflow seems to suggest deleting the kubenetes cluster and rebuilding it...... is there no better solutions?
This could probably due to some firewall rules or security settings that were introduced. I would encourage you to check it along with the following troubleshooting steps -
Ensure all EKS nodes are in running state.
Restart nodes as required
Checking networking configuration and see if other kubectl commands are running.
I have a server which is running inside of a kubernetes pod.
Its log output can be retrieved using "kubectl logs".
The application goes through some start up before it is ready to process incoming messages.
It indicates its readiness through a log message.
The "kubectl logs" command is not available from within the pod. I think it would be insecure to even try to install it.
Is there a way of either:
getting the log from within the container? or
running a readiness probe that is executed outside of the container?
(rather than as a docker exec)
Here are some options I've considered:
Redirecting the output to a log file loses it from "Kubectl log"
Teeing it to a log file avoids that limitation but creates an unnecessary duplicate log.
stdout and stderr of the application are anonymous pipes (to kubernetes) so eavesdropping on /proc/1/fd/1 or /proc/1/fd/2 will not work.
A better option may be to use the http API. For example this question
kubectl proxy --port=8080
And from within the container:
curl -XGET http://127.0.0.1:8080/api
However I get an error:
Starting to serve on 127.0.0.1:8080
I0121 17:05:38.928590 49105 log.go:172] http: Accept error: accept tcp 127.0.0.1:8080: accept4: too many open files; retrying in 5ms
2020/01/21 17:05:38 http: proxy error: dial tcp 127.0.0.1:8080: socket: too many open files
Does anyone have a solution or a better idea?
You can actually do what you want. Create a kubernetes "serviceaccount" object with permissions just to do what you want, use the account for your health check pod, and just run kubectl logs as you described. You install kubectl, but limit the permissions avaialable to it.
However, there's a reason you don't find examples of that- its not a great way of building this thing. Is there really no way that you can do a health check endpoint in your app? That is so much more convenient for your purposes.
Finally, if the answer to that really is "NO", could you have your app write a ready file? Instead of print "READY" do touch /app/readyfile. then your health check can just check if that file exists. (to make this work, you would have to create a volume and mount it at /app in both your app container and the health check container so they can both see the generated file)
Too many open files was because I did not run kubectl with sudo.
So the log can be retrieved via the http API with:
sudo kubectl proxy --port 8080
And then from inside the app:
curl -XGET http://127.0.0.1:8080/api/v1/namespaces/default/pods/mypodnamehere/log
That said, I agree with #Paul Becotte that having the application created a ready file would be a better design.
I have downloaded/installed Kubernetes, Virtual Box and MiniKube. Later, I started minikube on VM. When I try running kubectl version command from my terminal I receive the below error message. Could anybody tell me what is the reason behind this error. I have explored everywhere but I couldn't find right resolution for this problem. I am new to this and just taking baby steps. Any help would be appreciated. Thank you.
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?
Could anybody tell me what is the reason behind this error
It is because your kubectl configuration file, housed at $HOME/.kube/config, points at that IP address and port but there is nothing listening on that IP and port.
Using the minikube status command will tell you what it thinks is going on, and minikube ssh will place you inside the virtual machine and allow you to look around for yourself, which can be helpful to get the docker logs to say why there is nothing listening on the port you were expecting.
A good place to start is to run minikube ip and see if it matches the IP address kubectl is expecting (as seen in the error message). If not, update your kubeconfig accordingly.
minikube ssh
and then
journalctl -u kubelet
The above should provide you with additional information about why the server is refusing connections.
This answer might also be helpful: How to diagnose Kubernetes not responding on API
If you are running behind a proxy, make sure to export the NO_PROXY env variable or permanently set it in your /etc/environment file.
export NO_PROXY=192.168.99.0/24,127.0.0.1,...
Yesterday, my kubernetes cluster is suddenly down
I tried to investigate as the follows but not sure what the reason was:
Unable to access Kube Dashboard, it returns HTTP ERROR 502
Unable to access deployed apps on cluster, it also returns 502 error
Cannot use kubectl command, it shows the message: "Unable to connect
to the server: x509: certificate has expired or is not yet valid"
With this error, I googled and got the article.
But I'm not sure if this is correct or not.
Can you please help to advise.
Thank you so much.
Environment:
Kubernetes 1.5
Kube-aws