Today my pod could not start and show this error:
2021-04-22 12:41:26.325 WARN 1 --- [ngPollService-1] c.c.f.a.i.RemoteConfigLongPollService : Long polling failed, will retry in 64 seconds. appId: 0010010006, cluster: default, namespaces: TEST1.RABBITMQ_CONFIG_REPORT+TEST1.RABBITMQ-CONFIG+application+TEST1.EUREKA+TEST1.DATASOURCE-DRUID+TEST1.COMMON_CONFIG+TEST1.REDIS-CONFIG, long polling url: null, reason: Get config services failed from http://service-apollo-config-server-test-alpha.sre.svc.cluster.local:8080/services/config?appId=0010010006&ip=172.30.184.11 [Cause: Could not complete get operation [Cause: Connection refused (Connection refused)]]
this error tell me this pod could not access the config service, and fetch config failed from config center, so it could not start. Then I login to another node(work fine node) pod and curl the config pod like this:
curl http://service-apollo-config-server-test-alpha.sre.svc.cluster.local:8080
works fine. so the config service is ok. now I run the same command in the problem node pod:
bash-4.4# curl http://service-apollo-config-server-test-alpha.sre.svc.cluster.local:8080
curl: (7) Failed to connect to service-apollo-config-server-test-alpha.sre.svc.cluster.local port 8080: Connection refused
bash-4.4# curl http://service-apollo-config-server-test-alpha.sre.svc.cluster.local:8080
and I ping the config node like this from problem node, works fine :
ping service-apollo-config-server-test-alpha.sre.svc.cluster.local
then I scan the config node using nmap from problem node:
bash-4.4# nmap service-apollo-config-server-test-alpha.sre.svc.cluster.local
Starting Nmap 7.70 ( https://nmap.org ) at 2021-04-22 12:45 CST
Nmap scan report for service-apollo-config-server-test-alpha.sre.svc.cluster.local (10.254.82.131)
Host is up (0.000010s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
22/tcp open ssh
111/tcp open rpcbind
3306/tcp open mysql
8443/tcp open https-alt
did not found the 8080 port. seems network is fine but could not access the service from node. why the problem node pod could not access the config service? what should I do to find out the problem and fix it? I found on the problem node using pod ip it could work, for example:
# pod ip access works
curl 172.30.112.2:11025
# service ip failed
curl 10.254.94.209:11025
# service name failed
curl soa-illidan-superhub.dabai-fat.svc.cluster.local:11025
Finally I found the kube-proxy process was exit, in CentOS 7.6, using this command to start:
systemctl start kube-proxy
fix it.
Related
I am attempting to reverse port forward information from a Kubernetes pod 192.168.49.2:30085 to my locoalhost:8000 but I am having issues.
The goal is to get the same response as curl 192.168.49.2:30085 on localhost:8000
One thing I have tried has been using ktunnel with the command ktunnel expose <service_name> 30085:8000 to try to forward this info. Doing this I get the output
INFO[0000] Exposed service's cluster ip is: 10.101.77.99
.INFO[0000] waiting for deployment to be ready
.....
INFO[0001] port forwarding to https://192.168.49.2:8443/api/v1/namespaces/default/pods/thisisatest-59d5584c84-b5qrc/portforward
INFO[0001] Waiting for port forward to finish
INFO[0001] Forwarding from 127.0.0.1:28688 -> 28688
INFO[2022-08-03 20:23:48.605] starting tcp tunnel from source 30085 to target 8000
Which seems normal and I am able to get a response when using curl http://10.101.77.99:30085 on local host 8000 but not the correct response.
I have also attempted to run a tcp server with the command nc 127.0.0.1 8000 | kubectl exec -i <pod_name> tcpserver 127.0.0.1 30085 cat But am having poor results as well. Am I using these services incorrectly? Or is there a better way to do this all together?
i have 2 pods(pod1,pod2) with 2 services(service1 links pod1, service2 links pod2) in the same namespace.
Within pod1 i performed below testing. i entered into pod1 using kubectl exec ... and
Invoked restapi exposed in pod2 using wget pod2ipaddress - success
Invoked restapi exposed in pod2 using wget service2ipaddress - success
Invoked restapi exposed in pod2 using wget service2Name - failure with message Could not resolve host: pcgamesapp-service
curl https://www.google.com - failure with message Could not resolve host: www.google.com
Executed below nslookup command , but received connection timeout error as shown in below screenshot
Setup
Initalised k8s cluster (v1.22.9) with 1 masterNode and 1 workerNode on ubuntu 20.04 using kubeadm in AWS EC2 virtual machines
weavenet plugin for networking
i tracked the kube-dns logs but the request information of curl serviceName doesnot showup in the kube-dns logs
kubectl logs --namespace=kube-system -l k8s-app=kube-dns
i followed the steps mentioned in dns-debugging-resolution documentation, but i'm not able to figure out a solution
From reading the previous answers i assume that there is some issue with coredns or iptables ...etc . But i'm unable to figure out a solution for this issue.
Please suggest
i received the solution from a member named felix-dpg(#fdpg in stackoverflow) through another platform. i have added the solution details below
The information related to "which ports has to be opened for master node and worker node" are in this link
The weavenet port details are in this link
Initially i didnot open (6783,6784)TCP,(6783,6784)UDP ports for Weavenet and port 53 for UDP(to enable inter-communication among the nodes in the cluster). The issue which i reported in this question got resolved by opening the ports for Weavenet and UDP
Ports opened for Master node
Ports opened for Worker node
Please let me know if you have any doubts or concerns with the above details. Thank you
I'm trying to setup k8s locally on my own mac, and after installing all the dependencies, I try to run minikube start, but get the following error message:
😄 minikube v1.4.0 on Darwin 10.14.6
💡 Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🏃 Using the running virtualbox "minikube" VM ...
⌛ Waiting for the host to be provisioned ...
🐳 Preparing Kubernetes v1.16.0 on Docker 18.09.9 ...
E1006 09:57:30.975647 22071 cache_images.go:79] CacheImage k8s.gcr.io/kube-apiserver:v1.16.0 -> /Users/chrisbao/.minikube/cache/images/k8s.gcr.io/kube-apiserver_v1.16.0 failed: fetching image: Get https://k8s.gcr.io/v2/: dial tcp [2404:6800:4008:c04::52]:443: i/o timeout
E1006 09:57:30.976341 22071 cache_images.go:79] CacheImage gcr.io/k8s-minikube/storage-provisioner:v1.8.1 -> /Users/chrisbao/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1 failed: fetching image: Get https://gcr.io/v2/: dial tcp [2404:6800:4008:c00::52]:443: i/o timeout
and minikube status command returns the following status info:
host: Running
kubelet:
apiserver: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
so how to debug and fix it? what's the potential reason?
E1006 09:57:30.975647 22071 cache_images.go:79] CacheImage k8s.gcr.io/kube-apiserver:v1.16.0 -> /Users/chrisbao/.minikube/cache/images/k8s.gcr.io/kube-apiserver_v1.16.0 failed: fetching image: Get https://k8s.gcr.io/v2/: dial tcp [2404:6800:4008:c04::52]:443: i/o timeout
Looks like you aren't able to pull the k8s api server image from GCR. You can try use one of the available image mirrors by using the --image-repository or --image-mirror-country flags. E.g., if you are based in China, you can start minikube with:
minikube start --image-mirror-country=cn
You're getting a connection timeout when trying to pull images.
"Get https://k8s.gcr.io/v2/: dial tcp [2404:6800:4008:c04::52]:443: i/o timeout"
Can you confirm that you're able to access the internet from within your minikube VM?
minikube ssh
ping google.com
you should see something like
I was able to cluster 2 nodes together in Kubernetes. The master node seems to be running fine but running any command on the worker node results in the error: "The connection to the server localhost:8080 was refused - did you specify the right host or port?"
From master (node1),
$ kubectl get nodes
NAME STATUS AGE VERSION
node1 Ready 23h v1.7.3
node2 Ready 23h v1.7.3
From worker (node 2),
$ kubectl get nodes
The connection to the server localhost:8080 was refused - did you specify the right host or port?
$ telnet localhost 8080
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
$ ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.032 ms
I am not sure how to fix this issue. Any help is appreciated.
On executing,"journalctl -xeu kubelet" I see:
"CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container", but this seems to be related to installing a pod network ... which I am not able to because of the above error.
Thanks!
kubectl interfaces with kube-apiserver for cluster management. The command works on the master node because that's where kube-apiserver runs. On the worker nodes, only kubelet and kube-proxy is running.
In fact, kubectl is supposed to be run on a client (eg. laptop, desktop) and not on the kubernetes nodes.
from master you need ~/.kube/config pass this file as argument for kubectl command. Copy the config file to other server or laptop then pass the argument as for kubectl command
eg:
kubectl --kubeconfig=~/.kube/config
This worked for me after executing following commands:
$ sudo mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
As a hint, the message being prompted indicates its related to network.
So one potential answer could also be, which worked for my resolution, is to have a look at the key cluster value for context within contexts.
My error was that I had placed an incorrect cluster name there.
Having the appropriate cluster name is crucial to finding it for respective context and the error will disappear.
To solve the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?, you may be missing a step.
My Fix:
On MacOS if you install K8s with brew, you still need to brew install minikube, afterwards you should run minikube start. This will start your cluster.
Run the command kubectl cluster-info and you should get a happy path response similar to:
Kubernetes control plane is running at https://127.0.0.1:63000
KubeDNS is running at https://127.0.0.1:63308/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Kubernetes install steps: https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/
Minikube docs: https://minikube.sigs.k8s.io/docs/start/
Ensure what context is selected if you're running Kubernetes in the Docker Desktop.
Once you've selected it right, you'll be able to run the kubectl commands without any exception:
% kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
CoreDNS is running at https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
% kubectl get nodes
NAME STATUS ROLES AGE VERSION
docker-desktop Ready control-plane,master 2d11h v1.22.5
I've set up and deployed a Kubernetes stateful set containing three CockroachDB pods, as per docs. My ultimate objective is to query the database without requiring use of kubectl. My intermediate objective is to query the database without actually shelling into the database pod.
I forwarded a port from a pod to my local machine, and attempted to connect:
$ kubectl port-forward cockroachdb-0 26257
Forwarding from 127.0.0.1:26257 -> 26257
Forwarding from [::1]:26257 -> 26257
# later, after attempting to connect:
Handling connection for 26257
E0607 16:32:20.047098 80112 portforward.go:329] an error occurred forwarding 26257 -> 26257: error forwarding port 26257 to pod cockroachdb-0_mc-red, uid : exit status 1: 2017/06/07 04:32:19 socat[40115] E connect(5, AF=2 127.0.0.1:26257, 16): Connection refused
$ cockroach node ls --insecure --host localhost --port 26257
Error: unable to connect or connection lost.
Please check the address and credentials such as certificates (if attempting to
communicate with a secure cluster).
rpc error: code = Internal desc = transport is closing
Failed running "node"
Anyone manage to accomplish this?
From inside the Kubernetes cluster, you can talk to the database by connecting the cockroachdb-public DNS name. In the docs, that corresponds to the example command:
kubectl run cockroachdb -it --image=cockroachdb/cockroach --rm --restart=Never -- sql --insecure --host=cockroachdb-public
While that command is using the CockroachDB image, any Postgres client driver you use should be able to connect to cockroachdb-public when running with the Kubernetes cluster.
Connecting to the database from outside of the Kubernetes cluster will require exposing the cockroachdb-public service. The details will depend somewhat on how your Kubernetes cluster was deployed, so I'd recommend checking out their docs on that:
https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-the-service
And in case you're curious, the reason forwarding port 26257 isn't working for you is because port forwarding from a pod only works if the process in the pod is listening on localhost, but the CockroachDB process in the statefulset configuration is set up to listen on the pod's hostname (as configured via the --host flag).