How to resolve Kubernetes API from minikube? - minikube

I can't seem to be able to find the Kubernetes API IP address from within the minikube machine, using minikube ssh.
nslookup kubernetes.default.svc.cluster.local 10.0.0.1
Server: 10.0.0.1
Address 1: 10.0.0.1
nslookup: can't resolve 'kubernetes.default.svc.cluster.local'
Same thing happens with just kubernetes. This happens even though it seems that the kube-dns pods are up-and-running just fine:
# kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
kube-addon-manager-minikube 1/1 Running 0 9m
kube-dns-910330662-16bvw 3/3 Running 0 8m
kubernetes-dashboard-vbn9m 1/1 Running 0 8m
There is no KubeDNS info in cluster-info (but nothing major in the dump):
# kubectl cluster-info
Kubernetes master is running at https://192.168.64.26:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Version for kubectl:
# kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-07-26T00:12:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
and minikube:
# minikube version
minikube version: v0.21.0
This happens on OSX and both with xhyve and virtualbox as drivers.
There are quite a few minikube/dns issues in the minikube issue tracker, but none of them quite like this. What should the kubectl cluster-info output look like for minikube that has kube-dns working? What about the nslookup, shouldn't that work?
EDIT. It seems like kubectl cluster-info for minikube is returning what it should, but I can't even resolve kubernetes from inside the kube-dns pod either:
# kubectl -n kube-system exec -it kube-dns-910330662-1n64f -c kubedns -- nslookup kubernetes.default.svc.cluster.local localhost
Server: 127.0.0.1
Address 1: 127.0.0.1 localhost
nslookup: can't resolve 'kubernetes.default.svc.cluster.local': Name does not resolve
which really should work?

Related

Kubernetes - unable to run echoserver on minikube

I am following this example
When I run the following command I get the error:
➜ kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10
Error from server (NotFound): the server could not find the requested resource
I just installed fresh minikube, kubectl and the data is below:
Development/tools/k8s
➜ kubectl get nodes
NAME STATUS AGE
minikube Ready 2m
Development/tools/k8s
➜ kubectl get pods
No resources found.
Development/tools/k8s
➜ kubectl get rc --all-namespaces
No resources found.
Development/tools/k8s
➜ kubectl cluster-info
Kubernetes master is running at https://192.168.99.101:8443
KubeDNS is running at https://192.168.99.101:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
➜ kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:40:50Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
➜ minikube version
minikube version: v1.7.3
commit: 436667c819c324e35d7e839f8116b968a2d0a3ff
What am I doing wrong?
Your kubectl is version 1.5 which is very old. It’s trying to use an outdated format for the Deployment resource which no longer exists in the server.

unable to launch kubectl dashboard on Mac

I have deployed minikube on MacOS using the instructions here https://kubernetes.io/docs/tasks/tools/install-minikube/
The brew install was ok and the minikube status shows
$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.102
I am able to interact with the cluster using kubectl
$kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.10 --port=8080
Viewing the pods is also ok
$kubectl get pod
NAME READY STATUS RESTARTS AGE
hello-minikube-856979d68c-glhsx 1/1 Running 0 18m
But when i try to launch the kubectl dashboard, i get 503 error
$minikube dashboard
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
The Dashboard service seems to present
$kubectl -n kube-system get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3h19m k8s-app=kube-dns
kubernetes-dashboard ClusterIP 10.109.210.119 <none> 80/TCP 119m app=kubernetes-dashboard
below is the kubectl version info
$kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T16:57:42Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Any pointers on what is missing ? How to get the dashboard working
Thanks
Praveen
Check kubectl cluster-info, you can find more here
kubectl -n kube-system port-forward svc/kubernetes-dashboard 8080:80
Your dashboard should be accessible on http://localhost:8080, keep in mind that the dashboard is deprecated so you can check octant.
Try $ minikube dashboard command. It will open a new tab in your default browser showing the minikube dashbord.
Finally able to solve this issue with instructions from GitHub issues
https://github.com/kubernetes/minikube/issues/4352
Basically, with these commands
minikube stop
minikube start --extra-config=apiserver.authorization-mode=RBAC
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
minikube dashboard

Get ServiceUnavailable from `kubectl top` with heapster

I have a managed kubernetes setup, called Cluster Container Engine (CCE), in the Open Telekom Cloud. Their documentation can be found online.
My CCE has one master and three nodes which run k8s version 1.9.2 (more details below). I can access the CCE through kubectl and deploy new pods onto it.
The CCE has a deployment of heapster preinstalled. However, attempting to inspect node resource usage fails (I can observe the same effect for pod usage):
$ kubectl top pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)
I've attempted all debugging steps I could think of (see below) and I'm still lost when it comes to fixing this. Any advice?
The deployment, pod and service items for heapster are present (outputs filtered to include only heapster):
$ kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
heapster-apiserver-84b844ffcf-lzh4b 1/1 Running 0 47m
$ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heapster ClusterIP 10.247.150.244 <none> 80/TCP 19d
$ kubectl get deploy -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
heapster-apiserver 1/1 1 1 19d
To check that heapster does indeed collect metrics properly, I've ssh'd into one of the nodes and executed:
$ curl -k http://10.247.150.244:80/api/v1/model/metrics/
[
"cpu/usage_rate",
"memory/usage",
"cpu/request",
"cpu/limit",
"memory/request",
"memory/limit"
]
Pod Log Output
Finally, I checked the log output from the heapster-apiserver-84b844ffcf-lzh4b pod:
$ kubectl logs -n kube-system heapster-apiserver-84b844ffcf-lzh4b
I0311 13:38:18.334525 1 heapster.go:78] /heapster --source=kubernetes.summary_api:''?kubeletHttps=true&inClusterConfig=false&insecure=true&auth=/srv/config --api-server --secure-port=6443
I0311 13:38:18.334718 1 heapster.go:79] Heapster version v1.5.3
I0311 13:38:18.340912 1 configs.go:61] Using Kubernetes client with master "https://192.168.1.228:5443" and version <nil>
I0311 13:38:18.340996 1 configs.go:62] Using kubelet port 10255
I0311 13:38:18.358918 1 heapster.go:202] Starting with Metric Sink
I0311 13:38:18.510751 1 serving.go:327] Generated self-signed cert (/var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key)
E0311 13:38:18.540860 1 heapster.go:128] Could not create the API server: missing clientCA file
I0311 13:38:18.558944 1 heapster.go:112] Starting heapster on port 8082
Cluster Info
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-CCE2.0.7-B003", GitCommit:"302f471a1e2caa114c9bb708c077fbb363aa2f13", GitTreeState:"clean", BuildDate:"2018-06-20T03:27:16Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get nodes
192.168.1.163 Ready worker 19d v1.9.2-CCE2.0.7-B003
192.168.1.211 Ready nfs-server 19d v1.9.2-CCE2.0.7-B003
192.168.1.227 Ready worker 19d v1.9.2-CCE2.0.7-B003
All nodes use EulerOS_2.0_SP2 with kernel version 3.10.0-327.59.59.46.h38.x86_64.
I0311 13:38:18.510751 1 serving.go:327] Generated self-signed cert (/var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key)
It seems like your API server is running on HTTP but heapster has a https url configured. You need to set the --source parameter to override the Kubernetes master like described here:
--source=kubernetes:http://master-ip?inClusterConfig=false&useServiceAccount=true&auth=
BTW: heapster has been deprecated and it is advised to switch to metrics server.

Can not connect to service in 192.168.99.101

I am new on kubernetes and need your help.
I followed installation instruction on this page and everything worked fine. Here are some Info after installation:
~ > kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:10Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T06:59:37Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
~ > kubectl cluster-info
Kubernetes master is running at https://192.168.99.101:8443
~ > minikube ip
192.168.99.101
Then I try to run a simple hello world service:
~ > minikube start
~ > kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready <none> 18h v1.12.4
~ > kubectl run hw --image=karthequian/helloworld --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/hw created
~ > kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hw 1 1 1 1 21s
~ > kubectl get rs
NAME DESIRED CURRENT READY AGE
hw-854c64787 1 1 1 71s
~ > kubectl get pods
NAME READY STATUS RESTARTS AGE
hw-854c64787-8xhw6 1/1 Running 0 59s
~ > kubectl expose deployment hw --type=NodePort
service/hw exposed
~> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hw NodePort 10.106.4.194 <none> 80:31269/TCP 6m23s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20h
~ > minikube service hw
Opening kubernetes service default/hw in default browser...
After the last line I got this output:
Did I do something wrong? Is there anything I have not configured right? Thank you.
Here is how I solved this problem. First I go to Docker -> Preferences -> Kubernetes and Tick "Enable Kubernetes". It will take around 5 mins to install.
Then I delete my current minikube with minikube delete and repeat the whole process(minikube start, kubectl run hw --image=karthequian/helloworld --port=80, kubectl expose deployment hw --type=NodePort, minikube service hw
) and it works :).

kubernetes, prompt freezes at port forward command

I want to set up port forwarding from my local ports (nodes) to the pod redis-master after the online guide from kubernetes.
At the moment my prompt is frozen for more than 5 minutes at the port-forward command.
[root#k8s-master deployments]# kubectl create -f https://k8s.io/docs/tasks/access-application-cluster/redis-master.yaml
pod "redis-master" created
[root#k8s-master deployments]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-master 2/2 Running 0 1m
[root#k8s-master deployments]# kubectl get pods redis-master --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
6379
[root#k8s-master deployments]# kubectl port-forward redis-master 6379:6379
Forwarding from 127.0.0.1:6379 -> 6379
^C
I don't know why my prompt is frozen. In my logs aren't some error or warn entries.
journalctl -u kubelet.service -f --since "2018-02-19 10:30:00" --priority 0
-- Logs begin at Sa 2018-02-03 21:21:50 CET. --
kubectl version
[root#k8s-master deployments]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
os-release
[root#k8s-master deployments]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Ports
[root#k8s-master deployments]# ss -atun | grep 6379
[root#k8s-master deployments]#
The behaviour you see is expected. This command does not get daemonized by default. It will be forwarding the port until you kill the command with CTRL-C or other similar methods.
You could try using & at the end of the command if you want to continue using that prompt. Personally I would use a terminal multiplexer like tmux or screen.