How to connect to Dask Jupyter notebook running in minikube from internet? - kubernetes

I have a dask cluster running in minikube in a remote VM (Oracle Linux, 64GB, 8 core). The VM connects through proxy to external networks.
I want to access the Jupyter notebook provided by Dask from my browser running in my local Mac.
I would like to understand what are the options available for me to setup this connection.
Here is what I tried:
minikube start --driver=docker --base-image="gcr.io/k8s-minikube/kicbase:v0.0.29" --memory 32768 --cpus 6
minikube tunnel
helm install mydask dask/dask --set scheduler.serviceType=LoadBalancer --set jupyter.serviceType=LoadBalancer
While this does provide external-ip to the 'mydask-jupyter' service, the IP is not in the same subnet as my VM. Therefore it is not publicly accessible.
Next I tried starting minikube as below:
minikube start --driver=none
However running into other errors:
💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.23.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem": exit status 1
stdout:
[init] Using Kubernetes version: v1.23.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
stderr:
[WARNING Firewalld]: firewalld is active, please ensure ports [8443 10250] are open or your cluster may not function correctly
[WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
[WARNING Hostname]: hostname "my-dask" could not be reached
[WARNING Hostname]: hostname "my-dask": lookup my-dask on <<IP>> no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.23.1: output: Trying to pull repository k8s.gcr.io/kube-apiserver ...
Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
...
My Questions:
What configurations are required to fix this?
Is there a better alternative to minikube that will be convenient for this use-case?
Thank you.

Related

How can I rename master nodes in a HA kubernetes cluster?

I have a kubernetes cluster with 3 master nodes. They are named master-1, master-2 and master-3. I would like to rename them as control-plane-n.
I could not find a clear procedure to do this. The closest one is how to rename a node in a cluster. So I just tried that. Here is what I did (my hosts are running ubuntu 18.04, and kubernetes v1.16.2):
On master-1:
kubectl drain master-3 --ignore-daemonsets
kubectl delete node master-3
Run "kubeadm token create --print-join-command" and copy the output
On master-3:
sudo kubeadm reset
sudo hostnamectl set-hostname control-plane-3
Modify /etc/cloud/cloud.cfg to set preserve_hostname to true
Reboot the VM
Paste in the join command from master-1, with --control-plane option added
Here is the log I got:
sudo kubeadm join 172.22.19.188:6443 --control-plane --token nxxzby.zsfdx86e7cv1rq0e --discovery-token-ca-cert-hash sha256:553366c2f91fd3abffe3e3d1c39d9314e2d73e8a6181f4da9938a8e24fd77456
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/data/kubernetes/pki"
error execution phase control-plane-prepare/certs: error creating PKI assets: failed to write or validate certificate "apiserver": certificate apiserver is invalid: x509: certificate is valid for master-3, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not control-plane-3
To see the stack trace of this error execute with --v=5 or higher
How can I proceed? Or is there a better approach?
Thanks in advance for any idea or suggestion you can offer.
Based on #zerkms comment, you can create a 4th node with a proper name, join, then remove one of the old from the cluster.
Doing this 3 times you will be able to have all node with the desired name.

When I run sudo minikube start --vm-driver=none it gives me error

When I run sudo minikube start --vm-driver=none it gives me this error and I am using Ubuntu 16.0.4.
Error starting cluster: cmd failed: sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
output: [init] Using Kubernetes version: v1.16.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING FileExisting-ebtables]: ebtables not found in system path
[WARNING FileExisting-socat]: socat not found in system path
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
[WARNING Hostname]: hostname "minikube" could not be reached
[WARNING Hostname]: hostname "minikube": lookup minikube on 127.0.1.1:53: no such host
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[WARNING Port-10250]: Port 10250 is in use
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Port-10251]: Port 10251 is in use
[ERROR Port-10252]: Port 10252 is in use
[ERROR Port-2380]: Port 2380 is in use
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
: running command: sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
.: exit status 1
The none driver makes a lot of assumptions that would normally be handled by the VM setup process used by all other drivers. In this case you can see that some of the ports it expects to use are already in use so it won't continue. You would need to remove whatever is using those ports. The none driver is generally used for very niche situations, almost always in an ephemeral CI environment, though maybe also check out KinD as a newer tool that might address that use case better. If you just want to run a local dev environment on Linux without an intermediary VM, maybe try k3s or microk8s instead.
In the documentation there is a note about the need of having these ports free. You need to sort it out what is using them to get rid of these errors:
[WARNING Port-10250]: Port 10250 is in use
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Port-10251]: Port 10251 is in use
[ERROR Port-10252]: Port 10252 is in use
[ERROR Port-2380]: Port 2380 is in use
This is preventing you system to load and it's described on the documentation. To investigate you can use the following command:
$ sudo lsof -i :10251
If you don't have this command installed you can install it using:
$ sudo apt-get install lsof
Running minikube with --vm-driver=none is possible but not recommended. By using none Driver, you have decreased reliability, decreased security, risk of data loss and more. Refer to documentation for more details.
There is an issue opened to address these security problems.
Full list of open ‘none’ driver issues
The best way of doing it on Linux is using kvm. To accomplish that you can follow these instructions (assuming you have Docker CE installed):
$ sudo apt-get install -y qemu-kvm libvirt-clients libvirt-daemon-system
$ sudo adduser $USER libvirt
And run minikube with kvm2 driver:
$ minikube start --vm-driver kvm2 --kubernetes-version v1.15.0

The connection to the server xxxx:6443 was refused - did you specify the right host or port?

I follow this to install kubernetes on my cloud.
When I run command kubectl get nodes I get this error:
The connection to the server localhost:6443 was refused - did you specify the right host or port?
How can I fix this?
If you followed only mentioned docs it means that you have only installed kubeadm, kubectl and kubelet.
If you want to run kubeadm properly you need to do 3 steps more.
1. Install docker
Install Docker ubuntu version. If you are using another system chose it from left menu side.
Why:
If you will not install docker you will receive errror like below:
preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": e
xecutable file not found in $PATH
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
2. Initialization of kubeadm
You have installed properly kubeadm and docker but now you need to initialize kubeadm. Docs can be found here
In short version you have to run command
$ sudo kubeadm init
After initialization you will receive information to run commands like:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
and token to join another VM to cluster. It looks like
kubeadm join 10.166.XX.XXX:6443 --token XXXX.XXXXXXXXXXXX \
--discovery-token-ca-cert-hash sha256:aXXXXXXXXXXXXXXXXXXXXXXXX166b0b446986dd05c1334626aa82355e7
If you want to run some special action in init phase please check this docs.
3. Change node status to Ready
After previous step you will be able to execute
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-kubeadm NotReady master 4m29s v1.16.2
But your node will be in NotReady status. If you will describe it $ kubectl describe node you will see error:
Ready False Wed, 30 Oct 2019 09:55:09 +0000 Wed, 30 Oct 2019 09:50:03 +0000 KubeletNotReady runtime network not ready: Ne
tworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
It means that you have to install one of CNIs. List of them can be found here.
EDIT
Also one thing comes to my mind.
Sometimes when you turned off and on VM you need to restart
kubelet and docker service. You can do it by using
$ service docker restart
$ systemctl restart kubelet
Hope it helps.
Looks like kubeconfig file is missing.. Did you copy admin.conf file to ~/.kube/config ?
Verify if there are any proxies set like "http_proxy" or "https_proxy", mostly we set it as environment variables. If yes, then remove the proxies and it should work for you.
I did the following 2 steps. The kubectl works now.
$ service docker restart
$ systemctl restart kubelet

kubeadm fails to initialize when kubeadm init is called

I am new to kubernetes and trying to configure kubernetes master node, I have installed kubeadm, kubectl and kubelet following
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
but when I try to start kubeadm by typing kubeadm init, it gives me the following error
[init] Using Kubernetes version: v1.14.0
[preflight] Running pre-flight checks
[WARNING Firewalld]: no supported init system detected, skipping checking for services
[WARNING Service-Docker]: no supported init system detected, skipping checking for services
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: no supported init system detected, skipping checking for services
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
It seems you have stale data present on the system. To remove that data (/etc/kubernetes) directory run:
kubeadm reset
Now you need to set ip_forward content with 1 by following command:
echo 1 > /proc/sys/net/ipv4/ip_forward
This should resolve your issue.
The br_netfilter module is required for kubernetes installation. Enable this kernel module so that the packets traversing the bridge are processed by iptables for filtering and for port forwarding, and the kubernetes pods across the cluster can communicate with each other.
Run the command below to enable the br_netfilter kernel module.
#modprobe br_netfilter
#echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
or
#nano /proc/sys/net/ipv4/ip_forward
you should see 0
delete 0 and write 1

Minikube on Windows with VirtualBox: Connection attempt fail

I got Kubernetes Minikube on my laptop (4cores, 8 GB RAM). I just performed the basic installation steps (got miniKube and kubectl, enabled the BIOS virtualization) and I am able to start the cluster:
C:\Users\me>minikube start
Starting local Kubernetes cluster...
Starting VM...
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.
However, when I try to interact with the cluster, I allways get the same error, sample:
C:\Users\me>kubectl get pods --context=minikube
Unable to connect to the server: dial tcp 192.168.99.100:8443: connectex: A connection attempt failed because the connected party
did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
I execute minikube ip and I ping the result IP and I get a response. Also I tried to give more memory (3Gb vs the standard 2Gb) and nothing changed.
Am I doing something wrong here?
Thanks!
I had same issue as above. I found out that kubectl couldn't connect to the cluster and would throw up the error when i'm on a VPN connection. When I turned off my VPN client, it started working as fine.
I think it could be some problem with the cluster, when I run minikube status I've got the mixed results of cluster running and cluster stopped:
First run:
c:\> minikube status
minikube: Running
cluster: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
Second run:
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
Third run:
minikube: Running
cluster: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100
The service is flapping.
UPDATED:
Connecting to the minikube vm using minikube ssh I realized the kubeconfig file have wrong path separator for certificates generated by minikube automatic configuration. The path on kubeconfig file stands for \var\lib\localkube\certs\ca.cert and it have to be /var/lib/localkube/certs/ca.cert and so on...
To update the file I have to copy the content of the orignal file to my desktop, fix the directory separators and save the correct file to /var/lib/localkube/kubeconfig and restart the service using:
sudo systemclt restart localkube.
I hope everyone can use minikube with this tip.
If it keep to hit 8443 connection issue when changed work environment, would simplify turn off TLS verification for minikube local cluster if there is not clue.
https://github.com/robertluwang/docker-hands-on-guide/blob/master/minikube-no-tls-verify.md
Hope it is helpful for you.
BR/
Robert
from the documentation:
for Troubleshooting
Run minikube start --alsologtostderr -v=7 to debug crashes
I had the same problem:
check if a some service of a VPN is running by checking the task management, for me, I had a running service of my VPN, so kill the task and try to run the command showed above