Problem
When I connect a k3s agent to the server and run "kubectl get nodes" on the server. I get the following error:
root#k3s-master:/home/marc# kubectl get nodes
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
But if the server is standalone, I can easily run "kubectl get nodes".
CPU utilisation on the server stays about 30-40%. And RAM usage is at 583M of 3.74G.
Specs
2x Raspberry Pi 4b with 4GB RAM
Fresh install of raspbian lite (buster)
Enabled legacy iptables and cgroups
K3S Installation
On the server, I've done:
curl -sfL https://get.k3s.io | sh -
And on the agent:
curl -sfL https://get.k3s.io | K3S_URL=https://k3s-master:6443 K3S_TOKEN=<token> sh -
Thanks in advance, it drives me crazy!
Thanks to stack overflows related question feature, I've stumbled upon this question: PI4 k3s install server currently unable to handle the request
There seems to be an issue regarding cgroup memory failures with buster kernel 5.4.x
Edit
Adding the cgroup didnt helped, so I switched to Ubuntu 20.05 and its working now.
Related
I'm total beginner for kubernetes and tried various methods in stackoverflow, youtube and blogs. None of them worked for me.
When running kubectl command randomly getting following error:
Eg: kubectl get nodes
The connection to the server 192.168.1.105:6443 was refused - did you specify the right host or port?
Name : kubelet
Version : 1.25.3
OS : Rocky Linux with vmware player
Working nodes: Rocky Linux 2x with same software versions and specs
tried
swapoff -a
Restarting services (docker, kubectl)
Completely re-installed cluster
Config file added : ~/home/.kube/config
daemon.json added
Below command will help to identify isssue
crictl info
Install CRI
The reason is we should tell which CRI should use for the cluster.
Every node should have same CRI. I prefer docker engine.
https://kubernetes.io/docs/setup/production-environment/container-runtimes/
https://github.com/Mirantis/cri-dockerd
kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock
kubeadm join 192.168.1.200:6443 --token 49tl1j.0txot1q9s94o6wu7 --discovery-token-ca-cert-hash sha256:98a02f3a60fc4c00f399726a5d2b61213a714f8c9be3700eb648315094fa9ee0 --cri-socket=unix:///var/run/cri-dockerd.sock
I'm currently testing OS and version upgrades for a ceph cluster. Starting info:
The cluster is currently on Centos 7 and Ceph version Nautilus. I'm trying to change OS with ubuntu 20.04 and version with Octopus. I started with upgrading mon1 first. I will write down the things done in order.
First of I stopped monitor service - systemctl stop ceph-mon#mon1
Then I removed the monitor from cluster - ceph mon remove mon1
Then installed ubuntu 20.04 on mon1. Updated the system and configured ufw.
Installed ceph octopus packages.
Copied ceph.client.admin.keyring and ceph.conf to mon1 /etc/ceph/
Copied ceph.mon.keyring to mon1 to a temporary folder and changed ownership to ceph:ceph
Got the monmap ceph mon getmap -o ${MONMAP} - The thing is i did this after removing the monitor.
Created /var/lib/ceph/mon/ceph-mon1 folder and changed ownership to ceph:ceph
Created the filesystem for monitor - sudo -u ceph ceph-mon --mkfs -i mon1 --monmap /folder/monmap --keyring /folder/ceph.mon.keyring
After noticing I got the monmap after the monitors removal I added it manually - ceph mon add mon1 <ip> --fsid <fsid>
After starting manually and checking cluster state with ceph -s I can see mon1 is listed but is not in quorum. The monitor daemon runs fine on the said mon1 node. I noticed on logs that mon1 is stuck in "probe" state and on other monitor logs there is an output such as mon1 (rank 2) addr [v2:<ip>:3300/0,v1:<ip>:6789/0] is down (out of quorum) , as i said the the monitor daemon is running on mon1 without any visible errors just stuck in probe state.
I wondered if it was caused by os&version change so i first tried out configuring manager, mds and radosgw daemons by creating the respective folders in /var/lib/ceph/... and copying keyrings. All these services work fine, i was able to reach to my buckets, was able to open the Octopus version dashboard, and metadata server is listed as active in ceph -s. So evidently my problem is only with monitor configuration.
After doing some checking found this on red hat ceph documantation:
If the Ceph Monitor is in the probing state longer than expected, it
cannot find the other Ceph Monitors. This problem can be caused by
networking issues, or the Ceph Monitor can have an outdated Ceph
Monitor map (monmap) and be trying to reach the other Ceph Monitors on
incorrect IP addresses. Alternatively, if the monmap is up-to-date,
Ceph Monitor’s clock might not be synchronized.
There is no network error on the monitor, I can reach all the other machines in the cluster. The clocks are synchronized. If this problem is caused by the monmap situation how can I fix this?
Ok so as a result, directly from centos7-Nautilus to ubuntu20.04-Octopus is not possible for monitor services only, apparently the issue is about hostname resolution with different Operating systems. The rest of the services is fine. There is a longer way to do this without issue and is the correct solution. First change os from centos7 to ubuntu18.04 and install ceph-nautilus packages and add the machines to cluster (no issues at all). Then update&upgrade the system and apply "do-release-upgrade". Works like a charm. I think what eblock mentioned was this.
I'm trying to install and run a single-node lightweight kubernetes cluster, to play around with on my Raspberry pi4, of which I found k3s. However, from what I've read or seen, I'm probably missing something, but haven't found reference to the exact problem I'm getting (testing with simple kubectl command after installation):
$ kubectl get nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request
The installations that I've referenced:
Turing Pis, multi-node cluster
-> The part of knowing and using Ansible currently seems like bit of a overkill)
Pi setup & k3s install -> Good tutorial, but not having similar config responses?
$ sudo k3s server
INFO[2020-09-30T06:58:13.488363192+01:00] Starting k3s v1.18.9+k3s1 (630bebf9)
INFO[2020-09-30T06:58:13.489450500+01:00] Cluster bootstrap already complete
FATA[2020-09-30T06:58:13.535582640+01:00] starting kubernetes: preparing server: start cluster and https: listen tcp :6443: bind: address already in use
Presumed that this isn't necessary anymore then, based on the newer installation version.
complete k3s 101 youtube -> Still not magically working, as shown.
So if anyone is able to please help me, or guide me in a direction to better debug and display the problem so that I understand and can fix the problem.
Feedback from the installation didn't display that anything went wrong:
$ sudo curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 664" sh -
[INFO] Finding release for channel stable
[INFO] Using v1.18.9+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.9+k3s1/sha256sum-arm.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.9+k3s1/k3s-armhf
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
After that, trying commands:
$ k3s --version
k3s version v1.18.9+k3s1 (630bebf9)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9+k3s1", GitCommit:"630bebf94b9dce6b8cd3d402644ed023b3af8f90", GitTreeState:"clean", BuildDate:"2020-09-17T19:04:57Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/arm"}
Error from server (ServiceUnavailable): the server is currently unable to handle the request
$ sudo kubectl get nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request
$ sudo k3s kubectl get nodes
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
And looking with htop, definitely 'something' is happening with k3s servers:
Not sure if anything is missing, or must be changed to hosts, for k3s server + agent on device:
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 raspberrypi
... No clue what to debug further??
After learning a bit more of the installation process, by watching this video (k3s install on Pi4 - live walkthrough), I noticed that k3s runs as a service on raspbian.
meaning you're able to:
# see all listed services, to find the name of the running k3s service
$ systemctl --type=service
# service name ironically being 'k3s', and being able to follow the logs for service
$ journalctl -u k3s -f
However, looking in '/boot/cmdline.txt', these cgroup values where in the file, but after a endline-character, which prohibited the k3s service sufficiently reading from the file. File content required to be:
$ sudo cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=1 cgroup_memory=1 cgroup_enable=memory
With that done, I checked journalctl again for the logs, and noticed significantly other logs, regarding pod's containers etc. Master node being functional!:
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
raspberrypi Ready master 3m52s v1.18.9+k3s1
If this still doesn't work, I also saw a recent blog post regarding the same issue (due to raspbian kernal update), where fix is also suggested -> post
I am trying to install Kubernetes on windows server 2016.
I tried to install minikube, and got some errors.
This is the tutorial that I followed:
https://www.assistanz.com/installing-minikube-on-windows-2016-server/
This is the command + error that I got:
PS C:\Windows\system32> minikube start –vm-driver=hyperv –hyperv-virtual-switch=Minikube
Starting local Kubernetes v1.10.0 cluster...
Starting VM... Downloading Minikube ISO
170.78 MB / 170.78 MB [============================================] 100.00% 0s
E1106 19:29:10.616564 11852 start.go:168] Error starting host: Error creating host: Error executing step: Running precreate checks.
: VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path.
Retrying.
E1106 19:29:10.689675 11852 start.go:174] Error starting host: Error creating host: Error executing step: Running precreate checks.
: VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
minikube config set WantReportErrorPrompt false
================================================================================
Please enter your response [Y/n]:
Someone knows how to solve it?
I googled it, but no luck.
Thanks!
I was never able to get the config parameters to work with minikube start.
I was able to get past this error using the minikube config commands in PowerShell (should also work at a command prompt):
minikube config set vm-driver hyperv
minikube config set hyperv-virtual-switch ExternalSwitch
minikube config view
minikube delete
minikube start
For more information on the command run: minikube config -h
Looking at the documentation you have provided, I have noticed that the screenshot shows a slight difference to the one they've quote.
I have also found this command in another piece of documentation from kubernetes here, showing the same command as that from the screenshot.
I suggest you try the following command;
minikube start --vm-driver=hyperv --hyperv-virtual-switch=Minikube
It is true that OP has pasted the incorrect command, because there is - instead of --. I tried to pass this arguments to minikube and all you get is an instant error. So the issue must be somewhere else. I remember having similar issue and it got resolved after deleting the .kube and .minikube folders and trying to run it again.
After taking a closer look this tutorial is destined for installation of minikube inside of a Windows Server 2016 Virtual Machine, so you have to have a Nested Virtualization able hardware:
Prerequisites The Hyper-V host and guest must both be Windows Server
2016/Windows 10 Anniversary Update or later. VM configuration version
8.0 or greater. An Intel processor with VT-x and EPT technology -- nesting is currently Intel-only. There are some differences with
virtual networking for second-level virtual machines. See "Nested
Virtual Machine Networking".
So the main question is, is that true in your scenario? Are you trying to perform your steps on Windows Server Hyper-V virtual machine with nested virtualization feature?
If you confirm that I have technical possibilities to check it in that scenario.
Otherwise I recommend using the "traditional way" of running minikube in Windows, according for example to this tutorial.
I've installed docker on Debian 9.3 and created a swarm using 4 computers.
Now I am trying to install Kubernetes locally and am having some trouble getting things to work.
$ uname -a
Linux tma 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3 (2017-12-03) x86_64 GNU/Linux
I'm trying to follow this guide:
https://kubernetes.io/docs/getting-started-guides/minikube/
I want to use Deb9 since that is what I use in our lab.
I am using KVM as the hypervisor.
Has anyone installed Kubernetes locally via Minikube successfully?
I get the following error: when I issue kubectl cluster-info as sudo and non-sudo
$kubectl cluster-info
Kubernetes master is running at localhost:8080 To further debug and diagnose cluster problems, use kubectl cluster-info dump. The connection to the server localhost:8080 was refused - did you specify the right host or port?
I have Minikube running on Debian.
I can reproduce your error if I don't have anything running. Most probably your cluster didn't start, you'll need to debug more.
This is a great Debian 9 + Minikube resource: https://medium.com/linagora-engineering/install-k8s-minikube-on-top-of-kvm-on-debian-9-9cd5b646063c