Background: Something happened yesterday which led to usage of kubectl commands rendering my macbook unusable ( network/internet breaking and freezing all applications).
I fixed it by using
sriramkumar#Srirams-MacBook-Pro ~ % brew link --overwrite kubernetes-cli
Linking /usr/local/Cellar/kubernetes-cli/1.25.0... 224 symlinks created.
sriramkumar#Srirams-MacBook-Pro ~ % ls -al /usr/local/bin/kubectl
lrwxr-xr-x 1 sriramkumar admin 43 8 Sep 23:21 /usr/local/bin/kubectl -> ../Cellar/kubernetes-cli/1.25.0/bin/kubectl
And today I discovered it is helm that is broken, so eg. when I use `helm ls -a`
It is doing the exact same thing ( network/internet breaking and freezing all applications)
The error I'm getting is
sriramkumar#Srirams-MacBook-Pro ~ % helm ls -a
Error: Kubernetes cluster unreachable: Get "https://something.ibm.com:25628/version?timeout=32s": dial tcp: i/o timeout
I have uninstalled helm, reinstalled set KUBECONFIG=/etc/rancher/k3s/k3s.yaml ( from the internet) but it's not solving the issue. Any help is much appreciated
Related
I have running k8s cluster using kops. the autoscaling policy terminate the master machine and recreated a new one since then every time i try to run kubectl command it returns "The connection to the server refused, did you specify the right host or port". i tried to ssh to the master machine but the did not found any of k8s services so i think the autoscale policy did not configure the master node correctly. so what should i do in this situation ?
update: also i found this log in syslog file:
E: Package 'ebtables' has no installation candidate
Jun 25 12:03:33 ip-172-20-35-193 nodeup[7160]: I0625 12:03:33.389286 7160 executor.go:145] No progress made, sleeping before retrying 2 failed task(s)
the issue was the kops was unable to install ebtables and conntrack so i installed it manually by :
sudo apt-get -o Acquire::Check-Valid-Until=false update
sudo apt-get install -y ebtables --allow-unauthenticated
sudo apt-get install --yes conntrack
and everything is running fine now
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
On my CentOS 8 server, many dnf and yum commands fail with this error:
Failed to download metadata for repo
This seems to apply only to repositories involving https connections, e.g.:
/etc/yum.repos.d $ cat epel-modular.repo
[epel-modular]
name=Extra Packages for Enterprise Linux Modular $releasever - $basearch/pub/epel/$releasever/Modular/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-modular-$releasever&arch=$basearch&infra=$infra&content=$contentdir
...
I have used wget to test connectivity to these sites and it is successful, e.g. wget https://mirrors.fedoraproject.org succeeds.
However, various dnf or yum commands fail:
$ dnf provides /bin/ls
Extra Packages for Enterprise Linux Modular 8 - x86_64 0.0 B/s | 0 B 00:00
Failed to download metadata for repo 'epel-modular'
Error: Failed to download metadata for repo 'epel-modular'
# dnf update --refresh
CentOS-8 - AppStream 20 kB/s | 4.3 kB 00:00
CentOS-8 - Base 19 kB/s | 3.8 kB 00:00
CentOS-8 - Extras 7.4 kB/s | 1.5 kB 00:00
CentOS-8 - PowerTools 20 kB/s | 4.3 kB 00:00
Remi's Modular repository for Enterprise Linux 8 - x86_64 4.1 kB/s | 3.5 kB 00:00
Safe Remi's RPM repository for Enterprise Linux 8 - x86_64 3.6 kB/s | 3.0 kB 00:00
Wazuh repository 0.0 B/s | 0 B 00:00
Failed to download metadata for repo 'wazuh_repo'
Error: Failed to download metadata for repo 'wazuh_repo'
How do I resolve this problem?
In my case
sudo rm -r /var/cache/dnf
solved my problem.
Source: https://access.redhat.com/discussions/4222851
I was getting the same error with the Centos8 build inside of a docker container. Fixed by running the below:
# Do on build
RUN dnf clean all && rm -r /var/cache/dnf && dnf upgrade -y && dnf update -y
Error before (i was running yum update and not dnf):
Step 4/5 : RUN yum clean all && yum update -y && echo hostname -a
---> Running in 10d319da361d
0 files removed
CentOS-8 - AppStream 0.0 B/s | 0 B 00:05
Failed to download metadata for repo 'AppStream'
Error: Failed to download metadata for repo 'AppStream'
Success after:
---> 0b96049ee5eb
Step 4/5 : RUN dnf clean all && rm -r /var/cache/dnf && dnf upgrade -y && dnf update -y & echo hostname
---> Running in f928c6da6cb0
hostname
Removing intermediate
I just faced this problem so I thought I'd add what fixed it for me since the resolution was pretty simple. For me, I was overlooking the fact that whenever I changed my hostname, it was wiping out everything in /etc/resolv.conf so I:
nano -w /etc/resolv.conf **
added the following lines:
nameserver 8.8.8.8
nameserver 8.8.4.4
Test it (e.g. yum update -y)
**note: I used nano in example, simply swap the nano portion for your own text editor of choice
If it does not work, check your Internet connection. This error appeared on my CentOS 8 installation, and the problem was that the Internet connection wasn't correctly configured.
Just add the following (DNS server) to /etc/resolv.conf file:
nameserver 9.9.9.9
Will fix the issue :)
I was facing this issue when I tried to run
yum update -y
from a docker container. I changed the version from centos:8 to centos:7 and it solved the issue for me.
You may not realize it, but yum and dnf require SSL certificates when communicating via secure HTTP (aka HTTPS). Have you checked the certificates under /etc/pki/tls/certs? There should be at least two, for example:
/etc/pki/tls/certs# ls -l
total 4
lrwxrwxrwx 1 root root 49 Jan 30 12:48 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root 55 Dec 11 13:19 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
If these links are missing you may be able to simply restore them. Otherwise, the files are part of the ca-certificates package. You can obtain them from another server running your operating system via yumdownloader then restore your certificates from the resulting RPM file using yum --nogpgcheck localinstall <RPM file>.
I had the same error after I was messing around with SD adapter to microSD card. Edit /etc/fstab to remove SD line solved the issue for me.
I will confirm I recently had this issue and what fixed it for me was a change in the DNS servers. I had a static IP set but DNS was automatic, I had to reconfigure DNS to use my gateway and dc. This is working now.
My issue was that the interface wasn't up after a fresh vm install.
Ran ifconfig to find my interface name and then ifup eXXX to get a dhcp ip address.
Also don't forget to change ONBOOT=no to ONBOOT=yes in /etc/sysconfig/network-scripts/ifcfg-eXXX so it comes up during next boot.
I follow this to install kubernetes on my cloud.
When I run command kubectl get nodes I get this error:
The connection to the server localhost:6443 was refused - did you specify the right host or port?
How can I fix this?
If you followed only mentioned docs it means that you have only installed kubeadm, kubectl and kubelet.
If you want to run kubeadm properly you need to do 3 steps more.
1. Install docker
Install Docker ubuntu version. If you are using another system chose it from left menu side.
Why:
If you will not install docker you will receive errror like below:
preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": e
xecutable file not found in $PATH
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
2. Initialization of kubeadm
You have installed properly kubeadm and docker but now you need to initialize kubeadm. Docs can be found here
In short version you have to run command
$ sudo kubeadm init
After initialization you will receive information to run commands like:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
and token to join another VM to cluster. It looks like
kubeadm join 10.166.XX.XXX:6443 --token XXXX.XXXXXXXXXXXX \
--discovery-token-ca-cert-hash sha256:aXXXXXXXXXXXXXXXXXXXXXXXX166b0b446986dd05c1334626aa82355e7
If you want to run some special action in init phase please check this docs.
3. Change node status to Ready
After previous step you will be able to execute
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-kubeadm NotReady master 4m29s v1.16.2
But your node will be in NotReady status. If you will describe it $ kubectl describe node you will see error:
Ready False Wed, 30 Oct 2019 09:55:09 +0000 Wed, 30 Oct 2019 09:50:03 +0000 KubeletNotReady runtime network not ready: Ne
tworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
It means that you have to install one of CNIs. List of them can be found here.
EDIT
Also one thing comes to my mind.
Sometimes when you turned off and on VM you need to restart
kubelet and docker service. You can do it by using
$ service docker restart
$ systemctl restart kubelet
Hope it helps.
Looks like kubeconfig file is missing.. Did you copy admin.conf file to ~/.kube/config ?
Verify if there are any proxies set like "http_proxy" or "https_proxy", mostly we set it as environment variables. If yes, then remove the proxies and it should work for you.
I did the following 2 steps. The kubectl works now.
$ service docker restart
$ systemctl restart kubelet
I have set up my master node and I am trying to join a worker node as follows:
kubeadm join 192.168.30.1:6443 --token 3czfua.os565d6l3ggpagw7 --discovery-token-ca-cert-hash sha256:3a94ce61080c71d319dbfe3ce69b555027bfe20f4dbe21a9779fd902421b1a63
However the command hangs forever in the following state:
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
Since this is just a warning, why does it actually fails?
edit: I noticed the following in my /var/log/syslog
Mar 29 15:03:15 ubuntu-xenial kubelet[9626]: F0329 15:03:15.353432 9626 server.go:193] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Mar 29 15:03:15 ubuntu-xenial systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Mar 29 15:03:15 ubuntu-xenial systemd[1]: kubelet.service: Unit entered failed state.
First if you want to see more detail when your worker joins to the master use:
kubeadm join 192.168.1.100:6443 --token m3jfbb.wq5m3pt0qo5g3bt9 --discovery-token-ca-cert-hash sha256:d075e5cc111ffd1b97510df9c517c122f1c7edf86b62909446042cc348ef1e0b --v=2
Using the above command I could see that my worker could not established connection with the master, so i just stoped the firewall:
systemctl stop firewalld
This can be solved by creating a new token
using this command:
kubeadm token create --print-join-command
and use the token generated for joining other nodes to the cluster
The problem had to do with kubeadm not installing a networking CNI-compatible solution out of the box;
Therefore, without this step the kubernetes nodes/master are unable to establish any form of communication;
The following task addressed the issue:
- name: kubernetes.yml --> Install Flannel
shell: kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
become: yes
environment:
KUBECONFIG: "/etc/kubernetes/admin.conf"
when: inventory_hostname in (groups['masters'] | last)
I did get the same error on CentOS 7 but in my case join command worked without problems, so it was indeed just a warning.
> [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker
> cgroup driver. The recommended driver is "systemd". Please follow the
> guide at https://kubernetes.io/docs/setup/cri/ [preflight] Reading
> configuration from the cluster... [preflight] FYI: You can look at
> this config file with 'kubectl -n kube-system get cm kubeadm-config
> -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
As the official documentation mentions, there are two common issues that make the init hang (I guess it also applies to join command):
the default cgroup driver configuration for the kubelet differs from
that used by Docker. Check the system log file (e.g. /var/log/message)
or examine the output from journalctl -u kubelet. If you see something
like the following:
First try the steps from official documentation and if that does not work please provide more information so we can troubleshoot further if needed.
I had a bunch of k8s deployment scripts that broke recently with this same error message... it looks like docker changed it's install. Try this --
previous install:
apt-get isntall docker-ce
updated install:
apt-get install docker-ce docker-ce-cli containerd.io
How /var/lib/kubelet/config.yaml is created?
Regarding the /var/lib/kubelet/config.yaml: no such file or directory error.
Below are steps that should occur on the worker node in order for the mentioned file to be created.
1 ) The creation of the /var/lib/kubelet/ folder. It is created when the kubelet service is installed as mentioned here:
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
2 ) The creation of config.yaml. The kubeadm join flow should take place so when you run kubeadm join, kubeadm uses the Bootstrap Token credential to perform a TLS bootstrap, which fetches the credential needed to download the kubelet-config-1.X ConfigMap and writes it to /var/lib/kubelet/config.yaml.
After a successful execution you should see the logs below:
.
.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
.
.
So, after these 2 steps you should have /var/lib/kubelet/config.yaml in place.
Failure of the kubeadm join flow
In your case, it seems that the kubeadm join flow failed which might happen due to multiple reasons like bad configuration of iptables, ports that are already in use, container runtime not installed properly, etc' - as described here and here.
As far as I know, the fact that no networking CNI-compatible solution was in place should not affect the creation of /var/lib/kubelet/config.yaml:
A) We can see the under the kubeadm preflight checks what issues will cause the join phase to fail.
B ) I also tested this issue by removing the current solution I used (Calico) and ran kubeadm reset and kubeadm join again and no errors appeared in the kubeadm logs (I've got the successful execution logs I mentioned above) and /var/lib/kubelet/config.yaml was created properly.
(*) Of course that the cluster can't function in this state - I just wanted to emphasize that I think the problem was one of the options mentioned in A.
I have created kubernetes cluster using minikube and installed Jenkins x on it.
I am not able to access the Jenkins x dashboard.
Error 503 Service Temporarily Unavailable nginx/1.13.9.
note: I have tried with restarting minikube cluster also.
As mentioned by James Rawlings in the comments, it's likely to be a resource issue. The recommendations in the manual are:
A known good configuration on a 2015 model Macbook Pro is to use 8 GB of RAM, 8 cores, a 150 GB disk size and hyperkit.
The disk size is particularly large as a number of images will need to be downloaded.
So we highly recommend using one of the public clouds above to try out Jenkins X. They all have free tiers so it should not cost you any significant cash and it’ll give you a chance to try out the cloud.
Default minikube installation uses only 2048 MB of RAM, 2 CPU cores and 20Gb disk space. You can adjust minikube VM size using command line options:
$ minikube start --cpus=8 --memory=8192 --disk-size=150g --vm-driver=hyperkit
or use jx tool for that:
For MacOS
$ brew install docker-machine-driver-hyperkit
# docker-machine-driver-hyperkit need root owner and uid
$ sudo chown root:wheel /usr/local/opt/docker-machine-driver-hyperkit/bin/docker-machine-driver-hyperkit
$ sudo chmod u+s /usr/local/opt/docker-machine-driver-hyperkit/bin/docker-machine-driver-hyperkit
$ brew tap jenkins-x/jx
$ brew install jx
$ jx create cluster minikube
For linux:
$ curl -L https://github.com/jenkins-x/jx/releases/download/v1.3.784/jx-linux-amd64.tar.gz | tar xzv
$ sudo mv jx /usr/local/bin
$ jx create cluster minikube