I try to join worker node to k8s kluser.
sudo kubeadm join 10.2.67.201:6443 --token x --discovery-token-ca-cert-hash sha2566 x
But i get error on this stage:
curl -sSL http://localhost:10248/healthz'
failed with error: Get http://localhost:10248/healthz: dial tcp
Error:
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I see that kubelet service is down:
journalctl -xeu kubelet
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.
Nov 22 15:49:00 s001as-ceph-node-03 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Nov 22 15:49:00 s001as-ceph-node-03 kubelet[286703]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag.
Nov 22 15:49:00 s001as-ceph-node-03 kubelet[286703]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag.
Nov 22 15:49:00 s001as-ceph-node-03 kubelet[286703]: F1122 15:49:00.224350 286703 server.go:251] unable to load client CA file /etc/kubernetes/ssl/ca.crt: open /etc/kubernetes/ssl/ca.cr
Nov 22 15:49:00 s001as-ceph-node-03 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 22 15:49:00 s001as-ceph-node-03 systemd[1]: Unit kubelet.service entered failed state.
Nov 22 15:49:00 s001as-ceph-node-03 systemd[1]: kubelet.service failed.
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Nov 22 15:49:10 s001as-ceph-node-03 kubelet[286717]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag.
Nov 22 15:49:10 s001as-ceph-node-03 kubelet[286717]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag.
Nov 22 15:49:10 s001as-ceph-node-03 kubelet[286717]: F1122 15:49:10.476478 286717 server.go:251] unable to load client CA file /etc/kubernetes/ssl/ca.crt: open /etc/kubernetes/ssl/ca.cr
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: Unit kubelet.service entered failed state.
Nov 22 15:49:10 s001as-ceph-node-03 systemd[1]: kubelet.service failed.
I fixed it.
Just copy /etc/kubernetes/pki/ca.crt into /etc/kubernetes/ssl/ca.crt
Related
I am trying to run Prometheus' standalone app on an RPI4 8GB. I am following the instructions laid out here: https://pimylifeup.com/raspberry-pi-prometheus/
My prometheus.service file is this:
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
[Service]
User=pi
Restart=on-failure
ExecStart=/home/pi/prometheus/prometheus \
--config.file=/home/pi/prometheus/prometheus.yml \
--storage.tsdb.path=/home/pi/prometheus/data
[Install]
WantedBy=multi-user.target
But when I try to run the service I get the following error.
● prometheus.service - Prometheus Server
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-11-24 18:42:51 GMT; 2s ago
Docs: https://prometheus.io/docs/introduction/overview/
Process: 485265 ExecStart=/home/pi/prometheus/prometheus --config.file=/home/pi/prometheus/prometheus.yml --storage.tsdb.path=/home/pi/prometheus/data (code=exited, status=2)
Main PID: 485265 (code=exited, status=2)
CPU: 160ms
Nov 24 18:42:51 master2 systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 5.
Nov 24 18:42:51 master2 systemd[1]: Stopped Prometheus Server.
Nov 24 18:42:51 master2 systemd[1]: prometheus.service: Start request repeated too quickly.
Nov 24 18:42:51 master2 systemd[1]: prometheus.service: Failed with result 'exit-code'.
Nov 24 18:42:51 master2 systemd[1]: Failed to start Prometheus Server.
What does Error Status 2 mean in this context? Is it a permission problem, or something else?
I am trying to deploy an springboot microservices in kubernetes cluster having 1 master and 2 worker node. When I am trying to get the node state using the command sudo kubectl get nodes, I am getting one of my worker node is not ready. It showing not ready in status.
When I am applying to troubleshoot the following command,
sudo journalctl -u kubelet
I am getting response like kubelet.service: Unit entered failed state and kubelet service stopped. The following is the response what I am getting when applying the command sudo journalctl -u kubelet.
-- Logs begin at Fri 2020-01-03 04:56:18 EST, end at Fri 2020-01-03 05:32:47 EST. --
Jan 03 04:56:25 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi
Jan 03 04:56:31 MILDEVKUB050 kubelet[970]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --confi
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.053962 970 server.go:416] Version: v1.17.0
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.084061 970 plugins.go:100] No cloud provider specified.
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.235928 970 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:32 MILDEVKUB050 kubelet[970]: I0103 04:56:32.280173 970 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curre
Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: I0103 04:56:38.107966 970 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Jan 03 04:56:38 MILDEVKUB050 kubelet[970]: F0103 04:56:38.109401 970 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable swa
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:38 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:56:48 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.901632 1433 server.go:416] Version: v1.17.0
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907654 1433 plugins.go:100] No cloud provider specified.
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.907806 1433 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:48 MILDEVKUB050 kubelet[1433]: I0103 04:56:48.947107 1433 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr
Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: I0103 04:56:49.263777 1433 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to
Jan 03 04:56:49 MILDEVKUB050 kubelet[1433]: F0103 04:56:49.264219 1433 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:49 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.712729 1500 server.go:416] Version: v1.17.0
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.714927 1500 plugins.go:100] No cloud provider specified.
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.715248 1500 server.go:821] Client rotation is on, will bootstrap in background
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.763508 1500 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-curr
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: I0103 04:56:59.956706 1500 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to
Jan 03 04:56:59 MILDEVKUB050 kubelet[1500]: F0103 04:56:59.957078 1500 server.go:273] failed to run Kubelet: running with swap on is not supported, please disable sw
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Unit entered failed state.
Jan 03 04:56:59 MILDEVKUB050 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 03 04:57:10 MILDEVKUB050 systemd[1]: Started kubelet: The Kubernetes Node Agent.
log file: service: Unit entered failed state
I tried by restarting the kubelet. But still there is no change in node state. Not ready state only.
Updates
When I am trying the command systemctl list-units --type=swap --state=active , then I am getting the following response,
docker#MILDEVKUB040:~$ systemctl list-units --type=swap --state=active
UNIT LOAD ACTIVE SUB DESCRIPTION
dev-mapper-MILDEVDCR01\x2d\x2dvg\x2dswap_1.swap loaded active active /dev/mapper/MILDEVDCR01--vg-swap_1
Important
When I am getting these kind of issue with node not ready, each time I need to disable the swap and need to reload the daemon and kubelet. After that node becomes ready state. And again I need to repeat the same.
How can I find a permanent solution for this?
failed to run Kubelet: running with swap on is not supported, please disable swap
You need to disable swap on the system for kubelet to work. You can disable swap with sudo swapoff -a
For systemd based systems, there is another way of enabling swap partitions using swap units which gets enabled whenever systemd reloads even if you have turned off swap using swapoff -a
https://www.freedesktop.org/software/systemd/man/systemd.swap.html
Check if you have any swap units using systemctl list-units --type=swap --state=active
You can permanently disable any active swap unit with systemctl mask <unit name>.
Note: Do not use systemctl disable <unit name> to disable the swap unit as swap unit will be activated again when systemd reloads. Use systemctl mask <unit name> only.
To make sure swap doesn't get re-enabled when your system reboots due to power cycle or any other reason, remove or comment out the swap entries in /etc/fstab
Summarizing:
Run sudo swapoff -a
Check if you have swap units with command systemctl list-units --type=swap --state=active. If there are any active swap units, mask them using systemctl mask <unit name>
Remove swap entries in /etc/fstab
The root cause is the swap space. To disable completely follow steps:
run swapoff -a: this will immediately disable swap but will activate on restart
remove any swap entry from /etc/fstab
reboot the system.
If the swap is gone, good. If, for some reason, it is still here, you
had to remove the swap partition. Repeat steps 1 and 2 and, after
that, use fdisk or parted to remove the (now unused) swap partition.
Use great care here: removing the wrong partition will have disastrous
effects!
reboot
This should resolve your issue.
Removing /etc/fstab will give the vm error, I think we should find another way to solve this issue. I tried to remove the fstab, all command (install, ping and other command) error.
I am new to container worrld and trying to setup a kubernetes cluster locally in two linux VMs. During the cluster initialization it got stuck at
[apiclient] Created API client, waiting for the control plane to
become ready
I have followed the pre-flight check steps,
[root#lm--kube-glusterfs--central ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.0
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] WARNING: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [lm--kube-glusterfs--central kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.99.7.215]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
OS : Red Hat Enterprise Linux Server release 7.4 (Maipo)
Kuberneter version :
kubeadm-1.6.0-0.x86_64.rpm
kubectl-1.6.0-0.x86_64.rpm
kubelet-1.6.0-0.x86_64.rpm
kubernetes-cni-0.6.0-0.x86_64.rpm
cri-tools-1.12.0-0.x86_64.rpm
How to debug the issue or is there any version mismatch of kube versions. Same was working before when i use google.cloud.repo to install yum -y install kubelet kubeadm kubectl .
I couldnot use repo due to some firewall issues. Hence using the rpms to install.
After executing the following command,journalctl -xeu kubelet
Jul 02 09:45:09 lm--son-config-cn--central kubelet[28749]: W0702 09:45:09.841246 28749 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet
Jul 02 09:45:09 lm--son-config-cn--central kubelet[28749]: I0702 09:45:09.841304 28749 kubelet.go:494] Hairpin mode set to "hairpin-veth"
Jul 02 09:45:09 lm--son-config-cn--central kubelet[28749]: W0702 09:45:09.845626 28749 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 02 09:45:09 lm--son-config-cn--central kubelet[28749]: I0702 09:45:09.857969 28749 docker_service.go:187] Docker cri networking managed by kubernetes.io/no-op
Jul 02 09:45:09 lm--son-config-cn--central kubelet[28749]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs
Jul 02 09:45:09 lm--son-config-cn--central systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Jul 02 09:45:09 lm--son-config-cn--central systemd[1]: Unit kubelet.service entered failed state.
Jul 02 09:45:09 lm--son-config-cn--central systemd[1]: kubelet.service failed.
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: kubelet.service holdoff time over, scheduling restart.
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: Started Kubernetes Kubelet Server.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: Starting Kubernetes Kubelet Server...
-- Subject: Unit kubelet.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun starting up.
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.251465 28772 feature_gate.go:144] feature gates: map[]
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.251889 28772 server.go:469] No API client: no api servers specified
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.252009 28772 docker.go:364] Connecting to docker on unix:///var/run/docker.sock
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.252049 28772 docker.go:384] Start docker client with request timeout=2m0s
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.259436 28772 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.275674 28772 manager.go:143] cAdvisor running in container: "/system.slice"
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.317509 28772 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial r
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.328881 28772 fs.go:117] Filesystem partitions: map[/dev/vda2:{mountpoint:/ major:253 mino
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.335711 28772 manager.go:198] Machine: {NumCores:8 CpuFrequency:2095078 MemoryCapacity:337
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: [7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unifie
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.338001 28772 manager.go:204] Version: {KernelVersion:3.10.0-693.11.6.el7.x86_64 Container
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.338967 28772 server.go:350] No api server defined - no events will be sent to API server.
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.338980 28772 server.go:509] --cgroups-per-qos enabled, but --cgroup-root was not specifie
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.342041 28772 container_manager_linux.go:245] container manager verified user specified cg
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.342071 28772 container_manager_linux.go:250] Creating Container Manager object based on N
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.346505 28772 kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.346571 28772 kubelet.go:494] Hairpin mode set to "hairpin-veth"
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: W0702 09:45:20.351473 28772 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: I0702 09:45:20.363583 28772 docker_service.go:187] Docker cri networking managed by kubernetes.io/no-op
Jul 02 09:45:20 lm--son-config-cn--central kubelet[28772]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: Unit kubelet.service entered failed state.
Jul 02 09:45:20 lm--son-config-cn--central systemd[1]: kubelet.service failed.
Related to issue
There are a few fixes shown there , all you need is to change the cgroup driver to systemd
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I created master node on GCE using this commands:
gcloud compute instances create master --machine-type g1-small --zone europe-west1-d
gcloud compute addresses create myexternalip --region europe-west1
gcloud compute target-pools create kubernetes --region europe-west1
gcloud compute target-pools add-instances kubernetes --instances master --instances-zone europe-west1-d
gcloud compute forwarding-rules create kubernetes-forward --address myexternalip --region europe-west1 --ports 1-65535 --target-pool kubernetes
gcloud compute forwarding-rules describe kubernetes-forward
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
and opened all firewalls.
After it i created aws ec2 instance, opened firewalls and using:
kubeadm join --token 55d287.b540e254a280f853 ip:6443 --discovery-token-unsafe-skip-ca-verification
to connecting instance to cluster.
But in the master node it is not displayed
Docker version: 17.12
Kubernetes version: 1.9.3
UPD:
Output from node on aws ec2
systemctl status kubelet.service:
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2018-02-24 20:23:53 UTC; 23s ago
Docs: http://kubernetes.io/docs/
Main PID: 30678 (kubelet)
Tasks: 5
Memory: 13.4M
CPU: 125ms
CGroup: /system.slice/kubelet.service
└─30678 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests -
Feb 24 20:23:53 ip-172-31-0-250 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Feb 24 20:23:53 ip-172-31-0-250 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 24 20:23:53 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.420375 30678 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.420764 30678 controller.go:114] kubelet config controller: starting controller
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.420944 30678 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: W0224 20:23:53.425410 30678 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.444969 30678 server.go:182] Version: v1.9.3
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.445274 30678 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:23:53 ip-172-31-0-250 kubelet[30678]: I0224 20:23:53.445565 30678 plugins.go:101] No cloud provider specified.
journalctl -u kubelet:
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:15:12 ip-172-31-0-250 kubelet[30243]: I0224 20:15:12.819249 30243 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:15:12 ip-172-31-0-250 kubelet[30243]: I0224 20:15:12.821054 30243 controller.go:114] kubelet config controller: starting controller
Feb 24 20:15:12 ip-172-31-0-250 kubelet[30243]: I0224 20:15:12.821243 30243 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 24 20:15:12 ip-172-31-0-250 kubelet[30243]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: kubelet.service: Unit entered failed state.
Feb 24 20:15:12 ip-172-31-0-250 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:15:23 ip-172-31-0-250 kubelet[30304]: I0224 20:15:23.186834 30304 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:15:23 ip-172-31-0-250 kubelet[30304]: I0224 20:15:23.187255 30304 controller.go:114] kubelet config controller: starting controller
Feb 24 20:15:23 ip-172-31-0-250 kubelet[30304]: I0224 20:15:23.187451 30304 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 24 20:15:23 ip-172-31-0-250 kubelet[30304]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: kubelet.service: Unit entered failed state.
Feb 24 20:15:23 ip-172-31-0-250 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:15:33 ip-172-31-0-250 kubelet[30311]: I0224 20:15:33.422948 30311 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:15:33 ip-172-31-0-250 kubelet[30311]: I0224 20:15:33.423349 30311 controller.go:114] kubelet config controller: starting controller
Feb 24 20:15:33 ip-172-31-0-250 kubelet[30311]: I0224 20:15:33.423525 30311 controller.go:118] kubelet config controller: validating combination of defaults and flags
Feb 24 20:15:33 ip-172-31-0-250 kubelet[30311]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: kubelet.service: Unit entered failed state.
Feb 24 20:15:33 ip-172-31-0-250 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Feb 24 20:15:43 ip-172-31-0-250 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Feb 24 20:15:43 ip-172-31-0-250 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 24 20:15:43 ip-172-31-0-250 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Feb 24 20:15:43 ip-172-31-0-250 kubelet[30319]: I0224 20:15:43.671742 30319 feature_gate.go:226] feature gates: &{{} map[]}
Feb 24 20:15:43 ip-172-31-0-250 kubelet[30319]: I0224 20:15:43.672195 30319 controller.go:114] kubelet config controller: starting controller
UPD:
Error is on aws ec2 instance side, but i can't find what is wrong.
PROBLEM SOLVED
Should initialize kubeadm with this flag --apiserver-advertise-address
After you create load balancer you need to type this command for show you external load balancer ip address
gcloud compute forwarding-rules describe kubernetes-forward
And the initialize cluster with this flag
--apiserver-advertise-address=external_load_balancer_ip
So your kubeadm command looks like this
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=external_load_balancer_ip
I installed the HAproxy(1.5.14-3.el7) from CentOS7.2 repository.
When I reload HAproxy service with a wrong haproxy.cfg,
the return code of reload is incorrect.
About HAproxy,OS,systemd information pls see below:
[root#unknown ~]# rpm -qa | egrep haproxy
haproxy-1.5.14-3.el7.x86_64
[root#unknown ~]#
[root#unknown ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root#unknown ~]#
[root#unknown ~]# rpm -qa | egrep systemd
systemd-libs-219-19.el7.x86_64
systemd-219-19.el7.x86_64
systemd-sysv-219-19.el7.x86_64
[root#unknown ~]#
The return code of reload is incorrect.
[root#unknown ~]# service haproxy status
Redirecting to /bin/systemctl status haproxy.service
●haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2016-06-07 11:24:41 UTC; 4s ago
[root#unknown ~]#
[root#unknown ~]#
[root#unknown ~]# more /etc/haproxy/haproxy.cfg
XXXX **--> I added an incorrect keyword into haproxy.cfg**
Global
....
[root#unknown ~]#
[root#unknown ~]#
[root#unknown ~]# service haproxy reload
Redirecting to /bin/systemctl reload haproxy.service
[root#unknown ~]#
[root#unknown ~]# echo $?
0 **--> It was sucessful !!!**
[root#unknown ~]#
[root#unknown ~]# service haproxy status
Redirecting to /bin/systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2016-06-07 11:24:41 UTC; 21s ago
Process: 16507 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 16464 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
tq16464 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
tq16465 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
mq16466 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Jun 07 11:24:41 unknown systemd[1]: Started HAProxy Load Balancer.
Jun 07 11:24:41 unknown systemd[1]: Starting HAProxy Load Balancer...
Jun 07 11:24:41 unknown haproxy-systemd-wrapper[16464]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy...id -Ds
Jun 07 11:24:57 unknown systemd[1]: Reloaded HAProxy Load Balancer.
Jun 07 11:24:57 unknown haproxy-systemd-wrapper[16464]: haproxy-systemd-wrapper: re-executing
Jun 07 11:24:57 unknown haproxy-systemd-wrapper[16464]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy... 16466
Jun 07 11:24:57 unknown haproxy-systemd-wrapper[16464]: [ALERT] 158/112457 (16508) : parsing [/etc/haproxy/haproxy.cfg:9]: unknown k...ction. **--> In fact, it was wrong**
Jun 07 11:24:57 unknown haproxy-systemd-wrapper[16464]: [ALERT] 158/112457 (16508) : Error(s) found in configuration file : /etc/hap...xy.cfg
Jun 07 11:24:57 unknown haproxy-systemd-wrapper[16464]: [ALERT] 158/112457 (16508) : Fatal errors found in configuration.
So I descided to use the sysv init.d script to start/reload/stop HAproxy service.
sysv init.d script:
[root#unknown ~]# cat /etc/init.d/haproxy
#!/bin/sh
#
# chkconfig: - 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \
# for high availability environments.
# processname: haproxy
# config: /etc/haproxy/haproxy.cfg
# pidfile: /var/run/haproxy.pid
# Script Author: Simon Matter <simon.matter#invoca.ch>
# Version: 2004060600
### BEGIN INIT INFO
# Provides: HA-Proxy
# Required-Start: $network $syslog sshd
# Required-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: HAProxy
### END INIT INFO
# Source function library.
if [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
. /etc/rc.d/init.d/functions
else
exit 0
fi
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
#[ ${NETWORKING} = "no" ] && exit 0
# This is our service name
BASENAME=`basename $0`
if [ -L $0 ]; then
BASENAME=`find $0 -name $BASENAME -printf %l`
BASENAME=`basename $BASENAME`
fi
[ -f /etc/$BASENAME/$BASENAME.cfg ] || exit 1
RETVAL=0
start() {
/usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with '$BASENAME check'."
return 1
fi
echo -n "Starting $BASENAME: "
daemon /usr/sbin/$BASENAME -D -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$BASENAME
return $RETVAL
}
stop() {
killproc $BASENAME -USR1
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$BASENAME
[ $RETVAL -eq 0 ] && rm -f /var/run/$BASENAME.pid
return $RETVAL
}
restart() {
/usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with '$BASENAME check'."
return 1
fi
stop
start
}
reload() {
/usr/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with '$BASENAME check'."
return 1
fi
/usr/sbin/$BASENAME -D -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid -sf $(cat /var/run/$BASENAME.pid)
}
check() {
/usr/sbin/$BASENAME -c -q -V -f /etc/$BASENAME/$BASENAME.cfg
}
rhstatus() {
status $BASENAME
}
condrestart() {
[ -e /var/lock/subsys/$BASENAME ] && restart || :
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
reload)
reload
;;
condrestart)
condrestart
;;
status)
rhstatus
;;
check)
check
;;
*)
echo $"Usage: $BASENAME {start|stop|restart|reload|condrestart|status|check}"
exit 1
esac
exit $?
When I reloaded HAproxy serivce with the correct haproxy.cfg,
the command(service haproxy reload) returned 0,
but HAproxy's status became failed.
[root#unknown ~]# service haproxy status
● haproxy.service - LSB: HAProxy
Loaded: loaded (/etc/rc.d/init.d/haproxy)
Active: active (running) since Tue 2016-06-07 11:33:22 UTC; 1h 14min ago
Docs: man:systemd-sysv-generator(8)
Process: 16636 ExecStart=/etc/rc.d/init.d/haproxy start (code=exited, status=0/SUCCESS)
Main PID: 16641 (haproxy)
CGroup: /system.slice/haproxy.service
mq16641 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
Jun 07 11:33:22 unknown systemd[1]: Starting LSB: HAProxy...
Jun 07 11:33:22 unknown haproxy[16636]: Starting haproxy: [ OK ]
Jun 07 11:33:22 unknown systemd[1]: Started LSB: HAProxy.
[root#unknown ~]#
[root#unknown ~]# service haproxy reload
Reloading haproxy configuration (via systemctl): [ OK ]
[root#unknown ~]# echo $?
0 **--> It was sucessful !!!**
[root#unknown ~]#
[root#unknown ~]# service haproxy status
● haproxy.service - LSB: HAProxy
Loaded: loaded (/etc/rc.d/init.d/haproxy)
Active: failed (Result: signal) since Tue 2016-06-07 12:48:01 UTC; 1s ago
Docs: man:systemd-sysv-generator(8)
Process: 16869 ExecStop=/etc/rc.d/init.d/haproxy stop (code=exited, status=0/SUCCESS)
Process: 16863 ExecReload=/etc/rc.d/init.d/haproxy reload (code=exited, status=0/SUCCESS)
Process: 16636 ExecStart=/etc/rc.d/init.d/haproxy start (code=exited, status=0/SUCCESS)
Main PID: 16868 (code=killed, signal=KILL)
Jun 07 11:33:22 unknown systemd[1]: Starting LSB: HAProxy...
Jun 07 11:33:22 unknown haproxy[16636]: Starting haproxy: [ OK ]
Jun 07 11:33:22 unknown systemd[1]: Started LSB: HAProxy.
Jun 07 12:48:00 unknown systemd[1]: Reloaded LSB: HAProxy.
Jun 07 12:48:00 unknown systemd[1]: haproxy.service: main process exited, code=killed, status=9/KILL **--> It was killed ,but I don't know which process killed it, Cgroup ?**
Jun 07 12:48:01 unknown haproxy[16869]: [FAILED]
Jun 07 12:48:01 unknown systemd[1]: Unit haproxy.service entered failed state.
Jun 07 12:48:01 unknown systemd[1]: haproxy.service failed.
[root#unknown ~]#
I used a newer systemd to get the detailed logs
Jun 07 13:02:59 elb systemd[1]: Starting LSB: HAProxy...
Jun 07 13:02:59 elb systemd[7010]: Executing: /etc/rc.d/init.d/haproxy start
Jun 07 13:02:59 elb haproxy[7010]: Starting haproxy: [ OK ]
Jun 07 13:02:59 elb systemd[1]: Child 7010 belongs to haproxy.service
Jun 07 13:02:59 elb systemd[1]: haproxy.service: control process exited, code=exited status=0
Jun 07 13:02:59 elb systemd[1]: haproxy.service got final SIGCHLD for state start
Jun 07 13:02:59 elb systemd[1]: Main PID loaded: 7015
Jun 07 13:02:59 elb systemd[1]: haproxy.service changed start -> running
Jun 07 13:02:59 elb systemd[1]: Job haproxy.service/start finished, result=done
Jun 07 13:02:59 elb systemd[1]: Started LSB: HAProxy. **--> start HAproxy successfully **
Jun 07 13:03:27 elb systemd[1]: Trying to enqueue job haproxy.service/reload/replace
Jun 07 13:03:27 elb systemd[1]: Installed new job haproxy.service/reload as 9504
Jun 07 13:03:27 elb systemd[1]: Enqueued job haproxy.service/reload as 9504
Jun 07 13:03:27 elb systemd[1]: About to execute: /etc/rc.d/init.d/haproxy reload
Jun 07 13:03:27 elb systemd[1]: Forked /etc/rc.d/init.d/haproxy as 7060
Jun 07 13:03:27 elb systemd[1]: haproxy.service changed running -> reload
Jun 07 13:03:27 elb systemd[7060]: Executing: /etc/rc.d/init.d/haproxy reload
Jun 07 13:03:27 elb systemd[1]: Child 7015 belongs to haproxy.service
Jun 07 13:03:27 elb systemd[1]: Main PID changing: 7015 -> 7065
Jun 07 13:03:27 elb systemd[1]: Child 7060 belongs to haproxy.service
Jun 07 13:03:27 elb systemd[1]: haproxy.service: control process exited, code=exited status=0
Jun 07 13:03:27 elb systemd[1]: haproxy.service got final SIGCHLD for state reload
Jun 07 13:03:27 elb systemd[1]: haproxy.service changed reload -> running
Jun 07 13:03:27 elb systemd[1]: Job haproxy.service/reload finished, result=done
Jun 07 13:03:27 elb systemd[1]: Reloaded LSB: HAProxy. **--> successful to reload HAproxy**
Jun 07 13:03:27 elb systemd[1]: Child 7065 belongs to haproxy.service
Jun 07 13:03:27 elb systemd[1]: haproxy.service: main process exited, code=killed, status=9/KILL **--> process 7065 has been killed unexpectly**
Jun 07 13:03:27 elb systemd[1]: haproxy.service changed running -> failed
Jun 07 13:03:27 elb systemd[1]: Unit haproxy.service entered failed state.
Jun 07 13:03:27 elb systemd[1]: haproxy.service failed.
Jun 07 13:03:27 elb systemd[1]: haproxy.service: cgroup is empty **-->Did cgroup killed process 7065? Is it a bug of systemd? **
In CentOS7.1, I use the sysv init script (pls see above) to reload haproxy ,and 'service haproxy reload' command could return correct result.
I don't know what is wrong in CentOS7.2. I just want to get following results of reloading HAproxy.
When haproxy.cfg file is incorrect, 'service haproxy reload' command returns 1
When haproxy.cfg file is correct, 'service haproxy reload' command returns 0
Can anyone help me ? Thanks
I would guess this is a SELinux issue. you can try change for below:
:
vi /etc/selinux/config
SELINUX=enforcing
SELINUXTYPE=targeted
SELINUX=disabled
:wq! #save and quit