I want to drain node on shutdown and uncordon on start, I wrote below unit file but i am getting error (Openshift 3.11 and Kubernetes 1.11.0)
[Unit]
Description=Drain Node at Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target
[Service]
Type=oneshot
ExecStart=/bin/sleep 60 && kubectl uncordon $HOSTNAME
ExecStop=kubectl drain $HOSTNAME --ignore-daemonsets --force --grace-period=30 && /bin/sleep 60
[Install]
WantedBy=halt.target reboot.target shutdown.target
its giving me error
error: no configuration has been provided
I set Environment variable but still no success
[Service]
Environment="KUBECONFIG=$HOME/.kube/config"
Following systemd unit is working, in ExecStop %H should be use for HOSTNAME
[Unit]
Description=Drain Node at Shutdown
After=network.target glusterd.service
[Service]
Type=oneshot
Environment="KUBECONFIG=/root/.kube/config"
ExecStart=/bin/true
ExecStop=/usr/bin/kubectl drain %H --ignore-daemonsets --force --grace-period=30 --delete-local-data
TimeoutStopSec=200
# This service shall be considered active after start
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
Related
I am trying to add prometheus jmx agent (jmx_prometheus_javaagent-0.3.1.jar) to an existing kafka cluster.
But when I run the java agent, I am not getting response on the port as it says-
curl http://localhost:8080
curl: (7) Failed connect to localhost:8080; Connection refused
Here is my configuration file "kafka.service":
[kafka#Kafka-dev prometheus]$ cat /etc/systemd/system/kafka.service
[Unit]
Description=Kafka
After=network.target
[Service]
User=kafka
Group=kafka
Environment="KAFKA_HEAP_OPTS=-Xmx256M -Xms128M"
Environment="KAFKA_OPTS=-javaagent:/home/kafka/prometheus/jmx_prometheus_javaagent-0.3.1.jar=8080:/home/kafka/prometheus/kafka-0-8-2.yml"
ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server.properties
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
Then when I start Kafka.service it looks that it works:
sudo systemctl restart kafka
But when I check the status I find that the service is inactive:
[kafka#Kafka-dev ~]$ sudo systemctl status kafka.service
● kafka.service - Kafka
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Thu 2019-12-05 10:00:17 UTC; 1min 0s ago
Process: 125469 ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh -daemon /home/kafka/kafka/config/server.properties (code=exited, status=0/SUCCESS)
Main PID: 125469 (code=exited, status=0/SUCCESS)
Note- firewalls on the machine are disabled.
I'm suspecting this has something to do with the configuration of jmx_prometheus_javaagent.
I ran command
systemctl stop kubelet
then try to start it
systemctl start kubelet
but can't able to start it
here is the output of systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2019-06-05 15:35:34 UTC; 7s ago
Docs: https://kubernetes.io/docs/home/
Process: 31697 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 31697 (code=exited, status=255)
Because of this i am not able to run any kubectl command
example kubectl get pods gives
The connection to the server 172.31.6.149:6443 was refused - did you specify the right host or port?
Worked
Need to disable swap using swapoff -a
then,
try to start it systemctl start kubelet
So i need to reset kubelete service
Here are the step :-
check status of your docker service.
If stoped,start it by cmd sudo systemctl start docker.
If not installed installed it
#yum install -y kubelet kubeadm kubectl docker
Make swap off by #swapoff -a
Now reset kubeadm by #kubeadm reset
Now try #kudeadm init
after that check #systemctl status kubelet
it will be working
Check nodes
kubectl get nodes
if Master Node is not ready ,refer following
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
if you not able to create pod ..check dns
kubectl get pods --namespace=kube-system
if dns pods are in pending state
i.e you need to use network service
i used calico
kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
Now your master node is ready .. now you can deploy pod
I have setup cluster with kubeadm its working fine and 6443 port is up. but after reboot my system cluster is not getting up.
What should I do?
please find the logs
node#node1:~$ sudo kubeadm init
[init] using Kubernetes version: v1.11.1
......
node#node1:~$
node#node1:~$ mkdir -p $HOME/.kube
node#node1:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
node#node1:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
node#node1:~$
node#node1:~$
node#node1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 NotReady master 4m v1.11.1
node#node1:~$ ps -ef | grep 6443
root 5542 5503 8 13:17 ? 00:00:17 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=172.16.2.171 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=PersistentVolumeLabel --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
node 6792 4426 0 13:20 pts/1 00:00:00 grep --color=auto 6443
node#node1:~$
node#node1:~$
node#node1:~$
node#node1:~$ sudo reboot
Connection to node1 closed by remote host.
Connection to node1 closed.
abc#xyz:~$ ssh node#node1
node#node1's password:
node#node1:~$ kubectl get nodes
No resources found.
The connection to the server 172.16.2.171:6443 was refused - did you specify the right host or port?`enter code here`
node#node1:~$
node#node1:~$ ps -ef | grep 6443
node 7083 1920 0 13:36 pts/0 00:00:00 grep --color=auto 6443
Your kubelet service is not running. Try to view its logs:
$ journalctl -u kubelet
To start the service:
$ sudo systemctl start kubelet
If you want to make kubelet running during the boot you'll need to enbale it. First of all check the kubelet service status:
$ systemctl status kubelet
There will be a line:
...
Loaded: loaded (/etc/systemd/system/kubelet.service; (enabled|disabled)
...
"disabled" entry means you should enable it:
$ sudo systemctl enable kubelet
But, highly likely it is already enabled, because this was done by "systemd vendor preset", so you will have to debug why kubelet falls. You can post logs output here and stackoverflow's community will help you.
I assume that you did not install Kubernetes from packages delivered to your
Linux distribution - as far as I know, installation on Ubuntu makes services
dependent on Kubernetes installed to avoid the situation you are describing.
The problem you are facing is the lack of support for starting kubelet by systemd or other runtime scripts.
Systemd is a system and a service manager. On behalf of them, Kubernetes starts
on system boot.
You may try to repair your installation by copying/creating required systemd configuration
kubernetes.service to your installation /etc/systemd directory.
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStart=/usr/bin/kubelet \
--api-servers=http://127.0.0.1:8080 \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
and enable service with systemctl:
sudo systemctl enable kubelet
The journalctl logs may provide information about problems with Kubernetes
Services if they still exist.
sudo journalctl -xeu kubelet
I have created a 3 node azure kubernetes cluster using the following commands
az group create --name ResourceGroup --location canadacentral
az provider register -n Microsoft.ContainerService
az provider register -n Microsoft.Compute
az provider register -n Microsoft.Network
az aks create --resource-group ResourceGroup --name ReplicaSet --node-count 3 --kubernetes-version 1.8.7 --node-vm-size Standard_A0 --generate-ssh-keys
kubectl create -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml
kubectl create -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml
Subsequently I have created a postgres stateful set as well which does not start since opensci is not installed on the kubelet.
Kubelet logs from Node-1 (where the pgset pod is scheduled)
I0313 05:42:41.910525 7845 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3")
I0313 05:42:41.910605 7845 operation_generator.go:416] MountVolume.WaitForAttach entering for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3") DevicePath ""
E0313 05:42:41.910744 7845 iscsi_util.go:207] iscsi: could not read iface default error:
E0313 05:42:41.910815 7845 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0\"" failed. No retries permitted until 2018-03-13 05:44:43.910784094 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.WaitForAttach failed for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3") : executable file not found in $PATH
E0313 05:43:12.080406 7845 kubelet.go:1628] Unable to mount volumes for pod "pgset-0_default(a9826973-2674-11e8-a384-0a58ac1f03e3)": timeout expired waiting for volumes to attach/mount for pod "default"/"pgset-0". list of unattached/unmounted volumes=[pgdata]; skipping pod
E0313 05:43:12.081262 7845 pod_workers.go:182] Error syncing pod a9826973-2674-11e8-a384-0a58ac1f03e3 ("pgset-0_default(a9826973-2674-11e8-a384-0a58ac1f03e3)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"pgset-0". list of unattached/unmounted volumes=[pgdata]
My question is whether there is a way to configure and ensure that the kubelet comes up by default with the openiscsi initiator utils installed and running
The following steps were followed to manually install iscsi-initiator in kubelet:
SSH into the Kubernetes Nodes
Identify the docker container running the kubelet using sudo docker ps.
Enter the kubelet container shell
sudo docker exec -it kubelet_container_id bash
Install open-iscsi.
apt-get update
apt install -y open-iscsi
When I run kubelet version I get an error message ending in:
error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
But when I check the config file located at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, I see the value IS systemd. I have changed the value and done a systemctl daemon-reload and systemctl restart kubelet in between each change and the error message is always the same.
I am guessing it must be reading from the wrong config file, but how can I find where it is reading from!
try this:
kubelet --cgroup-driver=systemd version
The "docker" package (1.13.1) already has "systemd" as the default cgroup-driver, see this.
The file driver is systemd changed by default cgroupfs, and docker file driver we installed is systemd caused by inconsistency, which causes the image to fail to start.
docker info
...
Cgroup Driver: systemd
There are two ways now, one is to modify docker, the other is to modify kubelet,
Modify docker: #
Modify or create /etc/docker/daemon.json and add the following:
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
Restart docker:
systemctl restart docker
systemctl status docker
Modify kubelet: #
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
Add the following content --cgroup-driver=systemd to $KUBELET_EXTRA_ARGS.
Or:
$ DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
$ echo $DOCKER_CGROUPS
$ cat >/etc/sysconfig/kubelet<<EOF
KUBELET_CGROUP_ARGS="--cgroup-driver=$DOCKER_CGROUPS"
EOF
#restart
$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl restart kubelet
Or:
DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
echo $DOCKER_CGROUPS
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS"
EOF
# restart
$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl restart kubelet