I need to provide a specific node name to my master node in kuberenetes. I am using kubeadm to setup my cluster and I know there is an option --node-name master which you can provide to kubeadm init and it works fine.
Now, the issue is I am using the config file to initialise the cluster and I have tried various ways to provide that node-name to the cluster but it is not picking up the name. My config file of kubeadm init is:
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
advertiseAddress: 10.0.1.149
controlPlaneEndpoint: 10.0.1.149
etcd:
endpoints:
- http://10.0.1.149:2379
caFile: /etc/kubernetes/pki/etcd/ca.pem
certFile: /etc/kubernetes/pki/etcd/client.pem
keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
podSubnet: 192.168.13.0/24
kubernetesVersion: 1.10.3
apiServerCertSANs:
- 10.0.1.149
apiServerExtraArgs:
endpoint-reconciler-type: lease
nodeRegistration:
name: master
Now I run kubeadm init --config=config.yaml and it timeouts with following error:
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-
config" in the "kube-system" Namespace
[markmaster] Will mark node ip-x-x-x-x.ec2.internal as master by
adding a label and a taint
timed out waiting for the condition
PS: This issue also comes when you don't provide --hostname-override to kubelet along with --node-name to kubeadm init. I am providing both. Also, I am not facing any issues when I don't use config.yaml file and use command line to provide --node-name option to kubeadm init.
I want to know how can we provide --node-name option in config.yaml file. Any pointers are appreciated.
I am able to resolve this issue using the following config file, Just updating if anyone encounters the same issue:
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
advertiseAddress: 10.0.1.149
controlPlaneEndpoint: 10.0.1.149
etcd:
endpoints:
- http://10.0.1.149:2379
caFile: /etc/kubernetes/pki/etcd/ca.pem
certFile: /etc/kubernetes/pki/etcd/client.pem
keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
podSubnet: 192.168.13.0/24
kubernetesVersion: 1.10.3
apiServerCertSANs:
- 10.0.1.149
apiServerExtraArgs:
endpoint-reconciler-type: lease
nodeName: master
This is the way you can specify --node-name in config.yaml
Related
I am trying to add the two flags below to apiserver in the /etc/kubernetes/manifests/kube-apiserver.yaml file:
spec:
containers:
- command:
- kube-apiserver
- --enable-admission-plugins=NodeRestriction,PodNodeSelector
- --admission-control-config-file=/vagrant/admission-control.yaml
[...]
I am not mounting a volume or mount point for the /vagrant/admission-control.yaml file. It is completely accessible from the node master, since it is shared by the VM created by vagrant:
vagrant#master-1:~$ cat /vagrant/admission-control.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodNodeSelector
path: /vagrant/podnodeselector.yaml
vagrant#master-1:~$
Kubernetes version:
vagrant#master-1:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Link to the /etc/kubernetes/manifests/kube-apiserver.yaml file being used by the running cluster Here
vagrant#master-1:~$ kubectl delete pods kube-apiserver-master-1 -n kube-system
pod "kube-apiserver-master-1" deleted
Unfortunately "kubectl describe pods kube-apiserver-master-1 -n kube-system" only informs that the pod has been recreated. Flags do not appear as desired. No errors reported.
Any suggestion will be helpful,
Thank you.
NOTES:
I also tried to make a patch on the apiserver's configmap.
The patch is applied, but it does not take effect in the new
running pod.
I also tried to pass the two flags in a file via kubeadm
init --config, but there is little documentation on how to put these
two flags and all the other ones of the apiserver that I need in a configuration file in order to reinstall the master node.
UPDATE:
I hope that be useful for everyone facing the same issue...
After 2 days of searching the internet, and lots and lots of tests, I only managed to make it work with the procedure below:
sudo tee ${KUBEADM_INIT_CONFIG_FILE} <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: "${INTERNAL_IP}"
bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: ${KUBERNETES_VERSION}
controlPlaneEndpoint: "${LOADBALANCER_ADDRESS}:6443"
networking:
podSubnet: "10.244.0.0/16"
apiServer:
extraArgs:
advertise-address: ${INTERNAL_IP}
enable-admission-plugins: NodeRestriction,PodNodeSelector
admission-control-config-file: ${ADMISSION_CONTROL_CONFIG_FILE}
extraVolumes:
- name: admission-file
hostPath: ${ADMISSION_CONTROL_CONFIG_FILE}
mountPath: ${ADMISSION_CONTROL_CONFIG_FILE}
readOnly: true
- name: podnodeselector-file
hostPath: ${PODNODESELECTOR_CONFIG_FILE}
mountPath: ${PODNODESELECTOR_CONFIG_FILE}
readOnly: true
EOF
sudo kubeadm init phase control-plane apiserver --config=${KUBEADM_INIT_CONFIG_FILE}
You need to create a hostPath volume mount like below
volumeMounts:
- mountPath: /vagrant
name: admission
readOnly: true
...
volumes:
- hostPath:
path: /vagrant
type: DirectoryOrCreate
name: admission
I'm trying to set which cri-o socket to use by kubeadm !
To achieve this I should use the flag --cri-socket /var/run/crio/crio.sock
The current command is in the form kubeadm init phase <phase_name>. I must add the --cri-socket flag to it.
I edited the command this way kubeadm init --cri-socket /var/run/crio/crio.sock phase <phase_name>.
Unfortunatly I am getting the error Error: unknown flag: --cri-socket.
=> It seems that the argument phase <phase_name> and the flag --cri-socket /var/run/crio/crio.sock is not compatible.
How do I fix that ?
Thx
##################Update 1######################
File : /etc/kubernetes/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.10.3.15
bindPort: 6443
certificateKey: 9063a1ccc9c5e926e02f245c06b8xxxxxxxxxxx
nodeRegistration:
name: p3kubemaster1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
criSocket: /var/run/crio/crio.sock
I see two things that may help:
Check /var/lib/kubelet/kubeadm-flags.env if it is properly configured.
In addition to the flags used when starting the kubelet, the file also
contains dynamic parameters such as the cgroup driver and whether to
use a different CRI runtime socket (--cri-socket).
More details can be found here.
Check your init config file (kubeadm init --config string will show you the path do the configuration file) and try to add something like this:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
criSocket: "unix:///var/run/crio/crio.sock"
Please let me know if that helped.
We am setting-up HA k8s environment on AWS. We have created a ami where docker, k8s are installed.
HA cluster with 3 master and 5 worker nodes are created with TLS enabled network load balancer. The certificate added to TLS with domain as *.amazonaws.com.
In my cluster ClusterConfiguration file, the controlPlaneEndpoint and certSANs are pointing to DNS of load balancer.
kubeadm installation fails, when checking the docker logs for k8s_kube-scheduler, I see wildcard certificate is not accepted.
Config file.
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
extraArgs:
cloud-provider: aws
clusterName: test
controlPlaneEndpoint: tf-k8s-t1-nlb-34390285259d3aac.elb.us-west-1.amazonaws.com
controllerManager:
extraArgs:
cloud-provider: aws
configure-cloud-routes: "false"
address: 0.0.0.0
kubernetesVersion: v1.13.2
networking:
dnsDomain: cluster.local
podSubnet: 10.168.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler:
extraArgs:
address: 0.0.0.0
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
cloud-provider: aws`
E0318 15:36:20.604025 1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: Get https://tf-k8s-t1-nlb-34390285259d3aac.elb.us-west-1.amazonaws.com:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: x509: certificate is valid for *.amazonaws.com, not tf-k8s-t1-nlb-34390285259d3aac.elb.us-west-1.amazonaws.com
Could you help me on how to pass wildcard character in my kubeadm configuration?
wildcard certificates only work for 1 sub-level.
say you have a *.example.com cert,
it is accepted for foo.example.com and foo2.example.com, but not for foo.boo.example.com, you'd need a *.boo.example.com cert for that.
I have the following conf
cat kubeadm-conf.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
apiServerExtraArgs:
enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
networking:
podSubnet: 192.168.0.0/16
but, when I do
ps -aux | grep admission
root 20697 7.4 2.8 446916 336660 ? Ssl 03:49 0:21 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=10.0.2.15 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction
I only see the NodeRestriction
Please, let me know if anyone can help me make sure that the admission-webhook is indeed running on my cluster.
I assume that MutatingAdmissionWebhook and ValidatingAdmissionWebhook have not being properly propagated through api-server as per your provided outputs.
I suggest to proceed with the following steps to achieve your goal:
Check and edit /etc/kubernetes/manifests/kube-apiserver.yaml manifest file by adding required admission control plugins to enable-admission-plugins Kubernetes API server flag:
--enable-admission-plugins=NodeRestriction,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
Delete current kube-apiserver Pod and wait until Kubernetes will respawn the new one with reflected changes:
kubectl delete pod <kube-apiserver-Pod> -n kube-system
Hope it will help you, I've successfully checked these steps on my environment.
More information about Kubernetes Admission Controllers you can find in the official documentation.
Thanks for the reply, even that works, posting the kubeadm answer just in case anyone needs it, following is the right kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
networking:
podSubnet: 192.168.0.0/16
apiServer:
extraArgs:
enable-admission-plugins: "NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook"
With reference to https://github.com/kubernetes/kubeadm/issues/1239. How do I configure and start the latest kubeadm successfully?
kubeadm_new.config is generated by config migration:
kubeadm config migrate --old-config kubeadm_default.config --new-config kubeadm_new.config. Content of kubeadm_new.config:
apiEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
apiVersion: kubeadm.k8s.io/v1alpha3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: khteh-t580
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1alpha3
auditPolicy:
logDir: /var/log/kubernetes/audit
logMaxAge: 2
path: ""
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
etcd:
local:
dataDir: /var/lib/etcd
image: ""
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.12.2
networking:
dnsDomain: cluster.local
podSubnet: ""
serviceSubnet: 10.96.0.0/12
unifiedControlPlaneImage: ""
I changed "kubernetesVersion: v1.12.2" in kubeadm_new.config and it seems to progress further and now stuck at the following error:
failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false.
How do I set fail-swap-on to FALSE to get it going?
Kubeadm comes with a command which prints default configuration, so you can check each of the assigned default values with:
kubeadm config print-default
In your case, if you want to disable swap check in the kubelet, you have to add the following lines to your current kubeadm config:
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
You haven't mentioned why you chose to disable swap.
I wouldn't consider it as a first option - not because memory swap is a bad practice (it is a useful and basic kernel mechanism) but because it seems the Kubelet is not designed to work properly with swap enabled.
K8S is very clear about this topic as you can see in the Kubeadm installation:
Swap disabled. You MUST disable swap in order for the kubelet to work
properly.
I would recommend reading about Evicting end-user Pods and the relevant features that K8S provides to prioritize memory of pods:
1 ) The 3 qos classes - Make sure that your high priority workloads are running with the Guaranteed (or at least Burstable) class.
2 ) Pod Priority and Preemption.