How do I find the join command for kubeadm on the master? - kubernetes

I've lost the original 'kubeadm join' command when I previously ran kubeadm init.
How can I retrieve this value again?

kubeadm token create --print-join-command

To print a join command for a new worker node use:
kubeadm token create --print-join-command
But if you need to join a new control plane node, you need to recreate a new key for the control plane join command. This can be done with three simple steps:
Re upload certificates in the already working master node with kubeadm init phase upload-certs --upload-certs. That will generate a new certificate key.
Print join command in the already working master node with kubeadm token create --print-join-command.
Join a new control plane node with $JOIN_COMMAND_FROM_STEP2 --control-plane --certificate-key $KEY_FROM_STEP1.
This might not work for the old Kubernetes versions but I tried with the new version and it worked for me.

To create kubeadm join command, please run the following commands:
Step 1 - Retrieve Token CA Hash:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex \
| sed 's/^.* //'
This command will provide you public key.
Step 2 - Retrieve bootstrap Tokens:
kubeadm token list
This will print all tokens, so copy the token value under TOKEN with the description "The default bootstrap token generated by kubeadm init."
Step 3 - Creates kubeadm init command:
Now use following syntax to create join command without creating a new token:
kubeadm join <ip-address>:6443\
--token=<token-from-step-2> \
--discovery-token-ca-cert-hash sha256:<ca-hash-from-step-1>
kubeadm token create command creates a new token, in this case without any description, so for you not to create any additional tokens, just pick the token which has a DESCRIPTION as mentioned in Step 2.

Run the below command on your master node machine.
kubeadm token create --print-join-command
This command will generate the new token as well as the join command which you can use at your worker node to join the cluster.

Building off #Abhishek Jain's answer, here's a script to print the kubeadm join command with a little help from jq:
# get the join command from the kube master
CERT_HASH=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex \
| sed 's/^.* //')
TOKEN=$(kubeadm token list -o json | jq -r '.token' | head -1)
IP=$(kubectl get nodes -lnode-role.kubernetes.io/master -o json \
| jq -r '.items[0].status.addresses[] | select(.type=="InternalIP") | .address')
PORT=6443
echo "sudo kubeadm join $IP:$PORT \
--token=$TOKEN --discovery-token-ca-cert-hash sha256:$CERT_HASH"

If you are joining control plane nodes, you will need a certificate key in the command too:
kubeadm token create \
--print-join-command \
--certificate-key \
$(kubeadm alpha certs certificate-key)
The kubeadm alpha certs certificate-key command will generate a new certificate key on demand as per the documentation here
To Join a worker node, the command kubeadm token create --print-join-command given in the accepted answer is sufficient

Here is a bash script that automate this task
read -p 'master ip address : ' ipaddr
sha_token = "$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')"
token = "$(kubeadm token list | awk '{print $1}' | sed -n '2 p')"
echo "kubeadm join $ipaddr:6443 --token=$token --discovery-token-ca-cert-hash sha256:$sha_token"

Related

HashiCorp vault error "vault: command not found"

I am trying to run some vault commands using a shell script and it gives me an error called,
line 13: vault: command not found
line 14: vault: command not found
But I have already installed vault and and stored a openssl key and certificate using KV secret store and successfully retrieved that key value pair using terminal commands.
This is the shell script that I have used,
#!/bin/sh
PARENT_DIR=sslcertnkeys
CERT_FILE=apache-cert.crt
KEY_FILE=apache-key.key
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN=s.8AreSPokOQPF9Rs1xp82TDz2
# make parent directory if not exists
[ -e $PARENT_DIR ] || mkdir -p $PARENT_DIR
# remove files if exists
[ -e $CERT_FILE ] && rm -f $CERT_FILE
[ -e $KEY_FILE ] && rm -f $KEY_FILE
# retrieve certificate and key from vault and store in disk
vault kv get -field=certificate certs/apache > $CERT_FILE
vault kv get -field=private_key certs/apache > $KEY_FILE
# unset vault address
unset VAULT_ADDR
# unset vault login token
unset VAULT_TOKEN
# start apache server
systemctl start httpd.service
Can someone help me to figure out what is the problem in here?

vault kv put/write binary data

I have a enc file that I generate using openssl like so.
openssl enc -aes-256-cbc -md sha1 -pass file:/tmp/keyfile.key \
-in /tmp/keyfile \
-out /tmp/keyfile.enc
I want to store the keyfile.enc file as secret in vault. I tried
cat keyfile.enc | vault kv put secret/mysecret key=-
Does kv put/write work with binary data ?
I want to use the vault mutating webhook to mount the secret into a container ultimately.
https://discuss.hashicorp.com/t/loading-storing-binary-files-as-secrets/22636
Got my answer. Yes we cannot use binary files, base64 encode them

Differences between generated x509 certificates in kubernetes v1.10.11 and v1.11.5

I have been creating an amazon AMI with kubernetes installed on it to use as a worker node in EKS, I install the kubelet binary from the amazon-eks s3 bucket.
After upgrading from k8s version 1.10.11 to 1.11.5 I noticed a difference in the x509 certificate that is generated when installing kubelet.
If I jump onto one of the worker nodes with 1.10.11 installed and run this command openssl s_client -connect localhost:10250 2>/dev/null | openssl x509 -noout -text I get the following output for X509v3 Subject Alternative Name:
DNS:ip-<my-ip>.eu-central-1.compute.internal, DNS:ip-<my-ip>, IP Address:<my-ip>
whereas, if I run the same command on a worker node with 1.11.5 installed I get the following output for X509v3 Subject Alternative Name:
DNS:ip-<my-ip>
The only change between the two nodes is the version of kubernetes installed.
Am I missing anything that is now required as of version 1.11.x to set the additional Subject Alternative Names as seemed to be previously done in v1.10.x? I require the IP address to be set in the certificate in the format IP Address:<my-ip> which I was getting for free in version 1.10.
FYI I am running kubelet with the following args:
ExecStart=/usr/bin/kubelet \
--address=0.0.0.0 \
--authentication-token-webhook \
--authorization-mode=Webhook \
--allow-privileged=true \
--cloud-provider=aws \
--cluster-dns=DNS_CLUSTER_IP \
--cluster-domain=cluster.local \
--cni-bin-dir=/opt/cni/bin \
--cni-conf-dir=/etc/cni/net.d \
--container-runtime=docker \
--max-pods=MAX_PODS \
--node-ip=INTERNAL_IP \
--network-plugin=cni \
--pod-infra-container-image=602401143452.dkr.ecr.REGION.amazonaws.com/eks/pause-amd64:3.1 \
--cgroup-driver=cgroupfs \
--register-node=true \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--feature-gates=RotateKubeletServerCertificate=true \
--anonymous-auth=false \
--client-ca-file=CLIENT_CA_FILE \
--node-labels=env=NODE_LABEL
As far as handling the certificates there are not Kubernetes specific differences between 1.10.11 and 1.11.5. It might be related to specific EKS AMI for the nodes that you are using (make sure they are matching)
If not you can manually create the certificates for the kubelet using the same CA as the one in your Kubernetes master. For example:
easyrsa
./easyrsa --subject-alt-name="IP:${MASTER_IP},"\
"IP:-<my-ip>,"\
"DNS:ip-<my-ip>.eu-central-1.compute.internal,"\
"DNS:ip-<my-ip>,"\
--days=10000 \
build-server-full server nopass
openssl
Config (csr.conf):
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = <country>
ST = <state>
L = <city>
O = <organization>
OU = <organization unit>
CN = <my-ip>
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = ip-<my-ip>.eu-central-1.compute.internal
DNS.2 = ip-<my-ip>
IP.1 = <my-ip>
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=#alt_names
Create CSR:
$ openssl req -new -key server.key -out server.csr -config csr.conf
Create certificate:
$ openssl x509 -req -in server.csr -CA cluster-ca.crt -CAkey cluster-ca.key \
-CAcreateserial -out server.crt -days 10000 \
-extensions v3_ext -extfile csr.conf
cfssl
In a similar fashion you can use cfssl, described here.

kubeadm: Explicitly set token for nodes to join with

I have set up my master nodes using kubeadm.
Now I want to run the join command on my nodes so that the later join the cluster.
All I have to do is run
kubeadm join --token <token> --discovery-token-ca-cert-hash <sha256>
where <token> and are values previously returned by the command below:
kubeadm init
I am also trying to script the above process and I see that parsing the actual tokens from the last command is kinda difficult;
So I was wandering whether there is a way to explicitly specify the <token> and the <sha256> during cluster initialization, to avoid having to perform hacky parsing of the init command.
I was trying to make a script for it as well.
In order to get the values needed I am using these commands:
TOKEN=$(sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no root#$MASTER_IP sudo kubeadm token list | tail -1 | cut -f 1 -d " ")
HASH=$(sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no root#$MASTER_IP openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' )
Basically I use this commands to ssh on master and get this values.
I have not found a easier way to achieve this.
Actually there seems to be a way around this:
(I am putting this in ansible tasks cause this is where I am planning to use it)
- name: kubernetes.yml --> Initiate kubernetes cluster
shell: 'kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={{ ansible_facts[if_name]["ipv4"]["address"] }}'
become: yes
when: inventory_hostname in groups['masters']
- name: kubernetes.yml --> Get the join command
shell: kubeadm token create --print-join-command
register: rv_join_command
when: inventory_hostname in (groups['masters'] | last)
become: yes
- name: kubernetes.yml --> Print the join command
debug:
var: rv_join_command.stdout
Output:
TASK [kubernetes.yml --> Print the join command] *******************************
ok: [kubernetes-master-1] =>
rv_join_command.stdout: 'kubeadm join 192.168.30.1:6443 --token ah0dbr.grxg9fke3c28dif3i --discovery-token-ca-cert-hash sha256:716712ca7f07bfb4aa7df9a8b30ik3t0k3t2259b8c6fc7b68f50334356078 '

How to access kubernetes API from the (non-master) nodes or remote

I'm having a trouble setting up access to kubernetes cluster from the outside. This is what I'm trying to achieve:
- Have ability to access to kube cluster from the outside (from nodes that are not "master" and even from any remote) to be able to do kube actions only on specific namespace.
My logic was to do following:
Create new namespace (let's call it testns)
Create service account (testns-account)
Create role which gives access for creating any type of kube resource inside testns namespace
Create role binding which binds service account with role
Generate token from service account
Now, my logic was that I need to have token + api server URL to access kube cluster with limited "permissions" but that doesnt seem like it is enough.
What would be the easiest way to achieve this? For start, I could have access with kubectl just to verify that limited permissions on namespace work but eventually, I would have some client side code which doing the access and creates kube resources with these limited permissions.
You need to generate a kubeconfig from the token. There are scripts to handle this. Here it is for posterity:
!/usr/bin/env bash
# Copyright 2017, Z Lab Corporation. All rights reserved.
# Copyright 2017, Kubernetes scripts contributors
#
# For the full copyright and license information, please view the LICENSE
# file that was distributed with this source code.
set -e
if [[ $# == 0 ]]; then
echo "Usage: $0 SERVICEACCOUNT [kubectl options]" >&2
echo "" >&2
echo "This script creates a kubeconfig to access the apiserver with the specified serviceaccount and outputs it to stdout." >&2
exit 1
fi
function _kubectl() {
kubectl $# $kubectl_options
}
serviceaccount="$1"
kubectl_options="${#:2}"
if ! secret="$(_kubectl get serviceaccount "$serviceaccount" -o 'jsonpath={.secrets[0].name}' 2>/dev/null)"; then
echo "serviceaccounts \"$serviceaccount\" not found." >&2
exit 2
fi
if [[ -z "$secret" ]]; then
echo "serviceaccounts \"$serviceaccount\" doesn't have a serviceaccount token." >&2
exit 2
fi
# context
context="$(_kubectl config current-context)"
# cluster
cluster="$(_kubectl config view -o "jsonpath={.contexts[?(#.name==\"$context\")].context.cluster}")"
server="$(_kubectl config view -o "jsonpath={.clusters[?(#.name==\"$cluster\")].cluster.server}")"
# token
ca_crt_data="$(_kubectl get secret "$secret" -o "jsonpath={.data.ca\.crt}" | openssl enc -d -base64 -A)"
namespace="$(_kubectl get secret "$secret" -o "jsonpath={.data.namespace}" | openssl enc -d -base64 -A)"
token="$(_kubectl get secret "$secret" -o "jsonpath={.data.token}" | openssl enc -d -base64 -A)"
export KUBECONFIG="$(mktemp)"
kubectl config set-credentials "$serviceaccount" --token="$token" >/dev/null
ca_crt="$(mktemp)"; echo "$ca_crt_data" > $ca_crt
kubectl config set-cluster "$cluster" --server="$server" --certificate-authority="$ca_crt" --embed-certs >/dev/null
kubectl config set-context "$context" --cluster="$cluster" --namespace="$namespace" --user="$serviceaccount" >/dev/null
kubectl config use-context "$context" >/dev/null
cat "$KUBECONFIG"