I have been creating an amazon AMI with kubernetes installed on it to use as a worker node in EKS, I install the kubelet binary from the amazon-eks s3 bucket.
After upgrading from k8s version 1.10.11 to 1.11.5 I noticed a difference in the x509 certificate that is generated when installing kubelet.
If I jump onto one of the worker nodes with 1.10.11 installed and run this command openssl s_client -connect localhost:10250 2>/dev/null | openssl x509 -noout -text I get the following output for X509v3 Subject Alternative Name:
DNS:ip-<my-ip>.eu-central-1.compute.internal, DNS:ip-<my-ip>, IP Address:<my-ip>
whereas, if I run the same command on a worker node with 1.11.5 installed I get the following output for X509v3 Subject Alternative Name:
DNS:ip-<my-ip>
The only change between the two nodes is the version of kubernetes installed.
Am I missing anything that is now required as of version 1.11.x to set the additional Subject Alternative Names as seemed to be previously done in v1.10.x? I require the IP address to be set in the certificate in the format IP Address:<my-ip> which I was getting for free in version 1.10.
FYI I am running kubelet with the following args:
ExecStart=/usr/bin/kubelet \
--address=0.0.0.0 \
--authentication-token-webhook \
--authorization-mode=Webhook \
--allow-privileged=true \
--cloud-provider=aws \
--cluster-dns=DNS_CLUSTER_IP \
--cluster-domain=cluster.local \
--cni-bin-dir=/opt/cni/bin \
--cni-conf-dir=/etc/cni/net.d \
--container-runtime=docker \
--max-pods=MAX_PODS \
--node-ip=INTERNAL_IP \
--network-plugin=cni \
--pod-infra-container-image=602401143452.dkr.ecr.REGION.amazonaws.com/eks/pause-amd64:3.1 \
--cgroup-driver=cgroupfs \
--register-node=true \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--feature-gates=RotateKubeletServerCertificate=true \
--anonymous-auth=false \
--client-ca-file=CLIENT_CA_FILE \
--node-labels=env=NODE_LABEL
As far as handling the certificates there are not Kubernetes specific differences between 1.10.11 and 1.11.5. It might be related to specific EKS AMI for the nodes that you are using (make sure they are matching)
If not you can manually create the certificates for the kubelet using the same CA as the one in your Kubernetes master. For example:
easyrsa
./easyrsa --subject-alt-name="IP:${MASTER_IP},"\
"IP:-<my-ip>,"\
"DNS:ip-<my-ip>.eu-central-1.compute.internal,"\
"DNS:ip-<my-ip>,"\
--days=10000 \
build-server-full server nopass
openssl
Config (csr.conf):
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = <country>
ST = <state>
L = <city>
O = <organization>
OU = <organization unit>
CN = <my-ip>
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = ip-<my-ip>.eu-central-1.compute.internal
DNS.2 = ip-<my-ip>
IP.1 = <my-ip>
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=#alt_names
Create CSR:
$ openssl req -new -key server.key -out server.csr -config csr.conf
Create certificate:
$ openssl x509 -req -in server.csr -CA cluster-ca.crt -CAkey cluster-ca.key \
-CAcreateserial -out server.crt -days 10000 \
-extensions v3_ext -extfile csr.conf
cfssl
In a similar fashion you can use cfssl, described here.
Related
I'm generating a keypair in a softHSM keystore, and uses it to sign a software.
Now I need to extract the public key and move it to my target.
I can sign and verify with pkcs11-tool, but I have not succeeded with using openssl verifying.
All I get is
80DB511C4A7F0000:error:02000068:rsa routines:ossl_rsa_verify:bad signature:../crypto/rsa/rsa_sign.c:430:
This is the series of commands I use to generate the keys.
softhsm2-util --init-token --free --label "token-label" --so-pin mysecret1 --pin mysecret1
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --login-type so --so-pin mysecret1 --init-pin --new-pin mysecret1
# create a public-private key pair.
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so -l --token-label token-label -k --key-type rsa:2048 --usage-sign --id 1002 --label rsatest --pin mysecret1
pkcs11-tool --modul /usr/lib/softhsm/libsofthsm2.so --id 1002 --read-object --type pubkey -o rsa.der
openssl ec -pubin -inform DER -in rsa.der -outform PEM -out rsa.pem
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin mysecret1 --sign --id 1002 -m RSA-PKCS --input text.txt --output /tmp/rsa.signature
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --id 1002 --verify -m RSA-PKCS --input-file text.txt --signature-file /tmp/rsa.signature
openssl dgst -sha512 -keyform PEM -verify rsa.pem -signature /tmp/rsa.signature text.txt
I have tried several combinations of PEM and DER files. Also to change the -sha512 parameter.
A demo project with Dockerfile can be found here: https://gitlab.com/kjeld.flarup/consoletest
A little inspired by https://stackoverflow.com/users/589259/maarten-bodewes I searched for RSA-PSS and found that I could specify SHA512 in the mechanism. Then it worked.
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --login --pin mysecret1 --sign --id 1002 -m SHA512-RSA-PKCS --input text.txt --output /tmp/rsa.signature
First of all, question is languate-agnostic. I'm trying to write a simple application that connects to PostgreSQL using SSL.
I created certificates using scripts:
# Create CA private key
openssl genrsa -des3 -out root.key 4096
#Remove a passphrase
openssl rsa -in root.key -out root.key
# Create a root Certificate Authority (CA)
openssl \
req -new -x509 \
-days 365 \
-subj "/CN=localhost" \
-key root.key \
-out root.crt
# Create server key
openssl genrsa -des3 -out server.key 4096
#Remove a passphrase
openssl rsa -in server.key -out server.key
# Create a root certificate signing request
openssl \
req -new \
-key server.key \
-subj "/CN=localhost" \
-text \
-out server.csr
# Create server certificate
openssl \
x509 -req \
-in server.csr \
-text \
-days 365 \
-CA root.crt \
-CAkey root.key \
-CAcreateserial \
-out server.crt
I created a database using:
init.sql
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE TABLE TESTING_DATA (
ID SERIAL PRIMARY KEY,
UUID UUID UNIQUE NOT NULL DEFAULT uuid_generate_v4(),
NAME TEXT NOT NULL,
INFO NUMERIC(3, 2)
);
INSERT INTO TESTING_DATA (NAME, INFO)
VALUES
('Notebook', 1),
('Office supplies', 2),
('Pencil', 2),
('Office supplies', 1),
('Eraser', 1),
('Coffee', 1),
('Cookies', 2),
('Puzzles', 5)
;
postgresql.conf
ssl = on
ssl_ca_file = '/etc/postgres/security/root.crt'
ssl_cert_file = '/etc/postgres/security/server.crt'
ssl_key_file = '/etc/postgres/security/server.key'
password_encryption = scram-sha-256
pg_hba.conf
local all all md5
host all all 127.0.0.1/32 md5
hostssl all all 0.0.0.0/0 cert clientcert=1
Dockerfile
FROM postgres:12-alpine
ENV POSTGRES_USER=pguser
ENV POSTGRES_PASSWORD=pgpassword
ENV POSTGRES_DB=securitylearning
COPY pg_hba.conf postgresql.conf /etc/postgresql/config/
COPY --chown=postgres:postgres root.crt server.crt server.key /etc/postgres/security/
COPY init.sql /docker-entrypoint-initdb.d/
EXPOSE 5432
CMD ["postgres", "-c", "config_file=/etc/postgresql/config/postgresql.conf", "-c", "hba_file=/etc/postgresql/config/pg_hba.conf"]
I launched a container, I ensured that from the container itself I can connect to database and select something from the table.
I created a simple program:
server.go
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
func main() {
connection := fmt.Sprint(
" host=localhost",
" port=5432",
" user=pguser",
" dbname=securitylearning",
" sslmode=verify-full",
" sslrootcert=root.crt",
" sslkey=client.key",
" sslcert=client.crt",
)
db, err := sql.Open("postgres", connection)
defer db.Close()
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
row := db.QueryRow("SELECT * FROM TESTING_DATA")
fmt.Println(row)
}
I tried to:
place files root.crt, server.crt, server.key next to the compiled binary and add to the connection string in go file sslrootcert, sslcert, sslkey respectively
place same files, but with names root.crt, postgresql.crt, postgresql.key in ~/.postgresql/ directory, because pq uses them by default.
For now, it's not working. I randomly get one of those two errors:
read: connection reset by peer
or
EOF
Could you please help? What am I missing here? Or could you point me to some resources? Thanks in advance.
Update 1
Thanks to suggestion in comments, I created client key and certificate, using
# Create client key
openssl genrsa -out client.key 4096
#Remove a passphrase
openssl rsa -in client.key -out client.key
# Create client certificate signing request
openssl \
req -new \
-key client.key \
-subj "/CN=172.17.0.2" \
-out client.csr
# Create client certificate
openssl \
x509 -req \
-in client.csr \
-CA root.crt \
-CAkey root.key \
-CAcreateserial \
-days 365 \
-text \
-out client.crt
I'm using 172.17.0.2 in CN, because it's host IP from docker container's perspective.
I've tried both:
using following keys in connection string from program
" sslrootcert=root.crt",
" sslkey=client.key",
" sslcert=client.crt",
copying root.crt, client.key, client.srt to ~/.postgresql/, trying psql with
psql "host=localhost port=5432 user=pguser dbname=securitylearning sslmode=verify-full sslrootcert=root.crt sslkey=client.key sslcert=client.crt"
or without password.
Both ways still fail to connect. In psql case I get error
psql: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
Thanks to suggestions in comments I managed to solve it.
First of all, as suggested, I stepped back and tried to proceed with smaller steps. Such as, securely connect with psql from host.
Mistake 1
I forgot to add the following property to postgresql.conf
listen_addresses = '*'
The documentation says:
If the list is empty, the server does not listen on any IP interface at all, in which case only Unix-domain sockets can be used to connect to it.
Mistake 2
I fell into a little misconception with certificates and their common names (CN). The following points should be applied to scripts that create certificates. In short:
CN for CA can be anything as long as it is different from the server's
CN. See this question and answer for details
CN for server must be IP/hostname by which we will call server from client (here it's localhost. But if the database would be located at
cooldatabase.com <- this would be server's CN)
CN for client must be username by which we will connect (here, it's
pguser)
When I fixed these two issues - I managed to connect via both psql and go program! Also, the default postgresql.conf is very informative!
I have set up my master nodes using kubeadm.
Now I want to run the join command on my nodes so that the later join the cluster.
All I have to do is run
kubeadm join --token <token> --discovery-token-ca-cert-hash <sha256>
where <token> and are values previously returned by the command below:
kubeadm init
I am also trying to script the above process and I see that parsing the actual tokens from the last command is kinda difficult;
So I was wandering whether there is a way to explicitly specify the <token> and the <sha256> during cluster initialization, to avoid having to perform hacky parsing of the init command.
I was trying to make a script for it as well.
In order to get the values needed I am using these commands:
TOKEN=$(sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no root#$MASTER_IP sudo kubeadm token list | tail -1 | cut -f 1 -d " ")
HASH=$(sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no root#$MASTER_IP openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' )
Basically I use this commands to ssh on master and get this values.
I have not found a easier way to achieve this.
Actually there seems to be a way around this:
(I am putting this in ansible tasks cause this is where I am planning to use it)
- name: kubernetes.yml --> Initiate kubernetes cluster
shell: 'kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={{ ansible_facts[if_name]["ipv4"]["address"] }}'
become: yes
when: inventory_hostname in groups['masters']
- name: kubernetes.yml --> Get the join command
shell: kubeadm token create --print-join-command
register: rv_join_command
when: inventory_hostname in (groups['masters'] | last)
become: yes
- name: kubernetes.yml --> Print the join command
debug:
var: rv_join_command.stdout
Output:
TASK [kubernetes.yml --> Print the join command] *******************************
ok: [kubernetes-master-1] =>
rv_join_command.stdout: 'kubeadm join 192.168.30.1:6443 --token ah0dbr.grxg9fke3c28dif3i --discovery-token-ca-cert-hash sha256:716712ca7f07bfb4aa7df9a8b30ik3t0k3t2259b8c6fc7b68f50334356078 '
I've lost the original 'kubeadm join' command when I previously ran kubeadm init.
How can I retrieve this value again?
kubeadm token create --print-join-command
To print a join command for a new worker node use:
kubeadm token create --print-join-command
But if you need to join a new control plane node, you need to recreate a new key for the control plane join command. This can be done with three simple steps:
Re upload certificates in the already working master node with kubeadm init phase upload-certs --upload-certs. That will generate a new certificate key.
Print join command in the already working master node with kubeadm token create --print-join-command.
Join a new control plane node with $JOIN_COMMAND_FROM_STEP2 --control-plane --certificate-key $KEY_FROM_STEP1.
This might not work for the old Kubernetes versions but I tried with the new version and it worked for me.
To create kubeadm join command, please run the following commands:
Step 1 - Retrieve Token CA Hash:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex \
| sed 's/^.* //'
This command will provide you public key.
Step 2 - Retrieve bootstrap Tokens:
kubeadm token list
This will print all tokens, so copy the token value under TOKEN with the description "The default bootstrap token generated by kubeadm init."
Step 3 - Creates kubeadm init command:
Now use following syntax to create join command without creating a new token:
kubeadm join <ip-address>:6443\
--token=<token-from-step-2> \
--discovery-token-ca-cert-hash sha256:<ca-hash-from-step-1>
kubeadm token create command creates a new token, in this case without any description, so for you not to create any additional tokens, just pick the token which has a DESCRIPTION as mentioned in Step 2.
Run the below command on your master node machine.
kubeadm token create --print-join-command
This command will generate the new token as well as the join command which you can use at your worker node to join the cluster.
Building off #Abhishek Jain's answer, here's a script to print the kubeadm join command with a little help from jq:
# get the join command from the kube master
CERT_HASH=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex \
| sed 's/^.* //')
TOKEN=$(kubeadm token list -o json | jq -r '.token' | head -1)
IP=$(kubectl get nodes -lnode-role.kubernetes.io/master -o json \
| jq -r '.items[0].status.addresses[] | select(.type=="InternalIP") | .address')
PORT=6443
echo "sudo kubeadm join $IP:$PORT \
--token=$TOKEN --discovery-token-ca-cert-hash sha256:$CERT_HASH"
If you are joining control plane nodes, you will need a certificate key in the command too:
kubeadm token create \
--print-join-command \
--certificate-key \
$(kubeadm alpha certs certificate-key)
The kubeadm alpha certs certificate-key command will generate a new certificate key on demand as per the documentation here
To Join a worker node, the command kubeadm token create --print-join-command given in the accepted answer is sufficient
Here is a bash script that automate this task
read -p 'master ip address : ' ipaddr
sha_token = "$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')"
token = "$(kubeadm token list | awk '{print $1}' | sed -n '2 p')"
echo "kubeadm join $ipaddr:6443 --token=$token --discovery-token-ca-cert-hash sha256:$sha_token"
I am using Kong 10.x with datastore postgresql 9.6.
I want to establish ssl connection between kong and it's datastore postgresql.
But I get the following errors:
Postgresql Error:
LOG: could not accept SSL connection: tlsv1 alert unknown ca
Error from kong:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: [postgres error] could not retrieve current migrations: [postgres error] connection refused
Below are my Kong and Postgresql Configurations:
Kong:
# Kong configuration file
# DATASTORE
database = postgres
pg_host = 10.0.1.191
pg_port = 5432
pg_user = kong
pg_password = kong
pg_database = kong
pg_ssl = on
pg_ssl_verify = on
# DEVELOPMENT & MISCELLANEOUS
lua_ssl_trusted_certificate = /opt/postgres_ssl/postgresql.crt # Absolute path to the certificate
Postgresql:
pg_hba.conf:
hostssl all all 10.0.1.191/32 md5 clientcert=1
postgresql.conf:
listen_addresses = '10.0.1.191'
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_ca_file = 'root.crt'
Certificate Generation Procedure:
openssl genrsa -passout pass:mypass -des3 -out server.key 1024
openssl rsa -passin pass:mypass -in server.key -out server.key
chmod 400 server.key
openssl req -new -key server.key -days 3650 -out server.crt -x509 -subj '/C=IN/ST=Maharastra/L=Mumbai/O=Development/CN=10.0.1.191'
cp server.crt root.crt
openssl genrsa -passout pass:iotadmin -des3 -out postgresql.key 1024
openssl rsa -in postgresql.key -out postgresql.key -passin pass:mypass
openssl req -new -key postgresql.key -days 3650 -out postgresql.csr -subj '/C=IN/ST=Maharastra/L=Mumbai/O=Development/CN=kong'
openssl x509 -req -in postgresql.csr -CA root.crt -CAkey server.key -out postgresql.crt -CAcreateserial
The error message says that Kong doesn't trust the CA which signed the certificate of the database. This doesn't surprise much, because it only knows the latter, but not the certificate of the CA.
Try using the root certificate for your lua_ssl_trusted_certificate config entry and it should work:
lua_ssl_trusted_certificate = /path/to/your/root.crt