I have two kubernetes cluster setup with kubeadm and im using haproxy to redirect and load balance traffic to the different clusters. Now I want to redirect the requests to the respective api server of each cluster.
Therefore, I need to decrypt the ssl requests, read the "Host" HTTP-Header and encrypt the traffic again. My example haproxy config file looks like this:
frontend k8s-api-server
bind *:6443 ssl crt /root/k8s/ssl/apiserver.pem
mode http
default_backend k8s-prod-master-api-server
backend k8s-prod-master-api-server
mode http
option forwardfor
server master 10.0.0.2:6443 ssl ca-file /root/k8s/ssl/ca.crt
If I now access the api server via kubectl, I get the following errors:
kubectl get pods
error: the server doesn't have a resource type "pods"
kubectl get nodes
error: the server doesn't have a resource type "nodes"
I think im using the wrong certificates for decryption and encryption.
Do I need to use the apiserver.crt , apiserver.key and ca.crt files in the directory /etc/kubernetes/pki ?
Your setup probably entails authenticating with your Kubernetes API server via client certificates; when your HAProxy reinitiates the connection it is not doing so with the client key and certificate on your local machine, and it's likely making an unauthenticated request. As such, it probably doesn't have permission to know about the pod and node resources.
An alternative is to proxy at L4 by reading the SNI header and forwarding traffic that way. This way, you don't need to read any HTTP headers, and thus you don't need to decrypt and re-encrypt the traffic. This is possible to do with HAProxy.
Related
I'm trying to understand the security implications for using self-signed certificates for a Kubernetes validating webhook.
If I'm understanding correctly, the certificate is simply used to be able to serve the validating webhook server over https. When the Kubernetes api-server receives a request that matches the configuration for a validating webhook, it'll first check with the validating webhook server over https. If your validating webhook server lives on the Kubernetes cluster (is not external) then this traffic is all internal to a Kubernetes cluster. If this is the case is it problematic that the cert is self-signed?
If I'm understanding correctly, the certificate is simply used to be
able to serve the validating webhook server over https.
Basically yes.
If your validating webhook server lives on the Kubernetes cluster (is
not external) then this traffic is all internal to a Kubernetes
cluster. If this is the case is it problematic that the cert is
self-signed?
If the issuing process is handled properly and in secure manner, self-signed certs shouldn't be a problem at all. Compare with this example.
On Google Kubernetes Engine (GKE) you can use the cloud.google.com/app-protocols annotation on a Service to specify what protocol is used on that port (HTTP or HTTPS) docs
When you create an External HTTP(S) Ingress, it will use this protocol between the Ingress and the Service.
How do I set things up so that the Service uses a certificate that is actually trusted by the Ingress?
Does it just trust any certificate signed by the Cluster Root CA? Manage tls in a cluster suggests you need to include the pod IP address in the CSR - does that mean generating the CSR and waiting for the signed certificate to be created should be part of my container startup process?
Turns out when the "GKE Ingress for HTTP(S) Load Balancing" uses HTTPS to connect to the service, it accepts any certificate valid (even a self-signed one), without further configuration.
Apparently it does not use TLS to protect against MITM attacks here (which I guess might be reasonable).
I want to proxy API Gateway requests to a container in an ECS cluster. As API Gateway can not be placed in a security group, I have to expose the service through an internet-facing ALB load balancer. To verify that the service only handles traffic from API GW, I'm looking to use client certificates.
Is there a way to have ELB verify the API Gateway client certificate for me and terminate the SSL connection before handing the request off to my HTTP backend?
If ELB cannot verify the certificate, it seems to me that I have to "downgrade" the proxy to a TCP proxy, and perform the SSL handshake in my service. However, to do this, I also need to serve the certificate from the service. This is a problem because I'm using AWS provisioned SSL certificates, and I do not possess the private key. Thus I would have to not only recreate the SSL handshake, but also get my SSL certificate from somewhere else, is this correct?
Hoping there's a low-effort way to set up the trust between API Gateway and an ECS service behind an ALB.
Unfortunately, ELB does not support validating the client certificate for you. You can implement SSL termination on your service and use the ELB in TCP mode (or use the newer NLB). You don't need the private key of the client certificate, you simply add the public key/cert to your trusted CA on the service so that the service only trusts the API GW client certificate.
Objective
I'm seeking clarification around the nuances of accessing the Kubelet API.
Context
I have the IP of the node (physical host's IP) that a pod is in. I would like to make calls to the Kubelet API (running on the node) e.g to ${node_ip}:10255
Question(s)
Can the protocol be HTTP?
If it can be HTTP, do I need provide any form of authentication e.g. a bearer token?
If it must be HTTPS, what forms of authentication must I provide?
Bearer token?
Certificates?
There are two ports the kubelet may listen on.
--read-only-port is the http read-only port for the Kubelet to serve on with no authentication/authorization (defaults to 10255, can set to 0 to disable). If enabled, this only serves read-only data, and doesn't expose the APIs that allow pod exec/attach/proxy, etc.
--port is the https port for the Kubelet to serve all its APIs on, with optional authentication/authorization. (defaults to 10250)
See http://kubernetes.io/docs/admin/kubelet-authentication-authorization/ for the authentication/authorization options for the secure port.
Authentication options include client certificate, API bearer token, and to allow anonymous requests.
Authorization options include allowing all requests and delegating authorization to the API server via the SubjectAccessReview API
I have a Kubernetes cluster running in High Availability mode with 3 master nodes. When I try to run the DNS cluster add-on as-is, the kube2sky application errors with an x509 signed by unknown certificate authority message for the API Server service address (which in my case is 10.100.0.1). Reading through some of the GitHub issues, it looked like Tim Hockin had fixed this type of issue via using the default service account tokens available.
All 3 of my master nodes generate their own certificates for the secured API port, so is there something special I need to do configuration-wise on the API servers to get the CA certificate included in the default service account token?
It would be ideal to have the service IP of the API in the SAN field of all your server certificates.
If this is not possible in your setup, set the clusters{}.cluster.insecure-skip-tls-verify field to true in your kubeconfig file, or the pass the --insecure-skip-tls-verify flag to kubectl.
If you are trying to reach the API from within a pod you could use the secrets mounted via the Service Account. By default, if you use the default secret, the CA certificate and a signed token are mounted to /var/run/secrets/kubernetes.io/serviceaccount/ in every pod, and any client can use them from within the pod to communicate with the API. This would help you solving the unknown certificate authority error and provide you with an easy way to authenticate against your API servers at the same time.