I have istio set up with mtls globally enabled. I have verifed it by bashing into a pot without an envoy sidecar, and running a curl command on http, which failed. Then run a curl on HTTPS using the Istio certs, which worked.
When I port-forward to a service, eq kubectl port-forward svc/my-svc 8080:80 I am able to access my app by going to http://localhost:8080
I would expect that this wouldn't work, since mtls is being enforced.
How does this port-forward with kubernetes work? Does it go directly to the node, thereby by-passing the sidecar?
I found out that Istio is working in parallel with standard Kubernetes network services and does not affect traffic in case of port-forwarding.
Istio network service will create iptables rule exactly the same as kubectl port-forward or any other standard command do.
Unfortunately, I did not find any official documentation with an explanation of how it works.
Related
I have make my deployment work with istio ingressgateway before. I am not aware of any changes made in istio or k8s side.
When I tried to deploy, I see an error in replicaset side that's why it cannot create new pod.
Error creating: Internal error occurred: failed calling webhook
"namespace.sidecar-injector.istio.io": Post
"https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp
10.104.136.116:443: connect: no route to host
When I try to go inside api-server and ping 10.104.136.116 (istiod service IP) it just hangs.
What I have tried so far:
Deleted all coredns pods
Deleted all istiod pods
Deleted all weave pods
Reinstalling istio via istioctl x uninstall --purge
turning all of VMs firewall
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -F
restarted all of the nodes
manual istio pod injection
Setup
k8s version: 1.21.2
istio: 1.10.3
HA setup
CNI: weave
CRI: containerd
In my case this was related to firewall. More info can be found here.
The gist of it is that on GKE at least you need to open another port 15017 in addition to 10250 and 443. This is to allow communication from your master node(s) to you VPC.
I don't have a definite answer unto why is this happening. But kube-apiserver cannot access istiod via service IP, wherein it can connect when I used the istiod pod IP.
Since I don't have the control over the VM and lower networking layer and not sure if they have changed something (because it is working before).
I made this work by changing my CNI from weave to flannel
In my case it was due to firewall. Following this Istio debug guide, I identified that the kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4 command was timing out while all other cluster internal calls were ok.
The best way to diagnose this is to open temporarly your AWS Security Groups involved to 0.0.0.0/0 for port 15017 and then try again.
If the errror won't show again, you know there's need to fix this part.
I am using EKS with Amazon VPC CNI v1.12.2-eksbuild.1
I have a Kubernetes cluster which doesn't need to expose ports to the public. I am installing monitoring and logging (Prometheus & Loki or Elastic) for in house use and would like to use their GUI. I could provision https ingress and limit IP access but port forwarding seems to work.
How Does port forwarding work, under the hood?
Is port forwarding as secure as my kubectl connection?
Is the connection as fast as an ingress load balancer based HTTPs connection?
In Kubernetes documentation you can find information that port-forward command allows you to access and interact with internal Kubernetes cluster processes from your localhost. Also it's one of the best tools to debugging.
Forward one or more local ports to a pod. This command requires the node to have 'socat' installed.
Use resource type/name such as deployment/mydeployment to select a pod. Resource type defaults to 'pod' if omitted.
If there are multiple pods matching the criteria, a pod will be selected automatically. The forwarding session ends when the selected pod terminates, and rerun of the command is needed to resume forwarding.
1. How Does port forwarding work, under the hood?
This information can be found in How Does Kubernetes Port Forwarding Work? article.
The whole process is simplified by the fact that kubectl already has a built-in port forwarding functionality.
A user interacts with Kubernetes using the kubectl command-line on their local machine.
The port-forward command specifies the cluster resource name and defines the port number to port-forward to.
As a result, the Kubernetes API server establishes a single HTTP connection between your localhost and the resource running on your cluster.
The user is now able to engage that specific pod directly, either to diagnose an issue or debug if necessary.
Port forwarding is a work-intensive method. However, in some cases, it is the only way to access internal cluster resources.
2. Is port forwarding as secure as my kubectl connection?
For this question, you can find answer in Is kubectl port-forward encrypted?. As pointed by #iomv
As far as I know when you port-forward the port of choice to your machine kubectl connects to one of the masters of your cluster so yes, normally communication is encrypted. How your master communicate to the pod though is dependent on how you set up internal comms.
or #neokyle
kubectl port-forward uses socat to make an encrypted TLS tunnel with port forwarding capabilities. The tunnel goes from you to the kube api-server to the pod so it may actually be 2 tunnels with the kube api-server acting as a pseudo router.
Kubecelt port-forward is encrypted.
3. Is the connection as fast as an ingress load balancer based HTTPs connection
As connection is inside the cluster, it should be faster than connection from outside the cluster to the cluster.
In addition, there was similar Stackoverflow thread about kubectl port-forward.
I would like to know if port-forward blocks other requests to the pod like redis, mongo, etc.
If I use kubectl port-foward command with mongo port for example, the mongo server will not receive the data ?
As per your comment, kubectl port-forward does not block any exposed traffic to the target pod. So adding portforward won't affect others.
What portforward does is simply making a specific request to the API server. (see doc)
Going further, I don't think port-forward makes pod more "dangerous" (vulnerable against security), however it is generally used for debugging to scope into the pod, not to expose a service in the pod. Use Nodeport service for production.
Plus, port-forward has default timeout setting in kubelet.
kubectl proxy and kubectl port-forwarding look similar and confusing to me, what are their main differences and use cases?
As mentioned in "How kubectl port-forward works?"
kubectl port-forward forwards connections to a local port to a port on a pod.
Compared to kubectl proxy, kubectl port-forward is more generic as it can forward TCP traffic while kubectl proxy can only forward HTTP traffic.
As an example, see "Kubernetes port forwarding simple like never before" from Alex Barashkov:
Port forwarding mostly used for the purpose of getting access to internal cluster resources and debugging.
How does it work?
Generally speaking, using port forwarding you could get on your ‘localhost’ any services launched in your cluster.
For example, if you have Redis installed in the cluster on 6379, by using a command like this:
kubectl port-forward redis-master-765d459796-258hz 7000:6379
you could forward Redis from the cluster to localhost:7000, access it locally and do whatever you want to do with it.
For a limited HTTP access, see kubectl proxy, and, as an example, "On Securing the Kubernetes Dashboard" from Joe Beda:
The easiest and most common way to access the cluster is through kubectl proxy. This creates a local web server that securely proxies data to the dashboard through the Kubernetes API server.
As shown in "A Step-By-Step Guide To Install & Use Kubernetes Dashboard" from Awanish:
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
Accessing Dashboard using the kubectl
kubectl proxy
It will proxy server between your machine and Kubernetes API server.
Now, to view the dashboard in the browser, navigate to the following address in the browser of your Master VM:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
I couldn't find any information on wherever a connection creation between cluster's pod and locahost is encrypted when running "kubectl port-forward" command.
It seems like it uses "socat" library which supports encryption, but I'm not sure if kubernetes actually uses it.
As far as I know when you port-forward the port of choice to your machine kubectl connects to one of the masters of your cluster so yes, normally communication is encrypted. How your master communicate to the pod though is dependent on how you set up internal comms.
kubectl port-forward uses socat to make an encrypted TLS tunnel with port forwarding capabilities.
The tunnel goes from you to the kube api-server to the pod so it may actually be 2 tunnels with the kube api-server acting as a pseudo router.
An example of where I've found it useful was that I was doing a quick PoC of a Jenkins Pipeline hosted on Azure Kubernetes Service and earlier in my Kubernetes studies I didn't know how to setup an Ingress, but I could reach the Server via port 80 unencrypted, but I knew my traffic could be snooped on. So I just did kubectl port-forward to temporarily login and securely to debug my POC. Also really helpful with RabbitMQ Cluster hosted on Kubernetes, you can go into the management webpage with kubectl port-forward and make sure that it's clustering the way you wanted it to.