I have built a K3s (https://k3s.io) cluster on a set of Raspberry Pi4 computers.
The controller (ctrl-1) node is a gateway in that it has 2 network interfaces. One is connected to my LAN and the other is connected to a network that it creates, e.g. K3S-LAN. The two nodes (node-1 and node-2) are deployed to the K3S-LAN.
I want to be able to access the applications running on the nodes through ctrl-1, e.g. from the LAN. This is because this cluster is meant to be portable so only the ctrl-1 node needs to be connected to the guest LAN. (Yes there are issues with DNS names etc to be sorted out, but I want to get the basics running first).
This means that I need to be able to "proxy" the ingress through ctrl-1. I thought I had the right idea for this in that I deployed "nginx-ingress" to the master, using Helm. However I forgot about the service for this - this has been scheduled on the nodes, whereas it needs to be on the controller so that the ports are opened up (I think). However I cannot find how to make the service run on the controller.
At the moment I have the service running with a type of NodePort. I could install MetalLB so that I have LoadBalancer capabilities. However with what I have seen I am not sure if this would help or not.
ctrl-1 does not have any taints setup on it, just the role of master.
Am I barking up the wrong tree here? I guess this might not be the intended use case of Kubernetes, but I am playing around with an idea. Thanks for any ideas that people have.
Update*
I have just thought that the way around this might be to run HAProxy on ctrl-1 (as another service on the host) and setup rules to proxy to the necessary services within the cluster. That would act as the bridge between the networks.
You just need to expose your pod via a Nodeport type service and it can be accessed via http://master-node-ip:nodeport. Make sure that kube-proxy is running on all master and worker nodes.
The ingress approach also should work as long as you have kube-proxy running on your master. You deploy nginx ingress on your cluster and it will get deployed into a worker node. Then you can expose nginx ingress controller itself using a NodePort service. After this you can create ingress resource for configuring the nginx ingress controller to route traffic to your backend pods and services running on worker nodes. The services for backend pods should be of type ClusterIP.
Deploy nginx ingress controller and expose it via NodePort service using kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.1/deploy/static/provider/baremetal/service-nodeport.yaml
Deploy nginx pod(nginx is an example..this should be your pod) kubectl run nginx --generator=run-pod/v1 --image=nginx
Expose nginx pod via ClusterIP service
apiVersion: v1
kind: Service
metadata:
labels:
run: nginx
name: nginx-service
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
run: nginx
Create ingress resource
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: mycha-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: nginx-service
servicePort: 80
With above setup I can now access nginx and get "Welcome to nginx! " via http://master-node-ip:NodePort of nginx ingress controller
Related
I am learning Kubernetes and trying to deploy an app using MiniKube.
I have managed to expose the service mapped to nginx pod on Minikube IP. I can access the nginx service on url $(minikube ip):$(serviceport). which is fine, however I am looking to expose this to the public network. Currently this service is only accessible via my local machine, any other machine on my wifi network is not able to access it as it is exposed only on minikube ip. I dont want to forward the port in my local linux via IPtables, and I am looking for a built in solution to expose the port to world (and not just on minikube ip). I know it can be achieved as minikube dashboard by default expose the service on localhost, this implies that minikube can talk to other network adapters and can register the port, I am not sure how.
Here is my service yaml:
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: nginxservice
labels:
app: nginxservice
spec:
type: NodePort
ports:
- port: 80
name: http
targetPort: 80
nodePort: 32756
selector:
app: nginxcontainer
#subudear is right - you need Ingress.
An API object that manages external access to the services in a
cluster, typically HTTP. Ingress may provide load balancing, SSL
termination and name-based virtual hosting.
Ingress exposes HTTP and
HTTPS routes from outside the cluster to services within the cluster.
Traffic routing is controlled by rules defined on the Ingress
resource.
To be able use regularly use ingress(Im not talking about minikube right now) - it is not enough simply create Ingress object. You should first install related ingress controller.
There are lot of them, most popular are:
NGINX Ingress Controller
Kubernetes Nginx Ingress Controller
Traefik
Istio Ingress Controller
First 2 are very similar, but use absolutely different annotations. It often happens people confuse them
Talking about minikube:
As per guidelines, in order to install ingress the only you have to do is
minikube addons enable ingress
Please note that by default, minikube installing exactly NGINX Ingress controller
nginx-ingress-controller-5984b97644-rnkrg 1/1 Running 0 1m
You have to create ingress.
Follow the steps in this doc - https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/
Can someone please explain how POD to POD works in AKS?
from the docs, I can see it uses kube proxy component to send the traffic to the desired POD.
But I have been told that I must use clusterIP service and bind all the relevant POD's together.
So what is real flow? Or I missed something. below a couple of questions to be more clear.
Questions:
how POD to POD inside one node can talk to each other? what is the flow?
how POD to POD inside a cluster (different nodes) can talk to each other? what is the flow?
if it's possible it will be highly appreciated if you can describe the flows for #1 and #2 in the deployment of kubenet and CNI.
Thanks a lot!
for pod to pod communication we use services. so first we need to understand,
why we need service: what actually do service for us that, they resolve the dns name and give us the the exact ip that we need to connect a specific pod. now as you want to communicate with pod to pod you need to create a ClusterIP service.
ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType. with ClusterIP service you can't access a pod from outside the cluster for this reason we use clusterip service if we want the communication between pod to pod only.
kube-proxy is the network proxy that runs on each node in your cluster.
it maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.
every service maintain iptables.And kube-proxy handled these ip tables for every service. so yes, kube-proxy is the most vital point for network setup in our k8s cluster.
how the network policy works in kubernetes:
all Pods can communicate with all other Pods without using network address translation (NAT).
all Nodes can communicate with all Pods without NAT.
the IP that a Pod sees itself as is the same IP that others see it as.
with those point:
Container-to-Container networking
Pod-to-Pod networking
Pod-to-Service networking
Internet-to-Service networking
It handles transmission of packets between pod to pods, and also with the outside world. It acts like a network proxy and load balancer for pods running on the node by implementing load-balancing using NAT in iptables.
The kube-proxy process stands in between the Kubernetes network and the pods that are running on that particular node. It is responsible for ensuring that communication is maintained efficiently across all elements of the cluster. When a user creates a Kubernetes service object, the kube-proxy instance is responsible to translate that object into meaningful rules in the local iptables rule set on the worker node. iptables is used to translate the virtual IP assigned to the service object to all of the pod IPs mapped by the service.
i hope it's clear your idea about kube proxy.
lets see a example how it's works.
here i used headless service so that i can connect a specific pod.
---
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: default
spec:
clusterIP: None
selector:
app: my-test
ports:
- port: 80
name: rest
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-sts
spec:
serviceName: my-service
replicas: 3
selector:
matchLabels:
app: my-test
template:
metadata:
labels:
app: my-test
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
---
this will create 3 pods. as : my-sts-0, my-sts-1, my-sts-2. now if we want to connect to the pod my-sts-0 just use this dns name my-sts-0.my-service.default.svc:80 . and the service will resolve the dns name and will provide the exact podip of my-sts-0. now if you need to comminucate from my-sts-1 to my-sts-0, you can just use this dns name.
The template is like my_pod_name.my_Service_Name.my_Namespace.svc.cluster-domain.example , but you can skip the cluster-domain.example part. Only Service_Name.Namespace.svc will work fine.
ref
To start playing with Kubernetes I created a small project where I create a three nodes cluster on my local machine using Vagrant and Vbox:
https://github.com/sasadangelo/k8s-cluster
Just typing vagrant up my cluster is up and running and I can start to play with it. I got familiar with the main concepts and commands and created a simple "Hello World" application composed by a docker image with an NGINX web server where you can connect via browser and a "Hello World" message appears with the hostname. The hostname helps me to understand which Pod responded to my request.
Here the project:
https://github.com/sasadangelo/k8s-tutorials/tree/master/hello-k8s
Currently, I can create a deployment with kubectl create exposing it as a service with kubectl expose and then increase replica with kubect scale. At the moment, I am able to use use NodePort service, so with kubectl describe service I can see on which IP the 5 Pods listen, I connect to them and everything works fine.
My problem is that now I want to load balance the traffic. I want to connect to a single IP and as soon as I click on the browser Reload button I want to see that different Pod names appear.
I know that Kubernetes by default offers only NodePort service and if I want a load balancer I need something like an Ingress. I also know that a simple reverse proxy like Ingress Nginx is a good implementation for my needs.
However, I read lot of tutorials but I have hard time understanding how I should configure it to achieve what I need.
Here code I have so far:
https://github.com/sasadangelo/k8s-cluster/tree/master/ingress
Can anyone help me how to fix Ingress code to achieve what I need?
Kube Proxy which is a kubernetes component provides load balancing at L4 layer via iptables load balancing.So with ClusterIP type service itself you get load balancing between pods. But with clusterIP type service you can not access it from outside the cluster i.e from a browser. Nodeport type service provides access to the pods from outside the cluster with load balancing at L4 layer by kube proxy.
LoadBalancer and Ingress provides load balancing and rule based routing at L7 layer if that's what you are looking for.
Looking at your code you have yamls for nginx ingress controller and exposed it as Nodeport. What you need next is to create an ingress resource
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /
backend:
serviceName: web
servicePort: 8080
Once you create above ingress and you have a ClusterIP type service for your pod with a name web which exposed port 8080, You should be able to access it.
Here is the doc on using nginx ingress for an hello world application on Minikube which you can follow to make ingress work in your setup as well.
Here is what I have done in detail to solve the issue. First of all, I installed Nginx Ingress resources:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml
then I create an Nginx Ingress resource in a ingress-nginx.yaml file:
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-k8s-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: hello-k8s.info
http:
paths:
- path: /
backend:
serviceName: hello-k8s
servicePort: 80
and installed it with the command:
kubectl apply -f ingress-nginx.yaml
You can see code here.
Inside the cluster (doing vagrant ssh on whatever node) I tested:
curl -H "Host: hello-k8s.info" IP
where IP is got from the command:
kubectl get ingress
Suppose it is 10.97.139.101. Outside the cluster, on my Mac (Running on 3 VirtualBox on my Mac) to access the application I needed to add a route:
sudo route add -net 10.97.0.0/16 192.168.205.10
where 192.168.205.10 is the IP of the master node. In /etc/hosts I added the line:
10.97.139.101 hello-k8s.info
Now typing hello-k8s.info in the browser I see the "Hello World" web page appear.
Thanks to Arghya Sadhu for help.
Is it possible to expose the service by ingress with the type of ClusterIP?
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-service
ports:
- name: my-service-port
port: 4001
targetPort: 4001
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: my.example.com
http:
paths:
- path: /my-service
backend:
serviceName: my-service
servicePort: 4001
I know the service can be exposed with the type of NodePort, but it may cost one more NAT connection, if someone could show me what's the fastest way to detect internal service from the world of internet in the cloud.
No, clusterIP is only reachable from within the cluster. An Ingress is essentially just a set of layer 7 forwarding rules, it does not handle the layer 4 requirements of exposing the internals of your cluster to the outside world. At least 1 NAT step is required.
For Ingress to work, though, you need to have at least one service involved that exposes your workload externally, so nodePort or loadBalancer. Your ingress controller and the infrastructure of your cluster will determine which of the two services you will need to use.
In the case of Nginx ingress, you need to have a single LoadBalancer service which the ingress will use to bridge traffic from outside the cluster to inside it. After that, you can use clusterIP services for each of your workloads.
In your above example, as long as the nginx ingress controller is correctly configured (with a loadbalancer), then the config you are using should work fine.
In short : YES
Now to the elaborate answer...
First thing first, let's have a look at what the official documentation says :
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
[...]
An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer...
What's confusing here is the term Load balancer. In the definition above, we are talking about the classic and well known in the web load balancer.
This one has nothing to do with kubernetes !
So back to the definition, to use an Ingress and make it work, we need a kubernetes resource called IngressController. And this resource happen to be a load balancer ! That's it.
However, you have to keep in mind that there is a difference between a load balancer in the outside world and a kubernetes service of type type:LoadBalancer.
So in summary (and in order to redirect the traffic from the outside world to your k8s clusterIp service) :
Do you need a Load balancer to make your kind:Ingress works ? Yes, this is the kind:IngressController kubernetes resource.
Do you need a kubernetes service type:LoadBalancer or type:NodePort to make your kind:Ingress works ? Definitely no ! A service type:ClusterIP works just fine !
I tested ingress in minikube successfully, no issue at all.
Then I deployed my app into ubuntu, if I am using service NodePort, it is also worked very well. After that, I was thinking to use Ingress as load balancer to router traffic, so that external url is no longer the ugly long port.
But unfortunately, I did not succeed, always failed.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dv
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /test
backend:
serviceName: ngsc
servicePort: 3000
kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
dv * 80 12s
root#kmaster:/home/ubuntu/datavisor# kubectl describe ing dv
Name: dv
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/ ngsc:3000 (192.168.1.14:3000,192.168.1.17:3000,192.168.1.18:3000)
Annotations:
ingress.kubernetes.io/rewrite-target: /
Events: <none>
Then when I tried to access, I got following error:
curl http://cluster-ip
curl: (7) Failed to connect to <cluster-ip> port 80: Connection refused
What I really hope is to let the url exposed outside is http://ipaddress, instead of http://ipaddress:30080
I know that I can easily use nginx out of kubernete to meet this requirement, but that is not ideal, I want kubernete to handle it so that even service port changed, everything is still up.
Can you check above output and tell me what is the error? I checked a lot doc, every place seemed focus only in minikube, nothing related to real cluster deployment. Do I need to install anything to make ingress working? when I use kubectl get all --all-namespaces I did not see the ingress-controller installed at all. How can I install it if needed?
Thanks for your advice
Well, actually Kubernetes does not provide any Ingress controller out of box. You have to install Nginx Ingress or Traefik Ingress or anything else. Ingress controller must run somewhere in your cluster, it's a must. Actually ingress controller is the actual proxy that proxies traffic to your applications.
And I think you should know that minikube under the hood also uses nginx-ingress-controller (see https://github.com/kubernetes/minikube/tree/master/deploy/addons/ingress).
In a cloud environments ingress controllers run behind the cloud load balancer that performs load balancing between cluster nodes.
If you run on-prem cluster - then usually your ingress-controller is run as NodePort service and you may create DNS record pointing to your node IP addresses. It is also possible to run ingress controller on dedicated nodes and use hostNetwork: true. That will allow to use standard 80/443 ports. So there are many options here.