I have a docker image that serves a simple static web page.
I have a working Kubernetes cluster of 4 nodes (physical servers not in the cloud anywhere).
I want to run that docker image on 2 of the 4 Kubernetes nodes and have it be accessible to the world outside the cluster and load balanced and have it move it to another node if one dies.
Do I need to make a pod then a replication controller then a kube proxy something?
Or do I need to just make a replication controller and expose it somehow?
Do I need to make service?
I don't need help with how to make any of those things, that seems well documented, but what I can't tell what I need to make.
What you need is to expose your service (that consists of pods which are run/scaled/restarted by your replication controller). Using deployment instead of replication controller has additional benefits (mainly for updating the app).
If you are on bare metal then you probably wish to expose your service via type: NodePort - so every node in your cluster will open a static port that routes traffic to pods.
You can then either point your load balancer to that nodes on that port, or make a DNS entry with all Kubernetes nodes.
Docs: http://kubernetes.io/docs/user-guide/quick-start/
You'll need:
1) A load balancer on one of your nodes in your cluster, that is a reverse proxy Pod like nginx to proxy the traffic to an upstream.
This Pod will need to be exposed to the outside using hostPort like
ports:
- containerPort: 80
hostPort: 80
name: http
- containerPort: 443
hostPort: 443
name: https
2) A Service that will use the web server selector as target.
3) Set the Service name (which will resolve to the Service IP) as the upstream in nginx config
4) Deploy your web server Pods, which will have the selector to be targeted by the Service.
You might also want to look at External IP for the Service
http://kubernetes.io/docs/user-guide/services/#external-ips
but I personally never managed to get that working on my bare metal cluster.
Related
I am a bit new to Kubernetes and I am working with EKS.
I have two main apps for which there is a number of pods and I have set up a ELB for external access.
I also have a small app with say 1-2 pods. I don't want to set up a ELB just for this small app. I checked the node port, but in that case, I can't use the default HTTPS port 443.
So I feel the best thing to do in this case would be to bring the small app outside the cluster, then maybe set it up in a EC2 instance. Or is there some other way to expose the small app while keeping it inside the cluster itself?
You can try to use the Host network (Node) like hostport (Not recommended in k8s to use in prod)
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
hostPort: 443
The hostPort feature allows to expose a single container port on the
host IP. Using the hostPort to expose an application to the outside of
the Kubernetes cluster has the same drawbacks as the hostNetwork
approach discussed in the previous section. The host IP can change
when the container is restarted, two containers using the same
hostPort cannot be scheduled on the same node and the usage of the
hostPort is considered a privileged operation on OpenShift.
Extra
I don't want to set up a elb just for this small app.
Ideally, you have to use the deployments with the ingress and ingress controller. So there will be single ELB for the whole EKS cluster and all services will be using that single point.
All PODs or deployment will be running into a single cluster if you want. Single point ingress will work as handling the traffic into EKS cluster.
https://kubernetes.io/docs/concepts/services-networking/ingress/
You can read this article how to setup the ingress in EKS aws so you will get an idea.
You can use a different domains for exposing services.
Example :
https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/
Basically I have Following Hdfs Cluster setup using docker-compose:
Node 1 with IP: 192.168.1.1 having service deployed as below:
Namenode1:9000
HMaster1: 8300
ZooKeeper1:1291
Node 2 with IP: 192.168.1.2 having service deployed as below:
Namenode2:9000
ZooKeeper2:1291
How does Traefik / Ngnix - (Ingress Controllers) forwards request to two different services having configured with same port number?
There are several great tutorials on how ingress and load balancing works in kubernetes, e.g. this one by Mark Betz. As a general rule, it helps to think in terms of services and workloads instead of specific nodes where your workloads are running on.
A workload deployed in Kubernetes (a so called Pod) has its own internal IP address, called a ClusterIP. That pod can have one or more ports open, just on that pod-owned ip address.
If you now have several pods to distribute the load, e.g. like 5 web server processes or backend logic, it would be hard for a client (inside the cluster) to keep track of all those pod IPs, because they also change when a pod is updated or just restarted due to a crash. This is why Kubernetes has a so called concept of services. Those provide a stable DNS name and IP which then transparently "forwards" to one of the healthy pods. So your client only needs to know the DNS name and not keep track of the specific pod IPs.
If you now want to expose such a service to the public, there are different ways. Either you set your service to type: LoadBalancer which then sets up some load balancer infrastructure on your cloud provider and routes traffic to the nodes and then to the pods - or - you already have an ingress controller in place and just define the routing based on host names and paths. An ingress controller itself is such a loadbalanced service with an attached cloud load balancer and also has some pods (with e.g. a traefik or nginx container) which then route your packets accordingly.
So coming back to your initial question: If you want to expose a service with several pods of the same kind, then you would first create a Service resource that matches your Pods using the selector and then you create one single ingress resource that provides a hostname/path and references this service. The ingress controller will pick up those ingress resources and configure the traefik or nginx accordingly. The ingress controller doesn't really care about the host IPs and port numbers, because it acts on the internal kubernetes ClusterIPs, so you even don't need (and shouldn't) expose such a service directly when you have an ingress in place.
I hope this answers your question regarding exposing two workloads over an ingress controller. For details, check the Kubernetes docs on Ingresses. Based on the services you named (zookeeper, hdfs) load balancing and ingresses might not be what you need for that case. Zookeeper instances should be internal in most cases and need to be adressed individually, so you might want to check out headless services, for this use case. Also check the Kubernetes docs for a way to run zookeeper.
I'm trying to use Kubernetes to make configurations and deployments explicitly defined and I also like Kubernetes' pod scheduling mechanisms. There are (for now) just 2 apps running on 2 replicas on 3 nodes. But Google's Kubernetes Engine's load balancer is extremely expensive for a small app like ours (at least for the moment) at the same time I'm not willing to change to a single instance hosting solution on a container or deploying the app on Docker swarm etc.
Using node's IP seemed like a hack and I thought that it might expose some security issues inside the cluster. Therefore I configured a Træfik ingress and an ingress controller to overcome Google's expensive flat rate for load balancing but turns out an outward facing ingress spins up a standart load balancer or I'm missing something.
I hope I'm missing something since at this rates ($16 a month) I cannot rationalize using kubernetes from start up for this app.
Is there a way to use GKE without using Google's load balancer?
An Ingress is just a set of rules that tell the cluster how to route to your services, and a Service is another set of rules to reach and load-balance across a set of pods, based on the selector. A service can use 3 different routing types:
ClusterIP - this gives the service an IP that's only available inside the cluster which routes to the pods.
NodePort - this creates a ClusterIP, and then creates an externally reachable port on every single node in the cluster. Traffic to those ports routes to the internal service IP and then to the pods.
LoadBalancer - this creates a ClusterIP, then a NodePort, and then provisions a load balancer from a provider (if available like on GKE). Traffic hits the load balancer, then a port on one of the nodes, then the internal IP, then finally a pod.
These different types of services are not mutually exclusive but actually build on each other, and it explains why anything public must be using a NodePort. Think about it - how else would traffic reach your cluster? A cloud load balancer just directs requests to your nodes and points to one of the NodePort ports. If you don't want a GKE load balancer then you can already skip it and access those ports directly.
The downside is that the ports are limited between 30000-32767. If you need standard HTTP port 80/443 then you can't accomplish this with a Service and instead must specify the port directly in your Deployment. Use the hostPort setting to bind the containers directly to port 80 on the node:
containers:
- name: yourapp
image: yourimage
ports:
- name: http
containerPort: 80
hostPort: 80 ### this will bind to port 80 on the actual node
This might work for you and routes traffic directly to the container without any load-balancing, but if a node has problems or the app stops running on a node then it will be unavailable.
If you still want load-balancing then you can run a DaemonSet (so that it's available on every node) with Nginx (or any other proxy) exposed via hostPort and then that will route to your internal services. An easy way to run this is with the standard nginx-ingress package, but skip creating the LoadBalancer service for it and use the hostPort setting. The Helm chart can be configured for this:
https://github.com/helm/charts/tree/master/stable/nginx-ingress
One option is to completely disable this feature on your GKE cluster. When creating the cluster (on console.cloud.google.com) under Add-ons disable HTTP load balancing. If you are using gcloud you can use gcloud beta container clusters create ... --disable-addons=HttpLoadBalancing.
Alternatively, you can also inhibit the GCP Load Balancer by adding an annotation to your Ingress resources, kubernetes.io/ingress.class=somerandomstring.
For newly created ingresses, you can put this in the yaml document:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: somerandomstring
...
If you want to do that for all of your Ingresses you can use this example snippet (be careful!):
kubectl get ingress --all-namespaces \
-o jsonpath='{range .items[*]}{"kubectl annotate ingress -n "}{.metadata.namespace}{" "}{.metadata.name}{" kubernetes.io/ingress.class=somerandomstring\n"}{end}' \
| sh -x
Now using Ingresses is pretty useful with Kubernetes, so I suggest you check out the nginx ingress controller and after deployment, annotate your Ingresses accordingly.
If you specify the Ingress class as an annotation on the Ingress object
kubernetes.io/ingress.class: traefik
Traefik will pick it up while the Google Load Balancer will ignore it. There is also a bit of Traefik documentation on this part.
You could deploy the nginx ingress controller using NodePort mode (e.g. if using the helm chart set controller.service.type to NodePort) and then load-balance amongst your instances using DNS. Just make sure you have static IPs for the nodes or you could even create a DaemonSet that somehow updates your DNS with each node's IP.
Traefik seems to support a similar configuration (e.g. through serviceType in its helm chart).
I have deployment that hosts a website on port 9001 and a service attached to it. I want to allow anyone (from outside cluster) to be able to connect to that site.
Any help would be appreciated.
I want to allow anyone (from outside cluster) to be able to connect to that site
There are many ways to do this using kubernetes services to expose port 9001 of the website to the outside world:
Service type LoadBalancer if you have an external, cloud-provider's load-balancer.
ExternalIPs. The website can be hit at ExternalIP:Port.
Service type NodePort if the cluster's nodes are reachable from the users. The website can be hit at NodeIP:NodePort.
Ingress controller and ingress resource.
As you wrote that this is not a cloud deployment, you need to consider how to correctly expose this to the world in a decent fashion. First and formost, create a NodePort type service for your deployment. With this, your nodes will expose that service on a high port.
Depending on your network, at this point you either need to configure a loadbalancer in your network to forward traffic for some IP:80 to your node(s) high NodePort, or for example deploy HAProxy in a DeamonSet with hostNetwork: true that will proxy 80 to your NodePort.
A bit more complexity can be added by deployment of Nginx IngressController (exposed as above) and use of Ingress to make the Ingress Controller expose all your services without the need to fiddle with NodePort/LB/HAProxy for each of them individualy any more.
We presently have a setup where applications within our mesos/marathon cluster want to reach out to services which may or may not reside in our mesos/marathon cluster. Ingress for external traffic into the cluster is accomplished via an Amazon ELB sitting in front of a cluster of Traefik instances, which then chooses the appropriate set of container instances to load-balance to via the incoming HTTP Host header compared against essentially a many-to-one association of configured host headers against a particular container instance. Internal-to-internal traffic is actually handled by this same route as well, as the DNS record that is associated with a given service is mapped to that same ELB both internal to and external to our mesos/marathon cluster. We also give the ability to have multiple DNS records pointing against the same container set.
This setup works, but causes seemingly unnecessary network traffic and load against our ELBs as well as our Traefik cluster, as if the applications in the containers or another component were able to self-determine that the services they wished to call out to were within the specific mesos/marathon cluster they were in, and make an appropriate call to either something internal to the cluster fronting the set of containers, or directly to the specific container itself.
From what I understand of Kubernetes, Kubernetes provides the concept of services, which essentially can act as the front for a set of pods based on configuration for which pods the service should match over. However, I'm not entirely sure of the mechanism by which we can have applications in a Kubernetes cluster know transparently to direct network traffic to the service IPs. I think that some of this can be helped by having Envoy proxy traffic meant for, e.g., <application-name>.<cluster-name>.company.com to the service name, but if we have a CNAME that maps to that previous DNS entry (say, <application-name>.company.com), I'm not entirely sure how we can avoid exiting the cluster.
Is there a good way to solve for both cases? We are trying to avoid having our applications' logic have to understand that it's sitting in a particular cluster and would prefer a component outside of the applications to perform the routing appropriately.
If I am fundamentally misunderstanding a particular component, I would gladly appreciate correction!
When you are using service-to-service communication inside a cluster, you are using Service abstraction which is something like a static point which will road traffic to the right pods.
Service endpoint available only from inside a cluster by it's IP or internal DNS name, provided by internal Kubernetes DNS server. So, for communicating inside a cluster, you can use DNS names like <servicename>.<namespace>.svc.cluster.local.
But, what is more important, Service has a static IP address.
So, now you can add that static IP as a hosts record to the pods inside a cluster for making sure that they will communicate each other inside a cluster.
For that, you can use HostAlias feature. Here is an example of configuration:
apiVersion: v1
kind: Pod
metadata:
name: hostaliases-pod
spec:
restartPolicy: Never
hostAliases:
- ip: "10.0.1.23"
hostnames:
- "my.first.internal.service.example.com"
- ip: "10.1.2.3"
hostnames:
- "my.second.internal.service.example.com"
containers:
- name: cat-hosts
image: busybox
command:
- cat
args:
- "/etc/hosts"
So, if you will use your internal Service IP in combination with service's public FQDN, all traffic from your pod will be 100% inside a cluster, because the application will use internal IP address.
Also, you can use upstream DNS server which will contain same aliases, but an idea will be the same.
With Upstream DNS for the separate zone, resolving will work like that:
With a new version of Kubernetes, which using Core DSN for providing DNS service, and has more features it will be a bit simpler.