I'm working on an application deployment environment using Kubernetes where I want to be able to spin up copies of my entire application stack based on a Git reference for the primarily web application, e.g. "master" and "my-topic-branch". I want these copies of the app stack to coexist in the same cluster. I can create Kubernetes services, replication controllers, and pods that use a "gitRef" label to isolate the stacks from each other, but some of the pods in the stack depend on each other (via Kubernetes services), and I don't see an easy, clean way to restrict the services that are exposed to a pod.
There a couple ways to achieve it that I can think of, but neither are ideal:
Put each stack in a separate Kubernetes namespace. This provides the cleanest isolation, in that there are no resource name conflicts and the applications can have the DNS hostnames for the services they depend on hardcoded, but seems to violate what the documentation says about namespaces†:
It is not necessary to use multiple namespaces just to separate slightly different resources, such as different versions of the same software: use labels to distinguish resources within the same namespace.
This makes sense, as putting the app stacks in different resources would negate all the usefulness of label selectors. I'd just name the namespace with the Git reference and all the other resources wouldn't need to be filtered at all.
Create a copy of each service for each copy of the application stack, e.g. "mysql-master" and "mysql-my-topic-branch". This has the advantage that all the app stacks can coexist in the same namespace, but the disadvantage of not being able to hardcode the DNS hostname for the service in the applications that need them, e.g. having a web app target the hostname "mysql" regardless of which instance of the MySQL Kubernetes service it actually resolves to. I would need to use some mechanism of injecting the correct hostname into the pods or having them figure it out for themselves somehow.
Essentially what I want is a way of telling Kubernetes, "Expose this service's hostname only to pods with the given labels, and expose it to them with the given hostname" for a specific service. This would allow me to use the second approach without having to have application-level logic for determining the correct hostname to use.
What's the best way to achieve what I'm after?
[†] http://kubernetes.io/v1.1/docs/user-guide/namespaces.html
The documentation on putting different versions in different namespaces is a bit incorrect I think. It is actually the point of namespaces to separate things completely like this. You should put a complete version of each "track" or deployment stage of your app into its own namespace.
You can then use hardcoded service names - "http://myservice/" - as the DNS will resolve on default to the local namespace.
For ingresses I have copied my answer here from the GitHub issue on cross-namespace ingresses.
You should use the approach that our group is using for Ingresses.
Think of an Ingress not as much as a LoadBalancer but just a document specifying some mappings between URLs and services within the same namespace.
An example, from a real document we use:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
namespace: dev-1
spec:
rules:
- host: api-gateway-dev-1.faceit.com
http:
paths:
- backend:
serviceName: api-gateway
servicePort: 80
path: /
- host: api-shop-dev-1.faceit.com
http:
paths:
- backend:
serviceName: api-shop
servicePort: 80
path: /
- host: api-search-dev-1.faceit.com
http:
paths:
- backend:
serviceName: api-search
servicePort: 8080
path: /
tls:
- hosts:
- api-gateway-dev-1.faceit.com
- api-search-dev-1.faceit.com
- api-shop-dev-1.faceit.com
secretName: faceitssl
We make one of these for each of our namespaces for each track.
Then, we have a single namespace with an Ingress Controller which runs automatically configured NGINX pods. Another AWS Load balancer points to these pods which run on a NodePort using a DaemonSet to run at most and at least one on every node in our cluster.
As such, the traffic is then routed:
Internet -> AWS ELB -> NGINX (on node) -> Pod
We keep the isolation between namespaces while using Ingresses as they were intended. It's not correct or even sensible to use one ingress to hit multiple namespaces. It just doesn't make sense, given how they are designed. The solution is to use one ingress per each namespace with a cluster-scope ingress controller which actually does the routing.
All an Ingress is to Kubernetes is an object with some data on it. It's up to the Ingress Controller to do the routing.
See the document here for more info on Ingress Controllers.
Related
I have an application running on Kubernetes, that uses nginx as the ingress controller that created a load balancer in AWS. I noticed that by default the application is open to the World, with 0.0.0.0/32 is added to the inbound rules of the AWS Security group that is attached to the load balancer.
I want to allow only certain IPs to access the application. That makes me use nginx.ingress.kubernetes.io/whitelist-source-range annotation in the ingress controller.
But I wouldn't know the IPs of the entities that must be allowed to access the application beforehand. An upstream process (Jenkins job) that creates certain containers, which try to talk to the application that's running on Kube.
How can I dynamically modify the ingress controller annotation to add and remove IPs without causing any downtime? And No, I do not have a common IP range that I can add. I have several different VPCs which have their own CIDR blocks.
Short answer: you don't put the whitelist annotation on the controller, you put it on the ingress resource. And updating that does not require any downtime.
--
Long answer: Yes, by default, the controller loadbalancer is open to the world, and that is expected. All traffic comes into the ingress controller's loadbalancer and the controller then determines how to route it within the cluster.
It determines this routing via the use of ingress resources. Here is an example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 12.0.0.0/8,10.0.0.0/8
name: ingress-whitelist-example
spec:
rules:
- host: somehost.com
http:
paths:
- backend:
service:
name: service-for-somehost
port:
number: 80
tls:
- hosts:
- somehost.com
secretName: tls-secret
The ingress controller will get traffic for (in this example) the domain host somehost.com and it will route the request to the service service-for-somehost in the cluster, on port 80
If a request comes into the controller outside of the ranges 12.0.0.0/8 or 10.0.0.0/8 (as described by the annotation), then the controller will reject the request with a Forbidden error.
The ingress resource is not a resource that needs to be respun or taken down for updates, like Deployments.
The AWS load balancer is external to the cluster. You can, of course, choose to block/whitelist traffic within the AWS cloud before it reaches the cluster and that is fine, but that is not managed within the nginx controller.
Some further reading in Ingresses is available here
I have a working Nexus 3 pod, reachable on port 30080 (with NodePort): http://nexus.mydomain:30080/ works perfectly from all hosts (from the cluster or outside).
Now I'm trying to make it accessible at the port 80 (for obvious reasons).
Following the docs, I've implemented it like that (trivial):
[...]
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nexus-ingress
namespace: nexus-ns
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nexus.mydomain
http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: nexus-service
servicePort: 80
Applying it works without errors. But when I try to reach http://nexus.mydomain, I get:
Service Unavailable
No logs are shown (the webapp is not hit).
What did I miss ?
K3s Lightweight Kubernetes
K3s is designed to be a single binary of less than 40MB that completely implements the Kubernetes API. In order to achieve this, they removed a lot of extra drivers that didn't need to be part of the core and are easily replaced with add-ons.
As I mentioned in comments, K3s as default is using Traefik Ingress Controller.
Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.
This information can be found in K3s Rancher Documentation.
Traefik is deployed by default when starting the server... To prevent k3s from using or overwriting the modified version, deploy k3s with --no-deploy traefik and store the modified copy in the k3s/server/manifests directory. For more information, refer to the official Traefik for Helm Configuration Parameters.
To disable it, start each server with the --disable traefik option.
If you want to deploy Nginx Ingress controller, you can check guide How to use NGINX ingress controller in K3s.
As you are using specific Nginx Ingress like nginx.ingress.kubernetes.io/rewrite-target: /$1, you have to use Nginx Ingress.
If you would use more than 2 Ingress controllers you will need to force using nginx ingress by annotation.
annotations:
kubernetes.io/ingress.class: "nginx"
If mention information won't help, please provide more details like your Deployment, Service.
I do not think you can expose it on port 80 or 443 over a NodePort service or at least it is not recommended.
In this configuration, the NGINX container remains isolated from the
host network. As a result, it can safely bind to any port, including
the standard HTTP ports 80 and 443. However, due to the container
namespace isolation, a client located outside the cluster network
(e.g. on the public internet) is not able to access Ingress hosts
directly on ports 80 and 443. Instead, the external client must append
the NodePort allocated to the ingress-nginx Service to HTTP requests.
-- Bare-metal considerations - NGINX Ingress Controller
* Emphasis added by me.
While it may sound tempting to reconfigure the NodePort range using
the --service-node-port-range API server flag to include unprivileged
ports and be able to expose ports 80 and 443, doing so may result in
unexpected issues including (but not limited to) the use of ports
otherwise reserved to system daemons and the necessity to grant
kube-proxy privileges it may otherwise not require.
This practice is therefore discouraged. See the other approaches
proposed in this page for alternatives.
-- Bare-metal considerations - NGINX Ingress Controller
I did a similar setup a couple of months ago. I installed a MetalLB load balancer and then exposed the service. Depending on your provider (e.g., GKE), a load balancer can even be automatically spun up. So possibly you don't even have to deal with MetalLB, although MetalLB is not hard to setup and works great.
I'm working on a SaaS app that will be running in Kubernetes. We're using a Helm chart that deploys all the components into the cluster (for simplicity's sake let's assume it's a frontend service, a backend and a database). App architecture is multi-tenant (we have a single instance of each service that are being shared by all tenants) and we would like to keep it that way. What I'm currently struggling with and would like to ask for advice/best practice on is how does one go about automating the provisioning of custom sub-domains for the tenants?
Imagine the app is hosted at exampleapp.com.
A brand new customer comes and registers a new organisation some-company. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain some-company.exampleapp.com. I would like this provisioning to be done automatically and not require any manual intervention.
What options do I have for implementing automated sub-domain provisioning in Kubernetes?
How does our (exampleapp.com) domain registrar/nameserver provider fit into the solution?
Does it have to provide an API for dynamic DNS record creation/modification?
I appreciate that the questions I'm asking are quite broad so I'm not expecting anything more than a high-level conceptual answer or pointers to some services/libraries/tools that might help me achieve this.
Note: Since this is more of a theoretical question, I'll give you some points from a Kubernetes Engineer, I divided your question in blocks to ease the understanding.
About your multi-tenancy architecture:
Keeping "it that way" is achievable. It simplifies the Kubernetes structure, on the other hand it relies more on your app.
Question 1:
Imagine the app is hosted at exampleapp.com. A brand new customer comes and registers a new organisation some-company. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain some-company.exampleapp.com. I would like this provisioning to be done automatically and not require any manual intervention.
Suggestion:
For that, you will have to give your app admin privileges and the tools required for it to add Ingress Rules entries to your Ingress when a new client is added. A script using kubectl patch is the simpler solution from my viewpoint.
For this approach I suggest installing the Nginx Ingress Controller for it's versatility.
Here is an Example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: <ingress-name>
spec:
rules:
- host: client1.exampleapp.com
http:
paths:
- path: /client1
backend:
serviceName: <main-service>
servicePort: <main-service-port>
- host: client2.exampleapp.com
http:
paths:
- path: /client2
backend:
serviceName: <main-service>
servicePort: <main-service-port>
And here is the one-liner command using kubectl patch on how to add new rules:
kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'
POC:
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
demo-ingress client1.exampleapp.com,client2.exampleapp.com 192.168.39.39 80 15m
$ kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'
ingress.extensions/demo-ingress patched
$ kubectl describe ingress demo-ingress
Rules:
Host Path Backends
---- ---- --------
client1.exampleapp.com
/client1 main-service:80 (<none>)
client2.exampleapp.com
/client2 main-service:80 (<none>)
client3.exampleapp.com
/client3 main-service:80 (<none>)
This rule redirects the traffic incoming from the subdomains to subpaths inside your main app.
Also, to add TLS handling, refer to: Nginx Ingress TLS Guide
Question2 :
How does our (exampleapp.com) domain registrar/nameserver provider fit into the solution? Does it have to provide an API for dynamic DNS record creation/modification?
Suggestion:
I believe you already has something similar, but you need a wildcard record in your DNS to point *.example.app to the IP of the ingress, I don't believe you need anything more than that, because it redirects all to the ingress and the ingress forwards it internally.
Question 3:
If there are some strong arguments why multi-tenancy + Kubernetes don't go along very well, those are welcome as well.
Opinion:
I don't see any major reason why it would be a problem. You just have, once again, to adequate your app to handle scaling because I believe in the long run you would want your app to be able to scale to multi-pod structure to provide elastic availability.
These are my 2 cents to your question, I hope it helps you!
I have two different deployments creating two different pods that spins up two different containers serves different purposes. but as a coincidence the port being exposed by both of those containers is 8080.
I created a single service with two ports 8080 and 8081(type=LoadBalancer) to expose both of those deployments. when I hit the LoadBalancer url I get back response from container 1 and after hitting refresh few times I get back the response from container 2. This behavior is same on both ports.
I know that changing the port exposed on the dockerfile of one of those containers would solve this problem. but just out of curiosity as a newbie to the kubernetes, is there any different approach to handle this scenario?
You could use Ingress. Here is an example.
Instead of creating one Service for both pods. Create one Service per pod. Make sure the selector labels are different for both. Set type to NodePort. Then create an Ingress with rules like.
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
Now there are many ingress solutions out there. Ingress in k8s is just a networking spec. All it is, is a data model that represents your networking logic. The various ingress controllers take that spec and implement the logic with their given solution. Here is a link to the docs for nginx ingress controller. https://www.nginx.com/products/nginx/kubernetes-ingress-controller/
I'm building a service where users can build web apps - these apps will be hosted under a virtual DNS name *.laska.io
For example, if Tom and Jerry both built an app, they'd have it hosted under:
tom.laska.io
jerry.laska.io
Now, suppose I have 1000 users. Should I create one big ingress that looks like this?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: tom.laska.io
http:
paths:
- backend:
serviceName: nginx-service
servicePort: 80
- host: jerry.laska.io
http:
paths:
- backend:
serviceName: nginx-service
servicePort: 80
...and so forth
I'm worried about downtime - if I have an app with websockets for example. Also the file will become huge with 1000 users. Will reloading the ingress go fast enough? Also, how should I reload it?
A second option in my mind is to simply create one ingress for every web app. My worry about that is, can ingress-nginx handle many ingresses? Or is this an anti-pattern?
Which one is better?
You can create one ingress resource for each web app. If you search the official public charts repo, you'll see that many of the charts define an ingress resource within them. It's normal for each app to define its own ingress resource.
It's worth being clear that an ingress resource is just a definition of a routing rule. (It doesn't add an extra ingress controller or any other extra runtime component.) So it makes a lot of sense for an app to define its own routing rule.
The example you've given has all the ingress routing in one resource definition. This approach is easy to grasp and makes a lot of sense when you've got several related applications as then you might want to see their routing configuration together. You can see this also in the fanout ingress example in the kubernetes docs.
I can't see any performance concerns with defining the rules separately in distinct resources. The ingress controller will listen for new rules and update its configuration only when a new rule is created so there shouldn't be a problem with reading the resources. And I'd expect the combined vs separated options to result in the same routing rules being set in the background in nginx.
The most common pattern in the official charts repo is that the chart for each app defines its ingress resource and also exposes it through the values.yaml so that users can choose to enable or customise it as they wish. You can then compose multiple charts together and configure the rules for each in the relevant section of the values.yaml. (Here's an example I've worked on that does this with wildcard dns.) Or you can deploy each app separately under its own helm release.
An ingress ressource is just a rule. It's better to split your ingress ressources in different files and just reapply the ressources that need change.
I never noticed a downtime when applying a ressource.