what's the difference between openshift route and k8s ingress? - kubernetes

I'm new to openshift and k8s. I'm not sure what's the difference between these two terms, openshift route vs k8s ingress ?

Ultimately they are intended to achieve the same end. Originally Kubernetes had no such concept and so in OpenShift the concept of a Route was developed, along with the bits for providing a load balancing proxy etc. In time it was seen as being useful to have something like this in Kubernetes, so using Route from OpenShift as a starting point for what could be done, Ingress was developed for Kubernetes. In the Ingress version they went for a more generic rules based system so how you specify them looks different, but the intent is to effectively be able to do the same thing.

The following code implementation will create a route in OCP.
The OCP will consider the ingress as a route in the same way.
// build the ingress/route object
func (r *ReconcileMobileSecurityService) buildAppIngress(m *mobilesecurityservicev1alpha1.MobileSecurityService) *v1beta1.Ingress {
ls := getAppLabels(m.Name)
hostName := m.Name + "-" + m.Namespace + "." + m.Spec.ClusterHost + ".nip.io"
ing := &v1beta1.Ingress{
TypeMeta: v1.TypeMeta{
APIVersion: "extensions/v1beta1",
Kind: "Ingress",
},
ObjectMeta: v1.ObjectMeta{
Name: m.Name,
Namespace: m.Namespace,
Labels: ls,
},
Spec: v1beta1.IngressSpec{
Backend: &v1beta1.IngressBackend{
ServiceName: m.Name,
ServicePort: intstr.FromInt(int(m.Spec.Port)),
},
Rules: []v1beta1.IngressRule{
{
Host: hostName,
IngressRuleValue: v1beta1.IngressRuleValue{
HTTP: &v1beta1.HTTPIngressRuleValue{
Paths: []v1beta1.HTTPIngressPath{
{
Backend: v1beta1.IngressBackend{
ServiceName: m.Name,
ServicePort: intstr.FromInt(int(m.Spec.Port)),
},
Path: "/",
},
},
},
},
},
},
},
}
// Set MobileSecurityService instance as the owner and controller
controllerutil.SetControllerReference(m, ing, r.scheme)
return ing
}

Related

deploy kubernetes ingress with terraform

I'm trying deploy kubernetes ingress with terraform.
As described here link and my own variant:
resource "kubernetes_ingress" "node" {
metadata {
name = "node"
}
spec {
ingress_class_name = "nginx"
rule {
host = "backend.io"
http {
path {
path = "/"
backend {
service_name = kubernetes_service.node.metadata.0.name
service_port = 3000
}
}
}
}
}
}
error:
╷
│ Error: Failed to create Ingress 'default/node' because: the server could not find the requested resource (post ingresses.extensions)
│
│ with kubernetes_ingress.node,
│ on node.tf line 86, in resource "kubernetes_ingress" "node":
│ 86: resource "kubernetes_ingress" "node" {
│
╵
it works:
kubectl apply -f file_below.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: node
spec:
ingressClassName: nginx
rules:
- host: backend.io
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: node
port:
number: 3000
Need some ideas about how to deploy kubernetes ingress with terraform.
The issue here is that the example in YML is using the proper API version, i.e., networking.k8s.io/v1, hence it works as you probably have a version of K8s higher than 1.19. It is available since that version, the extensions/v1beta1 that Ingress was a part of was deprecated in favor of networking.k8s.io/v1 in 1.22, as you can read here. As that is the case, your current Terraform code is using the old K8s API version for Ingress. You can see that on the left-hand side of the documentation menu:
If you look further down in the documentation, you will see networking/v1 and in the resource section kubernetes_ingress_v1. Changing the code you have in Terraform to use Ingress from the networking.k8s.io/v1, it becomes:
resource "kubernetes_ingress_v1" "node" {
metadata {
name = "node"
}
spec {
ingress_class_name = "nginx"
rule {
host = "backend.io"
http {
path {
path = "/*"
path_type = "ImplementationSpecific"
backend {
service {
name = kubernetes_service.node.metadata.0.name
port {
number = 3000
}
}
}
}
}
}
}
}

URL of remoteEntry in Kubernetes Cluster

I am trying to build a series of Micro-Frontends using Webpack 5 and the ModuleFederationPlugin.
In the webpack config of my container app I have to configure how the container is going to reach out to the other microfrontends so I can make use of those micro-frontends.
This all works fine when I am serving locally, not using Docker and Kubernetes and my Ingress Controller.
However because I am using Kubernetes and an Ingress Controller, I am unsure what the remote host would be.
Link to Repo
Here is my container webpack.dev.js file
const { merge } = require("webpack-merge");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const ModuleFederationPlugin = require("webpack/lib/container/ModuleFederationPlugin");
const commonConfig = require("./webpack.common");
const packageJson = require("../package.json");
const devConfig = {
mode: "development",
devServer: {
host: "0.0.0.0",
port: 8080,
historyApiFallback: {
index: "index.html",
},
compress: true,
disableHostCheck: true,
},
plugins: [
new ModuleFederationPlugin({
name: "container",
remotes: {
marketing:
"marketing#https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:8081/remoteEntry.js",
},
shared: packageJson.dependencies,
}),
],
};
module.exports = merge(commonConfig, devConfig);
and here is my Ingress Config
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: ticketing.dev
http:
paths:
- path: /api/users/?(.*)
pathType: Prefix
backend:
service:
name: auth-srv
port:
number: 3000
- path: /marketing?(.*)
pathType: Prefix
backend:
service:
name: marketing-srv
port:
number: 8081
- path: /?(.*)
pathType: Prefix
backend:
service:
name: container-srv
port:
number: 8080
and here is my marketing webpack.dev.js file
const { merge } = require("webpack-merge");
const ModuleFederationPlugin = require("webpack/lib/container/ModuleFederationPlugin");
const commonConfig = require("./webpack.common");
const packageJson = require("../package.json");
const devConfig = {
mode: "development",
devServer: {
host: "0.0.0.0",
port: 8081,
historyApiFallback: {
index: "index.html",
},
compress: true,
disableHostCheck: true, // That solved it
},
plugins: [
new ModuleFederationPlugin({
name: "marketing",
filename: "remoteEntry.js",
exposes: {
"./core": "./src/bootstrap",
},
shared: packageJson.dependencies,
}),
],
};
module.exports = merge(commonConfig, devConfig);
I am totally stumped as to what the remote host would be to reach out to my marketing micro-frontend
serving it as usual without running it in a docker container or kubernetes cluster, the remote host would be
https://localhost:8081/remoteEntry.js
but that doesn't work in a kubernetes cluster
I tried using the ingress controller and namespace, but that too, does not work
https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:8081/remoteEntry.js
This is the error I get
https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:8081/remoteEntry.js
If your client and the node are on the same network (eg. can ping each other), do kubectl get service ingress-nginx --namespace ingress-nginx and take note of the nodePort# (TYPE=NodePort, PORT(S) 443:<nodePort#>/TCP). Your remote entry will be https://<any of the worker node IP>:<nodePort#>/remoteEntry.js
If you client is on the Internet and your worker node has public IP, your remote entry will be https://<public IP of the worker node>:<nodePort#>/remoteEntry.js
If you client is on the Internet and your worker node doesn't have public IP, you need to expose your ingress-nginx service with LoadBalancer. Do kubectl get service ingress-nginx --namespace ingress-nginx and take note of the EXTERNAL IP. Your remote entry become https://<EXTERNAL IP>/remoteEntry.js

Kubernetes rewrite-target paths for appending path to match

I'm using the OSS ingress-nginx Ingress controller and trying to create a rewrite-target rule such that I can append a path string before my string match.
If I wanted to create a rewrite rule with regex that matches /matched/path and rewrites that to /prefix/matched/path, how might I be able to do that?
I've tried something like the following but it's no good, and I'm just confused about the syntax of this ingress definition:
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- path: /(/prefix/)(/|$)(/matched/path)(.*)
backend:
serviceName: webapp1
If I wanted to create a rewrite rule with regex that matches
/matched/path and rewrites that to /prefix/matched/path, how might
I be able to do that?
In order to achieve this you have add /prefix into your rewrite-target.
Here's a working example with ingress syntax from k8s v1.18:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress-v118
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /prefix/$1
spec:
rules:
http:
paths:
- path: /(matched/path/?.*)
backend:
serviceName: test
servicePort: 80
Since the syntax for the new ingress changed in 1.19 (see release notes and some small info at the end) I`m placing also an example with it:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress-v119
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /prefix/$1
spec:
rules:
- http:
paths:
- path: /(matched/path/?.*)
pathType: Prefix
backend:
service:
name: test
port:
number: 80
Here is a test with http echo server:
➜ ~ curl 172.17.0.4/matched/path
{
"path": "/prefix/matched/path",
"headers": {
"host": "172.17.0.4",
"x-request-id": "011585443ebc6adcf913db1c506abbe6",
"x-real-ip": "172.17.0.1",
"x-forwarded-for": "172.17.0.1",
"x-forwarded-host": "172.17.0.4",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
This rule will also ignore the / at the end of the request:
➜ ~ curl 172.17.0.4/matched/path/
{
"path": "/prefix/matched/path/",
"headers": {
"host": "172.17.0.4",
"x-request-id": "0575e9022d814ba07457395f78dbe0fb",
"x-real-ip": "172.17.0.1",
"x-forwarded-for": "172.17.0.1",
"x-forwarded-host": "172.17.0.4",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
Worth to mention some notable differences/changes in the new ingress syntax:
spec.backend -> spec.defaultBackend
serviceName -> service.name
servicePort -> service.port.name (for string values)
servicePort -> service.port.number (for numeric values) pathType no longer has a default value in v1; "Exact", "Prefix", or
"ImplementationSpecific" must be specified Other Ingress API updates
backends can now be resource or service backends
path is no longer required to be a valid regular expression (#89778,
#cmluciano) [SIG API Machinery, Apps, CLI, Network and Testing]

Force kubernetes ingress cname format

With Kubernetes, in a multi-tenant env., controlled by RBAC, when creating a new Ingress cname, I would like to force cname format like:
${service}.${namespace}.${cluster}.kube.infra
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ${servce}
spec:
tls:
- hosts:
- ${service}.${namespace}.${cluster}.kube.infra
secretName: conso-elasticsearch-ssl
rules:
- host: ${service}.${namespace}.${cluster}.kube.infra
http:
paths:
- path: /
backend:
serviceName: ${service}
servicePort: 9200
Is it possible?
You can do by it by writing a validating admission webhook which validates the ingress yaml and rejects it if the cname format is not as per the way you want. A better way to is to use Open Policy agent(OPA) and write rego policy. Here is a guide on how to perform policy driven validation of ingress using OPA.
package kubernetes.admission
import data.kubernetes.namespaces
operations = {"CREATE", "UPDATE"}
deny[msg] {
input.request.kind.kind == "Ingress"
operations[input.request.operation]
host := input.request.object.spec.rules[_].host
not fqdn_matches_any(host, valid_ingress_hosts)
msg := sprintf("invalid ingress host %q", [host])
}
valid_ingress_hosts = {
// valid hosts
}
fqdn_matches_any(str, patterns) {
fqdn_matches(str, patterns[_])
}
fqdn_matches(str, pattern) {
// validation logic
}
fqdn_matches(str, pattern) {
not contains(pattern, "*")
str == pattern
}

Kubernetes Websockets using Socket.io, ExpressJS and Nginx Ingress

I want to connect a React Native application using Socket.io to a server that is inside a Kubernetes Cluster hosted on Google Cloud Platform (GKE).
There seems to be an issue with the Nginx Ingress Controller declaration but I cannot find it.
I have tried adding nginx.org/websocket-services; rewriting my backend code so that it uses a separate NodeJS server (a simple HTTP server) on port 3004, then exposing it via the Ingress Controller under a different path than the one on port 3003; and multiple other suggestions from other SO questions and Github issues.
Information that might be useful:
Cluster master version: 1.15.11-gke.15
I use a Load Balancer managed with Helm (stable/nginx-ingress) with RBAC enabled
All deployments and services are within the namespace gitlab-managed-apps
The error I receive when trying to connect to socket.io is: Error: websocket error
For the front-end part, the code is as follows:
App.js
const socket = io('https://example.com/app-sockets/socketns', {
reconnect: true,
secure: true,
transports: ['websocket', 'polling']
});
I expect the above to connect me to a socket.io namespace called socketdns.
The backend code is:
app.js
const express = require('express');
const app = express();
const server = require('http').createServer(app);
const io = require('socket.io')(server);
const redis = require('socket.io-redis');
io.set('transports', ['websocket', 'polling']);
io.adapter(redis({
host: process.env.NODE_ENV === 'development' ? 'localhost' : 'redis-cluster-ip-service.gitlab-managed-apps.svc.cluster.local',
port: 6379
}));
io.of('/').adapter.on('error', function(err) { console.log('Redis Adapter error! ', err); });
const nsp = io.of('/socketns');
nsp.on('connection', function(socket) {
console.log('connected!');
});
server.listen(3003, () => {
console.log('App listening to 3003');
});
The ingress service is:
ingress-service.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-connect-timeout: "7200"
nginx.ingress.kubernetes.io/proxy-read-timeout: "7200"
nginx.ingress.kubernetes.io/proxy-send-timeout: "7200"
nginx.org/websocket-services: "app-sockets-cluster-ip-service"
name: ingress-service
namespace: gitlab-managed-apps
spec:
tls:
- hosts:
- example.com
secretName: letsencrypt-prod
rules:
- host: example.com
http:
paths:
- backend:
serviceName: app-cms-cluster-ip-service
servicePort: 3000
path: /?(.*)
- backend:
serviceName: app-users-cluster-ip-service
servicePort: 3001
path: /app-users/?(.*)
- backend:
serviceName: app-sockets-cluster-ip-service
servicePort: 3003
path: /app-sockets/?(.*)
- backend:
serviceName: app-sockets-cluster-ip-service
servicePort: 3003
path: /app-sockets/socketns/?(.*)
The solution is to remove the nginx.ingress.kubernetes.io/rewrite-target: /$1 annotation.
Here is a working configuration: (please note that apiVersion has changed since the question has been asked)
Ingress configuration
ingress-service.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "64m"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
name: ingress-service
namespace: default
spec:
tls:
- hosts:
- example.com
secretName: letsencrypt-prod
rules:
- host: example.com
http:
paths:
- backend:
service:
name: app-sockets-cluster-ip-service
port:
number: 3003
path: /app-sockets/?(.*)
pathType: Prefix
On the service (Express.js):
app.js
const redisAdapter = require('socket.io-redis');
const io = require('socket.io')(server, {
path: `${ global.NODE_ENV === 'development' ? '' : '/app-sockets' }/sockets/`,
cors: {
origin: '*',
methods: ['GET', 'POST'],
},
});
io.adapter(redisAdapter({
host: global.REDIS_HOST,
port: 6379,
}));
io.of('/').adapter.on('error', err => console.log('Redis Adapter error! ', err));
io.on('connection', () => {
//...
});
The global.NODE_ENV === 'development' ? '' : '/app-sockets' bit is related to an issue in development. If you change it here, you must also change it in the snippet below.
In development the service is under http://localhost:3003 (sockets endpoint is http://localhost:3003/sockets).
In production the service is under https://example.com/app-sockets (sockets endpoint is https://example.com/app-sockets/sockets).
On frontend
connectToWebsocketsService.js
/**
* Connect to a websockets service
* #param tokens {Object}
* #param successCallback {Function}
* #param failureCallback {Function}
*/
export const connectToWebsocketsService = (tokens, successCallback, failureCallback) => {
//SOCKETS_URL = NODE_ENV === 'development' ? 'http://localhost:3003' : 'https://example.com/app-sockets'
const socket = io(`${ SOCKETS_URL.replace('/app-sockets', '') }`, {
path: `${ NODE_ENV === 'development' ? '' : '/app-sockets' }/sockets/`,
reconnect: true,
secure: true,
transports: ['polling', 'websocket'], //required
query: {
// optional
},
auth: {
...generateAuthorizationHeaders(tokens), //optional
},
});
socket.on('connect', successCallback(socket));
socket.on('reconnect', successCallback(socket));
socket.on('connect_error', failureCallback);
};
Note: I wasn't able to do it on the project mentioned in the question, but I have on another project which is hosted on EKS, not GKE. Feel free to confirm if this works for you on GKE as well.
Just change annotations to
nginx.ingress.kubernetes.io/websocket-services: "app-sockets-cluster-ip-service"
instead of
nginx.org/websocket-services: "app-sockets-cluster-ip-service"
Mostly it will resolve your issue.