How to set up a custom HTTP error in Kubernetes - kubernetes

Question

I want to create a custom 403 error page.
Currently I already have an Ingress created and in the annotations I have something like this:
"nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,88.100.01.01"
So any attempt to access my web app outside that IP range receives a 403 error.
In order to create a custom page I tried adding the following annotations:
"nginx.ingress.kubernetes.io/custom-http-errors": "403",
"nginx.ingress.kubernetes.io/default-backend": "default-http-backend"
where default-http-backend is the name of an app already deployed.
the ingress has this:
{
"kind": "Ingress",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "my-app-ingress",
"namespace": "my-app-test",
"selfLink": "/apis/extensions/v1beta1/namespaces/my-app-test/ingresses/my-app-ingress",
"uid": "8f31f2b4-428d-11ea-b15a-ee0dcf00d5a8",
"resourceVersion": "129105581",
"generation": 3,
"creationTimestamp": "2020-01-29T11:50:34Z",
"annotations": {
"kubernetes.io/ingress.class": "nginx",
"nginx.ingress.kubernetes.io/custom-http-errors": "403",
"nginx.ingress.kubernetes.io/default-backend": "default-http-backend",
"nginx.ingress.kubernetes.io/rewrite-target": "/",
"nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,90.108.01.012"
}
},
"spec": {
"tls": [
{
"hosts": [
"my-app-test.retail-azure.js-devops.co.uk"
],
"secretName": "ssl-secret"
}
],
"rules": [
{
"host": "my-app-test.retail-azure.js-devops.co.uk",
"http": {
"paths": [
{
"path": "/api",
"backend": {
"serviceName": "my-app-backend",
"servicePort": 80
}
},
{
"path": "/",
"backend": {
"serviceName": "my-app-frontend",
"servicePort": 80
}
}
]
}
}
]
},
"status": {
"loadBalancer": {
"ingress": [
{}
]
}
}
}
Yet I always get the default 403.
What am I missing?

Answer 1

I've reproduced your scenario and that worked for me.
I will try to guide you in steps I've followed.
Cloud provider: GKE
Kubernetes Version: v1.15.3
Namespace: default
I'm using 2 deployments of 2 images with a service for each one.
Service 1: default-http-backend - with nginx image, it will be our default backend.
Service 2: custom-http-backend - with inanimate/echo-server image, this service will be displayed if the request become from a whitelisted ip.
Ingress: Nginx ingress with annotations.
Expected behavior: The ingress will be configured to use default-backend, custom-http-errors and whitelist-source-range annotations. If the request was made from a whitelisted ip the ingress will redirect to custom-http-backend, if not it will be redirect to default-http-backend.
Deployment 1: default-http-backend
Create a file default-http-backend.yaml with this content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
spec:
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
containers:
- name: default-http-backend
image: nginx
ports:
- name: http
containerPort: 80
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
spec:
selector:
app: default-http-backend
ports:
- protocol: TCP
port: 80
targetPort: 80
Apply the yaml file: k apply -f default-http-backend.yaml
Deployment 2: custom-http-backend
Create a file custom-http-backend.yaml with this content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-http-backend
spec:
selector:
matchLabels:
app: custom-http-backend
template:
metadata:
labels:
app: custom-http-backend
spec:
containers:
- name: custom-http-backend
image: inanimate/echo-server
ports:
- name: http
containerPort: 8080
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: custom-http-backend
spec:
selector:
app: custom-http-backend
ports:
- protocol: TCP
port: 80
targetPort: 8080
Apply the yaml file: k apply -f custom-http-backend.yaml
Check if services is up and running
I'm using the alias k for kubectl
➜ ~ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
custom-http-backend ClusterIP 10.125.5.227 <none> 80/TCP 73s
default-http-backend ClusterIP 10.125.9.218 <none> 80/TCP 5m41s
...
➜ ~ k get pods
NAME READY STATUS RESTARTS AGE
custom-http-backend-67844fb65d-k2mwl 1/1 Running 0 2m10s
default-http-backend-5485f569bd-fkd6f 1/1 Running 0 6m39s
...
You could test the service using port-forward:
default-http-backend
k port-forward svc/default-http-backend 8080:80
Try to access http://localhost:8080 in your browse to see the nginx default page.
custom-http-backend
k port-forward svc/custom-http-backend 8080:80
Try to access http://localhost:8080 in your browse to see the custom page provided by the echo-server image.
Ingress configuration
At this point we have both services up and running, we need to install and configure the nginx ingress. You can follow the official documentation, this will not covered here.
After installed let's deploy the ingress, based in the code you posted i did some modifications: tls removed, added other domain and removed the path /api for tests purposes only and add my home ip to whitelist.
Create a file my-app-ingress.yaml with the content:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-app-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/custom-http-errors: '403'
nginx.ingress.kubernetes.io/default-backend: default-http-backend
nginx.ingress.kubernetes.io/whitelist-source-range: 207.34.xxx.xx/32
spec:
rules:
- host: myapp.rabello.me
http:
paths:
- path: "/"
backend:
serviceName: custom-http-backend
servicePort: 80
Apply the spec: k apply -f my-app-ingress.yaml
Check the ingress with the command:
➜ ~ k get ing
NAME HOSTS ADDRESS PORTS AGE
my-app-ingress myapp.rabello.me 146.148.xx.xxx 80 36m
That's all!
If I test from home with my whitelisted ip, the custom page is showed, but if i try to access using my cellphone in 4G network, the nginx default page is displayed.
Note I'm using ingress and services in the same namespace, if you need work with different namespace you need to use ExternalName.
I hope that helps!
References:
kubernetes deployments
kubernetes service
nginx ingress
nginx annotations

Answer 2

I want to create a custom 403 error page. Currently I already have an Ingress created and in the annotations.
So any attempt to access my web app outside that IP range receives a 403 error.
In order to create a custom page I tried adding the following annotations:
kind: Ingress
metadata:
name: my-app-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/custom-http-errors: '403'
nginx.ingress.kubernetes.io/default-backend: default-http-backend
nginx.ingress.kubernetes.io/whitelist-source-range: 125.10.156.36/32
spec:
rules:
- host: venkat.dev.vboffice.com
http:
paths:
- path: "/"
backend:
serviceName: custom-http-backend
servicePort: 80
where default-http-backend is the name of an app already deployed with default nginx page.
If I test from home with my whitelisted ip, the custom page is showed, but if i try to access using my cellphone in 4G network, it will display default backend 404
i need to add any nginx config change custom-http-backend pod????
Deployment 1:default-http-backend
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
spec:
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
containers:
- name: default-http-backend
image: nginx
ports:
- name: http
containerPort: 80
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
spec:
selector:
app: default-http-backend
ports:
- protocol: TCP
port: 80
targetPort: 80
Deployment 2: custom-http-backend
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-http-backend
spec:
selector:
matchLabels:
app: custom-http-backend
template:
metadata:
labels:
app: custom-http-backend
spec:
containers:
- name: custom-http-backend
image: inanimate/echo-server
ports:
- name: http
containerPort: 8080
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: custom-http-backend
spec:
selector:
app: custom-http-backend
ports:
- protocol: TCP
port: 80
targetPort: 8080

Answer 3

One can customize the 403 error page for ingress-nginx (/etc/nginx/template), just by editing the nginx.tmpl file. Then mounting it to ingress nginx controller deployment.Below is the part of the nginx.tmpl need to be edited:
{{/* Build server redirects (from/to www) */}}
{{ range $redirect := .RedirectServers }}
## start server {{ $redirect.From }}
server {
server_name {{ $redirect.From }};
{{ buildHTTPListener $all $redirect.From }}
{{ buildHTTPSListener $all $redirect.From }}
ssl_certificate_by_lua_block {
certificate.call()
}
error_page 403 /403.html;
{{ if gt (len $cfg.BlockUserAgents) 0 }}
if ($block_ua) {
return 403;
}
{{ end }}
{{ if gt (len $cfg.BlockReferers) 0 }}
if ($block_ref) {
return 403;
}
{{ end }}
location = /403.html {
root /usr/local/nginx/html/;
internal;
}
set_by_lua_block $redirect_to {
local request_uri = ngx.var.request_uri
if string.sub(request_uri, -1) == "/" then
request_uri = string.sub(request_uri, 1, -2)
end
{{ if ne $all.ListenPorts.HTTPS 443 }}
{{ $redirect_port := (printf ":%v" $all.ListenPorts.HTTPS) }}
return string.format("%s://%s%s%s", ngx.var.scheme, "{{ $redirect.To }}", "{{ $redirect_port }}", request_uri)
{{ else }}
return string.format("%s://%s%s", ngx.var.scheme, "{{ $redirect.To }}", request_uri)
{{ end }}
}
return {{ $all.Cfg.HTTPRedirectCode }} $redirect_to;
}
## end server {{ $redirect.From }}
{{ end }}
{{ range $server := $servers }}
## start server {{ $server.Hostname }}
server {
server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};
error_page 403 /403.html;
{{ if gt (len $cfg.BlockUserAgents) 0 }}
if ($block_ua) {
return 403;
}
{{ end }}
{{ if gt (len $cfg.BlockReferers) 0 }}
if ($block_ref) {
return 403;
}
{{ end }}
location = /403.html {
root /usr/local/nginx/html/;
internal;
}
{{ template "SERVER" serverConfig $all $server }}
{{ if not (empty $cfg.ServerSnippet) }}
# Custom code snippet configured in the configuration configmap
{{ $cfg.ServerSnippet }}
{{ end }}
{{ template "CUSTOM_ERRORS" (buildCustomErrorDeps "upstream-default-backend" $cfg.CustomHTTPErrors $all.EnableMetrics) }}
}
## end server {{ $server.Hostname }}
{{ end }}
In the above snippet error_apge 403 /403.html; is declared before we return 403. Then the location of /403.html is defined. The root path is same where one should mount the 403.html page. In this case its /usr/local/nginx/html/.
Below snippet will help you mount the volume with custom pages.
volumes:
- name: custom-errors
configMap:
# Provide the name of the ConfigMap you want to mount.
name: custom-ingress-pages
items:
- key: "404.html"
path: "404.html"
- key: "403.html"
path: "403.html"
- key: "50x.html"
path: "50x.html"
- key: "index.html"
path: "index.html"
This solution doesn't require you to spawn another/extra service or pod of any kind to work.
For more info: https://engineering.zenduty.com/blog/2022/03/02/customizing-error-pages

Answer 4

You need to create and deploy custom default backend which will return a custom error page.Follow the doc to deploy a custom default backend and configure nginx ingress controller by modifying the deployment yaml to use this custom default backend.
The deployment yaml for the custom default backend is here and the source code is here.