I'm running a single node k3s installation. The machine it's running on has two NICs installed with two IP addresses assigned. I want the ingress only bind to one of these, but no matter what I tried, nginx will always bind to both/all interfaces.
I'm using the official Helmchart for ingress-nginx and modified the following values:
clusterIP: ""
...
externalIPs:
- 192.168.1.200
...
externalTrafficPolicy: "Local"
...
The following doesn't look that bad to me, but nginx is still listening on the other inferface (192.168.1.123) too...
❯ k get service -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.43.30.231 192.168.1.200,192.168.1.200 80:30047/TCP,443:30815/TCP 9m14s
This is the service as generated by Helm:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
...
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.45.0
helm.sh/chart: ingress-nginx-3.29.0
annotations:
meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
managedFields:
...
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: http
nodePort: 30047
- name: https
protocol: TCP
port: 443
targetPort: https
nodePort: 30815
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
clusterIP: 10.43.30.231
clusterIPs:
- 10.43.30.231
type: LoadBalancer
externalIPs:
- 192.168.1.200
sessionAffinity: None
externalTrafficPolicy: Local
healthCheckNodePort: 32248
status:
loadBalancer:
ingress:
- ip: 192.168.1.200
You'll want to set the bind-address in the configmap settings.
Sets the addresses on which the server will accept requests instead of *. It should be noted that these addresses must exist in the runtime environment or the controller will crash loop.
Related
I have a node server which runs in a container inside a pod. The pod is up and running and is working fine.
Then I have a NodePort service which is used to serve the pod. The service is curl-able from inside the container and returns a response successfully.
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: server
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-12-20T23:46:09Z"
labels:
app: server
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
chart: server-0.1.0
helm.sh/chart: nginx-13.2.19
heritage: Helm
release: server
name: server
namespace: default
resourceVersion: "86325"
uid: 9eea1b02-af20-4e79-b713-6be2683cfcf7
spec:
clusterIP: 10.100.138.182
clusterIPs:
- 10.100.138.182
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 80
nodePort: 30001
protocol: TCP
targetPort: 3001
selector:
app: server
release: server
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
Then I have an NGINX Ingress Controller running and an Ingress resource.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: server
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-12-20T23:46:09Z"
generation: 2
labels:
app.kubernetes.io/instance: server
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: server
helm.sh/chart: server-0.1.0
name: server
namespace: default
resourceVersion: "87296"
uid: e4a597c3-3d85-402d-bf56-19858c22e7bf
spec:
defaultBackend:
service:
name: server
port:
number: 80
ingressClassName: nginx
status:
loadBalancer: {}
After all this when I do kubectl get pods:
NAME READY STATUS RESTARTS AGE
nginxserver-876bf6f87-ncnfl 1/1 Running 0 5h23m
server-7466b8f5d8-jmm57 1/1 Running 0 12m
kubectl get service:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 9h
nginxserver LoadBalancer 10.100.129.213 aa486e3c823d54f4fbe8da864b55d394-674371728.us-east-2.elb.amazonaws.com 80:31573/TCP 5h23m
server NodePort 10.100.138.182 <none> 80:30001/TCP 13m
kubectl get ingress:
NAME CLASS HOSTS ADDRESS PORTS AGE
server nginx * 80 14m
This currently shows 14m age but before this I had waited for 30m too but no address popped up.
I have tried searching a lot and found that this was associated with wrong ingress settings or wrong namespaces but none of that helped me.
What can I do to so that I can get an address for the ingress?
Thankyou!
I gave up. As a last option I thought maybe my ingress controller is not working properly. So I
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update
helm install ingress-nginx nginx-stable/nginx-ingress --set rbac.create=true
after deleting the previous one and my ingress got an idress within seconds.
I can't apply an ingress configuration.
I need access a jupyter-lab service by it's DNS
http://jupyter-lab.local
It's deployed to a 3 node bare metal k8s cluster
node1.local (master)
node2.local (worker)
node3.local (worker)
Flannel is installed as the Network controller
I've installed nginx ingress for bare metal like this
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml
When deployed the jupyter-lab pod is on node2 and the NodePort service responds correctly from http://node2.local:30004 (see below)
I'm expecting that the ingress-nginx controller will expose the ClusterIP service by its DNS name ...... thats what I need, is that wrong?
This is the CIP service, defined with symmetrical ports 8888 to be as simple as possible (is that wrong?)
---
apiVersion: v1
kind: Service
metadata:
name: jupyter-lab-cip
namespace: default
spec:
type: ClusterIP
ports:
- port: 8888
targetPort: 8888
selector:
app: jupyter-lab
The DNS name jupyter-lab.local resolves to the ip address range of the cluster, but times out with no response. Failed to connect to jupyter-lab.local port 80: No route to host
firewall-cmd --list-all shows that port 80 is open on each node
This is the ingress definition for http into the cluster (any node) on port 80. (is that wrong ?)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jupyter-lab-ingress
annotations:
# nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io: /
spec:
rules:
- host: jupyter-lab.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: jupyter-lab-cip
port:
number: 80
This the deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jupyter-lab-dpt
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: jupyter-lab
template:
metadata:
labels:
app: jupyter-lab
spec:
volumes:
- name: jupyter-lab-home
persistentVolumeClaim:
claimName: jupyter-lab-pvc
containers:
- name: jupyter-lab
image: docker.io/jupyter/tensorflow-notebook
ports:
- containerPort: 8888
volumeMounts:
- name: jupyter-lab-home
mountPath: /var/jupyter-lab_home
env:
- name: "JUPYTER_ENABLE_LAB"
value: "yes"
I can successfully access jupyter-lab by its NodePort http://node2:30004 with this definition:
---
apiVersion: v1
kind: Service
metadata:
name: jupyter-lab-nodeport
namespace: default
spec:
type: NodePort
ports:
- port: 10003
targetPort: 8888
nodePort: 30004
selector:
app: jupyter-lab
How can I get ingress to my jupyter-lab at http://jupyter-lab.local ???
the command kubectl get endpoints -n ingress-nginx ingress-nginx-controller-admission returns :
ingress-nginx-controller-admission 10.244.2.4:8443 15m
Am I misconfiguring ports ?
Are my "selector:appname" definitions wrong ?
Am I missing a part
How can I debug what's going on ?
Other details
I was getting this error when applying an ingress kubectl apply -f default-ingress.yml
Error from server (InternalError): error when creating "minnimal-ingress.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-contr
oller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": context deadline exceeded
This command kubectl delete validatingwebhookconfigurations --all-namespaces
removed the validating webhook ... was that wrong to do?
I've opened port 8443 on each node in the cluster
Ingress is invalid, try the following:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jupyter-lab-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: jupyter-lab.local
http: # <- removed the -
paths:
- path: /
pathType: Prefix
backend:
service:
# name: jupyter-lab-cip
name: jupyter-lab-nodeport
port:
number: 8888
---
apiVersion: v1
kind: Service
metadata:
name: jupyter-lab-cip
namespace: default
spec:
type: ClusterIP
ports:
- port: 8888
targetPort: 8888
selector:
app: jupyter-lab
If I understand correctly, you are trying to expose jupyternb through ingress nginx proxy and to make it accessible through port 80.
Run the folllowing command to check what nodeport is used by nginx ingress service:
$ kubectl get svc -n ingress-nginx ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.240.73 <none> 80:30816/TCP,443:31475/TCP 3h30m
In my case that is port 30816 (for http) and 31475 (for https).
Using NodePort type you can only use ports in range 30000-32767 (k8s docs: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). You can change it using kube-apiserver flag --service-node-port-range and then set it to e.g. 80-32767 and then in your ingress-nginx-controller service set nodePort: 80
apiVersion: v1
kind: Service
metadata:
annotations: {}
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 0.44.0
helm.sh/chart: ingress-nginx-3.23.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 80 # <- HERE
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 443 # <- HERE
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
Although this is genereally not advised to change service-node-port-range since you may encounter some issues if you use ports that are already open on nodes (e.g. port 10250 that is opened by kubelet on every node).
What might be a better solution is to use MetalLB.
EDIT:
How can I get ingress to my jupyter-lab at http://jupyter-lab.local ???
Assuming you don't need a failure tolerant solution, download the https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml file and change ports: section for the deployment object like following:
ports:
- name: http
containerPort: 80
hostPort: 80 # <- add this line
protocol: TCP
- name: https
containerPort: 443
hostPort: 443 # <- add this line
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
and apply the changes:
kubectl apply -f deploy.yaml
Now run:
$ kubectl get po -n ingress-nginx ingress-nginx-controller-<HERE PLACE YOUR HASH> -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-controller-67897c9494-c7dwj 1/1 Running 0 97s 172.17.0.6 <node_name> <none> <none>
Notice the <node_name> in NODE column. This is a node's name where the pod got scheduled. Now take this nodes IP and add it to your /etc/hosts file.
It should work now (go to http://jupyter-lab.local to check it), but this solution is fragile and if nginx ingress controller pod gets rescheduled to other node it will stop working (and it will stay lik this until you change the ip in /etc/hosts file). It's also generally not advised to use hostPort: field unless you have a very good reason to do so, so don't abuse it.
If you need failure tolerant solution, use MetalLB and create a service of type LoadBalancer for nginx ingress controller.
I haven't tested it but the following should do the job, assuming that you correctly configured MetalLB:
kubectl delete svc -n ingress-nginx ingress-nginx-controller
kubectl expose deployment -n ingress-nginx ingress-nginx-controller --type LoadBalancer
I don't mean being able to route to a specific port, I mean to actually change the port the ingress listens on.
Is this possible? How? Where is this documented?
No. From the kubernetes documentation:
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.
It may be possible to customize a LoadBalancer on a cloud provider like AWS to listen on other ports.
I assume you are using NGINX Ingress Controller. In this case, during installation, instead of doing a kubectl apply in the official yaml like this is one, you can try downloading the yaml and changing the port. The file above, which is used for an L4 AWS ELB, would become like this:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- port: {custom port 1}
targetPort: http
- port: {custom port 2}
targetPort: https
An alternative is to use a more powerful ingress controller.
Here is a list of different controllers.
My personal choice is Ambassador. If you follow the getting-started page, you just need to change the service definition for the port of your choice:
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: {custom port}
targetPort: 8080
selector:
service: ambassador
An Ingress definition is backed by an ingress controller. The ingress controller is deployed with normal Kubernetes objects so will have a Service associated with it that exposes ports for the ingress controller.
The kubernetes/ingress-nginx static deploys have a deploy.yaml with a Service type LoadBalancer:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Modify the ports the load balancer is configured with, in spec.ports[*].port in the external service, however that is deployed.
If you're using Helm to deploy the Kubernetes ingress-nginx controller, you can do this to change the ports from their defaults of 80 and 443 in a Helm values override file:
ingress-nginx:
enabled: true
...
controller:
service:
ports:
http: 8123
https: 9456
See here: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#values
In this example, any services exposed through the ingress are now available on ports 8123 (http) and 9456 (https).
Hi i have use Nginx controller both community and Nginx repo both works for in cloud with LoadBalancer service. Where as Nginx controller both community and Nginx repo both dose not work with service Nodeport "error 404".
404 Not Found
Service Type Nodeport
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
The default behavior of a NGINX Ingress Controller is to reply with a default backed that replies with a 404 - default backend: basically, it's a running Deployment resource returning that default page.
You can try overriding it passing to Helm a variable override using the path defaultBackend.image.repository as described in the README
I don't mean being able to route to a specific port, I mean to actually change the port the ingress listens on.
Is this possible? How? Where is this documented?
No. From the kubernetes documentation:
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.
It may be possible to customize a LoadBalancer on a cloud provider like AWS to listen on other ports.
I assume you are using NGINX Ingress Controller. In this case, during installation, instead of doing a kubectl apply in the official yaml like this is one, you can try downloading the yaml and changing the port. The file above, which is used for an L4 AWS ELB, would become like this:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- port: {custom port 1}
targetPort: http
- port: {custom port 2}
targetPort: https
An alternative is to use a more powerful ingress controller.
Here is a list of different controllers.
My personal choice is Ambassador. If you follow the getting-started page, you just need to change the service definition for the port of your choice:
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: {custom port}
targetPort: 8080
selector:
service: ambassador
An Ingress definition is backed by an ingress controller. The ingress controller is deployed with normal Kubernetes objects so will have a Service associated with it that exposes ports for the ingress controller.
The kubernetes/ingress-nginx static deploys have a deploy.yaml with a Service type LoadBalancer:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Modify the ports the load balancer is configured with, in spec.ports[*].port in the external service, however that is deployed.
If you're using Helm to deploy the Kubernetes ingress-nginx controller, you can do this to change the ports from their defaults of 80 and 443 in a Helm values override file:
ingress-nginx:
enabled: true
...
controller:
service:
ports:
http: 8123
https: 9456
See here: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#values
In this example, any services exposed through the ingress are now available on ports 8123 (http) and 9456 (https).