Kubernetes nginx Ingress configuration not working for Grafana - kubernetes

Question

I am new to configuring Ingress rules for my Kubernetes cluster.
My Kubernetes cluster is deployed on Bare Metal. No cloud.
I followed this link to set up my nginx-controller with RBAC in my cluster.
This is what I have deployed :
# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/default-http-backend-7c5bc89cc9-ks6kd 1/1 Running 0 2h
pod/nginx-ingress-controller-5b6864749-8xbhf 1/1 Running 0 2h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/default-http-backend ClusterIP 10.233.15.56 <none> 80/TCP 2h
service/ingress-nginx NodePort 10.233.38.84 <none> 80:31118/TCP,443:32003/TCP 2h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/default-http-backend 1 1 1 1 2h
deployment.apps/nginx-ingress-controller 1 1 1 1 2h
NAME DESIRED CURRENT READY AGE
replicaset.apps/default-http-backend-7c5bc89cc9 1 1 1 2h
replicaset.apps/nginx-ingress-controller-5b6864749 1 1 1 2h
Given that I have my setup, I want to access my grafana dashboard using a URL.
My grafana setup is working perfectly fine.
# kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/grafana-67c6585fbd-4jl7p 1/1 Running 0 2h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana NodePort 10.233.5.111 <none> 3000:32093/TCP 2h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1 1 1 1 2h
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-67c6585fbd 1 1 1 2h
I can access the dashboard using http://10.27.239.145:32093 which is the IP of one of my K8S worker nodes.
Now rather than accessing via IP:NodePort, I want to access via URL e.g. grafana.test.mydomain.com
So the ingress rule that I configured in my default namespace is :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
creationTimestamp: 2018-09-25T20:32:24Z
generation: 5
name: grafana
namespace: default
resourceVersion: "28485"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins-tls
uid: 1c51cece-c102-11e8-bf0f-02000a1bef39
spec:
rules:
- host: grafana.test.mydomain.com
http:
paths:
- backend:
serviceName: grafana
servicePort: 3000
path: /
On my local laptop from where I am testing, Ive added to my /etc/hosts the following entry :
10.27.239.145 grafana.test.mydomain.com
And in my browser, I am trying to access http://grafana.test.mydomain.com but I only get This site can’t be reached
grafana.test.mydomain.com refused to connect.
I have a strong feeling that I am missing out on something but can't figure it out.
I changed the NodePort to ClusterIP but no luck.
I know that my ingress controller is working since everytime I make a change to my ingress rules, I get logs from my ingress controller.
I0925 21:00:19.041440 9 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"grafana", UID:"1c51cece-c102-11e8-bf0f-02000a1bef39", APIVersion:"extensions/v1beta1", ResourceVersion:"28485", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/grafana
I0925 21:00:19.041732 9 controller.go:171] Configuration changes detected, backend reload required.
I0925 21:00:19.216044 9 controller.go:187] Backend successfully reloaded.
I0925 21:00:19.217645 9 controller.go:204] Dynamic reconfiguration succeeded.
Any help will strongly be appreciated regarding what might I have missed.

Answer 1

From what I see, you need to set grafana.test.mydomain.com to point to 10.233.38.84.
Basically, your nginx controller service is directing the traffic to your ingress and then your ingress forwards it to the backend on the nodePort (this is implicit in the ingress). It works for me, but I'm using an AWS ELB, I basically set grafana.test.mydomain.com to point to aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/default-http-backend-6586bc58b6-snxbv 1/1 Running 0 1h
pod/grafana-5b969bb7f9-tsv5k 1/1 Running 0 52m
pod/nginx-ingress-controller-6bd7c597cb-lfwcf 1/1 Running 0 1h
pod/prometheus-server-5dbf9f4fc9-mnwn4 1/1 Running 0 53m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/default-http-backend ClusterIP 10.x.x.x <none> 80/TCP 1h
service/grafana NodePort 10.x.x.x <none> 3000:30073/TCP 52m
service/ingress-nginx LoadBalancer 10.x.x.x aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com 80:30276/TCP,443:32011/TCP 1h
service/prometheus-server NodePort 10.x.x.x <none> 9090:32419/TCP 53m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/default-http-backend 1 1 1 1 1h
deployment.apps/grafana 1 1 1 1 52m
deployment.apps/nginx-ingress-controller 1 1 1 1 1h
deployment.apps/prometheus-server 1 1 1 1 53m
NAME DESIRED CURRENT READY AGE
replicaset.apps/default-http-backend-6586bc58b6 1 1 1 1h
replicaset.apps/grafana-5b969bb7f9 1 1 1 52m
replicaset.apps/nginx-ingress-controller-6bd7c597cb 1 1 1 1h
replicaset.apps/prometheus-server-5dbf9f4fc9 1 1 1 53m
$ kubectl describe ingress grafana-ingress -n ingress-nginx
Name: grafana-ingress
Namespace: ingress-nginx
Address: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-0000000000.us-west-2.elb.amazonaws.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
grafana.test.mydomain.com
/ grafana:3000 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"grafana-ingress","namespace":"ingress-nginx"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000},"path":"/"}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 40m nginx-ingress-controller Ingress ingress-nginx/grafana-ingress
Normal UPDATE 22m (x2 over 40m) nginx-ingress-controller Ingress ingress-nginx/grafana-ingress

Answer 2

As far as I can see, you only have a NodePort Service on port 32093.
Your NodePort publishes the port 3000 to 32093 to any external node address as you have already proven, but you configured Ingress to contact port 3000 on grafana service.
Either add the targetPort, port and nodePort to the service for your Grafana instance and point targetPort and port to 3000 and leave nodePort empty/set it to 32092. Then the ingress should work as you posted. Snippet:
nodePort: 32093
port: 3000
protocol: TCP
targetPort: 3000
Or try to set servicePort: 3000 in your ingress configuration to 32093. Warning: I never tested this. I do not know if Ingress supports that. According to the documentation it should as NodePort is a superset of ClusterIP:
NodePort: Exposes the service on each Node’s IP at a static port (the NodePort). A ClusterIP service, to which the NodePort service will route, is automatically created. You’ll be able to contact the NodePort service, from outside the cluster, by requesting :.
Edit
Btw: http://grafana.test.mydomain.com:32093 should then already work with your configuration (NodePort)