nginx-ingress tcp services - connection refused - kubernetes

Question

I'm currently deploying a new kubernetes cluster and I want to expose a mongodb service from outside the cluster using an nginx-ingress.
I know that nginx-ingress is usually for layer 7 applications but also capable to work on layer 4 (TCP/UDP) according to the official documentation.
https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
My mongodb service is a ClusterIP serivce which is accssible on port 11717 (internal namespace):
kubectl get svc -n internal
mongodb ClusterIP 10.97.63.154 <none> 11717/TCP 3d20h
telnet 10.97.63.154 11717
Trying 10.97.63.154...
Connected to 10.97.63.154.
I literally tried every possible combination to achieve this goal but with no success.
I'm using the nginx-ingress helm chart (daemonset type).
My nginx-ingress/templates/controller-daemonset.yaml file:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress-nginx-ingress
namespace: default
labels:
app.kubernetes.io/name: nginx-ingress-nginx-ingress
helm.sh/chart: nginx-ingress-0.13.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress-nginx-ingress
template:
metadata:
labels:
app: nginx-ingress-nginx-ingress
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9113"
prometheus.io/scheme: "http"
spec:
serviceAccountName: nginx-ingress-nginx-ingress
terminationGracePeriodSeconds: 30
hostNetwork: false
containers:
- name: nginx-ingress-nginx-ingress
image: "nginx/nginx-ingress:2.2.0"
imagePullPolicy: "IfNotPresent"
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
- name: mongodb
containerPort: 11717
hostPort: 11717
- name: prometheus
containerPort: 9113
- name: readiness-port
containerPort: 8081
readinessProbe:
httpGet:
path: /nginx-ready
port: readiness-port
periodSeconds: 1
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
{}
args:
- /nginx-ingress-controller
- -nginx-plus=false
- -nginx-reload-timeout=60000
- -enable-app-protect=false
- -tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- -publish-service=$(POD_NAMESPACE)/ingress-nginx
- -annotations-prefix=nginx.ingress.kubernetes.io
- -enable-app-protect-dos=false
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-ingress-nginx-ingress
- -default-server-tls-secret=$(POD_NAMESPACE)/nginx-ingress-nginx-ingress-default-server-tls
- -ingress-class=nginx
- -health-status=false
- -health-status-uri=/nginx-health
- -nginx-debug=false
- -v=1
- -nginx-status=true
- -nginx-status-port=8080
- -nginx-status-allow-cidrs=127.0.0.1
- -report-ingress-status
- -external-service=nginx-ingress-nginx-ingress
- -enable-leader-election=true
- -leader-election-lock-name=nginx-ingress-nginx-ingress-leader-election
- -enable-prometheus-metrics=true
- -prometheus-metrics-listen-port=9113
- -prometheus-tls-secret=
- -enable-custom-resources=true
- -enable-snippets=false
- -enable-tls-passthrough=false
- -enable-preview-policies=false
- -enable-cert-manager=false
- -enable-oidc=false
- -ready-status=true
- -ready-status-port=8081
- -enable-latency-metrics=false
My nginx-ingress/templates/controller-service.yaml file:
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-nginx-ingress
namespace: default
labels:
app.kubernetes.io/name: nginx-ingress-nginx-ingress
helm.sh/chart: nginx-ingress-0.13.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/instance: nginx-ingress
spec:
externalTrafficPolicy: Local
type: LoadBalancer
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
- name: mongodb
port: 11717
targetPort: 11717
protocol: TCP
selector:
app: nginx-ingress-nginx-ingress
My nginx-ingress/templates/tcp-services.yaml file:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: default
data:
"11717": internal/mongodb:11717
kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ingress-nginx-ingress-d5vms 1/1 Running 0 61m
nginx-ingress-nginx-ingress-kcs4p 1/1 Running 0 61m
nginx-ingress-nginx-ingress-mnnn2 1/1 Running 0 61m
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d1h <none>
nginx-ingress-nginx-ingress LoadBalancer 10.99.176.220 <pending> 80:31700/TCP,443:31339/TCP,11717:31048/TCP 61m app=nginx-ingress-nginx-ingress
telnet 10.99.176.220 80
Trying 10.99.176.220...
Connected to 10.99.176.220.
Escape character is '^]'.
telnet 10.99.176.220 11717
Trying 10.99.176.220...
telnet: Unable to connect to remote host: Connection refused
I can't understand why the connection is getting refused on port 11717.
How can I achieve this scenario:
mongo.myExternalDomain:11717 --> nginx-ingress service --> nginx-ingress pod --> mongodb service --> mongodb pod
Thanks in advance!
I would appreciate any kind of help!

Answer 1

I had simmiliar issue. Maybe this will help you. In my case it was in tcp-services configmap:
Shortly. Instead of this:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: default
data:
"11717": internal/mongodb:11717
please change to:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: default
data:
"11717": internal/mongodb:11717:PROXY
Details:
Edit the 'tcp-services' configmap to add a tcp .service 8000: namespace/service:8000.
edit the nginx-controller service to add a port (port:8000 --> targetPort:8000) for the tcp service in step1
Check /etc/nginx/nginx.conf in nginx controller pod and confirm it contains a 'server' block with correct listen 8000; directive for the tcp/8000 service.
Edit the 'tcp-services' configmap again to add the proxy-protocol decode directive and now the k/v for the tcp/8000 service becomes 8000: namespace/service:8000:PROXY
Check /etc/nginx/nginx.conf in nginx controller pod and there isn't any change comparing that from step3, it is still listen 8000;
Edit some ingress rule (make some change like updating the host)
Check /etc/nginx/nginx.conf in nginx controller pod again and now the listen directive for the tcp/8000 service becomes listen 8000 proxy_protocol; which is correct.