Connection refused when calling ingress from pod - kubernetes

I'm running a .NET 5 API application on Kubernetes as well as Identity Server in separate pods on my laptop using Docker Desktop. Both applications are made available for consumers using the nginx ingress controller. When I call the ingress in the browser from outside of the cluster, it works well. Also, when I run the API from Visual Studio, the connection to Identity Server via the ingress is successful.
But when I deploy the API to Kubernetes, the pod that runs the API can't connect to the ingress. The API uses the ingress to call Identity Server because it has to happen over https. SSL is terminated by the ingress and forwarding traffic to the pods over http.
The ingress address to IdentityServer is https://k8s-local.com/identityserver. From the API pod, the DNS is known; when I execute a ping, it goes ok. When I perform a curl command to this address from the pod, I'm getting a Connection Refused error:
curl: (7) Failed to connect to k8s-local.com port 443: Connection refused
How do I make the ingress DNS available to the pods?
This is what my ingress.yml looks like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
spec:
tls:
- hosts:
- k8s-local.com
secretName: local-tls
rules:
- host: k8s-local.com
http:
paths:
- pathType: Prefix
path: /identityserver/
backend:
service:
name: identityserver-service
port:
number: 5000
- pathType: Prefix
path: /customer-api/
backend:
service:
name: customer-api-service
port:
number: 5000
This is what I get when I run kubectl get services -n ingress-nginx:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.109.3.50 localhost 80:31964/TCP,443:32210/TCP 11d
ingress-nginx-controller-admission ClusterIP 10.105.158.51 <none> 443/TCP 11d
And kubectl get ingress:
NAME CLASS HOSTS ADDRESS PORTS AGE
my-ingress <none> k8s-local.com localhost 80, 443 6d11h

Related

Can I get response of curl through POD IP address (instead of host address) while using nginx ingress

I am using nginx ingress controller below is the ingress rule file for 2 services:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
namespace: kube-system
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
tls:
- hosts:
- rewrite.bar.com.com
secretName: ingress-tls
rules:
- host: rewrite.bar.com.com
- http:
paths:
- path: /my-service-1/(.*)
pathType: Prefix
backend:
service:
name: my-service-1
port:
number: 36995
- path: /my-service-2/(.*)
pathType: Prefix
backend:
service:
name: my-service-2
port:
number: 32243
Now using below command through shell of service-2 I can curl to the service-1 api endpoint, here I need to pass host ('wire.com') which is TLS enabled as well,
curl --resolve wire.com:443:10.22.148.179 https://wire.com:32243/GetData
Above curl using host address give me response successfully, no issue here!
Now I am using IP address of the POD instead of host address, but this won't give me response, it's always give error like curl: (52) Empty reply from server. Here 10.22.148.179 is my ingress public IP address and 10.2.0.58 is my POD IP address.
curl --resolve enabledservices-dev-aks.honeywell.com:443:10.22.148.179 http//10.2.0.58:32243/GetData
My goal to hit the POD/service api end point through IP address, is this possible with context of Ingress integrated?
Moving this from comments to answer.
The issue was curl request and HTTP protocol used while the server is serving by HTTPS. This is the reason of (52) Empty reply from server error.
Request by curl should be done by specifying the protocol like:
curl https://test.example.com:8888
Ingress is used as a single entry point to the cluster so all inside services can be exposed internally in the cluster using cluster-ip service type - see kubernetes service types.
If any inside service/pod is required to be tested from inside the cluster, request should be executed from the cluster to be able to hit a cluster-ip since cluster-ip is only accessible within the cluster.

Exposing kubernetes Dashboard with clusterIP service externally using Ingress rules

I am trying to expose kubernetes-dashboard app externally using Ingress resource. I have installed Nginx Controller and a service called Kubernetes-dashboard is clusterIP type service with port 443.
I have created Ingress resource with YAML file and pointing to backend service which is kubernetes-dashboard but somehow I am not getting the IP address of my host (dashboard.com) so that I can add this entry in /etc/hosts file. what is the resolution here. I am not able to paste the yaml file here as this website complain about code formatting.
I tried to put YAML file here in various ways but it does not work.
yaml file of kubernetes-dashboard as below:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: dashboard-ingress namespace: kubernetes-dashboard spec: tls: - hosts: - dashboard.com secretName: kubernetes-dashboard-certs rules: - host: dashboard.com http: paths: - pathType: ImplementationSpecific path: / backend: service: name: kubernetes-dashboard port: number: 443
Kubernetes-dashboard service config as below: Name: kubernetes-dashboard Namespace: kubernetes-dashboard Labels: k8s-app=kubernetes-dashboard Annotations: Selector: k8s-app=kubernetes-dashboard Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.106.1.186 IPs: 10.106.1.186 Port: 443/TCP TargetPort: 8443/TCP Endpoints: 10.44.0.3:8443 Session Affinity: None Events:
I am not getting the IP address of my host
You have to use the Nginx ingress controller service IP everywhere so traffic gets forwarded and managed via Nginx ingress.
you can check the IP of Nginx controller using the
Kubectl get svc -n ingress-nginx
Nginx controller service will be exposed as the type LoadBalancer you can use this IP into the DNS route as A or CNAME record.
Any request coming to your domain will get forwarded to
ingress > Nginx ingress controller > K8s service > K8s PODs

Could not access Kubernetes Ingress in Browser on Windows Home with Minikube?

I am facing the problem which is that I could not access the Kubernetes Ingress on the Browser using it's IP. I have installed K8s and Minikube on Windows 10 Home.
I am following this official document - https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/
First I created the deployment by running this below command on Minikube.
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
The deployment get created which can be seen on the below image:
Next, I exposed the deployment that I created above. For this I ran the below command.
kubectl expose deployment web --type=NodePort --port=8080
This created a service which can be seen by running the below command:
kubectl get service web
The screenshot of the service is shown below:
I can now able to visit the service on the browser by running the below command:
minikube service web
In the below screenshot you can see I am able to view it on the browser.
Next, I created an Ingress by running the below command:
kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
By the way the ingress yaml code is:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: hello-world.info
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 8080
The ingress gets created and I can verify it by running the below command:
kubectl get ingress
The screenshot for this is given below:
The ingress ip is listed as 192.168.49.2. So that means if I should open it in the browser then it should open, but unfortunately not. It is showing site can't be reached. See the below screeshot.
What is the problem. Please provide me a solution for it?
I also added the mappings on etc\hosts file.
192.168.49.2 hello-world.info
Then I also tried opening hello-world.info on the browser but no luck.
In the below picture I have done ping to hello-world.info which is going to IP address 192.168.49.2. This shows etc\hosts mapping is correct:
I also did curl to minikube ip and to hello-world.info and both get timeout. See below image:
The kubectl describe services web provides the following details:
Name: web
Namespace: default
Labels: app=web
Annotations: <none>
Selector: app=web
Type: NodePort
IP: 10.100.184.92
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
NodePort: <unset> 31880/TCP
Endpoints: 172.17.0.4:8080
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
The kubectl describe ingress example-ingress gives the following output:
Name: example-ingress
Namespace: default
Address: 192.168.49.2
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
hello-world.info
/ web:8080 172.17.0.4:8080)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1
Events: <none>
Kindly help. Thank you.
Having same issue as OP and things only work in minikube ssh, sharing the ingress.yaml below.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
defaultBackend:
service:
name: default-http-backend
port:
number: 80
rules:
- host: myapp-com # domain (i.e. need to change host table)
http:
paths: # specified path below, only be working when there is more than 1 path; If only having 1 path, it's always using / as path
- path: /
pathType: Prefix
backend:
service:
name: frontend-service # internal service
port:
number: 8080 # port number that internal service exposes
- path: /e($|/)(.*)
pathType: Prefix
backend:
service:
name: express-service # internal service
port:
number: 3000 # port number that internal service exposes
In my case (win10 + minikube + ingress minikube addon) the following helped:
Set custom domain IP to 127.0.01 in %WINDIR%\System32\drivers\etc\hosts file, i.e. by adding line 127.0.0.1 my-k8s.com
Get ingress pod name: kubectl get pods -n ingress-nginx
Start port forwarding: kubectl -n ingress-nginx port-forward pod/ingress-nginx-controller-5d88495688-dxxgw --address 0.0.0.0 80:80 443:443, where you should replace ingress-nginx-controller-5d88495688-dxxgw with your ingress pod name.
Enjoy using ingress on custom domain in any browser (but only when port forwarding is active)
Make sure pod to pod communication is open in minikube cluster. You can enable it by running below commands
minikube ssh
sudo ip link set docker0 promisc on
Make sure to install minikube ingress, ingress dns.
minikube addons enable ingress
minikube addons enable ingress-dns
For those wondering, this is a known issue with minikube, ingress is supported out-of-the-box on linux only.
minikube tunnel is a good fix, see this answer.
Try removing this annotation.
nginx.ingress.kubernetes.io/rewrite-target: /$1
And add this annotation:
annotations:
nginx.ingress.kubernetes.io/default-backend: ingress-nginx-controller
kubernetes.io/ingress.class: nginx
## tells ingress to check for regex in the config file
nginx.ingress.kubernetes.io/use-regex: "true"
Also, update your route as:
- path: /?(.*) ## instead of just '/'
backend:
serviceName: web
servicePort: 8080
I was in the same problem, the easiest solution that I found was modified the host windows file, but instead using the "minikube ip" use 127.0.0.1, and in ahotner terimnal run
$ minikube tunnel
With this you can open hello-world.info in the browser
I believe that if you check the ingress details you will find the right IP
kubectl describe ingress example-ingress
Check the Docs for more details about ingress
If the above doesn't help try this manifest. Check this Source
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
# If the class annotation is not specified it defaults to "gce".
kubernetes.io/ingress.class: "gce"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: test
servicePort: 1111
If you're running an Ingress controller on any OS other than Linux you need to pay attention to the message displayed when you enable the Ingress addon. To wit...
PS C:\Development\kubernetes\service\ingress> minikube addons enable ingress
� ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
� After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "
127.0.0.1"
▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
� Verifying ingress addon...
� The 'ingress' addon is enabled
PS C:\Development\kubernetes\service\ingress>
The thing to take away from this is that - on an O/S other than Linux - the IP address is 127.0.0.1 NOT whatever IP you see when you run > kubectl get ingress. This is because - on an OS other than Linux - you need minikube tunnel running as a 'bridge' between 127.0.0.1 and whatever IP the Ingress controller is using. It's 127.0.0.1 you need to reference in your hosts file, not the IP shown in > kubectl get ingress. Luck.

Kubernetes Loadbalancer redirecting to HTTPs when application is on port 80

I've deployed a series of deployments and services to a Kubernetes cluster with a load balancer. When I try to access my app this does not work as my application is exposed on port 80 but the URL is always redirected to port 443 (HTTPS). I suspect this is to do with the fact that the cluster IP is on port 443.
Any ideas on how I can fix this?
db NodePort 10.245.175.203 <none> 5432:30029/TCP 25m
kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 8m
redis NodePort 10.245.197.157 <none> 6379:31277/TCP 25m
web LoadBalancer 10.245.126.122 123.12.123.123 80:31430/TCP 25m
This is likely due to your application itself redirecting to port 443. What type of application is it?
This service exposed on port 443 has nothing to do with your application:
kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 8m
It's basically an internal service that allows you to access the kube-apiserver within your cluster.
You could try just setting up the LoadBalancer to listen on port 443 directly. Only you would have to port 80 traffic wouldn't work. If you want the port 80 redirects to work I suggest you use an Ingress controller like nginx. Something like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: your-ingress
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- hosts:
- yourhostname.com
secretName: tls-secret
rules:
- host: yourhostname.com
http:
paths:
- path: /
backend:
serviceName: web
servicePort: 443
You will also have to create a TLS secret holding your cert and key:
$ kubectl create secret tls tls-secret --key /tmp/tls.key --cert /tmp/tls.crt

How to verify working Traefik installation?

I'm in the process of setting up Traefik on a Kubernetes cluster, but I can't get it to work, so I need some troubleshooting help. The first thing I would like to verify is that the basic installation is successful.
The guide I'm following is this one:
https://docs.traefik.io/user-guide/kubernetes/
But, I'm installing on a 3-machine cluster (Master + 2x Nodes).
I have setup RBAC and create a Deployment / Service for Traefik. The Pod is up and running:
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
traefik-ingress-controller-7cf98d69cf-n2trx 1/1 Running 0 1h
This is the Service:
$ kubectl get services --namespace kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik-ingress-service NodePort 10.107.17.76 <none> 80:30820/TCP,8080: 31362/TCP 1h
Should I be able to access the Traefik Web UI now?
I tried to access "http://192.168.1.11:31362" from a web browser and it behaves a bit strange. I get a "404 page not found" error in the browser window, but the address bar in the browser changes to: "http://192.168.1.11:31362/dashboard/". That tells me that something is responding at that address / port.
This is the result of a Curl to the same address:
$ curl http://192.168.1.11:31362/
Found.
Is this normal behaviour at this step in the process?
I have also tried to test with an Service / Ingress like this:
apiVersion: v1
kind: Service
metadata:
name: test-service
spec:
selector:
app: homeautomationweb
ports:
- port: 80
targetPort: 31047
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: test.no
http:
paths:
- backend:
serviceName: test-service
servicePort: 80
I have a working web application running in the cluster exposed on a node port and is accessible outside the cluster at http://http://192.168.1.11:31047/.
The DNS name "test.no" is defined in /etc/hosts as 192.168.1.11
But, when I try to access http://test.no, I get:
"test.no refused to connect"
The details of what I'm doing and the exact content of the Kubernetes Yaml files can be found at the end of this article:
https://github.com/olavt/KubernetesRaspberryPI