Allow pod to access domain names of service through ingress - kubernetes-helm

I have a service and ingress setup on my minikube kubernetes cluster which exposes the domain name hello.life.com
Now I need to access this domain from inside another pod as
curl http://hello.life.com
and this should return proper html
My service is as follows:
apiVersion: v1
kind: Service
metadata:
labels:
app: bulging-zorse-key
chart: key-0.1.0
heritage: Tiller
release: bulging-zorse
name: bulging-zorse-key-svc
namespace: abc
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
name: bulging-zorse-key
type: ClusterIP
status:
loadBalancer: {}
My ingress is as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
labels:
app: bulging-zorse-key
chart: key-0.1.0
heritage: Tiller
release: bulging-zorse
name: bulging-zorse-key-ingress
namespace: dev
spec:
rules:
- host: hello.life.com
http:
paths:
- backend:
serviceName: bulging-zorse-key-svc
servicePort: 80
path: /
status:
loadBalancer:
ingress:
- {}
Can someone please help me out as to what changes do I need to make to get it working?
Thanks in advance!!!

I found a good explanation of your problem and the solution in the Custom DNS Entries For Kubernetes article:
Suppose we have a service, foo.default.svc.cluster.local that is available to outside clients as foo.example.com. That is, when looked up outside the cluster, foo.example.com will resolve to the load balancer VIP - the external IP address for the service. Inside the cluster, it will resolve to the same thing, and so using this name internally will cause traffic to hairpin - travel out of the cluster and then back in via the external IP.
The solution is:
Instead, we want foo.example.com to resolve to the internal ClusterIP, avoiding the hairpin.
To do this in CoreDNS, we make use of the rewrite plugin. This plugin can modify a query before it is sent down the chain to whatever backend is going to answer it.
To get the behavior we want, we just need to add a rewrite rule mapping foo.example.com to foo.default.svc.cluster.local:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
rewrite name foo.example.com foo.default.svc.cluster.local
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2019-01-09T15:02:52Z"
name: coredns
namespace: kube-system
resourceVersion: "8309112"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: a2ef5ff1-141f-11e9-9043-42010a9c0003
Note: In your case, you have to put ingress service name as a destination for the alias.
(E.g.: rewrite name hello.life.com ingress-service-name.ingress-namespace.svc.cluster.local) Make sure you're using correct service name and namespace name.
Once we add that to the ConfigMap via kubectl edit configmap coredns -n kube-system or kubectl apply -f patched-coredns-deployment.yaml -n kube-system, we have to wait 10-15 minutes. Recent CoreDNS versions includes reload plugin.
reload
Name
reload - allows automatic reload of a changed Corefile.
Description
This plugin periodically checks if the Corefile has changed by reading it and calculating its MD5 checksum. If the file has changed, it reloads CoreDNS with the new Corefile. This eliminates the need to send a SIGHUP or SIGUSR1 after changing the Corefile.
The reloads are graceful - you should not see any loss of service when the reload happens. Even if the new Corefile has an error, CoreDNS will continue to run the old config and an error message will be printed to the log. But see the Bugs section for failure modes.
In some environments (for example, Kubernetes), there may be many CoreDNS instances that started very near the same time and all share a common Corefile. To prevent these all from reloading at the same time, some jitter is added to the reload check interval. This is jitter from the perspective of multiple CoreDNS instances; each instance still checks on a regular interval, but all of these instances will have their reloads spread out across the jitter duration. This isn't strictly necessary given that the reloads are graceful, and can be disabled by setting the jitter to 0s.
Jitter is re-calculated whenever the Corefile is reloaded.
Running our test pod, we can see this works:
$ kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
If you don't see a command prompt, try pressing enter.
/ # host foo
foo.default.svc.cluster.local has address 10.0.0.72
/ # host foo.example.com
foo.example.com has address 10.0.0.72
/ # host bar.example.com
Host bar.example.com not found: 3(NXDOMAIN)
/ #

Related

Kubernetes Nginx ingress appears to remove headers before sending requests to backend service

I am trying to deploy a SpringbBoot Java application hosted on Apache Tomcat on a Kubernetes cluster using Ngninx Ingress for URL routing. More specifically, I am deploying on Minikube on my local machine, exposing the SpringBoot application as a Cluster IP Service, and executing the
minikube tunnel
command to expose services to my local machine. Visually, the process is the following...
Browser -> Ingress -> Service -> Pod hosting docker container of Apache Tomcat Server housing SpringBoot Java API
The backend service requires a header "SM_USER", which for the moment can be any value. When running the backend application as a Docker container with port forwarding, accessing the backend API works great. However, when deploying to a Kubernetes cluster behind an Nginx ingress, I get 403 errors from the API stating that the SM_USER header is missing. I suspect the following. My guess is that the header is included with the request to the ingress, but removed when being routed to the backend service.
My setup is the following.
Deploying on Minikube
minikube start
minikube addons enable ingress
eval $(minikube docker-env)
docker build . -t api -f Extras/DockerfileAPI
docker build . -t ui -f Extras/DockerfileUI
kubectl apply -f Extras/deployments-api.yml
kubectl apply -f Extras/deployments-ui.yml
kubectl expose deployment ui --type=ClusterIP --port=8080
kubectl expose deployment api --type=ClusterIP --port=8080
kubectl apply -f Extras/ingress.yml
kubectl apply -f Extras/ingress-config.yml
edit /etc/hosts file to resolve mydomain.com to localhost
minikube tunnel
Ingress YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
# Tried forcing the header by manual addition, no luck (tried with and without)
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header SM_USER $http_Admin;
spec:
rules:
- host: mydomain.com
http:
paths:
- path: /ui
pathType: Prefix
backend:
service:
name: ui
port:
number: 8080
- path: /api
pathType: Prefix
backend:
service:
name: api
port:
number: 8080
Ingress Config YAML
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: default
labels:
app.kubernetes.io/name: my-ingress
app.kubernetes.io/part-of: my-ingress
data:
# Tried allowing underscores and also using proxy protocol (tried with and without)
enable-underscores-in-headers: "true"
use-proxy-protocol: "true"
When navigating to mydomain.com/api, I expect to receive the ROOT API interface but instead receive the 403 error page indicating that the SM_USER is missing. Note, this is not a 403 forbidden error regarding the ingress or access from outside the cluster, the specific SpringBoot error page I receive is from within my application and custom to indicate that the header is missing. In other words, my routing is definitely correct, and I am able to access the API, its just that the header is missing.
Are there configs or parameters I am missing? Possibly an annotation?
This is resolved. The issue was that the config was being applied in the Application namespace. Note, even if the Ingress object is in the application namespace, if you are using minikubes built in ingress functionality, the ConfigMap must be applied in the Nginx ingress namespace.

Is there a reversal of `ExternalName` service in Kubernetes?

I'm trying to move my development environment to Kubernetes to be more in line with existing deployment stages. In that context I need to call a service by its Ingress DNS name internally, while this DNS name resolves to an IP unreachable from the cluster itself. I would like to create a DNS alias inside the cluster which would point to the service, basically a reversal of a ExternalName service.
Example:
The external DNS name is my-service.my-domain.local, resolving to 127.0.0.1
Internal service is my-service.my-namespace.svc.cluster.local
A process running in a pod can't reach my-service.my-domain.local because of the resolved IP, but could reach my-service.my-namespace.svc.cluster.local, but needs to be accessing the former by name
I would like to have a cluster-internal DNS name my-service.my-domain.local, resolving to the service my-service.my-namespace.svc.cluster.local (ExternalName service would do the exact opposite).
Is there a way to implement this in Kubernetes?
You can use the core dns and add the entry over there using configmap
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
labels:
eks.amazonaws.com/component: coredns
k8s-app: kube-dns
name: coredns
namespace: kube-system
data:
Corefile: |
domain-name:port {
errors
cache 30
forward . <IP or custom DNS>
reload
}
To test you can start one busy box pod
kubectl run busybox --restart=Never --image=busybox:1.28 -- sleep 3600
hit the domain name from inside of busy box
kubectl exec busybox -- nslookup domain-name
Official doc ref : https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
Nice article for ref : https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/
Or
you can map the domain to the service name using rewrite, rewrite name example.io service.default.svc.cluster.local
Use the Rewrite plug-in of CoreDNS to resolve a specified domain name
to the domain name of a Service.
apiVersion: v1
data:
Corefile: |-
.:5353 {
bind {$POD_IP}
cache 30
errors
health {$POD_IP}:8080
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
rewrite name example.io service.default.svc.cluster.local
loadbalance round_robin
prometheus {$POD_IP}:9153
forward . /etc/resolv.conf
reload
}
kind: ConfigMap
metadata:
labels:
app: coredns
k8s-app: coredns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: CoreDNS
release: cceaddon-coredns
name: coredns
namespace: kube-system

ingress nginx how to debug 502 page even though the ports in service and Ingress are correct?

i have a web application running inside cluster ip on worker node on port 5001,i'm also using k3s for cluster deployment, i checked the cluster connection it's running fine
the deployment has the container port set to 5001:
ports:
- containerPort:5001
Here is the service file:
apiVersion: v1
kind: Service
metadata:
labels:
io.kompose.service: user-ms
name: user-ms
spec:
ports:
- name: http
port: 80
targetPort: 5001
selector:
io.kompose.service: user-ms
status:
loadBalancer: {}
and here is the ingress file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: user-ms-ingress
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: user-ms
port:
number: 80
i'm getting 502 Bad Gateway error whenever i type in my worker or master ip address
expected: it should return the web application page
i looked online and most of them mention wrong port for service and ingress, but my ports are correct yes i triple check it:
try calling user-ms service on port 80 from another pod -> worked try
calling cluster ip on worker node on port 5001 -> worked
the ports are running correct, why is the ingress returning 502?
here is the ingress describe:
and here is the describe of nginx ingress controller pod:
the nginx ingress pod running normally:
here is the logs of the nginx ingress pod:
sorry for the images, but i'm using a streaming machine to access the terminal so i can't copy paste
How should i go with debugging this error?
ok i managed to figure out this, in the default setting of K3S it uses traefik as it default ingress, so that why my nginx ingress log doesn't show anything from 502 Bad GateWay
I decided to tear down my cluster and set it up again, now with suggestion from this issue https://github.com/k3s-io/k3s/issues/1160#issuecomment-1058846505 to create cluster without traefik:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -
now when i call kubectl get pods --all-namespaces i no longer see traefik pod running, previously it had traefik pods runining.
once i done all of it, run apply on ingress once again -> get 404 error, i checked in the nginx ingress pod logs now it's showing new error of missing Ingress class, i add the following to my ingress configuration file under metadata:
metadata:
name: user-ms-ingress
annotitations:
kubernetes.io/ingress.class: "nginx"
now i once more go to the ip of the worker node -> 404 error gone but got 502 bad gateway error, i checked the logs get connection refused errors:
i figured out that i was setting a network policy for all of my micro services, i delete the network policy and remove it's setting from all my deployment files.
Finally check once more and i can see that i can access my api and swagger page normally.
TLDR:
If you are using nginx ingress on K3S, remember to disable traefik first when created the cluster
don't forget to set ingress class inside your ingress configuration
don't setup network policy because nginx pod won't be able to call the other pods in that network
You can turn on access logging on nginx, which will enable you to see more logs on ingress-controller and also trace every requests routing through ingress, if you are trying to load UI/etc, it will show you that the requests are coming in from browser or if you accessing a particular endpoint, the calls will be visible on the nginx-controller logs. You can conclude, if the requests coming in are actually routing to the proper service using this and then start debugging the service (ex: check to see if you can curl the endpoint from any pod within the cluster etc)
Noticed that you are using the image(k8s.gcr.io/ingress-nginx/controller:v1.2.0), if you have installed using helm, there must be a kubernetes-ingress configmap with ingress controller, by default "disable-access-log" will be true, change it false and you should start seeing more logs on ingress-controller, you might want to bounce ingress controller pods if you do not see detailed logs.
Kubectl edit cm -n namespace kubernetes-ingress
apiVersion: v1
data:
disable-access-log: "false" #turn this to false
map-hash-bucket-size: "128"
ssl-protocols: SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3
kind: ConfigMap

Exposing simple application in gcloud with Kubernetes

I have a simple app which I am following from a basic tutorial :
https://medium.com/#bhargavshah2011/hello-world-on-kubernetes-cluster-6bec6f4b1bfd
I create a cluster hello-world2-cluster
I "connect" to the cluster using :
gcloud container clusters get-credentials hello-world2-cluster --zone us-central1-c --project strange-vortex-286312
I perform a git clone of the "hellow world" project, which is basically some kind of snake game :
$ git clone https://github.com/skynet86/hello-world-k8s.git
$ cd hello-world-k8s/
Next I run "create" on the YAML:
kubectl create -f hello-world.yaml
Deplyments and services are created as expected :
Fine! ALl good!
But.... now what????
What do I do now?
I want to access this app from the outside world. Where is the URL where I can access the application?
How can I get my friend Bob in Timbuktu to call some kind of URL (no need for DNS) to access my application?
I know the answer has something to do with LoadBalancer nodes and Ingress, but there seems to be no proper documentation out there on how to do this for a simple hello world app.
GKE hello-world with Ingress Official Example: https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress
The ingress controller is built-in in GKE, you just need a kind: Ingress object that points to your Service. Addressing your question:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-ingress
annotations:
kubernetes.io/ingress.class: "gce"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: hello-world
servicePort: 80
Once deployed, query kubectl get ingress hello-ingress -oyaml to find out the resulting load-balancer's IP in the status: field.
If you don't want to set up a DNS and/or an ingress, you can simply do it with what you already have using the node external ip.
It seems that you already have the deployement and the NodePort service is mapped to port 30081.
So, below are the remaining steps to make it work with what you have :
Get your node IP :
kubectl get nodes -o wide
Now that you have your node external IP, you need to allow tcp traffic to this port
gcloud compute firewall-rules create allow-tcp-30081 --allow tcp:30081
That's it ! You can just tell you're friend Bob that the service should be accessible at node-ip:30081
You can have a look at this page to see the other possibilities depending on your service type.
Also, when you use a NodePort the port is mapped to all the nodes of your cluster, so you can use any IP. No need to look for the node where your replicas is deployed.
If you don't want to use Ingress or anything. You can expose your deployment as LoadBalancer then using external IP your friend bob can access application. So to use
kubectl expose deployment hello-world-deployment --name=hello-world-deployment --type=LoadBalancer --port 80 --target-port 80
This will take around 1 minute to assign an external IP address to the service.
to get external IP use
kubectl get services
Get the External IP address in-front of your service and open your browser and enter the IP address that you copied.
Done !! This is the easiest and simplest way to get start with !!
You can expose your Kubernetes services to the internet with an Ingress.
GKE allows you to use different kinds of Ingress (nginx based, and so on) but, by default, it will expose your service through the Cloud Load Balancing service.
As indicated by #MaxLobur, you can configure this default Ingress applying a configuration file similar to this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
spec:
backend:
serviceName: hello-world
servicePort: 80
Or, equivalently, if you want to prepare your Ingress for various backend services and configure it based on rules:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: hello-world
servicePort: 80
This Ingress will be assigned an external IP address.
You can find which with the following command:
kubectl get ingress hello-world-ingress
This command will output something similar to the following:
NAME HOSTS ADDRESS PORTS AGE
hello-world-ingress * 203.0.113.12 80 2m
But, if you give this IP address to your friend Bob in Timbuktu, he will call you back soon telling you that he cannot access the application...
Now seriously: this external IP address assigned by GCP is temporary, an ephemeral one subject to change.
If you need a deterministic, fixed IP address for your application, you need to configure a static IP for your Ingress.
This will also allow you to configure, if necessary, a DNS record and SSL certificates for your application.
In GCP, the first step in this process is reserving a static IP address. You can configure it in the GCP console, or by issuing the following gcloud command:
gcloud compute addresses create hello-world-static-ip --global
This IP should be later assigned to your Ingress by modifying its configuration file:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "hello-world-static-ip"
spec:
backend:
serviceName: hello-world
servicePort: 80
This static IP approach only adds value because as long as it is associated with a Google Cloud network resource, you will not be charged for it.
All this process is documented in this GCP documentation page.

Nginx Ingress returns 502 Bad Gateway on Kubernetes

I have a Kubernetes cluster deployed on AWS (EKS). I deployed the cluster using the “eksctl” command line tool. I’m trying to deploy a Dash python app on the cluster without success. The default port for Dash is 8050. For the deployment I used the following resources:
pod
service (ClusterIP type)
ingress
You can check the resource configuration files below:
pod-configuration-file.yml
kind: Pod
apiVersion: v1
metadata:
name: dashboard-app
labels:
app: dashboard
spec:
containers:
- name: dashboard
image: my_image_from_ecr
ports:
- containerPort: 8050
service-configuration-file.yml
kind: Service
apiVersion: v1
metadata:
name: dashboard-service
spec:
selector:
app: dashboard
ports:
- port: 8050 # exposed port
targetPort: 8050
ingress-configuration-file.yml (host based routing)
kind: Ingress
metadata:
name: dashboard-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: dashboard.my_domain
http:
paths:
- backend:
serviceName: dashboard-service
servicePort: 8050
path: /
I followed the steps below:
kubectl apply -f pod-configuration-file.yml
kubectl apply -f service-configuration-file.yml
kubectl apply -f ingress-confguration-file.yml
I also noticed that the pod deployment works as expected:
kubectl logs my_pod:
and the output is:
Dash is running on http://127.0.0.1:8050/
Warning: This is a development server. Do not use app.run_server
in production, use a production WSGI server like gunicorn instead.
* Serving Flask app "annotation_analysis" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
You can see from the ingress configuration file that I want to do host based routing using my domain. For this to work, I have also deployed an nginx-ingress. I have also created an “A” record set using Route53
that maps the “dashboard.my_domain” to the nginx-ingress:
kubectl get ingress
and the output is:
NAME HOSTS ADDRESS. PORTS. AGE
dashboard-ingress dashboard.my_domain nginx-ingress.elb.aws-region.amazonaws.com 80 93s
Moreover,
kubectl describe ingress dashboard-ingress
and the output is:
Name: dashboard-ingress
Namespace: default
Address: nginx-ingress.elb.aws-region.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
host.my-domain
/ dashboard-service:8050 (192.168.36.42:8050)
Annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: false
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: false
Events: <none>
Unfortunately, when I try to access the Dash app on the browser, I get a
502 Bad Gateway error from the nginx. Could you please help me because my Kubernetes knowledge is limited.
Thanks in advance.
It had nothing to do with Kubernetes or AWS settings. I had to change my python Dash code from:
if __name__ == "__main__":
app.run_server(debug=True)
to:
if __name__ == "__main__":
app.run_server(host='0.0.0.0',debug=True).
The addition of host='0.0.0.0' did the trick!
I think you'll need to check whether any other service is exposed at path / on the same host.
Secondly, try removing rewrite-target annotation. Also can you please update your question with output of kubectl describe ingress <ingress_Name>
I would also suggest you to use backend-protocol annotation with value as HTTP (default value, you can avoid using this if dashboard application is not SSL Configured, and only this application is served at the said host.) But, you may need to add this if multiple applications are served at this host, and create one Ingress with backend-protocol: HTTP for non SSL services, and another with backend-protocol: HTTPS to serve traffic to SSL enabled services.
For more information on backend-protocol annotation, kindly refer this link.
I have often faced this issue in my Ingress Setup and these steps have helped me resolve it.