Unable to reach the URL AWS provides for my LoadBalancer service in EKS - kubernetes

Why is my LoadBalancer service in Kubernetes not reachable?
I have deployed an nginx-ingress-controller helm chart and it has a service of LoadBalancer type in EKS. This service receives a url (EXTERNAL-IP) and this url has an IP but when I'm trying the reach this url it's not reachable.
I did kubectl port-forward -n ingress-nginx services/ingress-nginx-controller8080:80 and then I can reach nginx in localhost:8080 so I know the problem is to reach the service itself from the internet. I've checked and VPC and subnets security-groups and inbound/outbound rules and it seems ok. .
can anyone provide some guidance on how to troubleshoot this issue?
This is the definition of the
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:xxx:certificate/xxx
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-type: nlb
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.5.1
helm.sh/chart: ingress-nginx-4.4.2
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
allocateLoadBalancerNodePorts: true
clusterIP: xxx
clusterIPs:
- xxx
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: xxx
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: xxx
port: 443
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
sessionAffinity: None
type: LoadBalancer
This is the command I'm using to deploy the nginx helm chart:
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace --version 4.4.2 -f values.yaml
And this is my values.yaml:
controller:
config:
allow-snippet-annotations: "true"
http-snippet: |
server {
listen 2443;
return 308 https://$host$request_uri;
}
use-forwarded-headers: "false"
service:
enabled: true
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:xxx:certificate/xxx
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-type: nlb
ports:
http: 80
https: 443
targetPorts:
http: http
https: http
type: LoadBalancer
to reach the nginx service I either go to it on browser or just do:
curl xxx-xxx.elb.eu-central-1.amazonaws.com
but I always get This site can’t be reached

First and foremost: give up. Nginx controller won't just work with ACM properly, I've wasted enourmous hours to accept this and move on.
Now that you did, here's approach I employed just yesterday and it worked brilliant.
Install nginx controller as helm release, without making any changes to controller service, this will create CLB (instead of NLB) and this is fine. NLB is mentioned in the guides on the internet as a crutch to get ACM certificate working, but all it does is produces redirect loops.
MOST IMPORTANT - Please, go through this to install cert-manager to manage LE certificates for you - ACM won't just let you export certificate. If you have external one already - good, just put it as secret for TLS definition below, otherwise after installation of cert-manager (if it takes too long to install helm for it, you didn't specify true for CRD installation - THIS IS CRUCIAL and if it gets stuck you will need to uninstall release and retry again properly).
Here's example of nginx ingress you can adapt to your needs:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: yeaboi-lb
namespace: yeaboi
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
rewrite ^/(/?)$ /yeaboi$1 break;
nginx.ingress.kubernetes.io/ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt"
acme.cert-manager.io/http01-edit-in-place: "true"
spec:
tls:
- secretName: yeaboi-tls
hosts:
- yeaboi.io
rules:
- host: yeaboi.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: yeaboi-web
port:
number: 80
ingressClassName: nginx
Here's example of yeaboi-web service you can use in conjunction with abovementioned ingress (obviously make sure to specify targetPort exposed for your deployment):
apiVersion: v1
kind: Service
metadata:
name: yeaboi-web
namespace: yeaboi
labels:
app: yeaboi-web
spec:
ports:
- port: 80
targetPort: 9000
protocol: TCP
selector:
app: yeaboi-web
Point your domain to CLB created by nginx (just in case make sure it has your EC2 in inService status etc).
Enjoy having working nginx ingress you can customize a lot (unlike ALB ingress which is severely limited in comparison).

Related

Helm - how to set custom service port for ingress controller [duplicate]

I don't mean being able to route to a specific port, I mean to actually change the port the ingress listens on.
Is this possible? How? Where is this documented?
No. From the kubernetes documentation:
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.
It may be possible to customize a LoadBalancer on a cloud provider like AWS to listen on other ports.
I assume you are using NGINX Ingress Controller. In this case, during installation, instead of doing a kubectl apply in the official yaml like this is one, you can try downloading the yaml and changing the port. The file above, which is used for an L4 AWS ELB, would become like this:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- port: {custom port 1}
targetPort: http
- port: {custom port 2}
targetPort: https
An alternative is to use a more powerful ingress controller.
Here is a list of different controllers.
My personal choice is Ambassador. If you follow the getting-started page, you just need to change the service definition for the port of your choice:
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: {custom port}
targetPort: 8080
selector:
service: ambassador
An Ingress definition is backed by an ingress controller. The ingress controller is deployed with normal Kubernetes objects so will have a Service associated with it that exposes ports for the ingress controller.
The kubernetes/ingress-nginx static deploys have a deploy.yaml with a Service type LoadBalancer:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Modify the ports the load balancer is configured with, in spec.ports[*].port in the external service, however that is deployed.
If you're using Helm to deploy the Kubernetes ingress-nginx controller, you can do this to change the ports from their defaults of 80 and 443 in a Helm values override file:
ingress-nginx:
enabled: true
...
controller:
service:
ports:
http: 8123
https: 9456
See here: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#values
In this example, any services exposed through the ingress are now available on ports 8123 (http) and 9456 (https).

Ingress -> Cluster IP back-end - got ERR_CONNECTION_REFUSED

I got ingress defined as:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wp-ingress
namespace: wordpress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 25m
spec:
rules:
- host: my.domain.com
http:
paths:
- path: /
backend:
serviceName: wordpress
servicePort: 6002
And the back-end, defined as Cluster IP running on 6002 port.
When I am trying to reach ingress by its ADDRESS in the browser I get ERR_CONNECTION_REFUSED.
I suspect it has to do with the back-end?
Q: What could be a problem? How to analyse it? to make it work.
See the picture below, it is on GCP, all ips are resolved. All seems connected to each other.
The nginx-ingress (ingress-controller, default backed) was installed as helm chart.
helm install --namespace wordpress --name wp-nginx-ingress stable/nginx-ingress --tls
UPDATE:
I do not use yet (https) for back-end, tried to remove http redirect from the ingress yml: nginx.ingress.kubernetes.io/ssl-redirect: "true" removed - did not help.
UPDATE2: wordpress yaml - got from running service at yaml tab in GCP->KE->Services
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-03-30T04:11:12Z"
labels:
app.kubernetes.io/instance: wordpress
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: wordpress
helm.sh/chart: wordpress-9.0.4
name: wordpress
namespace: wordpress
resourceVersion: "2518308"
selfLink: /api/v1/namespaces/wordpress/services/wordpress
uid: 7dac1a73-723c-11ea-af1a-42010a800084
spec:
clusterIP: xxx.xx.xxx.xx
ports:
- name: http
port: 6002
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/instance: wordpress
app.kubernetes.io/name: wordpress
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
UPDATE: 3
I tried:
kubectl -n wordpress exec -it wordpress-xxxx-xxxx -- /bin/bash
curl http://wordpress.wordpress.svc.cluster.local:6002 and it works - it gets me the html from the wordpress.

Bare metal Nginx controller with Services as Node port not working

Hi i have use Nginx controller both community and Nginx repo both works for in cloud with LoadBalancer service. Where as Nginx controller both community and Nginx repo both dose not work with service Nodeport "error 404".
404 Not Found
Service Type Nodeport
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
The default behavior of a NGINX Ingress Controller is to reply with a default backed that replies with a 404 - default backend: basically, it's a running Deployment resource returning that default page.
You can try overriding it passing to Helm a variable override using the path defaultBackend.image.repository as described in the README

Can I set custom ports for a Kubernetes ingress to listen on besides 80 / 443?

I don't mean being able to route to a specific port, I mean to actually change the port the ingress listens on.
Is this possible? How? Where is this documented?
No. From the kubernetes documentation:
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.
It may be possible to customize a LoadBalancer on a cloud provider like AWS to listen on other ports.
I assume you are using NGINX Ingress Controller. In this case, during installation, instead of doing a kubectl apply in the official yaml like this is one, you can try downloading the yaml and changing the port. The file above, which is used for an L4 AWS ELB, would become like this:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- port: {custom port 1}
targetPort: http
- port: {custom port 2}
targetPort: https
An alternative is to use a more powerful ingress controller.
Here is a list of different controllers.
My personal choice is Ambassador. If you follow the getting-started page, you just need to change the service definition for the port of your choice:
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: {custom port}
targetPort: 8080
selector:
service: ambassador
An Ingress definition is backed by an ingress controller. The ingress controller is deployed with normal Kubernetes objects so will have a Service associated with it that exposes ports for the ingress controller.
The kubernetes/ingress-nginx static deploys have a deploy.yaml with a Service type LoadBalancer:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Modify the ports the load balancer is configured with, in spec.ports[*].port in the external service, however that is deployed.
If you're using Helm to deploy the Kubernetes ingress-nginx controller, you can do this to change the ports from their defaults of 80 and 443 in a Helm values override file:
ingress-nginx:
enabled: true
...
controller:
service:
ports:
http: 8123
https: 9456
See here: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx#values
In this example, any services exposed through the ingress are now available on ports 8123 (http) and 9456 (https).

How to get around specifying "Host" header to access services pointed by Ingress controllers?

My ingress controller is working and I am able to access the service outside the cluster using http://(externalIP)/path using an HTTP GET request from a RestClient. However, I had to specify the "Host" header with value = "host" (value of my Ingress Resource) for this to work. Because of this, I am not able to hit http://(externalIP)/path from my web browser. Is there any way I can enable access from my external web browser without having to specify "Host" in the request header?
Ingress Resource :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
namespace: ingress-nginx
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: master1.saas.com
http:
paths:
- backend:
serviceName: gen-devops
servicePort: 10311
path: /*
Ingress Service :
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
externalIPs:
- 172.16.32.85
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
I assume you want to use this for testing.
If you are using any *nix flavor OS (MacOS, Linux) you can add an entry to your /etc/hosts file, something like this:
172.16.32.85 master1.saas.com
If you are using any Windows box you can add the same entry in C:\Windows\System32\Drivers\etc\hosts