I am struggling to get my nginx ingress (on AWS EKS) working with path rules and TLS.
The ingress is from
here
A snippet from the Ingress looks like:
spec:
tls:
- hosts:
- example.com
secretName: ingress-tls
rules:
- host: example.com
- http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 443
This ingress creates the AWS network load balancer, with a URL like
https://xyz.elb.us-west-1.amazonaws.com/
I am updating the
ingress-tls
secret with a certificate using
cert-manager.
When I access the ingress using the NLB URL
https://xyz.elb.us-west-1.amazonaws.com/api, I get
GOOD: Correct routing based on the path rules from the ingress definition (i.e. it goes to my
api-service as expected)
BAD: Certificate errors since I'm not accessing the ingress with the domain that the certificate is for.
When I access the ingress using the correct domain e.g.
https://example.com/api which is what I want to do, I get:
BAD:
404, it doesn't respect my path rules, and goes to
upstream-default-backend instead.
GOOD: certificate all good, it’s the one for
example.com that
cert-manager configured.
I tried removing the
host: example.com from the
rules:, which gives me:
GOOD: Correct routing based on the path rules from the ingress definition
BAD: Certificate errors, it serves up the default ingress “Fake” certificate instead of the one for
example.com, I guess since the
host is missing from the rules, though not sure of the exact reason.
Can someone please help me get
GOOD
GOOD
I’m at a loss here.
After staring at this for several more hours, and digging through the nasty chunk of lua that is the
nginx.conf for this, I found it! Maybe someday someone will have this problem, and might find this useful.
The problem was:
rules:
- host: example.com
- http:
This is defining (I think) a
host with no forwarding rules, then then some
http forwarding rules without a host. What I had intended was obviously that the forwarding rules would be for the host.
And that would be:
rules:
- host: example.com
http:
I have to say that I'm now even less of a fan of YAML than I was previously, if that's even possible.
Related
I've deployed the Duende IdentityServer to Okteto Cloud: https://id6-jeff-tian.cloud.okteto.net/.
Although the endpoint is https from the outside, the inside pods still think they are behind HTTP protocol. You can check the discovery endpoint to find out: https://id6-jeff-tian.cloud.okteto.net/.well-known/openid-configuration
That causes issues during some redirecting. So how to let the inner pods know that they are hosted in https scheme?
Can we pass some headers to the IdP to tell it the original https schema?
These headers should be forwarded to the inner pods:
X-Forwarded-For: Holds information about the client that initiated the request and subsequent proxies in a chain of proxies. This parameter may contain IP addresses and, optionally, port numbers.
X-Forwarded-Proto: The value of the original scheme, should be https in this case.
X-Forwarded-Host: The original value of the Host header field.
I searched from some aspnet documentations and found this: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?source=recommendations&view=aspnetcore-6.0, however, I don't know how to configure the headers in Okteto, or in any k8s cluster.
Is there anyone who can shed some light here?
My ingress configurations is as follows (https://github.com/Jeff-Tian/IdentityServer/blob/main/k8s/app/ingress.yaml):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: id6
annotations:
dev.okteto.com/generate-host: id6
spec:
rules:
- http:
paths:
- backend:
service:
name: id6
port:
number: 80
path: /
pathType: ImplementationSpecific
The headers that you mention are being added to the request when it’s forwarded to your pods.
Could you dump the headers on the receiving end?
Not familiar with Duende, but does it have a setting to specify the “public URL”? That’s typically what I’ve done in the past for similar setups.
I am fairly new to Kubernetes and have just deployed my first cluster to IBM Cloud. When I created the cluster, I get a dedicated ingress subdomain, which I will be referring to as <long-k8subdomain>.cloud for the scope of this post. Now, this subdomain works for my app. For example: <long-k8subdomain>.cloud/ping works from my browser/curl just fine- I get the expected JSON response back. But, if I add this subdomain to a CNAME record on my domain provider's DNS settings (I have used Bluehost and IBM Cloud's Internet Services), I get a 404 response back from all routes. However this response is the default nginx 404 response (it says "nginx" under "404 Not Found"). I believe this means that this means the ingress load balancer is being reached, but the request does not get routed right. I am using Kubernetes version 1.20.12_1561 on VPC gen 2 and this is my ingress-config.yaml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Host: <long-k8subdomain>.cloud";
spec:
rules:
- host: <long-k8subdomain>.cloud
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
I am pretty sure this problem is due to the annotations. Maybe I am using the wrong ones or I do not have enough. Ideally, I would like something like this: api..com/ to route correctly. I have also read a little bit about default backends, but I have not dove too much into that just yet. Any help would be greatly appreciated, as I have spent multiple hours trying to fix this.
Some sources I have used:
https://cloud.ibm.com/docs/containers?topic=containers-cs_network_planning
https://cloud.ibm.com/docs/containers?topic=containers-ingress-types
https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotations#annotations
Note: The reason why I have the second annotation is because for some reason, requests without that header were not being routed directly. So that was part of my debugging process and I just ended up leaving it as I am not sure if that annotation solves that, so I left it for now.
For the NGINX ingress controller to route requests for your own domain's CNAME record to the service instead of the IBM Cloud one, you need a rule in the ingress where the host identifies your domain.
For instance, if your domain's DNS entry is api.example.com, then change the resource YAML to:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
You should not need the second annotation for this to work.
If you want both of the hosts to work, then you could add a second rule instead of replacing host in the existing one.
I have a K8s setup with traefik being exposed like this
kubernetes:
ingressClass: traefik
service:
nodePorts:
http: 32080
serviceType: NodePort
Behind, I forward some requests to different services
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-name
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: my-host.com
http:
paths:
- path: /my-first-path
backend:
serviceName: my-nodeJs-services
servicePort: 3000
When the DNS is set directly to resolve to my ip, the application works fine with HTTP
http://my-host.com:32080/my-first-path
But when some one add SSL through AWS ALB / API Gateway, the application fail to be reached with 404-NotFound error
The route is like this
https://my-host.com/my-first-path
On the AWS size, they configured something like this
https://my-host.com => SSL Termination and => Forward all to 43.43.43.43:32080
I think this fail because traefik is expecting http://my-host.com but not https://my-host.com which lead to its failure to find the matching route? Or maybe at the ssl termination time, the hostname is lost so that traefik can not find a route?
What should I do in this situation?
I am not very familiar with ALB but what is probably happening is that the requests received by the loadbalancer contain the header Host: my-host.com and when it gets forwarded to your ingress controller, the header is replaced by Host: 43.43.43.43. If this is the case, I see 3 solutions:
ALB might be able to pass the original Host header to the target. (You will have to check in the doc if it's possible)
If the application behind your ingress doesn't check the host header, you can write an ingress that doesn't check a specific host. For example on these examples you can see that the host field is not specified.
If the name resolution works internally, you can define a name for your target, use this name in your ALB and in your ingress.
I have a subset of paths that I expose with an ingress.
I'd like to block them for anyone coming through the ingress.
I'm trying this with a GCE Ingress
Adding a rule like:
- host: my.example.com
http:
paths:
- backend:
serviceName: dead-end-backend
servicePort: 80
path: /private
This backend is designated dead-end using Nginx default backend deployment/service but it's not working well.
I'm not asking how to use the default-backend (but it solve it).
I'm asking for a proper way to do this.
I'm not sure it's the best thing to do but what I did was use a "default-backend" service which returns error for anything.
Then I added each route I wanted to block to get to that default backend.
Like the saying - if it's silly, but it's working, then it's not silly.
I'm trying to use cert-manager to issue a certificate via LetsEncrypt.
I've followed through with the steps here http://docs.cert-manager.io/en/latest/getting-started/index.html
However, my existing ingress is not being modified (I assume it needs to modify it due to adding a path for .well-known/....
Instead I see an ingress created for this with a name like: cm-acme-http-solver-kgpz6? Which is rather confusing?
If I get the yaml for that ingress I see the following for rules:
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: cm-acme-http-solver-2dd97
servicePort: 8089
path: /.well-known/acme-challenge/2T2D_XK1-zIJJ9_f2ANlwR-AcNTm3-WenOExNpmUytY
How exactly is this meant to work? As the documentation seems rather sparse.
The record you are seeing is for the challenge. It needs to succeed to configure the cert. If you are using "example.com" as the domain then it will not succeed. To get this to work you'll need to configure a DNS record for a valid hostname so that LetsEncrypt can resolve the domain and complete the check.
Usually you will not even see the challenge ingress resource. It usually runs the challenge and then removes itself as long as DNS and the hostname have been configured correctly. After it is removed the resource you created will get loaded into your ingress controller.
There are a few ingress controllers that do not support multiple ingress resources per hostname. They will load one ingress resource and ignore the other, so this is sort of a workaround/fix to the issue.