I've deployed the Duende IdentityServer to Okteto Cloud: https://id6-jeff-tian.cloud.okteto.net/.
Although the endpoint is https from the outside, the inside pods still think they are behind HTTP protocol. You can check the discovery endpoint to find out: https://id6-jeff-tian.cloud.okteto.net/.well-known/openid-configuration
That causes issues during some redirecting. So how to let the inner pods know that they are hosted in https scheme?
Can we pass some headers to the IdP to tell it the original https schema?
These headers should be forwarded to the inner pods:
X-Forwarded-For: Holds information about the client that initiated the request and subsequent proxies in a chain of proxies. This parameter may contain IP addresses and, optionally, port numbers.
X-Forwarded-Proto: The value of the original scheme, should be https in this case.
X-Forwarded-Host: The original value of the Host header field.
I searched from some aspnet documentations and found this: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?source=recommendations&view=aspnetcore-6.0, however, I don't know how to configure the headers in Okteto, or in any k8s cluster.
Is there anyone who can shed some light here?
My ingress configurations is as follows (https://github.com/Jeff-Tian/IdentityServer/blob/main/k8s/app/ingress.yaml):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: id6
annotations:
dev.okteto.com/generate-host: id6
spec:
rules:
- http:
paths:
- backend:
service:
name: id6
port:
number: 80
path: /
pathType: ImplementationSpecific
The headers that you mention are being added to the request when it’s forwarded to your pods.
Could you dump the headers on the receiving end?
Not familiar with Duende, but does it have a setting to specify the “public URL”? That’s typically what I’ve done in the past for similar setups.
Related
I'm fairly new to Kubernetes and I have played around with it for a few days now to get a feeling for it. Trying out to set up an Nginx Ingress controller on the google-cloud platform following this guide, I was able to set everything up as written there - no problems, I got to see the hello-app output.
However, when I tried replicating this in a slightly different way, I encountered a weird behavior that I am not able to resolve. Instead of using the image --image=gcr.io/google-samples/hello-app:1.0 (as done in the tutorial) I wanted to deploy a standard nginx container with a custom index page to see if I understood stuff correctly. As far as I can tell, all the steps should be the same except for the exposed port: While the hello-app exposes port 8080 the standard port for the nginx container is 80. So, naively, I thought exposing (i.e., creating a service) with this altered command should do the trick:
kubectl expose deployment hello-app --port=8080 --target-port=80
where instead of having target-port=8080 as for the hello-app, I put target-port=80. As far as I can tell, all other thins should stay the same, right? In any way, this does not work and when I try to access the page I get a "404 - Not Found" although the container is definitely running and serving the index page (I checked by using port forwarding from the google cloud which apparently directly makes the page accessible for dev purposes). In fact, I also tried several other combinations of ports (although I believe the above one should be the correct one) to no avail. Can anyone explain to my why the routing does not work here?
If you notice the tutorial inside the ingress configuration path: "/hello"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-resource
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: "34.122.88.204.nip.io"
http:
paths:
- pathType: Prefix
path: "/hello"
backend:
service:
name: hello-app
port:
number: 8080
you might have updated port number and service name config however if path /hello which means you request is going to Nginx container but not able to file the page hello.html so it's giving you 404.
You hit endpoint IP/hello (Goes to Nginx ingress controller)-->
checked for path /hello and forwarded request to service -->
hello-app (service forwarded request to PODs) --> Nginx POD (it
doesn't have anything at path /hello so 404)
404 written by Nginx side, in your case either it will be Nginx ingress controller or else container(POD) itself.
So try you ingress config without setting path path: "/" and hit the endpoint you might see the output from Nginx.
I am fairly new to Kubernetes and have just deployed my first cluster to IBM Cloud. When I created the cluster, I get a dedicated ingress subdomain, which I will be referring to as <long-k8subdomain>.cloud for the scope of this post. Now, this subdomain works for my app. For example: <long-k8subdomain>.cloud/ping works from my browser/curl just fine- I get the expected JSON response back. But, if I add this subdomain to a CNAME record on my domain provider's DNS settings (I have used Bluehost and IBM Cloud's Internet Services), I get a 404 response back from all routes. However this response is the default nginx 404 response (it says "nginx" under "404 Not Found"). I believe this means that this means the ingress load balancer is being reached, but the request does not get routed right. I am using Kubernetes version 1.20.12_1561 on VPC gen 2 and this is my ingress-config.yaml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Host: <long-k8subdomain>.cloud";
spec:
rules:
- host: <long-k8subdomain>.cloud
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
I am pretty sure this problem is due to the annotations. Maybe I am using the wrong ones or I do not have enough. Ideally, I would like something like this: api..com/ to route correctly. I have also read a little bit about default backends, but I have not dove too much into that just yet. Any help would be greatly appreciated, as I have spent multiple hours trying to fix this.
Some sources I have used:
https://cloud.ibm.com/docs/containers?topic=containers-cs_network_planning
https://cloud.ibm.com/docs/containers?topic=containers-ingress-types
https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotations#annotations
Note: The reason why I have the second annotation is because for some reason, requests without that header were not being routed directly. So that was part of my debugging process and I just ended up leaving it as I am not sure if that annotation solves that, so I left it for now.
For the NGINX ingress controller to route requests for your own domain's CNAME record to the service instead of the IBM Cloud one, you need a rule in the ingress where the host identifies your domain.
For instance, if your domain's DNS entry is api.example.com, then change the resource YAML to:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
You should not need the second annotation for this to work.
If you want both of the hosts to work, then you could add a second rule instead of replacing host in the existing one.
I am struggling to get my nginx ingress (on AWS EKS) working with path rules and TLS.
The ingress is from
here
A snippet from the Ingress looks like:
spec:
tls:
- hosts:
- example.com
secretName: ingress-tls
rules:
- host: example.com
- http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 443
This ingress creates the AWS network load balancer, with a URL like
https://xyz.elb.us-west-1.amazonaws.com/
I am updating the
ingress-tls
secret with a certificate using
cert-manager.
When I access the ingress using the NLB URL
https://xyz.elb.us-west-1.amazonaws.com/api, I get
GOOD: Correct routing based on the path rules from the ingress definition (i.e. it goes to my
api-service as expected)
BAD: Certificate errors since I'm not accessing the ingress with the domain that the certificate is for.
When I access the ingress using the correct domain e.g.
https://example.com/api which is what I want to do, I get:
BAD:
404, it doesn't respect my path rules, and goes to
upstream-default-backend instead.
GOOD: certificate all good, it’s the one for
example.com that
cert-manager configured.
I tried removing the
host: example.com from the
rules:, which gives me:
GOOD: Correct routing based on the path rules from the ingress definition
BAD: Certificate errors, it serves up the default ingress “Fake” certificate instead of the one for
example.com, I guess since the
host is missing from the rules, though not sure of the exact reason.
Can someone please help me get
GOOD
GOOD
I’m at a loss here.
After staring at this for several more hours, and digging through the nasty chunk of lua that is the
nginx.conf for this, I found it! Maybe someday someone will have this problem, and might find this useful.
The problem was:
rules:
- host: example.com
- http:
This is defining (I think) a
host with no forwarding rules, then then some
http forwarding rules without a host. What I had intended was obviously that the forwarding rules would be for the host.
And that would be:
rules:
- host: example.com
http:
I have to say that I'm now even less of a fan of YAML than I was previously, if that's even possible.
I have a K8s setup with traefik being exposed like this
kubernetes:
ingressClass: traefik
service:
nodePorts:
http: 32080
serviceType: NodePort
Behind, I forward some requests to different services
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-name
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: my-host.com
http:
paths:
- path: /my-first-path
backend:
serviceName: my-nodeJs-services
servicePort: 3000
When the DNS is set directly to resolve to my ip, the application works fine with HTTP
http://my-host.com:32080/my-first-path
But when some one add SSL through AWS ALB / API Gateway, the application fail to be reached with 404-NotFound error
The route is like this
https://my-host.com/my-first-path
On the AWS size, they configured something like this
https://my-host.com => SSL Termination and => Forward all to 43.43.43.43:32080
I think this fail because traefik is expecting http://my-host.com but not https://my-host.com which lead to its failure to find the matching route? Or maybe at the ssl termination time, the hostname is lost so that traefik can not find a route?
What should I do in this situation?
I am not very familiar with ALB but what is probably happening is that the requests received by the loadbalancer contain the header Host: my-host.com and when it gets forwarded to your ingress controller, the header is replaced by Host: 43.43.43.43. If this is the case, I see 3 solutions:
ALB might be able to pass the original Host header to the target. (You will have to check in the doc if it's possible)
If the application behind your ingress doesn't check the host header, you can write an ingress that doesn't check a specific host. For example on these examples you can see that the host field is not specified.
If the name resolution works internally, you can define a name for your target, use this name in your ALB and in your ingress.
I am working on a GKE cluster. I have a simple server running. There are three routes on the server.
route 1 - /
route 2 - /ping
route 3 - /health
These paths return 200 response with generic but different "ok" messages.
This is what my ingress yaml looks like -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: basic-ingress
spec:
tls:
- hosts:
- www.simple.com
secretName: simple-server-tls
rules:
- host: www.simple.com
http:
paths:
- path: /ping
backend:
serviceName: simple-server
servicePort: 8080
I have my server exposed as a load balancer. I can access all three routes using loadbalancer as
https://<loadbalancer_ip>:8080/
https://<loadbalancer_ip>:8080/ping
https://<loadbalancer_ip>:8080/health
However, when trying to use an ingress, I only receive a 200 response on https://www.simple.com/ping
The other two routes i.e. https://www.simple.com/health and https://www.simple.com/ return default backend - 404 error.
I can confirm that the server is running and is serving requests perfectly and I have given ingress enough time(upwards of 30 minutes) to finish setting-up. https://www.simple.com/healthz returns "OK".
I think I am making a mistake in configuring the ingress correctly. Any help/suggestions are appreciated.
I cannot provide the ingress logs as I am noticing that kubectl describe ingress/basic-ingress returns the error Error from server (NotFound): the server could not find the requested resource However, kubectl get ingress returns basic-ingress www.simple.com <ingress_ip> 80, 443 31m
Figured out the issue. You need to add a wild card to the path. I only had / in my path and hence it was rejecting (read throwing 404 error) for all my other urls. I added /* and removed the rewrite-target annotation as #bserdar correctly suggested and it worked. Link to the issue on github that helped a lot - https://github.com/kubernetes/ingress-nginx/issues/1120