I am new to AKS and trying to set up the cluster and expose it via an app gateway ingress controller. While I was able to set up the cluster using az commands and was able to deploy and hit it using HTTP. I am having some challenges in enabling HTTPS over 443 in-app gateway ingress and looking to get some help.
Below is our workflow and I am trying to setup app gateway listener on port 443
Below is the k8 we used for enabling the ingress. If I apply is without ssl cert it woks but if I give ssl cert I get a 502 bad gateway.
Cert is uploaded to KV and Cluster has KV add-on installed. But I am not sure how to attach this specific kv to cluster and whether the cert should be uploaded to gateway or Kubernetes.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80
This link can help you with KV add-on certificate on App GW: https://azure.github.io/application-gateway-kubernetes-ingress/features/appgw-ssl-certificate/
I use different configuration to set certs on Appgw.
I'm getting certificates via the akv2k8s tool. This creates secrets on k8s cluster.
Then I use those certs in the ingress configuration. Please check tls definition under spec.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- yourdomain.com
secretName: your-tls-secret-name
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80
Related
I have a k8s cluster where I deploy some containers.
The cluster is accessible at microk8s.hostname.internal.
At this moment I have an application/container deployed that is accessible here: microk8s.hostname.internal/myapplication with the help of a service and an ingress.
And this works great.
Now I would like to deploy another application/container but I would like it accessible like this: otherapplication.microk8s.hostname.internal.
How do I do this?
Currently installed addons in microk8s:
aasa#bolsrv0891:/snap/bin$ microk8s status
microk8s is running
high-availability: no
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
ingress # (core) Ingress controller for external access
metrics-server # (core) K8s Metrics Server for API access to service metrics
Update 1:
If I portforward to my service it works.
I have tried this ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
namespace: jupyter-notebook
annotations:
kubernetes.io/ingress.class: public
spec:
rules:
- host: jupyter.microk8s.hostname.internal
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: jupyter-service
port:
number: 7070
But I cant access it nor ping it. Chrome says:
jupyter.microk8s.hostname.internal’s server IP address could not be found.
My service looks like this:
apiVersion: v1
kind: Service
metadata:
name: jupyter-service
namespace: jupyter-notebook
spec:
ports:
- name: 7070-8888
port: 7070
protocol: TCP
targetPort: 8888
selector:
app: jupyternotebook
type: ClusterIP
status:
loadBalancer: {}
I can of course ping microk8s.hostname.internal.
Update 2:
The ingress that is working today that has a context path: microk8s.boliden.internal/myapplication looks like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: public
nginx.ingress.kubernetes.io/rewrite-target: /$1
name: jupyter-ingress
namespace: jupyter-notebook
spec:
rules:
- http:
paths:
- path: "/jupyter-notebook/?(.*)"
pathType: Prefix
backend:
service:
name: jupyter-service
port:
number: 7070
This is accessible externally by accessing microk8s.hostname.internal/jupyter-notebook.
To do this you would have to configure a kube service, kube ingress and the configure your DNS.
Adding an entry into the hosts file would allow DNS resolution to otherapplication.microk8s.hostname.internal
You could use dnsmasq to allow for wildcard resolution e.g. *.microk8s.hostname.internal
You can test the dns reoslution using nslookup or dig
You can copy the same ingress and update name of it and Host inside it, that's all change you need.
For ref:
kind: Ingress
metadata:
name: second-ingress <<- make sure to update name else it will overwrite if the same
spec:
rules:
- host: otherapplication.microk8s.hostname.internal
http:
paths:
- path: /
backend:
serviceName: service-name
servicePort: service-port
You can create the subdomain with ingress just update the Host in ingress and add the necessary serviceName and servicePort to route traffic to specific service.
Feel free to append the necessary fields, and annotation if any to the above ingress from the existing ingress which is working for you.
If you are running it locally you might have to map the IP to the subdomain locally in /etc/hosts file
/etc/hosts
otherapplication.microk8s.hostname.internal <IP address>
I am trying to use 'Kubernetes Ingress with Traefik, CertManager, LetsEncrypt and HAProxy' for certificates management.
What I want to do is use certificates in my application which deployed on kubernetes.
My application contains following services:
my-app1 NodePort 10.43.204.206 16686:31149/TCP
my-app2 NodePort 10.43.204.208 2746:30972/TCP
So for my-app1 without certificates I was accessing it as http://{IP}:31149/app1. And with certificates I am now accessing it as https://my-dns.com/app1. For this I am using this link. I created following ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod-ingress
namespace: my-ns
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "my-dns.com"
secretName: prod-cert
rules:
- host: "my-dns.com"
http:
paths:
- path: /app1
pathType: Prefix
backend:
service:
name: my-app1
port:
number: 16686
But for my-app2 without certificates I was accessing it as https://{IP}:30972/app2. So as I am already using https for my-app2 but I want to also use certificates for this service.
Any idea how to do this?
So the issue was, I was deploying my application with self signed certificates. So because of this I am getting issue while accessing my dashboard.
So now I just disabled self signed certificates in my application. And now I am able
to access dashboard with domain name https://my-dns.com.
We are implementing kubernetes on GKE.
I got stuck with the following.
The staging application on is hosted on GKE 1.19.7-gke.1500.
We have an application where we have to set up webhooks with applications. When setting up the webhook I get the same error. Namely that the url is not available or does not exist.
The application on kubernetes has a valid certificate. The application works locally (minikube and skaffold) when we use ngrok.
In addition, it is possible to perform an oauth with a third party software. But we cannot activate the webhook because our webhook endpoint is not found.
I looked at the cert-manager documentation, which states that there are known issues between Cert-manager and GKE. They say that this has to do with the settings in GKE, I opened some firewall myself, but without result. https://cert-manager.io/docs/installation/compatibility/
In addition, I have looked on the internet to find out if this problem can be fixed. But I find a lot of information about webhook to be set up for Kubernetes itself but not the application that runs on it.
I think it has to do with a port incorrectly GKE configured?
See our Pod Service setting:
---
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
selector:
app: back-end
ports:
- name: back-end
protocol: TCP
port: 5000
targetPort: 5000
Our Ingress setting:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.global-static-ip-name: "**.**.***.***"
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
cert-manager.io/issuer: '****'
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- test.xxxxxx.com
secretName: xxxxx
rules:
- host: test.xxxxx.com
http:
paths:
- path: /api/?(.*)
backend:
serviceName: backend
servicePort: 5000
- path: /?(.*)
backend:
serviceName: client
servicePort: 3000
Updated 03/15/2021
For example we want to set up a webhook for the following endpoint:
"https://test.xxxx.nl/api/v1/webhook/person".
We make a request to PipeDrive to set up a webhook, but then PipeDrive gives the following error: "errors: { subscription_url: [ 'invalid or non-reachable URL' ] } }"
response: [cmd]: kubectl get ep backend
NAME: backend
ENDPOINTS: 10.0.0.21:5000, 10.0.0.22:5000
AGE: 81d
I'm trying to migrate from ingress to istio gateway + virtual service routing, but I keep receiving a 404 Not Found error.
The only link that the app should be accessed to is using my-todos.com, configured locally.
What am I missing here?
Note: the ingress controller works just fine. Initially, todo-lb.default.svc.cluster.local in the istio.yaml file was just set to todo-lb, representing the configured load balancer, still with no success.
Here is the ingress.yaml file (to migrate from):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: todo-ingress
spec:
rules:
- host: my-todos.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: todo-lb
port:
number: 3001
- path: /api
pathType: Prefix
backend:
service:
name: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
Here is the istio.yaml file (to migrate TO):
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: todo-istio-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my-todos.com
# - "*"
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: tls-secret
hosts:
- my-todos.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-lb
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default.svc.cluster.local
port:
number: 3001
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
From what I see you've wrong gateway configuration in your virtual service, that's why it might not work.
If gateway is not in the same namespace as virtual service, you have to specify that in virtual service
Check the spec.gateways section
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo-Mongo
spec:
gateways:
- some-config-namespace/my-gateway # can omit the namespace if gateway is in same
namespace as virtual service.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: my-gateway
namespace: some-config-namespace
There is related istio documentation about that.
So please move your todo-istio-gateway to default namespace.
or use
gateways:
- istio-system/todo-istio-gateway
Few things to check if that won't help:
Is your app deployed in default namespace?
Is your app injected?
Problem might be with your Gateway api:
404 errors occur when multiple gateways configured with same TLS certificate
Configuring more than one gateway using the same TLS certificate will cause browsers that leverage HTTP/2 connection reuse (i.e., most browsers) to produce 404 errors when accessing a second host after a connection to another host has already been established.
For example, let’s say you have 2 hosts that share the same TLS certificate like this:
Wildcard certificate .test.com installed in istio-ingressgateway
Gateway configuration gw1 with host service1.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
Gateway configuration gw2 with host service2.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw1
VirtualService configuration vs2 with host service2.test.com and gateway gw2
Since both gateways are served by the same workload (i.e., selector istio: ingressgateway) requests to both services (service1.test.com and service2.test.com) will resolve to the same IP. If service1.test.com is accessed first, it will return the wildcard certificate (.test.com) indicating that connections to service2.test.com can use the same certificate. Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to service2.test.com. Since the gateway (gw1) has no route for service2.test.com, it will then return a 404 (Not Found) response.
You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2). Then, simply bind both VirtualServices to it like this:
Gateway configuration gw with host *.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw
VirtualService configuration vs2 with host service2.test.com and gateway
checkout this link:
https://istio.io/latest/docs/ops/common-problems/network-issues/#404-errors-occur-when-multiple-gateways-configured-with-same-tls-certificate
In addition to #Jakub answer, there can be one more reason you can get 404 error. Your current ingress rules in virtual service looks like this:
Hostname
Path
Route
my-todos.com
/
Forward to todo-lb.default.svc.cluster.local
my-todos.com
/api
Forward to {{ .Values.api.apiName }}
In Istio, each of the above is an ingress rule. If in Istio ingress-gateway, the rules are added in the above order, then all the URL path including the prefix /api gets routed to the first service i.e. todo-lb.default. It is better to create a single Virtual service like this and then see if the routing works as expected.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
gateways:
- <namespace>/todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default
port:
number: 3001
I have a service (Node REST app, let's call it ABC) within the cluster. This app listens to two ports, one runs on HTTP and the other on HTTPS.
I use Ingress. I can see Ingress only use HTTP port to communicate with ABC. I got confirmed this by stopping the HTTP and running ABC on HTTPS only.
Do I have to make any specific setup in Ingress so to use HTTPS instead of HTTP to communicate with ABC? Or is it a pattern to run services within a cluster on HTTP when use Ingress?
Ingress yaml :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
spec:
rules:
- host: abc.containers.myhost.net
http:
paths:
- backend:
serviceName: my-test-node
servicePort: 9080
path: my-test-node/xyz
tls:
- hosts:
- abc.containers.myhost.net
secretName: abc1