Facing an issue when accessing a service deployed in AKS through Istio using Host Header.
Below is the configuration for gateway and VirtualService in Istio (based out of the sample from httpbin) and trying to test the sample service using Host Header as given in this URL: https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/#before-you-begin
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: httpbin-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "httpbin.example.com"
- "40.76.148.29"
- "52.171.230.140"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "httpbin.example.com"
- "40.76.148.29"
- "52.171.230.140"
gateways:
- httpbin-gateway
http:
- match:
- uri:
prefix: /status
- uri:
prefix: /delay
route:
- destination:
port:
number: 8000
host: httpbin
The issue is that when I access the above service in CURL without host header works fine, whereas including the Host Header throws HTTP 403 error:
curl -s -I "http://52.171.230.140:80/status/200"
curl -s -I -HHost:httpbin.example.com "http://52.171.230.140:80/status/200"
Note: There is no Mutual TLS (PeerAuthentication) set on any of the namespaces.
Can someone point if anything is wrong in the setup/steps to fix the issue?
I'm trying to migrate from ingress to istio gateway + virtual service routing, but I keep receiving a 404 Not Found error.
The only link that the app should be accessed to is using my-todos.com, configured locally.
What am I missing here?
Note: the ingress controller works just fine. Initially, todo-lb.default.svc.cluster.local in the istio.yaml file was just set to todo-lb, representing the configured load balancer, still with no success.
Here is the ingress.yaml file (to migrate from):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: todo-ingress
spec:
rules:
- host: my-todos.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: todo-lb
port:
number: 3001
- path: /api
pathType: Prefix
backend:
service:
name: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
Here is the istio.yaml file (to migrate TO):
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: todo-istio-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my-todos.com
# - "*"
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: tls-secret
hosts:
- my-todos.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-lb
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default.svc.cluster.local
port:
number: 3001
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
From what I see you've wrong gateway configuration in your virtual service, that's why it might not work.
If gateway is not in the same namespace as virtual service, you have to specify that in virtual service
Check the spec.gateways section
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo-Mongo
spec:
gateways:
- some-config-namespace/my-gateway # can omit the namespace if gateway is in same
namespace as virtual service.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: my-gateway
namespace: some-config-namespace
There is related istio documentation about that.
So please move your todo-istio-gateway to default namespace.
or use
gateways:
- istio-system/todo-istio-gateway
Few things to check if that won't help:
Is your app deployed in default namespace?
Is your app injected?
Problem might be with your Gateway api:
404 errors occur when multiple gateways configured with same TLS certificate
Configuring more than one gateway using the same TLS certificate will cause browsers that leverage HTTP/2 connection reuse (i.e., most browsers) to produce 404 errors when accessing a second host after a connection to another host has already been established.
For example, let’s say you have 2 hosts that share the same TLS certificate like this:
Wildcard certificate .test.com installed in istio-ingressgateway
Gateway configuration gw1 with host service1.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
Gateway configuration gw2 with host service2.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw1
VirtualService configuration vs2 with host service2.test.com and gateway gw2
Since both gateways are served by the same workload (i.e., selector istio: ingressgateway) requests to both services (service1.test.com and service2.test.com) will resolve to the same IP. If service1.test.com is accessed first, it will return the wildcard certificate (.test.com) indicating that connections to service2.test.com can use the same certificate. Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to service2.test.com. Since the gateway (gw1) has no route for service2.test.com, it will then return a 404 (Not Found) response.
You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2). Then, simply bind both VirtualServices to it like this:
Gateway configuration gw with host *.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw
VirtualService configuration vs2 with host service2.test.com and gateway
checkout this link:
https://istio.io/latest/docs/ops/common-problems/network-issues/#404-errors-occur-when-multiple-gateways-configured-with-same-tls-certificate
In addition to #Jakub answer, there can be one more reason you can get 404 error. Your current ingress rules in virtual service looks like this:
Hostname
Path
Route
my-todos.com
/
Forward to todo-lb.default.svc.cluster.local
my-todos.com
/api
Forward to {{ .Values.api.apiName }}
In Istio, each of the above is an ingress rule. If in Istio ingress-gateway, the rules are added in the above order, then all the URL path including the prefix /api gets routed to the first service i.e. todo-lb.default. It is better to create a single Virtual service like this and then see if the routing works as expected.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
gateways:
- <namespace>/todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default
port:
number: 3001
I have istio configuration on which the docker container runs jasperreports on tomcat(bitnami jasperreports). the application itself could be accessed with context path at http://IP:8080/jasperserver.
I am planning to use istio in my kubernetes configuration, expose the service using http gateway and virtual service. can I add context path of "jasperserver" in addition to destination host and port number in virtual service configuration?
here is my current virtualservice configuration
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: http-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: jasper-virtual-services
spec:
hosts:
- "*"
gateways:
- http-gateway
http:
- match:
- uri:
exact: /report
route:
- destination:
host: jasperserver.reporting.svc.cluster.local
port:
number: 80
You are probably looking for rewrites
Something like
http:
- match:
- uri:
exact: /report
rewrite:
uri: /jasperserver
I have istio installed and can see it on Rancher. I have keycloak installed as well. I am trying to connect the two and have a gateway setup so I can access keycloak front-end through a URL.
In my keycloak manifest I have
# Source: keycloak/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: keycloak
.
. #Many other lines here
.
ports:
- name: http
containerPort: 8080
protocol: TCP
I then setup a gateway with command -
kubectl apply -f networking/custom-gateway.yaml
And in my custom-gateway.yaml file I have -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: keycloak-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: keycloak
spec:
hosts:
- "*"
gateways:
- keycloak-gateway
http:
- match:
- uri:
exact: /keycloak
rewrite:
uri: "/" # Non context aware backend
route:
- destination:
host: keycloak
port:
number: 80
websocketUpgrade: true
Now when I try to access the URL with http://node_ip_address:port/keycloak, I find that I am not able to access the front end. I have verified that keycloak is installed and the pod is up and running on Rancher.
I also have my istio instance connected to the bookinfo application and am able to run the bookinfo-gateway and connect to http://node_ip_address:port/productpage with a gateway that looks like the one described here. I am trying to setup the same gateway only for keycloak.
What am I doing wrong in my yaml files. How do I fix this? Any help is appreciated. Do I have the ports connected correctly?
As far as I can see, you should fix your Virtual Service.
I prepared small example with helm and keycloak helm chart.
Save this as keycloak.yaml, you can configure your keycloak password here.
keycloak:
service:
type: ClusterIP
password: mykeycloakadminpasswd
persistence:
deployPostgres: true
dbVendor: postgres
Install keycloak with helm and values prepared above.
helm upgrade --install keycloak stable/keycloak -f keycloak.yml
Create gateway and virtual service
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: keycloak-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: keycloak
spec:
hosts:
- "*"
gateways:
- keycloak-gateway
http:
- match:
- uri:
prefix: /auth
- uri:
prefix: /keycloak
rewrite:
uri: /auth
route:
- destination:
host: keycloak-http
port:
number: 80
virtual service route.host is name of kubernetes keycloak pod service.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
keycloak-http ClusterIP 10.0.14.36 <none> 80/TCP 22m
You should be able to connect to keycloak via your ingress_gateway_ip/keycloak or ingress_gateway_ip/auth and login with keycloak credentials, in my example it's login: keycloak and password: mykeycloakadminpasswd.
Note that you need to add prefix for /auth as it's default keycloak web to do everything. Keycloak prefix just rewrite to /auth here.
We have installed Istio on my Kubernetes cluster on AWS (we are using EKS). We have deployed several applications such as: Airflow, Jenkins, Grafana, etc. and we are able to reach them with port-forward. So, they are working as expected.
Now, what we would like to achieve is the possibility of mounting the applications on specific paths so that we can access them via an unique entrypoint.
Here an example to explain what I mean with "unique entrypoint":
http://a517f7.eu-west-1.elb.amazonaws.com/aiflow/ and you can see Airflow dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/jenkins/ and you can see Jenkins dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/grafana/ and you can see Grafana dahsboard
What we tried is the following
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: apps-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: airflow-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /airflow
route:
- destination:
host: webserver.airflow.svc.cluster.local
port:
number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
host: grafana.grafana.svc.cluster.local
port:
number: 3306
---
and so on
In this way but I keep having Aiflow 404 = lots of circles or similar depending on the service.
Do you know how to achieve such a result with Istio? We are also open to also use Nginx or Traefik.
So based on the comments:
The best way to expose a frontend & backend it's to separate both with a wildcard domain like *.domain.com.
For the frontend:
my-app.domain.com/
For the backend services it will always start with a subdomain like api:
api.domain.com/foo
For backend service you will need to use rewrite if your backend service it's no serving the path.
rewrite:
uri: "/"
Check Istio docs for more info about rewrite https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Destination