I've had a single VirtualService running through a simple http Gateway and everything works fine:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: satc-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: report
spec:
hosts:
- report-create.default.svc.cluster.local
gateways:
- satc-gateway
http:
- match:
- uri:
prefix: /v1/report
route:
- destination:
host: report-create
port:
number: 8080
I have since added a second VirtualService for a the second service in the cluster through the same Gateway
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: routing
spec:
hosts:
- routing-svc.default.svc.cluster.local
gateways:
- satc-gateway
http:
- match:
- uri:
prefix: /v1/routing
route:
- destination:
host: routing-svc.default.svc.cluster.local
port:
protcol: http
number: 8080
Whilst the first VirtualService still seems to serve, the second consistently 404s. I have added a simple endpoint "/v1/routing/test" to return a Hello World message to ensure the issue doesn't spawn from application logic.
Both VirtualServices seem to be running as expected, though only requests to report are returning anything other than a 404:
report [satc-gateway] [report-create.default.svc.cluster.local] 2h
routing [satc-gateway] [routing-svc.default.svc.cluster.local] 18m
I have tried removing the first deployment all together to ensure it is now hoovering up all the traffic coming in to the cluster and still get 404s. I've also tried executing the routes from inside the pod with a successful response, both services use container port 8080 which I have also triple checked.
I seem to have hit a bit of a wall with this, unsure what is the next best steps in order to debug correctly.
I have analyzed both your yamls and found few issues regarding the second one:
there are 2 extra spaces in the prefix: line
there is a typo in the 'protocol:' line (missed an o)
there are some differences in the destination: part
Please make the adjustments and let me know if that helped.
Related
I find that the Rewrite feature of my Virtual Service is not working very well. Here is my Virtual Service and DestinationRule yaml file:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: leads-http
namespace: seldon
spec:
gateways:
- istio-system/seldon-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /seldon/seldon/leads/
rewrite:
uri: /
route:
- destination:
host: leads-leads
port:
number: 8000
subset: leads
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- labels:
version: leads
name: leads
trafficPolicy:
connectionPool:
http:
idleTimeout: 60s
When I send an http request：
curl --location --request POST 'http://localhost/seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer'
I find that the istio-proxy service prints 404 not found in the logs:
"POST /seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1" 404
even though I expect:
POST /v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1
I am not sure what's happening. Does anyone have any idea? Thanks!
I think your issue is incorrectly configured DestinationRule or service name coneintion.
DestinationRule:
These rules specify configuration for load balancing, connection pool size from the sidecar, and outlier detection settings to detect and evict unhealthy hosts from the load balancing pool.
Version specific policies can be specified by defining a named subset and overriding the settings specified at the service level.
Note: Policies specified for subsets will not take effect until a route rule explicitly sends traffic to this subset.
DestinationRule-Subset:
It seems to me that name should go first in the structure. At least I havent seen/met another examples.
So in your case correct(at least I hope) DR is:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- name: leads
labels:
version: leads
However, if that wont help - I encourage you to check this self-resolved question:
Dont you have the same situation with named service port? I mean as per Explicit protocol selection you should add sufix in service name...
name: <protocol>[-<suffix>]
I'm trying to migrate from ingress to istio gateway + virtual service routing, but I keep receiving a 404 Not Found error.
The only link that the app should be accessed to is using my-todos.com, configured locally.
What am I missing here?
Note: the ingress controller works just fine. Initially, todo-lb.default.svc.cluster.local in the istio.yaml file was just set to todo-lb, representing the configured load balancer, still with no success.
Here is the ingress.yaml file (to migrate from):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: todo-ingress
spec:
rules:
- host: my-todos.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: todo-lb
port:
number: 3001
- path: /api
pathType: Prefix
backend:
service:
name: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
Here is the istio.yaml file (to migrate TO):
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: todo-istio-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my-todos.com
# - "*"
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: tls-secret
hosts:
- my-todos.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-lb
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default.svc.cluster.local
port:
number: 3001
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
# - "*"
gateways:
- todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
From what I see you've wrong gateway configuration in your virtual service, that's why it might not work.
If gateway is not in the same namespace as virtual service, you have to specify that in virtual service
Check the spec.gateways section
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo-Mongo
spec:
gateways:
- some-config-namespace/my-gateway # can omit the namespace if gateway is in same
namespace as virtual service.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: my-gateway
namespace: some-config-namespace
There is related istio documentation about that.
So please move your todo-istio-gateway to default namespace.
or use
gateways:
- istio-system/todo-istio-gateway
Few things to check if that won't help:
Is your app deployed in default namespace?
Is your app injected?
Problem might be with your Gateway api:
404 errors occur when multiple gateways configured with same TLS certificate
Configuring more than one gateway using the same TLS certificate will cause browsers that leverage HTTP/2 connection reuse (i.e., most browsers) to produce 404 errors when accessing a second host after a connection to another host has already been established.
For example, let’s say you have 2 hosts that share the same TLS certificate like this:
Wildcard certificate .test.com installed in istio-ingressgateway
Gateway configuration gw1 with host service1.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
Gateway configuration gw2 with host service2.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw1
VirtualService configuration vs2 with host service2.test.com and gateway gw2
Since both gateways are served by the same workload (i.e., selector istio: ingressgateway) requests to both services (service1.test.com and service2.test.com) will resolve to the same IP. If service1.test.com is accessed first, it will return the wildcard certificate (.test.com) indicating that connections to service2.test.com can use the same certificate. Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to service2.test.com. Since the gateway (gw1) has no route for service2.test.com, it will then return a 404 (Not Found) response.
You can avoid this problem by configuring a single wildcard Gateway, instead of two (gw1 and gw2). Then, simply bind both VirtualServices to it like this:
Gateway configuration gw with host *.test.com, selector istio: ingressgateway, and TLS using gateway’s mounted (wildcard) certificate
VirtualService configuration vs1 with host service1.test.com and gateway gw
VirtualService configuration vs2 with host service2.test.com and gateway
checkout this link:
https://istio.io/latest/docs/ops/common-problems/network-issues/#404-errors-occur-when-multiple-gateways-configured-with-same-tls-certificate
In addition to #Jakub answer, there can be one more reason you can get 404 error. Your current ingress rules in virtual service looks like this:
Hostname
Path
Route
my-todos.com
/
Forward to todo-lb.default.svc.cluster.local
my-todos.com
/api
Forward to {{ .Values.api.apiName }}
In Istio, each of the above is an ingress rule. If in Istio ingress-gateway, the rules are added in the above order, then all the URL path including the prefix /api gets routed to the first service i.e. todo-lb.default. It is better to create a single Virtual service like this and then see if the routing works as expected.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: todo-api
spec:
hosts:
- my-todos.com
gateways:
- <namespace>/todo-istio-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
host: {{ .Values.api.apiName }}
port:
number: {{ .Values.api.apiPort }}
- match:
- uri:
prefix: /
route:
- destination:
host: todo-lb.default
port:
number: 3001
I have kubernetes cluster with installed Istio. I have two pods, for example, sleep1 and sleep2 (containers with installed curl). I want to configure istio to permit traffic from sleep1 to www.google.com and forbid traffic from sleep2 to www.google.com.
So, I created ServiceEntry:
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: google
spec:
hosts:
- www.google.com
- google.com
ports:
- name: http-port
protocol: HTTP
number: 80
resolution: DNS
Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 80
name: http-port
protocol: HTTP
hosts:
- "*"
two virtualServices (mesh->egress, egress->google)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: mesh-to-egress
spec:
hosts:
- www.google.com
- google.com
gateways:
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
port:
number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: egress-to-google-int
spec:
hosts:
- www.google.com
- google.com
gateways:
- istio-egressgateway
http:
- match:
- gateways:
- istio-egressgateway
port: 80
route:
- destination:
host: google.com
port:
number: 80
weight: 100
As result, I can curl google from both pods.
And the question again: can i permit traffic from sleep1 to www.google.com and forbid traffic from sleep2 to www.google.com? I know that this is possible to do with kubernetes NetworkPolicy and black/white lists (https://istio.io/docs/tasks/policy-enforcement/denial-and-list/), but both methods are forbids (permits) traffic to specific ips or maybe I missed something?
You can create different service accounts for sleep1 and sleep2. Then you create an RBAC policy to limit access to the istio-egressgateway policy, so sleep2 will not be able to access any egress traffic through the egress gateway. This should work with forbidding any egress traffic from the cluster, that does not originate from the egress gateway. See https://istio.io/docs/tasks/traffic-management/egress/egress-gateway/#additional-security-considerations.
If you want to allow sleep2 access other services, but not www.google.com, you can use Mixer rules and handlers, see this blog post. It shows how to allow a certain URL path to a specific service account.
I think you're probably on the right track on the denial option.
It is also not limited to IP as we may see attribute-based example for Simple Denial and Attribute-based Denial
So, for example, if we write a simple denial rule for Sleep2 -> www.google.com:
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
name: denySleep2Google
spec:
compiledAdapter: denier
params:
status:
code: 7
message: Not allowed
---
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
name: denySleep2GoogleRequest
spec:
compiledTemplate: checknothing
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: denySleep2
spec:
match: destination.service.host == "www.google.com" && source.labels["app"]=="sleep2"
actions:
- handler: denySleep2Google
instances: [ denySleep2GoogleRequest ]
Please check and see if this helps.
Also, the "match" field in the "rule" entry is based on istio expression language around the attributes. Some vocabulary can be found in this doc.
We have installed Istio on my Kubernetes cluster on AWS (we are using EKS). We have deployed several applications such as: Airflow, Jenkins, Grafana, etc. and we are able to reach them with port-forward. So, they are working as expected.
Now, what we would like to achieve is the possibility of mounting the applications on specific paths so that we can access them via an unique entrypoint.
Here an example to explain what I mean with "unique entrypoint":
http://a517f7.eu-west-1.elb.amazonaws.com/aiflow/ and you can see Airflow dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/jenkins/ and you can see Jenkins dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/grafana/ and you can see Grafana dahsboard
What we tried is the following
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: apps-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: airflow-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /airflow
route:
- destination:
host: webserver.airflow.svc.cluster.local
port:
number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
host: grafana.grafana.svc.cluster.local
port:
number: 3306
---
and so on
In this way but I keep having Aiflow 404 = lots of circles or similar depending on the service.
Do you know how to achieve such a result with Istio? We are also open to also use Nginx or Traefik.
So based on the comments:
The best way to expose a frontend & backend it's to separate both with a wildcard domain like *.domain.com.
For the frontend:
my-app.domain.com/
For the backend services it will always start with a subdomain like api:
api.domain.com/foo
For backend service you will need to use rewrite if your backend service it's no serving the path.
rewrite:
uri: "/"
Check Istio docs for more info about rewrite https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Destination
Now I am testing the scenario running apps on Istio.
I cant access to legacy codes, so I cant change request urls.
For that, I made some simple apps.
I am not sure this scenario is available on Istio.
I have two applications(order and customer)
On the order app, there is code calling customer app with url "http://customer-app:8080/customer".
Now, I want to run two apps on K8S with Istio.
And I don't want to change my code especially calling urls.
(I know I can call each service with service-name.
but I want to make customer service name with "customer-service" not "customer-app")
I find there is a VirtualService which can register MESH_INTERNAL.
I make yaml file like that.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: customer-service-entry
spec:
hosts:
- http://customer-app:8080
location: MESH_INTERNAL
ports:
- number: 8080
name: http
protocol: http
endpoints:
- address: customer-service
ports:
http: 8080
is it possible scenario using virtual domain?
You should be able to do this using a VirtualService:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: customer-app
spec:
hosts:
- customer-app
http:
- match:
- port: 8080
route:
- destination:
host: customer-service
port:
number: 9080
You will also need to define a dummy K8S service for customer-app, so that it's resolvable:
apiVersion: v1
kind: Service
metadata:
name: customer-app
spec:
ports:
- port: 8080
name: http