I find that the Rewrite feature of my Virtual Service is not working very well. Here is my Virtual Service and DestinationRule yaml file:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: leads-http
namespace: seldon
spec:
gateways:
- istio-system/seldon-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /seldon/seldon/leads/
rewrite:
uri: /
route:
- destination:
host: leads-leads
port:
number: 8000
subset: leads
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- labels:
version: leads
name: leads
trafficPolicy:
connectionPool:
http:
idleTimeout: 60s
When I send an http request：
curl --location --request POST 'http://localhost/seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer'
I find that the istio-proxy service prints 404 not found in the logs:
"POST /seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1" 404
even though I expect:
POST /v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1
I am not sure what's happening. Does anyone have any idea? Thanks!
I think your issue is incorrectly configured DestinationRule or service name coneintion.
DestinationRule:
These rules specify configuration for load balancing, connection pool size from the sidecar, and outlier detection settings to detect and evict unhealthy hosts from the load balancing pool.
Version specific policies can be specified by defining a named subset and overriding the settings specified at the service level.
Note: Policies specified for subsets will not take effect until a route rule explicitly sends traffic to this subset.
DestinationRule-Subset:
It seems to me that name should go first in the structure. At least I havent seen/met another examples.
So in your case correct(at least I hope) DR is:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- name: leads
labels:
version: leads
However, if that wont help - I encourage you to check this self-resolved question:
Dont you have the same situation with named service port? I mean as per Explicit protocol selection you should add sufix in service name...
name: <protocol>[-<suffix>]
Related
In a project I'm currently working on, I'd like to create a DNS alias for a Kubernetes service located in another namespace. To do so, I created an ExternalName service such as the following:
kind: Service
apiVersion: v1
metadata:
name: connector
namespace: test
spec:
type: ExternalName
externalName: gateway.eventing.svc.cluster.local
So far, so good. When I request the 'connector' DNS, I successfully hit the external name, i.e. gateway.eventing.svc.cluster.local.
Now, I would like to add headers to all http requests sent to the connector ExternalName service, so I created an Istio VirtualService to do so:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
- connector.test.svc.cluster.local
http:
- match:
- uri:
prefix: /
route:
- destination:
host: connector
port:
number: 80
#headers config ignored for brevity
The problem is that the VirtualService is never called. It seems it does not intercepts request made to the connector DNS or to its fully qualified name, i.e. connector.test.svc.cluster.local.
I figured, after reading the documentation, that this happens because the Istio VirtualService checks the service registry, and the ExternalName service is not part of it, it's just some kind of DNS alias.
I therefore attempted to create an Istio ServiceEntry such as the following:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
endpoints:
- address: gateway.eventing.svc.cluster.local
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_INTERNAL
resolution: DNS
It works, and I can see in Kiali that instead of calling the PassthroughCluster when requesting connector, it is the connector ServiceEntry which is called, which is to my understanding what should be happening.
However, my connector VirtualService is still not called. Why is that? Is there a way to make it happen?
If not, what can I do to alias in a given namespace (i.e. test) a service located in another (i.e. eventing) and proxy http request thanks to an Istio VirtualService?
Thanks in advance for your help!
EDIT:
Sidecar injection is enabled namespace-wide (i.e. test)
So, it turns out that all that was missing to make it work was to both specify and name the port on the ExternalName service.
Here's the updated yaml:
kind: Service
apiVersion: v1
metadata:
name: connector
namespace: test
spec:
type: ExternalName
externalName: gateway.eventing.svc.cluster.local
ports:
- name: http
port: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
- connector.test.svc.cluster.local
http:
- match:
- uri:
prefix: /
route:
- destination:
host: connector
port:
number: 80
#headers config ignored for brevity
Naming the port is absolutely required, as it lets Istio know of the application protocol to use, as defined by the VirtualService.
No need to add a ServiceEntry, it will work with the BYON host specified in the VirtualService.
Note that the answer supplied by #Christoph Raab works as well, but is unhappilly too verbose to be marked as my prefered answer.
Update
I didn't see that the ports list was missing and am not sure how you could apply the yml, because the list should be required.
Anyways, I leave my answer. Maybe it will help someone else in the further.
Original Post (slightly modified)
Docs are not clear, but I think the header manipulation be could done by the receiving sidecar. As far as I understand your setup, the resource behind the ServiceEntry does not have a sidecar, so if that would be true, the manipulation wouldn't work.
In order to add custom headers you can use a EnvoyFilter of type lua that is applied to the sender's sidecar and can manipulate the traffic on the fly.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-custom-header-filter
namespace: test
spec:
configPatches:
- applyTo: CLUSTER
match:
context: SIDECAR_OUTBOUND
cluster:
service: connector
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_request(request_handle)
response_handle:logInfo("adding custom headers..."");
response_handle:headers():add("X-User-Header", "worked");
end
This filter is applied to every request to the service entry connector by every sidecar in the namespace test on outbound and adds a custom header before any other action done.
I'm trying to implement some sort of traffic routing using Istio in a Kubernetes cluster.
The situattion is the following one:
(customer service) => (preference service) => (recommendation service) which has two versions: v1 and v2.
I want to use a custom header, for example X-Svc-Env from an Istio VirtualService and to specify through this header the version of the Recommendation Service which I want to hit.
The configuration for the VirtualService is the following one:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: recommendation
namespace: online
spec:
hosts:
- recommendation
http:
- match:
- headers:
x-svc-env:
regex: v2
route:
- destination:
host: recommendation
subset: version-v2
- route:
- destination:
host: recommendation
subset: version-v1
Also, the DestinationRule that it's being used is the following one:
kind: DestinationRule
metadata:
name: recommendation
namespace: online
spec:
host: recommendation
subsets:
- labels:
version: v1
name: version-v1
- labels:
version: v2
name: version-v2
Well... this is not working because somehow, my custom header is not propagated through Envoy Proxy (I suppose).
I should mention that if I am using a well-known HTTP header, eg: baggage-user-agent (which is the User-Agent header from the OpenTracing specification, all the things are going pretty well).
Tx!
I have a confusion between Virtual Service and Destinationrule on which one is executed first?
Let’s say I have below configs,
Destinationrule -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: movies
namespace: aio
spec:
host: movies
subsets:
- labels:
version: v1
name: version-v1
- labels:
version: v2
name: version-v2
---
VirtualService
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: movies
namespace: aio
spec:
hosts:
- movies
http:
- route:
- destination:
host: movies
subset: version-v1
weight: 10
- destination:
host: movies
subset: version-v2
weight: 90
---
I read somewhere that,
A VirtualService defines a set of traffic routing rules to apply when a host is addressed.
DestinationRule defines policies that apply to traffic intended for service after routing has occurred.
Does this mean Destinationrules are invoked after Virtualservices?
I have a small diagram, is my understanding correct?
Yes,
According to istio documentation about DestinationRule:
DestinationRule defines policies that apply to traffic intended for a service after routing has occurred.
And for VirtualService:
A VirtualService defines a set of traffic routing rules to apply when a host is addressed.
There is an youtube video: Life of a Packet through Istio it explains in detail the order of processes that are applied to a packet going through the istio mesh.
I've had a single VirtualService running through a simple http Gateway and everything works fine:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: satc-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: report
spec:
hosts:
- report-create.default.svc.cluster.local
gateways:
- satc-gateway
http:
- match:
- uri:
prefix: /v1/report
route:
- destination:
host: report-create
port:
number: 8080
I have since added a second VirtualService for a the second service in the cluster through the same Gateway
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: routing
spec:
hosts:
- routing-svc.default.svc.cluster.local
gateways:
- satc-gateway
http:
- match:
- uri:
prefix: /v1/routing
route:
- destination:
host: routing-svc.default.svc.cluster.local
port:
protcol: http
number: 8080
Whilst the first VirtualService still seems to serve, the second consistently 404s. I have added a simple endpoint "/v1/routing/test" to return a Hello World message to ensure the issue doesn't spawn from application logic.
Both VirtualServices seem to be running as expected, though only requests to report are returning anything other than a 404:
report [satc-gateway] [report-create.default.svc.cluster.local] 2h
routing [satc-gateway] [routing-svc.default.svc.cluster.local] 18m
I have tried removing the first deployment all together to ensure it is now hoovering up all the traffic coming in to the cluster and still get 404s. I've also tried executing the routes from inside the pod with a successful response, both services use container port 8080 which I have also triple checked.
I seem to have hit a bit of a wall with this, unsure what is the next best steps in order to debug correctly.
I have analyzed both your yamls and found few issues regarding the second one:
there are 2 extra spaces in the prefix: line
there is a typo in the 'protocol:' line (missed an o)
there are some differences in the destination: part
Please make the adjustments and let me know if that helped.
We have installed Istio on my Kubernetes cluster on AWS (we are using EKS). We have deployed several applications such as: Airflow, Jenkins, Grafana, etc. and we are able to reach them with port-forward. So, they are working as expected.
Now, what we would like to achieve is the possibility of mounting the applications on specific paths so that we can access them via an unique entrypoint.
Here an example to explain what I mean with "unique entrypoint":
http://a517f7.eu-west-1.elb.amazonaws.com/aiflow/ and you can see Airflow dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/jenkins/ and you can see Jenkins dahsboard
http://a517f7.eu-west-1.elb.amazonaws.com/grafana/ and you can see Grafana dahsboard
What we tried is the following
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: apps-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: airflow-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /airflow
route:
- destination:
host: webserver.airflow.svc.cluster.local
port:
number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grafana-virtual-service
spec:
hosts:
- "*"
gateways:
- apps-gateway
http:
- match:
- uri:
prefix: /grafana
route:
- destination:
host: grafana.grafana.svc.cluster.local
port:
number: 3306
---
and so on
In this way but I keep having Aiflow 404 = lots of circles or similar depending on the service.
Do you know how to achieve such a result with Istio? We are also open to also use Nginx or Traefik.
So based on the comments:
The best way to expose a frontend & backend it's to separate both with a wildcard domain like *.domain.com.
For the frontend:
my-app.domain.com/
For the backend services it will always start with a subdomain like api:
api.domain.com/foo
For backend service you will need to use rewrite if your backend service it's no serving the path.
rewrite:
uri: "/"
Check Istio docs for more info about rewrite https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Destination