I'm studying how Istio works and in particular I'm looking at timeouts.
I followed this guide provided by Istio Docs, where the following VirtualService gets defined:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v2
timeout: 0.5s
If I understood well, every call to ratings service get a 0.5s timeout. I was wondering if it's possible to define timeout only in the communication beetween (for example) a pod and a service, and not for every call received by the service
Related
I find that the Rewrite feature of my Virtual Service is not working very well. Here is my Virtual Service and DestinationRule yaml file:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: leads-http
namespace: seldon
spec:
gateways:
- istio-system/seldon-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /seldon/seldon/leads/
rewrite:
uri: /
route:
- destination:
host: leads-leads
port:
number: 8000
subset: leads
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- labels:
version: leads
name: leads
trafficPolicy:
connectionPool:
http:
idleTimeout: 60s
When I send an http request：
curl --location --request POST 'http://localhost/seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer'
I find that the istio-proxy service prints 404 not found in the logs:
"POST /seldon/seldon/leads/v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1" 404
even though I expect:
POST /v2/models/leads-lgb/versions/v0.1.0/infer HTTP/1.1
I am not sure what's happening. Does anyone have any idea? Thanks!
I think your issue is incorrectly configured DestinationRule or service name coneintion.
DestinationRule:
These rules specify configuration for load balancing, connection pool size from the sidecar, and outlier detection settings to detect and evict unhealthy hosts from the load balancing pool.
Version specific policies can be specified by defining a named subset and overriding the settings specified at the service level.
Note: Policies specified for subsets will not take effect until a route rule explicitly sends traffic to this subset.
DestinationRule-Subset:
It seems to me that name should go first in the structure. At least I havent seen/met another examples.
So in your case correct(at least I hope) DR is:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: leads-leads
namespace: seldon
spec:
host: leads-leads
subsets:
- name: leads
labels:
version: leads
However, if that wont help - I encourage you to check this self-resolved question:
Dont you have the same situation with named service port? I mean as per Explicit protocol selection you should add sufix in service name...
name: <protocol>[-<suffix>]
In a project I'm currently working on, I'd like to create a DNS alias for a Kubernetes service located in another namespace. To do so, I created an ExternalName service such as the following:
kind: Service
apiVersion: v1
metadata:
name: connector
namespace: test
spec:
type: ExternalName
externalName: gateway.eventing.svc.cluster.local
So far, so good. When I request the 'connector' DNS, I successfully hit the external name, i.e. gateway.eventing.svc.cluster.local.
Now, I would like to add headers to all http requests sent to the connector ExternalName service, so I created an Istio VirtualService to do so:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
- connector.test.svc.cluster.local
http:
- match:
- uri:
prefix: /
route:
- destination:
host: connector
port:
number: 80
#headers config ignored for brevity
The problem is that the VirtualService is never called. It seems it does not intercepts request made to the connector DNS or to its fully qualified name, i.e. connector.test.svc.cluster.local.
I figured, after reading the documentation, that this happens because the Istio VirtualService checks the service registry, and the ExternalName service is not part of it, it's just some kind of DNS alias.
I therefore attempted to create an Istio ServiceEntry such as the following:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
endpoints:
- address: gateway.eventing.svc.cluster.local
ports:
- number: 80
name: http
protocol: HTTP
location: MESH_INTERNAL
resolution: DNS
It works, and I can see in Kiali that instead of calling the PassthroughCluster when requesting connector, it is the connector ServiceEntry which is called, which is to my understanding what should be happening.
However, my connector VirtualService is still not called. Why is that? Is there a way to make it happen?
If not, what can I do to alias in a given namespace (i.e. test) a service located in another (i.e. eventing) and proxy http request thanks to an Istio VirtualService?
Thanks in advance for your help!
EDIT:
Sidecar injection is enabled namespace-wide (i.e. test)
So, it turns out that all that was missing to make it work was to both specify and name the port on the ExternalName service.
Here's the updated yaml:
kind: Service
apiVersion: v1
metadata:
name: connector
namespace: test
spec:
type: ExternalName
externalName: gateway.eventing.svc.cluster.local
ports:
- name: http
port: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: connector
namespace: test
spec:
hosts:
- connector
- connector.test.svc.cluster.local
http:
- match:
- uri:
prefix: /
route:
- destination:
host: connector
port:
number: 80
#headers config ignored for brevity
Naming the port is absolutely required, as it lets Istio know of the application protocol to use, as defined by the VirtualService.
No need to add a ServiceEntry, it will work with the BYON host specified in the VirtualService.
Note that the answer supplied by #Christoph Raab works as well, but is unhappilly too verbose to be marked as my prefered answer.
Update
I didn't see that the ports list was missing and am not sure how you could apply the yml, because the list should be required.
Anyways, I leave my answer. Maybe it will help someone else in the further.
Original Post (slightly modified)
Docs are not clear, but I think the header manipulation be could done by the receiving sidecar. As far as I understand your setup, the resource behind the ServiceEntry does not have a sidecar, so if that would be true, the manipulation wouldn't work.
In order to add custom headers you can use a EnvoyFilter of type lua that is applied to the sender's sidecar and can manipulate the traffic on the fly.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-custom-header-filter
namespace: test
spec:
configPatches:
- applyTo: CLUSTER
match:
context: SIDECAR_OUTBOUND
cluster:
service: connector
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_request(request_handle)
response_handle:logInfo("adding custom headers..."");
response_handle:headers():add("X-User-Header", "worked");
end
This filter is applied to every request to the service entry connector by every sidecar in the namespace test on outbound and adds a custom header before any other action done.
I have an application that has some back-end services and SPA front-end build in React.
I want to have a canary release with istio. My concern is how to manage the release strategy in which-
I pass the traffic of certain volume to the front-end
When the back-end request is done from this new front-end, the traffic should be passed to new back-end services.
For this what will be the best approach?
It is explained in detail in istio documentation under VirtualService.
There is also nice simple explanation and example here:
Canary Deployments
A canary deployment is a strategy for safely rolling out a new version of a service. With Istio, you can use percentage-based traffic splitting to direct a small amount of traffic to the new version. Then you can run a canary analysis on v2 (like check latency and error rate), and finally direct more traffic at the new version until it's serving all traffic.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld
spec:
hosts:
- helloworld
http:
- route:
- destination:
host: helloworld
subset: v1
weight: 90
- destination:
host: helloworld
subset: v2
weight: 10
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: helloworld
spec:
host: helloworld
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
UPDATE:
Session affinity aka sticky session can be added with DestinationRule using hashed based loadbalancing according to istio documentation.
Hope it helps.
Is there a way to prevent a VirtualService from routing to empty/unhealthy destinations?
For example, consider the following VirtualService and DestinationRule:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpy
spec:
hosts:
- httpy
http:
- route:
- destination:
host: httpy
subset: prod
weight: 90
- destination:
host: httpy
subset: canary
weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpy
spec:
host: httpy
subsets:
- name: prod
labels:
isCanary: "false"
- name: canary
labels:
isCanary: "true"
Imagine that there are no healthy pods in the httpy service that have the isCanary: "true" label either because A) zero pods have the isCanary: "true" label or B) all of the canary pods are unready/unhealthy. This will result in 10% of requests to the httpy returning a "no healthy upstream" response.
Is there a good way to avoid this? Is it possible to make the VirtualService "smarter" and have it not route traffic to an obviously unusable destination?
If you want to try this out yourself, apply the following gist then delete the httpy-canary deployment: https://gist.github.com/llamasoft/aaca58fda0ec2e06f2fee4f272691460
Yes, in the destination rule You can configure failover with LocalityLoadBalancerSetting.
According to istio documentation:
If the goal of the operator is not to distribute load across zones and regions but rather to restrict the regionality of failover to meet other operational requirements an operator can set a ‘failover’ policy instead of a ‘distribute’ policy.
The following example sets up a locality failover policy for regions. Assume a service resides in zones within us-east, us-west & eu-west this example specifies that when endpoints within us-east become unhealthy traffic should failover to endpoints in any zone or sub-zone within eu-west and similarly us-west should failover to us-east.
failover:
- from: us-east
to: eu-west
- from: us-west
to: us-east
Failover requires outlier detection to be in place for it to work.
Hope it helps.
I have a confusion between Virtual Service and Destinationrule on which one is executed first?
Let’s say I have below configs,
Destinationrule -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: movies
namespace: aio
spec:
host: movies
subsets:
- labels:
version: v1
name: version-v1
- labels:
version: v2
name: version-v2
---
VirtualService
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: movies
namespace: aio
spec:
hosts:
- movies
http:
- route:
- destination:
host: movies
subset: version-v1
weight: 10
- destination:
host: movies
subset: version-v2
weight: 90
---
I read somewhere that,
A VirtualService defines a set of traffic routing rules to apply when a host is addressed.
DestinationRule defines policies that apply to traffic intended for service after routing has occurred.
Does this mean Destinationrules are invoked after Virtualservices?
I have a small diagram, is my understanding correct?
Yes,
According to istio documentation about DestinationRule:
DestinationRule defines policies that apply to traffic intended for a service after routing has occurred.
And for VirtualService:
A VirtualService defines a set of traffic routing rules to apply when a host is addressed.
There is an youtube video: Life of a Packet through Istio it explains in detail the order of processes that are applied to a packet going through the istio mesh.