Prevent VirtualService from routing to empty/unhealthy destinations? - kubernetes

Question

Is there a way to prevent a VirtualService from routing to empty/unhealthy destinations?
For example, consider the following VirtualService and DestinationRule:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpy
spec:
hosts:
- httpy
http:
- route:
- destination:
host: httpy
subset: prod
weight: 90
- destination:
host: httpy
subset: canary
weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: httpy
spec:
host: httpy
subsets:
- name: prod
labels:
isCanary: "false"
- name: canary
labels:
isCanary: "true"
Imagine that there are no healthy pods in the httpy service that have the isCanary: "true" label either because A) zero pods have the isCanary: "true" label or B) all of the canary pods are unready/unhealthy. This will result in 10% of requests to the httpy returning a "no healthy upstream" response.
Is there a good way to avoid this? Is it possible to make the VirtualService "smarter" and have it not route traffic to an obviously unusable destination?
If you want to try this out yourself, apply the following gist then delete the httpy-canary deployment: https://gist.github.com/llamasoft/aaca58fda0ec2e06f2fee4f272691460

Answer 1

Yes, in the destination rule You can configure failover with LocalityLoadBalancerSetting.
According to istio documentation:
If the goal of the operator is not to distribute load across zones and regions but rather to restrict the regionality of failover to meet other operational requirements an operator can set a ‘failover’ policy instead of a ‘distribute’ policy.
The following example sets up a locality failover policy for regions. Assume a service resides in zones within us-east, us-west & eu-west this example specifies that when endpoints within us-east become unhealthy traffic should failover to endpoints in any zone or sub-zone within eu-west and similarly us-west should failover to us-east.
failover:
- from: us-east
to: eu-west
- from: us-west
to: us-east
Failover requires outlier detection to be in place for it to work.
Hope it helps.